R8 — Coordination: consensus, leader election and distributed locks

This section is about making sure only one party acts at a time when there is no single machine to ask. It starts from the database lock you already trust, rebuilds it across machines as a lease, and shows where that leaks and how fencing tokens close the gap. It now also covers consensus — how the lock service itself stays one authority while running on several machines — so that the implementations at the end read as what they are: consensus-backed lock services.

1. What locks are for

Two workers read the same account balance of ₹100. Both subtract ₹30. Both write ₹70. One of the two debits has vanished — the balance should be ₹40.

That is a lost update, and a lock prevents it by making the second worker wait until the first is done. That is the entire purpose of a lock: stop two things touching the same data at the same time.

The distributed version of the same picture: you run two copies of a service for redundancy, a scheduled job fires at 2am on both, and both copies wake up, read the same rows and start the same work. Sometimes that is merely wasteful. Sometimes it debits an account twice.

Locks serve isolation, and essentially only isolation

Of the four ACID properties, a lock delivers isolation — the promise that concurrent work does not interleave badly, so the result looks as though the operations ran one after another. Locks produce that by making one party wait.

The others come from different machinery. Atomicity comes from the write-ahead and undo logs — the database can replay or reverse, and no lock is involved. Durability comes from the log plus fsync. Consistency comes from constraints, foreign keys and your own application logic; locks help you maintain it, but they do not implement it.

The two are entangled in one place worth noticing: locks are held until commit precisely so nobody observes your partial work before you have decided whether it is all happening. Locks implement isolation, but atomicity decides when they are released.

This matters for everything below, because a distributed lock is an attempt to get isolation without a transaction. You cannot stretch a database's isolation across services, so you approximate it by making everyone queue up — and the approximation leaks, which is what most of this section is about.

And atomicity, across machines?

If locks give you isolation, it is fair to ask what gives you each of the other three once a single transaction can no longer cover the work. There is no one answer — each property gets replaced by a different mechanism, and most of this pack is those mechanisms.

Two things follow from reading that column. First, atomicity is the property that degrades rather than disappears: you keep a weaker version of it through compensation, and you make the degradation visible and repairable instead of pretending it is not there. Second, isolation is the one you actually lose, which is why intermediate states leak and why so much domain modelling — holds, reservations, pending statuses — exists to hide them again.

2. Locks inside one database

Before the distributed version makes sense, it helps to know exactly what the database is doing, because the distributed version is an attempt to rebuild it by hand.

A lock in a database is bookkeeping, not a physical thing. When you lock a row, the database writes an entry into an in-memory structure — the lock manager — saying something like "row 42 of accounts is held exclusively by transaction 8817." Anyone else who wants that row is made to wait. Nothing is attached to the row itself.

BEGIN;
SELECT * FROM accounts WHERE id = 42 FOR UPDATE;  -- lock acquired here
UPDATE accounts SET balance = balance - 30 WHERE id = 42;
COMMIT;                                           -- lock released here

Modes. A shared lock lets many readers in and blocks writers. An exclusive lock lets exactly one writer in and blocks everyone. FOR UPDATE takes the exclusive one; FOR SHARE the other.

Granularity: row, page, or table — and you get whichever the access path implies. This is the part people assume rather than check. Look up by primary key or unique index and the database locks exactly one row. Run the same statement with no usable index and it must examine every row to find out which ones match, so it locks along the way — in MySQL/InnoDB with no index, that effectively means the table. Same syntax, wildly different blast radius. Locking granularity is really an indexing question, and EXPLAIN on your locking queries is how you find out what you actually have.

Three more ways the lock quietly grows beyond what you asked for:

• Gap and next-key locks. In InnoDB at REPEATABLE READ, a ranged FOR UPDATE locks the gaps between index entries too, so WHERE id BETWEEN 10 AND 20 FOR UPDATE blocks someone inserting id = 15 even though no such row exists. READ COMMITTED removes most of this.
• Foreign keys. Updating a child row takes a shared lock on the parent. Two transactions touching different payments of the same account can contend on the account row without either one mentioning it.
• Lock escalation. SQL Server converts many row locks into a table lock past a threshold. Postgres and InnoDB do not escalate.

Three properties make this work, and all three are free:

1. One authority. Every connection asks the same lock manager. Two components cannot disagree about who holds what.
2. The database knows whether the holder is alive. It owns the connection. If the client process dies, the TCP connection drops and the database finds out.
3. Automatic release. On commit, rollback, or a dropped connection, locks are released without anyone asking. Nothing gets held forever by accident.

The same single authority is what lets a database detect deadlocks: it can see every waiter, build a graph of who waits on whom, spot a cycle and kill one transaction to break it.

Two single-database variants worth knowing:

SELECT ... FOR UPDATE SKIP LOCKED is the same mechanism used the other way round: instead of making everyone wait, each worker takes rows nobody else holds. That is how the outbox relay in R4 shares work between instances.

Advisory locks lock a name rather than a row, for cases where there is no row to lock — "only one kick-off job at a time":

SELECT pg_try_advisory_xact_lock(hashtext('kickoff:ACH:2026-09-28'));

The _xact_ variant releases at the end of the transaction. Two cautions: the name is squeezed into a 64-bit integer, so unrelated strings can collide, and the session-scoped variant stays held until you release it or disconnect.

Practical rules for keeping single-database locks narrow: lock by primary key, one row at a time, in a deterministic order (ascending ID) to avoid deadlocks; use NOWAIT to fail fast or SKIP LOCKED to take different work; use FOR NO KEY UPDATE in Postgres when you are not touching the key; keep the transaction short, because duration is the real cost; and never lock a row and then make a network call. To verify rather than assume, query pg_locks joined to pg_stat_activity, or performance_schema.data_locks in MySQL 8, while holding an open transaction.

Hold onto those three free properties. The distributed version loses two of them.

3. What it takes to build a distributed lock

Now the two workers are separate processes on separate machines, possibly writing to different systems. There is no shared lock manager, so you have to rebuild the three properties yourself.

One authority — you need a single shared store that every participant talks to: a database table, Redis, ZooKeeper, etcd, DynamoDB. It must be one thing everyone sees identically.

Knowing the holder is alive — you cannot have this. Nobody can force a dead machine to let go, and no one can tell you it died. This is the hard one, and section 4 is about it.

Automatic release — since you cannot detect death, you attach a timer instead. The lock expires after N seconds whether or not the holder has finished. That single substitution is what makes a distributed lock a lease — a rental with an expiry — and there is no way to build one without it.

The algorithm, in five steps

1. Generate a unique ID for this attempt: pod-7:boot-abc123:req-99.
2. Atomically create a key only if it does not already exist, with your ID as the value and a TTL. Atomically is load-bearing — check-then-set as two operations is a race.
3. If the create succeeded, you hold the lock. If not, someone else does: back off and retry with jitter.
4. While working, extend the TTL periodically — a heartbeat.
5. When done, delete the key only if the value is still your ID, or you may delete a lock that has since been granted to someone else.

That is the whole mechanism. In Redis it is SET key myid NX PX 30000 plus a small Lua script to release. In SQL it is a lease table. The algorithm is easy; its correctness is not.

The lease table

The general pattern, and the same shape as the request ownership in R3:

CREATE TABLE lease (
  name          text PRIMARY KEY,      -- "kickoff:ACH", "recon:card:2026-09-28"
  owner         text NOT NULL,         -- pod name, so you can see who holds it
  fence_token   bigint NOT NULL,       -- see section 5
  acquired_at   timestamptz NOT NULL,
  expires_at    timestamptz NOT NULL
);

INSERT INTO lease (name, owner, fence_token, acquired_at, expires_at)
VALUES (:name, :me, 1, now(), now() + :ttl)
ON CONFLICT (name) DO UPDATE
   SET owner = :me,
       fence_token = CASE WHEN lease.owner = :me
                          THEN lease.fence_token        -- renewal: same token
                          ELSE lease.fence_token + 1    -- takeover: new token
                     END,
       acquired_at = now(),
       expires_at = now() + :ttl
 WHERE lease.expires_at < now()         -- only take over an expired lease
    OR lease.owner = :me                -- or renew my own
RETURNING fence_token;

Read it as one sentence: insert the lease; if it already exists, take it over only if it has expired or it is already mine, and either way tell me my token. Zero rows returned means somebody else holds it and you back off.

While you work, renew on a heartbeat at roughly a third of the TTL. The important part is what you do when a renewal comes back empty: you lost the lease, so stop immediately. Finishing the work and writing at the end is the failure mode fencing exists to catch.

Where the lease table lives

One table, in one shared database, used by every instance. Not one per node — a per-node table would be meaningless, because the whole point is that instances need a common place to find out whether someone else already holds the lease. If each had its own, all of them would always win.

Think of it as a whiteboard in a shared room. Anyone can look, anyone can write, and there is only one whiteboard.

There is one row per lockable thing, not one per node:

Three jobs, three rows, two pods. One pod holding two leases is fine — the rows are independent, and so are their token sequences. kickoff:ACH counts 411, 412, 413 across its own acquisitions while recon:card counts separately, because a token only ever gets compared against others from the same lease.

The store must be one thing that every instance sees identically. That rules out some tempting shortcuts. A local file is useless — each node sees only its own. A read replica is worse than useless: replication lag means two nodes can read different states and both conclude the lease is free. It has to be the primary, or a store with equivalent semantics.

Where to put it. If your services already share a database, put it there — cheapest correct answer, and the acquisition works precisely because the primary key on name makes the database the arbiter. If each service has its own database, you need a genuinely shared store: etcd, ZooKeeper or DynamoDB, or Redis if it is only an efficiency lock. Resist nominating one service's database to hold everyone's leases; that couples every service to that one, which is usually worse than running the shared store.

What this costs you operationally. The lease store becomes a dependency of every job that uses it. If it is unreachable, nobody acquires anything and the work stops. That is usually the right failure mode — stopping beats two writers — but it should be a conscious choice, and it is why people reach for a store designed for the job rather than bolting it onto a busy application database.

Scenario 10 is this section in one animation: a lease that expires while its holder is still alive, and a fencing token rejecting the zombie's write.

4. Why distributed locking is troublesome

The root problem: you cannot distinguish a dead process from a slow one. Both look identical from outside — no heartbeat arriving. Assume dead and be wrong, and two workers run. Assume alive and be wrong, and the lock is stuck forever and the work never happens. Every difficulty below follows from choosing "assume dead after N seconds".

Expiry while the holder is still alive

Here is the sequence that matters, slowly:

1. Worker A takes the lease on account 42. It has 30 seconds.
2. Worker A starts the work.
3. Worker A's JVM enters a stop-the-world garbage collection pause. It is frozen — not crashed, just not running. This lasts 40 seconds.
4. At second 30 the lease expires. From the lock service's point of view, A is gone.
5. Worker B takes the lease and starts working on account 42.
6. At second 40, A wakes up. It has no idea any time passed. As far as A is concerned it still holds the lock, and it carries on writing.

Now two workers are writing to the same account, which is exactly what the lock existed to prevent.

Nothing in the lock service can fix this. A pause can happen anywhere — GC, a hypervisor freezing the VM, a slow disk, a network partition delaying the release. The lock service made a reasonable decision on the information it had. The problem is that the lock service is not the thing being protected. The gap is between the lock service and the data.

The rest of the difficulties

• Time is not trustworthy. TTLs depend on clocks and on an assumption that pauses stay bounded. Distributed systems promise neither.
• Failover can lose the lock. Redis replicates asynchronously: the primary accepts your lock, dies before replicating, a replica is promoted with no record of it, and a second worker acquires it cleanly.
• Releasing the wrong lock. A naive DEL key after your lease has already expired deletes someone else's lock. Release must compare the owner ID and delete in one atomic step.
• The lock store is now in your availability path. Unreachable store means nobody acquires anything and all that work stops.
• Stampedes and deadlocks come back as your problem. No central lock manager means no deadlock detection; waiters all wake at the same instant unless you add jitter.

Efficiency locks and correctness locks

The difference decides how much of this you have to care about.

An efficiency lock stops duplicate work. If it fails, two workers rebuild the same cache or send the same report twice. You waste effort; nothing is corrupted. Best-effort is fine.

A correctness lock stops a second writer from corrupting state. If it fails, money moves twice.

Nearly every popular lock implementation is best-effort, whether or not its documentation says so, because of the expiry problem above. So the question to ask of any lock is not "is it reliable?" but "which kind is this, and does the thing I am protecting care?" A Redis lock is a very good efficiency lock. It is not what should stand between a customer and a double debit.

5. What to do about it

The real fix: fencing tokens

Stop trusting the lock and start checking at the place where the write lands.

Every time the lease is granted it comes with a number that only ever goes up. Worker A got 33. Worker B, taking over after the expiry, got 34. Every write carries its number, and the resource remembers the highest number it has seen and refuses anything lower.

sequenceDiagram
  autonumber
  participant A as Worker A
  participant L as Lock service
  participant R as The data, e.g. the ledger row
  participant B as Worker B
  A->>L: acquire lock on account 42, ttl 30s
  L-->>A: granted, token 33
  Note over A: frozen for 40 seconds
  L-->>L: lease expires
  B->>L: acquire lock on account 42
  L-->>B: granted, token 34
  B->>R: write with token 34
  R-->>B: accepted, highest seen is now 34
  Note over A: wakes up, still believes it holds the lock
  A->>R: write with token 33
  R-->>A: rejected, 33 is older than 34

Worker A is not asked to behave correctly, and it does not need to notice anything. Its write simply does not apply. That is the point: correctness stops depending on every participant being awake and honest.

What makes a token a valid fence

Three properties, and all three are required:

• Issued by the same authority that grants the lease, in the same atomic operation. A number your process picks is not a fence.
• Strictly increasing per lease. Each acquisition of kickoff:ACH must produce a higher number than the last. It need not be globally unique or gap-free — only monotonic within that one lease.
• Carried on every write the lease protects, not just the first.

What does not work as a token: a timestamp (clocks drift, and two machines can disagree by more than your lease), a random UUID (no order, so nothing to compare), a per-process counter (restarts reset it), or an attempt counter maintained by the worker.

Where the check has to happen

In the same statement as the write, as a predicate the database evaluates:

UPDATE account_state
   SET balance_minor = balance_minor - :amount,
       last_token    = :token
 WHERE account_id = :id
   AND last_token < :token;
-- zero rows updated means a newer owner exists

Not as an if in your application. Reading last_token, comparing it in code and then writing is a read-then-write race — the very thing you are trying to prevent, reintroduced one layer up. The comparison and the write must be one atomic operation, which is why it is a WHERE clause and not a branch.

Every protected resource keeps its own high-water mark. The token is meaningless in the abstract; it only means something against the thing that stores it. A column on the account row, a column on the job record, a small table — whatever the lease protects has to remember the highest token that has written to it. If the lease protects three different tables, all three need it.

What a rejected write means

Zero rows updated is not a transient error. It means another worker legitimately owns the lease now, which means you lost it while you thought you had it. Abort immediately: do not retry, do not re-acquire and continue from where you were, and do not attempt to compensate — the new owner is already handling this work, and a compensation would undo their correct write.

It is also worth alerting on. A rejected fence is proof that two workers genuinely overlapped. It should be rare, and a steady trickle usually means your TTL is shorter than your real worst-case work.

How you actually get a token

One rule governs all of this: the token must come out of the acquisition itself, in the same atomic operation that grants the lease. Acquire the lease and then fetch a number separately, and the two can interleave — you are back where you started.

From a lease table. The counter lives on the row and the acquisition returns it, using the INSERT ... ON CONFLICT ... RETURNING fence_token shown in section 3. Note the CASE in it: a renewal must keep the same token; only a change of holder increments it. Get this wrong and you fence yourself out — you renew, the number goes up, and a write still in flight with the old number is rejected by your own resource.

From a Postgres sequence, if you would rather not reason about the counter: SELECT nextval('fence_seq') as part of the acquiring transaction. Sequences are monotonic and gap-tolerant, which is all a fence needs. One sequence per lease name keeps the numbers small and readable.

From ZooKeeper. Create an ephemeral sequential node under the lock path:

create("/locks/kickoff-ACH/req-", EPHEMERAL_SEQUENTIAL)
  -> "/locks/kickoff-ACH/req-0000000412"

You hold the lock when your node has the lowest sequence number, and that number is your token — assigned by the ensemble, strictly increasing, impossible to forge.

From etcd. Acquire with a transaction that only succeeds if the key does not exist, then take the revision that write produced:

Txn().If(CreateRevision(key) == 0).Then(Put(key, me, WithLease(leaseID)))
-> response header revision, or the key's mod_revision

Revisions are cluster-wide and monotonic, so they fence naturally. The same call attaches a TTL lease, so expiry and token come from one operation.

From DynamoDB. The conditional update both grants and increments:

UpdateItem(
  Key: {pk: "lease#kickoff:ACH"},
  UpdateExpression: "SET #owner = :me, expires = :exp ADD fence :one",
  ConditionExpression: "attribute_not_exists(pk) OR expires < :now OR #owner = :me",
  ReturnValues: "UPDATED_NEW")

ReturnValues is the part that matters — it hands back the new counter in the same round trip.

From a Kubernetes Lease object. spec.leaseTransitions increments whenever the holder changes, which is exactly a fence; metadata.resourceVersion works too if you only need ordering.

From Redis. There is no native token. INCR followed by SET NX is two commands and not atomic, so the honest version is a Lua script that acquires and increments together and returns the counter. At that point, notice what you have built: a counter guarded by a store whose failover can lose both the lock and the counter. If you find yourself writing that script, ask whether this lock should live somewhere that fences for free.

Carrying the token afterwards. Hold it in the job's context and pass it to every write the lease protects. Two habits to avoid: re-reading the token from the lease table partway through the work — you might read the new holder's number and cheerfully fence yourself back in — and computing it locally from anything, including an attempt counter. The number you were handed at acquisition is the only one you may use until you acquire again.

When the resource cannot be fenced

Sometimes the thing you write to has no concept of a token: a payment rail, a partner API, an email provider. You cannot fence it, and no amount of locking fixes that. Two honest fallbacks, both already in this pack:

• An idempotency key at that boundary, derived deterministically, so a duplicate write from a zombie worker is recognised as a replay rather than a second instruction (R3).
• Reconciliation afterwards, for whatever still slips through (R6).

The third option, where available, is to funnel all writes through a component that can fence — a service that owns the row and applies the token — so the un-fenceable system is only ever called from one place.

Fencing without a lock at all

Strip the lease away and what remains — a monotonic number checked in the WHERE clause — is just optimistic concurrency control: a version column, compare-and-set, retry on conflict. Worth noticing, because if your protected work is a single write rather than a long multi-step job, you usually do not need the lock at all. The version check alone gives the same safety, and the loser simply retries.

Put the other way round: the lock only earns its place when you need exclusive time, not just a safe write. The fence protects the write; the lease is there to stop two workers duplicating minutes of effort.

Mistakes worth recognising

• Storing the token but never comparing it — a fence that is written and never read is decoration.
• Comparing it in application code instead of in the write predicate.
• Using a timestamp or UUID and calling it a fence.
• Fencing the first write of a job but not the later ones, so the zombie is blocked at step one and free at step four.
• Retrying after a rejected write, which turns a caught overlap back into a real one.

Better liveness: session-based locks

ZooKeeper and etcd tie the lock to a session rather than to a timeout you guessed. When the session drops, the lock vanishes. That is closer to the database's "I own the connection" guarantee than a TTL is, and it is the main reason to prefer them when correctness matters — along with the free fencing token.

The alternatives, which are usually better than the lock

Most of the time you do not need a distributed lock at all, and naming these before reaching for one is what separates a considered design from a reflex.

A unique constraint. The database can only accept one row with a given primary key. If both copies try to insert the same key, one wins and the other gets a conflict. That is mutual exclusion, decided by the database rather than by you. The idempotency insert in R3 works exactly this way.

A conditional update. UPDATE job SET state = 'running' WHERE state = 'pending' either changes one row or zero. The copy that gets one row owns the work; the copy that gets zero knows someone beat it. No lock service, and no way for both to think they won.

A database transaction with a row lock, if the protected state is in one database. All three free properties from section 2 come back.

Partitioning. If each account's work only ever lands on one consumer, one shard or one region, there is no contention to resolve. Nothing shared, nothing to lock. This is why single-writer-per-account (R10) and keying by account (R9) are architectural answers to problems people try to solve with locks — and it is what a Kafka consumer group does for you: one partition, one consumer, exclusivity without a lock anywhere.

A queue per key. Route all of one account's work to one queue and process it in order. You have serialised by where the work goes rather than by making things wait.

Idempotency plus reconciliation. Stop trying to prevent the second execution and make it harmless instead (R3, R6).

When none of those fit, you need a real lock. The cases that survive this filter are usually singleton jobs — the kick-off job for a rail, a reconciliation run for a date (R7), a nightly sweep — or taking over work that a dead process left half-finished.

6. Consensus: how a cluster agrees on a leader and a log

Section 3 said a lease needs one authority that every participant sees identically. Follow that requirement one step further and it turns back on itself.

If the authority is one machine, it is a single point of failure. When it dies, nobody acquires anything and every singleton job in the estate stops. So you replicate it. And the moment the lock service is three machines rather than one, those three have to agree among themselves about who holds kickoff:ACH — which is the very problem you were using them to solve.

That inner agreement is consensus: a group of nodes behaving like one reliable machine by agreeing on the same ordered log of operations, even while some of them fail. etcd, ZooKeeper and Consul are lock services only because they are consensus systems first. A lock service is only as trustworthy as the agreement protocol underneath it.

You have already seen what happens without it. The Redis failover in section 4 — the primary grants your lock, dies before replicating, and a promoted replica grants it again — is replication without consensus. Nothing made the replicas agree before the primary said yes.

What kind of failure it survives

The first question about any agreement protocol is what it assumes a broken node does.

Every node in your own data centre is yours, so the threat is crashes and pauses, not betrayal. CFT is the right model for a bank's internal coordination, and everything below is CFT.

Raft: roles and terms

Raft is the protocol to know, because it was designed to be explained, and most of what you run today uses it.

Every node is in one of three states:

Time is divided into terms: a number that only ever goes up, acting as a logical clock. Every message carries the sender's term. A node that sees a higher term than its own adopts it and steps down to follower; a message carrying a lower term is rejected as stale. Hold onto that rule — it comes back at the end of this section.

Leader election

1. Followers expect regular heartbeats from the leader — empty AppendEntries messages.
2. A follower that hears nothing within its election timeout assumes the leader is gone. It increments the term, becomes a candidate, votes for itself and sends RequestVote to everyone else.
3. A candidate that collects votes from a majority of the cluster becomes leader and starts heartbeating at once, which resets everyone else's timers.
4. Each node votes at most once per term. If two candidates split the vote and neither reaches a majority, both time out and try again in a new term.

The timeout is randomized, typically 150–300 ms per node, and that small detail is what makes elections converge. With identical timeouts every follower would become a candidate at the same instant, split the vote, and do it again forever. With random ones, one node almost always wakes first and wins before the others start.

Notice the same root problem as section 4: a follower cannot tell a dead leader from a slow one. Raft does not solve that. It makes being wrong safe — a spurious election costs a term, and the term rule stops the old leader doing harm.

Before reading further, play with it. Let a leader emerge, then crash it and watch the followers' timers run out, one become a candidate, and the votes arrive. Watch what the old leader's term is when it comes back.

Log replication

Electing a leader is half of it. The other half is making every node hold the same sequence of operations.

1. Clients write only to the leader. The leader appends the entry to its own log and sends AppendEntries to all followers in parallel.
2. Once the entry is stored on a majority, it is committed. The leader applies it to its state machine — "kickoff:ACH is now held by pod-7" — and only then acknowledges the client.
3. Each AppendEntries carries the index and term of the entry just before the new one. A follower whose log does not match at that point rejects it, and the leader steps back one entry at a time until the logs agree, then overwrites the follower's divergent tail. This is the log matching property: if two logs agree at one index and term, they agree on everything before it.

sequenceDiagram
  participant C as Client
  participant L as Leader
  participant F1 as Follower 1
  participant F2 as Follower 2
  C->>L: acquire kickoff:ACH
  L->>L: append to log
  L->>F1: AppendEntries
  L->>F2: AppendEntries
  F1-->>L: stored
  Note over L: 2 of 3 hold it, committed
  L-->>C: granted
  F2-->>L: stored, late

The part people miss: commit means majority, not all. The slow follower does not delay the client, and a crashed one does not block the cluster. That is the whole reason to pay for consensus rather than synchronous replication to every copy.

The safety rules

Elections and replication alone are not enough. A handful of rules close the gaps, and each one prevents a specific way to lose a committed write.

Quorum arithmetic

A Raft cluster needs a live majority to elect a leader or commit anything, so its size decides how many failures it survives.

A cluster of N tolerates ⌊(N−1)/2⌋ failures. Two things fall out of the table. Even sizes buy nothing: four nodes survive one failure, the same as three, while adding a node that can fail. And bigger is not free: every write waits for a majority to acknowledge, so seven nodes are slower than five. Five is the usual production answer — it survives one planned upgrade and one unplanned failure at the same time.

A minority partition stops; it does not split the brain.

Cut a five-node cluster into two and three. The side with three still has a majority and carries on. The side with two cannot elect a leader or commit, so it refuses writes. That is consensus choosing consistency over availability during a partition — the CP side of the trade R0 describes — and for a lock service it is exactly what you want. Two halves each granting the same lease is the outcome the whole system exists to prevent.

Terms are fencing tokens

Put the term next to section 5's three properties of a valid fence. It is issued by the authority. It strictly increases. It travels on every message. A term is a fencing token, and Raft uses it as one internally: a deposed leader that wakes from a GC pause and sends AppendEntries with term 7 is rejected by followers already at term 8. That is the zombie worker from section 4, defeated inside the cluster by the mechanism this section recommends outside it.

The consensus stores expose the same monotonic number to you:

Consensus fences the cluster's own leader; it cannot fence your worker. Inside etcd, a paused leader is harmless. Your application worker holding an etcd lease can still freeze past expiry and wake up writing. So the revision you were handed has to be carried to the ledger row and checked in the WHERE clause, exactly as section 5 describes. What consensus buys you is that the number is genuine — it survived a failover rather than being reissued by a promoted replica that never heard of it.

Paxos and ZAB, in one line each

• Paxos is the original crash-fault consensus protocol: proven correct, notoriously hard to implement in full, and used for a replicated log as Multi-Paxos — Google's Chubby is built on it. Raft is roughly equivalent in safety and performance; it won on understandability.
• ZAB (ZooKeeper Atomic Broadcast) is ZooKeeper's own leader-based protocol, with epochs playing the role of Raft's terms.

Who runs Raft

etcd, and therefore every Kubernetes cluster. Consul and Vault. CockroachDB and TiKV, which run a Raft group per range of data. Kafka's KRaft mode, which replaced ZooKeeper for Kafka's own metadata (R9 covers Kafka). Hazelcast's CP subsystem, behind FencedLock.

The trap to name: consensus is for small, critical metadata, not your payment volume. Every committed write costs a majority round trip through one leader. That is fine for who holds a lease, who is leader, and what the configuration is. It is the wrong place for every ledger posting — the databases that do use Raft for data split it into many independent groups for exactly this reason.

The defensible position: "I would not implement consensus, I would use it. When the lease itself must survive a node failure, it belongs in etcd or ZooKeeper, where a majority has to agree before I am told I hold it, and the revision I get back is a fencing token I carry to the data."

7. Real implementations

Worth being able to name a few, because "have you used one?" follows quickly.

Read them with section 6 in mind: Chubby, ZooKeeper, etcd and Hazelcast's CP subsystem run a consensus protocol underneath; Redis and the database-backed options rely on a single primary.

Google Chubby is the origin of the field: a lock service built on Paxos, used inside Google for master election in GFS and Bigtable. Its paper is also where fencing tokens come from, under the name sequencers.

ZooKeeper, usually through Apache Curator's InterProcessMutex. Ephemeral sequential znodes, lowest number wins, each waiter watching only its predecessor. HBase, Solr and older Kafka used it for controller and master election.

etcd, through its concurrency API — and this is the one you use daily without noticing. Kubernetes leader election (kube-scheduler, kube-controller-manager, most operators) runs on it, via Lease objects and client-go's leaderelection package.

Redis — SET NX PX directly, or Redisson's RLock in Java, which adds a watchdog that renews your lease while you work. Redlock is the multi-master variant, and the Kleppmann/antirez exchange about whether it is safe is the canonical debate in this area.

Hazelcast's FencedLock is the interesting one to cite: Raft-backed, and lock() hands you an explicit fencing token. A lock library that admits the problem in its API.

AWS DynamoDB Lock Client — a Java library implementing the lease table pattern on DynamoDB, with heartbeats and owner IDs, productised.

And the ones closest to a Spring stack: ShedLock, which exists purely so a @Scheduled job runs on one instance only, backed by a JDBC table, Redis or Mongo; Spring Integration's JdbcLockRegistry and RedisLockRegistry; and Quartz in clustered mode, which uses plain SELECT ... FOR UPDATE on its own tables. That last one is the honest summary of enterprise practice: most real "distributed locks" in a bank are a row in a database.

8. Choosing

9. Rules that keep it working

• Lock the smallest thing. Per account, per job key — never one global lock. A global lock quietly turns a distributed system into a serial one, and hides the fact that you never partitioned.
• Set the TTL against your worst case, not your average. Too short and leases expire during normal work, which produces exactly the split-brain fencing exists to catch. For variable work, extend the lease as you go rather than picking one huge number — the same reasoning as the SQS visibility timeout in R9.
• Try not to hold a lock across a network call. Someone else's slowest response becomes your lock duration. Commit a marker, release, and let whoever picks the work up next re-acquire.
• Release in a finally, but never depend on it. Correctness comes from expiry and fencing. Tidy shutdown is a nicety.
• Take multiple locks in a fixed global order, or do not take multiple locks.
• Add jitter to retries, or every waiter wakes at the same instant when a lease drops.
• Measure takeovers. Lock wait time, hold time against TTL, failed acquisitions, and how often a lease expired without being released. A rising takeover count means your TTL is too short or your work has slowed — an early warning for something else.
• Test the ugly case. SIGSTOP the holder for longer than the TTL, let another worker take over, then resume the first and check that its writes are rejected. Until you have run that, you do not know whether your fencing works.

10. What to say in the interview

"I would try not to need a distributed lock — a unique constraint and a conditional update cover most of it, and partitioning by account removes the rest. Where I genuinely need one, I use a lease with an expiry and a fencing token, because a lease can expire while the holder is still alive. The token gets checked at the resource, so a zombie writer is rejected rather than trusted. Redis is fine for stopping duplicate work; it is not what I would put between a customer and a double debit."