R0 — Foundations: what a distributed system can and cannot promise

Read this before anything else. Every section that follows is an answer to a problem stated here.

flowchart LR
  A["Exactly-once DELIVERY<br/>impossible — two generals"] -.-> B["Exactly-once PROCESSING<br/>possible inside one<br/>transactional system"]
  B -.-> C["Exactly-once EFFECT<br/>at-least-once + dedup<br/>+ idempotent ops"]
  C --> D["Residue handled by<br/>reconciliation"]

flowchart TD
  X["Message sent, no acknowledgement"] --> Y{"Retry?"}
  Y -->|"yes: at-least-once"| D["Possible duplicate<br/>receiver recognises and drops it"]
  Y -->|"no: at-most-once"| L["Possible loss<br/>no trace anywhere, found days later"]
  D --> OK["Recoverable"]
  L --> BAD["Not recoverable"]

The story, in one page

In an ideal world, correctness would be easy. A message would be delivered exactly once, processed exactly once, and its effect applied exactly once. There are three semantics for message delivery — at-most-once, at-least-once, and exactly-once — and in a perfect system, exactly-once delivery is the standard.

We do not live in that world. At any moment a machine can go down or a network connection can break, and because a distributed system is many components each doing their own piece of work, each one fails independently and unpredictably.

To account for those failures, systems are built with recovery mechanisms — primarily, replaying failed requests. Because of those retries, duplicate messages become common, and we are shifted into an at-least-once reality whether we chose it or not. That is the first thing to accept: duplicates are not a defect in your design, they are the cost of being able to recover at all.

But what happens when a system cannot recover even with retries? If a failure is unrecoverable, any partial work already done needs to be undone.

This was never an issue when everything was handled by a single system. In a single database we use a transaction to guarantee atomicity — the property that a group of connected operations either all happen, or none of them do. Undo is free, because the database does it for you.

Achieving that same atomicity becomes incredibly difficult when the connected operations are scattered across different machines. You need mechanisms that can orchestrate operations, undo them when a single node fails, and guarantee a consistent result across the board. That requirement is exactly what leads to distributed transactions and protocols like two-phase commit.

Fundamental concepts and principles

Everything in this pack leans on this vocabulary, and interviewers grade the precision of it. Read it once now; each item is expanded where it is first needed.

The three delivery guarantees

Everything a messaging system can offer falls into three buckets.

There is no fourth option, and the third one is not available at the level of delivery. Every real system picks at-least-once and then does something about the duplicates.

Delivery, processing and effect: three different claims

The phrase "exactly-once" gets used for three different things. Interviewers grade this distinction, so it is worth being precise.

Read the table left to right and the trap is obvious: the claims sound the same, the scopes are completely different. Kafka's transactional consume-process-produce is genuine exactly-once processing, and it stops at Kafka's edge — the moment your handler also writes to Postgres or calls a rail, you are back to building exactly-once effect by hand.

Say it in that order and the rest of your answer follows: delivery cannot be exactly-once, so I design for exactly-once effect.

ACID, and what it is a property of

The four guarantees a single database gives you inside one transaction:

Two things to be clear about early. ACID is a property of a transaction, not of a system — so "we need ACID across our microservices" is a sentence that usually has not been thought through. And inside one database all four are cheap and reliable; the moment an operation spans two databases, or a database and a broker, none of them stretches across the gap on its own. R2 is what that costs, R8 is the property you quietly lose first.

CAP and PACELC

CAP is usually mis-stated as "pick two of three". Partitions are not something you pick — the network will drop messages whether or not you planned for it. The real statement is narrower: when a partition happens, you choose between consistency and availability. Keep answering and risk disagreeing, or stop answering and stay correct.

PACELC covers the normal day. Partitions are rare, and CAP says nothing about the rest of the time, when the network is working. PACELC reads as a sentence: if there is a Partition, choose Availability or Consistency; Else — no partition — choose Latency or Consistency. That second half is the one you feel every day: keeping copies in agreement means waiting for other machines to confirm each write, and that wait is latency even when nothing is broken.

Where the main options sit:

R2 works through why that makes 2PC the expensive corner.

Synchronous and asynchronous

Not a style choice — it decides who waits, who knows about whom, and how failure spreads.

Synchronous means the caller blocks until it gets an answer: an HTTP or gRPC call. The caller must know the callee, both must be up at the same instant (temporal coupling), and the answer changes what the caller does next.

Asynchronous means the sender hands the message to a broker or queue and carries on. The two never have to be up simultaneously, the broker absorbs bursts, and adding a second receiver costs the sender nothing.

The trap to name: asynchronous does not mean "no answer" — it means the answer arrives as another message, so somebody has to correlate it, time it out, and decide what to do when it never comes. That bookkeeping is the real cost, and most of R4 and R5 is about paying it well.

Commands and events

The distinction that most bad event-driven designs get wrong.

A command says "please do this." An event says "this happened."

You send a command to one specific service, and it can refuse or fail. You announce an event to whoever is listening, and nobody can refuse it, because it is already true in the publisher's database.

The test when you are unsure: if nobody subscribed, would publishing it still be correct? For an event, yes — the fact stands. For a command, no — the work would simply not happen.

And the point people miss: transport is orthogonal to semantics. Putting "debit the account" on a queue gives you an asynchronous command, not an event. A webhook is an event delivered over HTTP. Choose the meaning first, then the transport. R4 takes this further, into why events are the thing that must never be lost.

Loss and duplicates: the asymmetry that decides every design

Duplicates are recoverable. Loss is not.

A duplicate arrives, the receiver recognises it, and throws it away. Cleaned up in one line of logic.

A lost message leaves no trace. Your side is committed and correct. The other side simply never heard. Nothing in either system knows anything is missing, and you find out days later from a reconciliation break or a customer.

Given a forced choice between the two failure modes, you pick duplicates every time. That single preference explains why at-least-once is the default, why retries are aggressive, and why every consumer has to be built to see the same thing twice.

What "throw it away" actually costs to build. Recognising a duplicate means remembering what you have already handled, and there are three usual implementations, in increasing order of strength:

• A UNIQUE constraint, which is the strongest because the database arbitrates. Insert the message ID or business key; a conflict is the duplicate check, and it is atomic with the work in the same transaction. This is the inbox table in R4 and the idempotency key in R3.
• A conditional write in a key-value store — DynamoDB's attribute_not_exists, or Redis SET key NX — where the store you already run does the arbitration.
• A distributed cache of keys, typically Redis with a TTL. Fast and cheap, but it is a cache: an eviction or a failover can lose a key, and losing a key means the duplicate gets processed. Fine as a first-line filter in front of a durable check, dangerous as the only check for money.

The rule that decides between them: the deduplication record must commit atomically with the effect it protects. If the two can diverge, you have moved the problem rather than solved it.

The principles that follow

Everything above collapses into a short list. These are the sentences to have ready, because each one is a defensible position rather than a fact to recall.

1. Prefer duplicates over loss, always. A duplicate is recognised and dropped in one line of logic. A lost message leaves no trace on either side and surfaces days later. Given a forced choice between the two failure modes, take the one you can clean up.
2. Therefore design for at-least-once delivery. Not as a compromise — as the deliberate choice that follows from principle 1. Retries are aggressive by default, and every consumer is built to see the same thing twice.
3. Therefore make effects idempotent, not deliveries unique. You cannot make the network deliver once. You can make the second delivery change nothing.
4. Keep the atomic part inside one database. Where two writes must agree, move the second one into the first system rather than trying to coordinate across both.
5. Fail locally where you can. A failure a component retries itself never becomes a distributed problem. Escalate only when the component genuinely cannot recover alone.
6. Unknown is not failed. A timeout is an absence of information. Do not report success, do not report failure, and do not blindly retry — record it and resolve it by comparing records.
7. Never order by wall-clock timestamp across machines. Order by a sequence the system itself issues.
8. Verify continuously; do not assume. Correctness is not asserted at write time, it is confirmed afterwards from independently produced records.

If you only keep one, keep the first. Principles 2, 3 and 8 are all consequences of it, and most of this pack is the engineering required to make it true.

The system we will keep coming back to

Rather than describe failures in the abstract, the whole pack uses one small topology. It is deliberately ordinary — a fan-out, some parallel work, a join, and an endpoint — because every failure worth understanding shows up in a graph this simple.

flowchart LR
  S(["start"]) --> A["1.1"]
  S --> B["1.2"]
  A --> T["2"]
  T --> C["3.1"]
  T --> D["3.2"]
  T --> E["3.3"]
  C --> F["4.1"]
  D --> F
  E --> G["4.2"]
  F --> Z(["Done"])
  G --> Z

Four features of it do all the work later:

• start fans out into two independent branches. 1.1 leads onward; 1.2 is a dead end — work that must happen but feeds nothing downstream. That asymmetry is what makes partial completion possible.
• 2 is a fan-out point. One inbound message becomes three outbound ones, which is why a replay aimed at one consumer inevitably reaches all three.
• 4.1 is a join. It needs both 3.1 and 3.2 before it can proceed, so a single missing input leaves it waiting indefinitely while looking perfectly healthy.
• Done can be read as an external system — a payment rail, a partner API — which is where the interesting failures stop being yours to fix.

Before reading further, play with it. The panel below runs ten failure scenarios on this exact graph, each narrated step by step. Watch scenario 1 to see the shape of the happy path, then jump to 7 and 8 — both end with a system that reports itself entirely healthy and is not. Everything this pack argues for exists because of what those two show. You can pause at any point and step through with Prev and Next.

Ten scenarios on one topology. Each is a failure this pack goes on to answer, and together they are the argument for everything after.

Scenarios 7 and 8 are the ones worth watching twice. Both end with a system that looks entirely healthy and is not — which is the whole reason the second half of this pack exists.

With those in mind, the rest of this section works through the first half of the chain: why exactly-once delivery is impossible, what you get instead, how anything stays ordered when no two machines agree on the time, and which failure mode you deliberately choose to live with. R2 picks it up at atomicity.

The two generals problem: why that third row is empty

Two armies on opposite hills must attack together. They can only communicate by messenger, and the messenger might be captured. The first general sends "attack at dawn." Did it arrive? He needs an acknowledgement. The acknowledgement might also be captured — so the second general needs an acknowledgement of the acknowledgement. There is no number of messages that ends this.

You have lived this without the armies. You text someone something that matters — "I've transferred the money, go ahead and pay the vendor" — from a train with one bar of signal, and no read receipts. Nothing comes back. Did the message fail to send? Did it arrive and their "sounds good" die on the way back? Send it again and they might pay twice. Leave it and they might never pay at all. You cannot tell from where you are standing, and no further message makes the uncertainty go away.

Your services are those generals. A service sends a request, the receiver processes it, and the response is lost. The sender now cannot distinguish two situations:

• the receiver never got it, or
• the receiver got it, did the work, and the reply died on the way back.

Whatever the sender does next is wrong in one of those two cases. Retry, and you might do the work twice. Give up, and you might not do it at all. The ambiguity cannot be removed — it can only be moved somewhere you can deal with it.

Ordering without a shared clock

A related question, and the one senior interviewers reach for next: if two machines disagree about the time, how does anything agree on what happened first?

Wall clocks cannot answer it. NTP drift, leap seconds and virtualised clocks mean two servers' timestamps can differ by more than the interval you are trying to order. Sorting events by created_at across services is a bug that looks like a design.

The answer is logical clocks, which order events by causality rather than by time. A Lamport clock is a counter each node increments on every event and sends with every message; the receiver takes the maximum of its own and the incoming value, plus one. That gives a total order consistent with causality — if A caused B, A's number is lower — though equal-ish numbers do not prove independence. Vector clocks carry one counter per node, which is heavier but can also tell you when two events were genuinely concurrent rather than ordered, which is what conflict detection in Dynamo-style stores needs.

In practice you rarely implement either by hand. You inherit their descendants: a Kafka partition offset is a per-partition Lamport clock, an etcd revision is a cluster-wide one, a database version column is a per-row one, and the sequence field on an event in R4 exists for the same reason. Spanner's TrueTime takes the other route — it bounds clock uncertainty with atomic clocks and GPS, then waits out the interval, which is why a hyperscaler can offer external consistency and you cannot.

The practical rule: never order by timestamp across machines. Order by a sequence the system itself issues, and treat wall-clock time as a human-readable annotation.

Identity without a coordinator: generating unique IDs

A payment request lands on one of forty instances of the payment service. Before anything is written, the payment needs a name — the ID the ledger rows, the outbox event, the rail message and the customer's receipt will all carry. On one database that name is free: an auto-increment column hands out the next number. Spread the writes across shards and regions and that counter becomes a coordinator — every insert waits on it, it is a single point of failure, and in a multi-region setup it puts a cross-region round trip on every write.

So the problem is the same one as the previous section, from the other side. Machines that do not share a clock also do not share a counter, and the ID has to be minted locally without asking anyone.

What a good ID has to be

The third property is the one people underrate. A random key inserts into a random leaf page of the index, so the working set is the whole index: page splits, cache misses, write amplification. A time-ordered key always inserts near the newest page, which stays in memory. On an append-heavy table like a ledger, that is the difference between an index that keeps up and one that does not.

The options

Read the table as two families. Random IDs (v4) buy perfect independence and pay for it in size and index locality. Time-prefixed IDs (ULID, UUIDv7, KSUID, Snowflake) put the timestamp first so that the ID sorts roughly by creation, and fill the rest with something that keeps two IDs from the same instant apart — randomness in ULID, a machine number and a counter in Snowflake.

The Snowflake layout

Snowflake is the one interviewers ask you to draw, because the whole design fits in one line.

| 1 sign | 41 timestamp (ms since custom epoch) | 10 machine id | 12 sequence |

Uniqueness comes from the combination, not from any one field: two IDs from different machines differ in the machine bits, and two IDs from the same machine in the same millisecond differ in the sequence. The sequence is what makes the scheme safe, not the clock. The timestamp only decides which bucket of 4,096 you are in.

Try the generator below. Mint several IDs quickly and watch the sequence field climb within a single millisecond, then reset when the millisecond ticks over; notice that the timestamp bits are the only part that moves the ID forward across time.

Tuning the bit split

The 1/41/10/12 split is a default, not a law. The bits are a budget, and you move them to where your system needs them.

The custom epoch is part of the same budget: start it at your launch date rather than 1970 and the 69 years start counting from when they are useful.

The trap: the clock goes backwards

A Snowflake generator reads the wall clock — the thing the previous section told you not to trust.

If NTP steps the clock back by a few milliseconds, the generator re-enters milliseconds it has already used, and the same timestamp-plus-sequence pairs come out again. That is a duplicate primary key, and on a payment table it is either a failed insert or, worse, two payments that look like one.

The defences are simple, and worth naming:

• Remember the last timestamp you used. If the clock now reads earlier, refuse to mint, or wait until it catches up. Never quietly continue.
• Exhaust the sequence, then wait. If 4,096 IDs are used inside one millisecond, block until the next millisecond rather than wrapping the counter.
• Make machine IDs genuinely unique. Two workers sharing a machine ID will collide on every millisecond they share. Assign them deliberately — never from something that two pods can both pick. The next two subsections cover how.

How machines get their IDs

Everything above assumed each generator already knows its 10-bit machine ID. That number is the one piece of coordination Snowflake cannot avoid, so the real design question is who hands it out, and how you know no two live workers hold the same one.

There are five ways to answer it, and they trade operational simplicity against how well they survive autoscaling.

The systems that describe theirs publicly cover most of the table:

The interviewer then asks three follow-ups, and each one maps to a row:

• "What stops two instances getting the same worker ID?" — The assignment has to be a single atomic decision in one place: a conditional write, a ZooKeeper node, a unique constraint. Anything an instance works out on its own (a random number, a hash of the hostname) can collide.
• "When can a worker ID be reused?" — Not the moment its owner dies. Only after the old owner can no longer be minting with it, plus a margin longer than any clock skew, so the new owner cannot re-enter milliseconds the old one used.
• "You run on ECS Fargate or plain autoscaling groups — there are no ordinals and the IPs are recycled." — Then derived IDs are out, and you need a lease: from ZooKeeper or etcd if you run one, or from the database you already have.

If none of that is worth it, change the ID, not the coordination: UUIDv7 or ULID need no machine ID at all, at the cost of 128 bits instead of 64.

The line to say: "The machine ID is the only coordination in Snowflake. I would lease it, not configure it — from etcd if we already run it, otherwise from a lease table in the database we already have — with a cooldown before reuse and a worker that stops minting if it cannot renew. If 128 bits is acceptable, UUIDv7 removes the problem."

The database lease table, in depth

For a payment service that already runs on Postgres, the lease table is usually the right answer: no new infrastructure, one SQL statement to claim, one to renew. It is the same lease R8 builds for a distributed lock, with one difference — instead of one named lock, it is a pool of 1,024 leases, one per worker ID, and a starting task takes whichever one is free.

The schema

CREATE TABLE worker_lease (
  worker_id  smallint    PRIMARY KEY CHECK (worker_id BETWEEN 0 AND 1023),
  owner      text,                                   -- task ID, NULL when free
  epoch      bigint      NOT NULL DEFAULT 0,         -- +1 on every claim
  expires_at timestamptz NOT NULL DEFAULT '-infinity'
);

INSERT INTO worker_lease (worker_id)
SELECT g FROM generate_series(0, 1023) AS g;        -- seeded once, never deleted

Three design choices carry the whole scheme:

Claiming

UPDATE worker_lease
   SET owner = :task, epoch = epoch + 1,
       expires_at = now() + interval '30 seconds'
 WHERE worker_id = (
         SELECT worker_id FROM worker_lease
          WHERE expires_at < now() - interval '10 seconds'   -- expired AND cooled down
          ORDER BY expires_at
          LIMIT 1
          FOR UPDATE SKIP LOCKED)
RETURNING worker_id, epoch;

• ORDER BY expires_at takes the row that has been free the longest, so a just-released ID is the last to be reused.
• FOR UPDATE SKIP LOCKED is what makes concurrent start-ups safe and fast: if twenty tasks start together, each locks a different row instead of queuing behind one.
• No row returned means the pool is exhausted. The task must fail start-up loudly. Falling back to a random or default worker ID is exactly how duplicates are born.

Renewing, and fencing yourself

The task renews every 10 seconds — a third of the TTL, so two renewals can fail before the lease is in danger:

UPDATE worker_lease
   SET expires_at = now() + interval '30 seconds'
 WHERE worker_id = :id AND owner = :task AND epoch = :epoch;
-- 1 row:  still mine
-- 0 rows: someone else owns this ID now — stop minting, claim a new one

A renewal that succeeds even after the lease has technically expired is still safe: a matching epoch proves nobody claimed the ID in the gap.

The hard case is not a task that dies — it is a task that pauses and wakes up.

A long GC pause or a network partition stops renewals while the process keeps running. R8 solves this for locks with a fencing token that the protected resource checks. Here there is no such resource: IDs are minted in memory, and nothing downstream can reject a write because it carries an old epoch. So the worker has to fence itself:

1. Before sending each renewal, note the time on the monotonic clock (System.nanoTime(), never wall time).
2. When the renewal succeeds, set a minting deadline of that start time plus 20 seconds — comfortably inside the 30-second TTL.
3. Every mint checks the deadline. Past it, minting throws until a renewal succeeds again.

Because the deadline is measured from before the request was sent, the worker always stops minting before the database could consider the lease expired, however long the request or the pause took. The cooldown then adds a further 10 seconds before anyone else can claim the ID. Together they guarantee that the old and new owners of worker 7 are never both minting, and never minting into the same milliseconds.

Why the cooldown exists

Expiry is judged on the database clock, but the IDs are stamped with each task's own clock. If the old owner's clock ran 3 seconds fast, its last IDs carry timestamps 3 seconds in the future. A new owner that started minting the instant the lease expired could produce those same timestamps under the same worker ID. The cooldown must be longer than the worst clock skew you tolerate between tasks, and that skew limit should be enforced — a task whose clock is more than a few seconds off from the database's now() at claim time should refuse to start.

What happens when the database is down

Size the pool so it never runs dry: 1,024 IDs must exceed peak running tasks plus everything sitting in expiry and cooldown after a deployment or a crash loop. Alert when free IDs fall below 10%.

Step through the full lifecycle below — the claim, a race between two start-ups, renewal, a crashed task, a paused one that wakes up as a zombie, and a graceful release. Watch the epoch column: it is the thing that turns the zombie's renewal into zero rows.

The Spring sketch

@Component
public class WorkerIdLease implements SmartLifecycle {
    private static final long SAFE_NANOS = Duration.ofSeconds(20).toNanos(); // TTL is 30 s
    private final JdbcTemplate jdbc;              // CLAIM_SQL, RENEW_SQL, RELEASE_SQL: the statements above
    private final String task = System.getenv().getOrDefault("TASK_ID", UUID.randomUUID().toString());
    private final ScheduledExecutorService timer = Executors.newSingleThreadScheduledExecutor();
    private volatile int workerId = -1;
    private volatile long epoch;
    private volatile long mintUntil;               // System.nanoTime() deadline
    private volatile boolean running;

    public WorkerIdLease(JdbcTemplate jdbc) { this.jdbc = jdbc; }

    @Override public void start() {
        claim();                                   // fail start-up if the pool is empty
        timer.scheduleAtFixedRate(this::renew, 10, 10, TimeUnit.SECONDS);
        running = true;
    }

    private record Lease(int id, long epoch) {}

    private void claim() {
        long sent = System.nanoTime();
        Lease got = jdbc.query(CLAIM_SQL,
                rs -> rs.next() ? new Lease(rs.getInt(1), rs.getLong(2)) : null, task);
        if (got == null) throw new IllegalStateException("no free worker ID");
        epoch = got.epoch();
        mintUntil = sent + SAFE_NANOS;
        workerId = got.id();                       // published last: minting can start
    }

    private void renew() {
        long sent = System.nanoTime();
        try {
            if (workerId >= 0 && jdbc.update(RENEW_SQL, workerId, task, epoch) == 1) {
                mintUntil = sent + SAFE_NANOS;
            } else {
                workerId = -1;                     // lost it: never resume with the old ID
                claim();
            }
        } catch (RuntimeException e) {
            // database down or pool empty: keep the old deadline, minting stops on its own,
            // and the next tick tries again (an escaping exception would cancel the schedule)
        }
    }

    /** Called by the Snowflake generator on every mint: one volatile read and a clock check. */
    public int workerIdForMinting() {
        int id = workerId;
        if (id < 0 || System.nanoTime() - mintUntil > 0) throw new WorkerIdUnavailableException();
        return id;
    }

    @Override public void stop() {
        running = false;
        int id = workerId;
        workerId = -1;                             // stop minting first
        timer.shutdownNow();
        if (id >= 0) jdbc.update(RELEASE_SQL, id, task, epoch);
    }

    @Override public boolean isRunning() { return running; }
}

Give the renewal a statement timeout shorter than the renewal interval, so a hung connection cannot hold the scheduler past the next tick. When the worker ID changes after a lost lease, the generator starts a fresh sequence space — the new ID makes the old millisecond buckets irrelevant.

The same pattern on DynamoDB or etcd

The defensible position: lease worker IDs from a pool in the database you already run, claim with one conditional update judged on the database clock, renew at a third of the TTL with an epoch check, stop minting on a local monotonic deadline before the lease can expire, and wait out a cooldown longer than clock skew before any ID is reused.

How this ties back to ordering

A time-ordered ID is tempting to use as an ordering key. Resist it. Within one machine, Snowflake IDs strictly increase. Across machines they are ordered by each machine's own clock, which is exactly the ordering the previous section ruled out. The timestamp inside an ID is for index locality and rough sorting, not for deciding what happened first. Where order matters — events on one payment, entries in one account — use a sequence the system issues, as principle 7 says.

IDs, idempotency keys and payments

Three different things get called "the ID" in a payment flow, and conflating them is a real bug.

The part people miss: an idempotency key is not an ID you generate on arrival. If the server mints a fresh payment ID on every attempt and treats that as the deduplication key, every retry becomes a new payment. The key has to exist before the first request leaves the client, so that the retry carries the same one. R3 covers how keys are scoped, stored and derived.

Two more consequences for the design:

• Sequential keys are kind to one B-tree and cruel to a range-sharded table. Every new write lands on the newest range, so one shard runs hot while the rest sit idle. R10 covers choosing a shard key that does not inherit this.
• Time-ordered IDs leak. A Snowflake or ULID reveals when it was created, a v1 UUID reveals the machine, and consecutive numbers reveal your volume. For identifiers you hand to customers or partners, decide whether that matters before you expose the internal key.

The defensible position: payment IDs are 64-bit Snowflake-style or 128-bit time-ordered IDs, minted inside the service with no central coordinator; idempotency keys come from the client and survive retries; and ordering comes from a sequence the system issues, never from the timestamp inside an ID.

Where the story continues

Each beat of the narrative above becomes a section of its own.

The pack falls into seven parts. Part I, the foundations (R0–R1), is the vocabulary and the parts: what can and cannot be promised, and the services and APIs every later diagram is drawn with. Part II, correctness across machines (R2–R8), is one argument, and reading it in order is the point: atomicity is unavailable, so you accept duplicates (R3) in order to eliminate loss (R4), and you undo by compensation rather than rollback (R5). R6 and R7 exist because none of that is ever complete, and R8 is the other property you lost — isolation. Part III, data and messaging (R9–R12), is where events travel and where state lives: the broker, the store, how to make it fast, and the cache in front of it. Part IV, scale and operations (R13–R16), is what running the result actually takes — volume, failing dependencies, knowing when it silently broke, and changing it without stopping it. Part V, the general design rounds (R17–R22), is where those mechanisms first get used: infrastructure first, then the product systems an interviewer reaches for outside payments, grouped by the engineering concern they share — stateful servers, reservations, large content, high-volume data flow, and lookup. Part VI, low-level design (R23–R24), prepares the object-oriented round: principles, patterns and ten case studies. Part VII, payments (R25–R28), turns the general mechanisms into rails, cutoffs and postings, and ends with the payments round itself.

What to say in the interview

"Exactly-once delivery isn't available — the two generals problem rules it out — so I design for at-least-once delivery and exactly-once effect, and I reconcile the residue. That single choice explains the rest: duplicates are recoverable and loss isn't, so I accept duplicates deliberately and then make them harmless."