R6 — Reconciliation: detecting the divergence nothing prevented

Outbox, idempotency and sagas are preventive — they reduce the chance that two systems diverge on any single operation. None of them detects divergence that has already happened. Three residues always remain:

flowchart TD
  L[("Your ledger<br/>what you intended")] --> M["Match on end-to-end reference,<br/>then amount + currency + date + counterparty"]
  RA[("Rail file / camt.053<br/>what actually moved")] --> M
  BK[("Bank / core account<br/>what you hold")] --> M
  M -->|matched| OK["Reconciled"]
  M -->|"in ledger, not in rail"| B1["Sent but not settled — age it, re-query"]
  M -->|"in rail, not in ledger"| B2["Money moved unrecorded — highest severity"]
  M -->|"amount differs"| B3["Fees, FX, partial — rule-based auto-repair"]
  M -->|"timing only"| B4["Cutoff crossing — resolves next cycle, never pages"]

• The unresolved unknown. A call times out at a boundary you do not control. You probe once, the answer is still unknown, and the ambiguity from the two generals problem is never resolved by messaging — only by comparing records afterwards.
• The boundary that will not cooperate. A third party with no idempotency keys, no status endpoint, and no transaction you can join. You cannot make their side atomic with yours, at any price.
• Bugs and operations. A manual data fix, a bad deploy, a replayed topic, a message dropped by a system you believed was safe. No pattern protects against a mistake in the pattern's own implementation.

So the general principle, which holds well beyond payments: in a distributed system, correctness is not something you assert at write time. It is something you verify continuously, from records produced independently.

flowchart LR
  subgraph PREVENT["Preventive — per operation"]
    A["Outbox: do not lose it"]
    B["Idempotency: do not repeat it"]
    C["Saga: undo it cleanly"]
  end
  subgraph DETECT["Detective — across the population"]
    D["Reconciliation: compare two<br/>independent records, classify<br/>every difference, drive it to zero"]
  end
  PREVENT -->|"residue: unknowns,<br/>uncooperative boundaries, bugs"| DETECT
  DETECT --> E["Repair automatically,<br/>or escalate to a human"]

This is not a payments-only concern. Anywhere one fact is represented in two places, the two drift: an order service against a warehouse system, billing against entitlements, a search index against its source database, a CDC pipeline against the tables it tails, a cache or read model against the ledger that owns the truth, a replica against its primary. The shape is identical — two records of the same fact, produced by different paths, compared on a schedule.

Independence is the requirement that makes it work. The comparison must not run through the code that produced the data, or a bug cancels itself out on both sides and the check proves nothing. Read the source of truth directly, parse the other party's record separately, and compare.

Comparing millions of rows without melting the network. Fetching both sides in full and diffing works until it does not. The standard escalation is hashing: bucket both datasets by a shared key — account, date, shard — hash each bucket's contents, and compare hashes rather than rows. Only buckets whose hashes disagree get opened, so a clean day costs almost nothing. Merkle trees take that one step further by arranging the hashes into a tree, so a single differing row is located in logarithmic time rather than by scanning the bucket; this is how Dynamo-style stores and Cassandra run anti-entropy repair between replicas. Two practical requirements: both sides must bucket and canonicalise identically — same rounding, same field order, same null handling — or every bucket differs for no reason, and the comparison should run where the data is (a batch job beside each store, exchanging hashes) rather than pulling everything to one place.

It converts hope into a number. "Eventually consistent" without a reconciler is an assumption. With one, you have a measurable position: how many records disagree, for how long, and how much value sits in the gap. That turns into an SLO with an owner, which is why reconciliation lag is a better top-line alert than any system metric.

The interview line. When you are asked "how do you know your system is correct?", preventive patterns are the wrong answer — they are how you try to be correct. Reconciliation is how you know. This section covers the concepts; R7 covers building one.

First, the record being compared

You cannot prove two records agree without being clear what your own record is. The essentials, condensed from the ledger model:

Why double-entry is engineering, not accounting. A single balance column has no memory: a double charge, a bad partial refund, and nothing in the data model objects. In double-entry every event is a set of entries whose debits equal credits, and an unbalanced posting is rejected at write time. That gives conservation (money is never created), auditability (history replays), and a runtime invariant you can assert continuously.

CREATE TABLE ledger_transaction (
  id              bigserial PRIMARY KEY,
  idempotency_key uuid UNIQUE,         -- the R3 key, at the record layer
  posted_at       timestamptz NOT NULL
);

CREATE TABLE ledger_entry (
  id             bigserial PRIMARY KEY,
  transaction_id bigint NOT NULL REFERENCES ledger_transaction,
  account_id     bigint NOT NULL,
  direction      text NOT NULL,   -- debit | credit
  amount_minor   bigint NOT NULL  -- integer minor units, never float
);

Four rules carry the design. Entries are append-only — a posted entry is never updated, and corrections are reversing entries. Amounts are integers in minor units, because floats lose cents at scale. Every transaction is idempotent by key. And accounts are typed, so a debit increases an asset and decreases a liability, which is what makes the balance equation checkable.

Pending versus posted. Authorisations and reservations need a pending state: debit customer:available, credit customer:pending. On capture, pending becomes settled; on expiry, it reverses. The customer-facing "available balance" is available minus pending — and that pending row is the semantic lock from R5, visible in the product.

Fast balances, and the hot spot. Summing entries stops working past a few million rows, so keep a materialised balance per account updated in the same transaction as the entries, with a version column for optimistic locking. That balance row then becomes your contention point, which is the scaling problem R13 picks up.

The invariant to name: the sum of all debits equals the sum of all credits, checked continuously. If that check ever fails, you stop the line.

Scenarios 7, 8 and 9 in that widget are this section's argument: a message lost with no trace, an outcome that stayed unknown at an external boundary, and the sweep that finds both of them a day later.

Reconciliation itself

Reconciliation is how you find out that money is wrong. It is the payments-specific answer to "how do you know your system is correct", and at manager level it is a stronger answer than any uptime metric.

The three-way match. Your ledger says what you intended and recorded. The rail's settlement file or statement says what actually moved. The bank or core account says what you hold. Reconcile ledger against rail, and ledger against bank. Matching only two of the three leaves a blind spot.

Matching. Match on your own end-to-end reference first, since you put it on the payment for this purpose. Fall back to a tuple of amount, currency, date window and counterparty. Expect fuzziness: rails truncate references, batch several payments into one settlement line, and net fees out.

Break classes and what each means.

Auto-repair versus queue. Encode the known, safe patterns (fee netting, rounding, known cutoff drift) as rules that post correcting entries automatically. Everything else goes to a human queue with full context. The metric that matters is not break count but break age: how long value sits unexplained.

Make it an SLO. "95% of payments reconciled within one settlement cycle; no break older than 48 hours." Reconciliation lag as an alerting signal is the single most credible thing you can say in an observability discussion, because it detects money loss rather than machine failure.

Scale. At volume this is a batch join over millions of rows against files that arrive on the rail's schedule, not yours. Partition by settlement date and rail, keep the previous cycle's unmatched items in the window, and make the whole run idempotent and replayable — you will rerun it after a fix, and it must not double-post corrections.

See R7 for how to build the system itself.

The same idea one layer down: replicas that drift

Everything above is reconciliation between records your business produced. The database underneath has the same problem with its own copies, and it solved it first.

Take a store with three replicas of every row — Cassandra, DynamoDB, Riak. A write arrives while one replica is restarting. The other two accept it, the client gets its acknowledgement, and the third replica comes back holding the old value. Nothing failed. Nothing was lost. One fact now lives in three places, and they disagree.

That is exactly the shape from the top of this section — two records of the same fact, produced by different paths — and the store answers it with the same move: compare, classify, repair. Read repair, anti-entropy and Merkle trees are the storage-layer twin of business reconciliation. It is worth knowing them for their own sake, because interviewers ask, and worth knowing them for the one thing they cannot do, because that is the argument for everything above.

flowchart TD
  subgraph STORE["Storage layer — copies of one record"]
    RA["Replica A"] <-->|"read repair,<br/>anti-entropy"| RB["Replica B"]
    RB <--> RC["Replica C"]
  end
  subgraph BIZ["Business layer — independent records"]
    L["Your ledger"] <-->|"matching,<br/>break classes"| F["Rail statement"]
  end
  STORE -.->|"agree with each other"| X["Says nothing about<br/>agreeing with the rail"]

First, knowing that two copies differ — and which one wins

Two questions come before any repair: are these copies different, and if so, which is right?

Detecting a difference. The cheap way is a hash or checksum: each replica hashes its copy, they swap hashes, and equal hashes mean equal data without shipping the data. The precise way is version metadata on every value. Vector clocks (and their refinement, dotted version vectors) carry one counter per node, which lets the store tell "B overwrote A" apart from "A and B were written concurrently and neither saw the other". R0 covers how logical clocks work; here they are the thing that tells a store whether it has a stale copy or a genuine conflict.

Resolving it. Once two copies disagree, something has to pick.

LWW does not resolve a conflict. It deletes one side of it.

The trap to name: LWW looks like a resolution strategy and behaves like data loss under concurrency. Two writes inside the clock-skew window, and the one stamped by the slower clock disappears — no error, no log line. That is tolerable for a "last seen" timestamp and never tolerable for a balance, which is one more reason the ledger is append-only and ordered by a sequence it issues itself, never by wall-clock time.

Read repair: fixing the copy you just touched

Read repair runs on the read path, which is what makes it proactive. In Cassandra at consistency level two:

1. The coordinator asks replica A for the full data and replica B for only a digest — a hash of its copy.
2. If the hash of A's data equals B's digest, the copies agree, and the coordinator returns the result.
3. If they differ, it fetches B's full data, picks the newest value (or merges), returns it to the client and writes it back to the stale replica.

The client gets the right answer, and the stale copy gets fixed as a side effect of someone asking. The price is latency: a higher consistency level consults more replicas on every read, and the read waits for the slowest of them.

Before reading further, play with it. Make one replica stale, issue a read, and watch the coordinator notice the digest mismatch and write the newer value back. Then notice which rows never get repaired.

The rows that never get repaired are the ones nobody reads. Read repair only touches data that is read, so a cold row can sit wrong on one replica indefinitely, until a read finds it or a replica holding the good copy is lost and the bad copy is all that remains.

Anti-entropy: sweeping everything in the background

Anti-entropy covers the rows read repair never sees. It runs in the background, independent of traffic, which is why it is described as passive.

Periodically a node picks another, often at random, and the two exchange state until both hold the union of their data, with conflicts resolved by whatever rule the store uses. Repeat across the cluster and every replica eventually converges. The run has three steps:

The comparison step is where the cost lives. Comparing by timestamps can mean moving everything. A checksum plus a list of recent updates is cheaper but misses older drift. A Merkle tree is the most efficient of the three, which is why serious stores use it.

Real systems run both. Read repair keeps hot data correct at read time; anti-entropy guarantees that everything else converges eventually.

Merkle trees: finding what differs without shipping it

A Merkle tree is a hash of hashes. Each leaf is the hash of one record or block. Each parent is the hash of its two children concatenated. The single hash at the top, the root, is a fingerprint of the entire dataset.

To compare two replicas, compare their roots. Equal roots mean the datasets are identical, and you are done after exchanging one hash. Different roots mean something differs, so compare the two children and descend only into the side whose hashes disagree. A single changed block in a million is found in O(log n) comparisons rather than a million.

Before reading further, play with it. Change one record on one replica, then compare the trees and watch the comparison descend only into the subtrees whose hashes disagree. Count how many hashes were exchanged against how many records exist.

The cost is on the write side: every update re-hashes the path from that leaf to the root. That is why stores maintain the tree per range or partition and often build it at repair time rather than continuously. The structure is everywhere once you look — Dynamo, Riak and Cassandra repair, Git, blockchains, Certificate Transparency logs.

What the database's reconciliation cannot tell you

Put the three mechanisms side by side with the business reconciliation from earlier in this section.

The last row is the whole argument. Storage reconciliation makes the copies agree with each other. It has no opinion on whether they agree with the world. If your service posts a wrong entry, all three replicas will hold that wrong entry, read repair will find their digests identical, anti-entropy will find their Merkle roots identical, and every check at the storage layer will pass.

A perfectly converged database can hold a ledger that disagrees with the rail statement, and nothing below the business layer will ever notice.

This is the independence requirement from the top of this section, seen from below. Replicas are copies produced by the same write path, so a bug in that path is faithfully replicated. The rail's statement is produced by someone else's system, which is precisely what makes comparing against it worth anything.

Borrowing the Merkle idea for a large reconciliation

The comparison technique transfers even though the guarantee does not. Earlier in this section, bucket hashing was the escalation for comparing millions of rows; a Merkle-style hierarchy is the same idea made recursive.

The design. Bucket both sides by a shared key — settlement date, then account range, then account. Hash each leaf bucket's canonicalised contents, hash the hashes upward, and compare top-down. A clean day costs one comparison per date. A day with one bad account opens one path of buckets, and row-level matching runs only on the leaf that disagrees.

Where it fits and where it does not.

Two rules keep it honest. Each side's hashes must be computed from that side's own source, read directly — hashing does not weaken independence, but computing both hashes through one shared code path does. And a matching hash proves equality only of what you chose to hash: leave the value date out of the canonical form and a value-date break will pass silently.

This is a technique you can propose, not a claim about how any particular bank runs its reconciliation. Present it as the scaling lever you would reach for once full row-by-row matching stops fitting in the batch window.

The defensible position. Databases reconcile their own replicas — read repair on the read path, anti-entropy in the background, Merkle trees to find what differs cheaply — and you should rely on that for copy-level consistency. It tells you the copies agree. Only business reconciliation, against records someone else produced, tells you they are right.

What to say in the interview

"Prevention is never complete — a timeout whose outcome stayed unknown, a counterparty who won't cooperate, a bug in my own pattern — so reconciliation is how I know the system is correct rather than how I hope it is. I match three ways: my ledger, the rail's statement, and the bank, on my own end-to-end reference first. The comparison runs on an independent path, never through the code that produced the data. And I report it as a number: how many records disagree, for how long, and how much value is sitting in the gap."