R28 — The JPMC senior manager round: design walkthroughs
Everything before this is machinery. This section is the round itself: five systems worked end to end — Venmo, UPI, FX and cross-border, a global payment platform, and a monolith migration — plus the question bank that surrounds them.
Four of the five are written as transcripts, because at this level the value is less in the final architecture than in how you handle the follow-up — the pushback, the trap question, the moment you have to say "I wouldn't do that" and defend it.
What this round is actually testing
At senior manager level the interviewer is not checking whether you know what a queue is. They are checking three things, and you can fail on any one of them while doing the other two well:
- Can you still design? A manager who cannot draw the system loses the room in ten minutes.
- Do you reason about failure and money rather than boxes? Every follow-up will be a failure mode, and the ones that matter are the silent ones.
- Can you zoom out? Cost per transaction, team size, first milestone, what you would buy instead of build, which decisions you would delegate.
A structure that works in 45 minutes, and saying it out loud at the start buys you credit:
- Clarify and scope, 5 min. Who are the actors, what volumes, what latency, which rails, what is explicitly out of scope.
- Capacity arithmetic, 2 min. Peak TPS, storage, and what that forces (R13). Do it aloud.
- The happy path, 10 min. One diagram, one payment, end to end.
- The data model, 5 min. Especially the ledger — errors hide in a bad one (R6).
- Failure modes, 15 min. This is where the round is won: duplicates, timeouts, partial failure, reconciliation.
- Scale and operations, 5 min. Sharding, ordering, observability, deployment.
- The manager layer, 3 min. Team, milestones, buy versus build, what you would measure.
Before you start — the words used in this section
The transcripts below use bank and payments vocabulary the way an interviewer would. This is the short version; R27 carries the full glossary — parties, rails, account types, ledger terms, identifiers, controls and operations. If any of it is new, read this once and the rest will follow.
About money movement
- Rail — a payment network you send through: UPI, ACH, SWIFT, the card networks. You do not control it; you follow its rules.
- Settlement — when the banks actually move the money between themselves. Often hours or days after the customer sees the payment complete.
- Clearing — the messaging and netting that happens before settlement: who owes whom, and how much.
- Irrevocable — once sent, it cannot be pulled back. UPI and instant card payouts are irrevocable; ACH is not.
- Return / chargeback — the money coming back later because the payer's bank reversed it, or the cardholder disputed it.
- Closed loop — both parties hold a balance with you, so a payment is just an entry in your own books and no bank is involved.
- Nostro account — your own bank account held at another bank, usually abroad. "Our money, over there."
- Netting — instead of wiring every payment, you total them up per partner and move the difference once.
- Value date — the date the customer wants the money to move, which may not be today.
- Business date — the accounting day a rail is currently working on. Not midnight, and not your server's date.
- Cut-off — the time a rail stops accepting work for the current window.
References that identify a payment
- Idempotency key — a value the caller sends with a request so that if they retry, you recognise it as the same request rather than a new one.
- UETR — a unique reference that travels with a cross-border payment through every bank that touches it, so anyone can ask "where is it?"
- RRN — the equivalent in UPI: the reference number NPCI assigns, which both banks and we can query by.
- VPA — a UPI address such as
ravi@okbank— a UPI address likename@bank, which resolves to a real bank account.
Engineering terms used loosely elsewhere, precisely here
- Ledger — the record of every movement of money, as paired entries. Not a balances table.
- Double-entry — every event is recorded twice, as a debit somewhere and a matching credit somewhere else, so the two always sum to zero.
- Minor units — the smallest unit of a currency: paise, cents. Money is stored as whole numbers of these, never as decimals.
- Hold — money that is in the account but not spendable, because something has reserved it.
- Saga — a business action split across several services, where each step has a defined way to be undone.
- Compensation — the undo. Not a rollback: you post an opposite entry rather than erasing the original.
- Pivot — the step in a saga after which nothing can be undone, usually the moment money leaves for a rail.
- Outbox — a table you write an outgoing message into, in the same transaction as the business change, so the message cannot be lost.
- CDC (change data capture) — reading a database's own change log to turn row changes into events, without the application publishing them.
- Optimistic locking — before writing, check that the version number you read is still current. If it is not, someone else changed the row and you retry.
- Partial index — an index over only the rows matching a condition, so a query for "everything stuck" stays fast without indexing everything.
- Shard — a slice of the data, held on its own database, so no single machine holds everything.
- Fence token — a number that only goes up, handed out with a lock, which the data itself checks so a stale writer is rejected.
Walkthrough 1 — Venmo, social peer-to-peer payments
Interviewer: Design Venmo.
You: Let me pin down the scope, because "Venmo" covers a few different products. I'm assuming a closed-loop wallet — users hold a balance with us — funded from bank accounts and cards, with a social feed as a first-class feature, instant transfer out as a paid option, and US money-transmitter regulation. For numbers I'll say 100 million users and 15 million payments a day, heavily peaked on Friday evenings.
Interviewer: Good. Where do you start?
You: With the observation that shapes everything else: most Venmo payments never touch a bank. If both parties hold a balance with us, a payment is two entries in one database transaction — debit the sender's available balance, credit the receiver's. One system, one commit. That is why it feels instant, and it means the interesting problems are not in the middle of the system, they are at the edges where money enters and leaves.
Here is the shape I'd draw — and it is worth walking rather than describing, so click any box for what it owns, and any concept underneath to see which components implement it.
Click a component
Every box below is a candidate microservice, a platform capability, or something you buy rather than build.
Concepts applied, and where
Interviewer: Why is the ledger in the middle rather than a balances table?
You: Because a balance column has no memory. A double charge or a bad partial refund leaves no evidence in it. Double-entry gives me conservation, an audit trail, and a runtime invariant I can assert continuously — sum of debits equals sum of credits. That is R6's argument, and in a money-transmitter business it is also a regulatory one.
Interviewer: Take me to the edges then. What goes wrong there?
You: There are two edges, money in and money out, and they fail in opposite directions.
Funding in — the risk is reversal after the fact
- Balance-funded is instant and safe. The money is already ours to move, so there is nothing to wait for.
- Card-funded settles quickly but carries chargeback exposure for months, and card acceptance costs are the highest of the three.
- ACH-funded — a bank transfer in the US, which takes days — is slow and, more importantly, reversible for days. A payment funded by ACH can be clawed back after the receiver has already spent the money — so the loss lands on us, not on either user.
- The choice is therefore a credit-risk decision, not an engineering one: either hold the funds until the ACH clears, or take the risk deliberately and price it. What I should not do is pretend the risk isn't there.
- Whichever we choose, the ledger has to model it: a
pendingversusavailablebalance, which is R5's semantic lock made visible in the product.
Cashing out — the risk is that it cannot be undone
- Standard ACH out is free and slow, and gives us a window in which a mistake is still recoverable.
- Instant transfer pushes to a debit card over the card rails for a fee, and it is irrevocable — once it lands, nothing brings it back.
- So all the risk work moves before the send: limits, velocity, device and account-takeover checks, screening. That is R25's rule about irrevocable rails, showing up inside our own product rather than on an external scheme.
- And it is where the fraud actually concentrates, because an attacker who takes over an account wants the money out, fast and unrecoverably. Instant transfer is the exit they aim for.
Interviewer: Show me what the ledger actually does across those two edges.
You: Four cases, and the postings say more than the prose does — money in with a hold, the return that lands the loss on us, and the two ways out.
The thing I would draw attention to is the middle column of the journal: every case is at least two postings, and one of them is about our own money rather than the customer's. A single balance column cannot express that, which is why the ledger is double-entry from day one rather than something we retrofit.
Interviewer: Show me the schema.
You: Five tables. Three are the ledger itself, one is the read optimisation, and one is the idempotency boundary.
-- 1. The chart of accounts. Every party and every internal pot is an account.
CREATE TABLE ledger_account (
id bigserial PRIMARY KEY,
name text NOT NULL, -- 'customer:8842:available'
owner_type text NOT NULL, -- customer | clearing | bank | revenue
owner_id bigint, -- null for internal accounts
currency char(3) NOT NULL,
normal_side text NOT NULL CHECK (normal_side IN ('debit','credit')),
status text NOT NULL DEFAULT 'open',
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (name, currency)
);
-- 2. The transaction: one real-world event, any number of legs.
CREATE TABLE ledger_transaction (
id bigserial PRIMARY KEY,
idempotency_key uuid NOT NULL UNIQUE, -- same key as the API request (R3)
kind text NOT NULL, -- p2p | ach_in | ach_out | instant_out | fee | reversal
reverses_id bigint REFERENCES ledger_transaction(id), -- set on corrections
external_ref text, -- ACH trace number, card auth code, UETR
posted_at timestamptz NOT NULL DEFAULT now(),
business_date date NOT NULL, -- not the same as posted_at (R26)
metadata jsonb
);
-- 3. The entries. Append-only. This table is never UPDATEd and never DELETEd.
CREATE TABLE ledger_entry (
id bigserial PRIMARY KEY,
transaction_id bigint NOT NULL REFERENCES ledger_transaction(id),
account_id bigint NOT NULL REFERENCES ledger_account(id),
direction text NOT NULL CHECK (direction IN ('debit','credit')),
amount_minor bigint NOT NULL CHECK (amount_minor > 0), -- paise; never a float
currency char(3) NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
) PARTITION BY RANGE (created_at);
CREATE INDEX ON ledger_entry (account_id, id); -- statement queries
CREATE INDEX ON ledger_entry (transaction_id); -- fetch a whole posting
-- 4. The read optimisation: current balance per account, updated in the
-- same transaction as the entries, guarded by a version for optimistic locking.
CREATE TABLE account_balance (
account_id bigint PRIMARY KEY REFERENCES ledger_account(id),
balance_minor bigint NOT NULL DEFAULT 0,
version bigint NOT NULL DEFAULT 0,
last_entry_id bigint, -- lets a rebuild verify itself
updated_at timestamptz NOT NULL DEFAULT now()
);
-- 5. Holds, so 'available' and 'current' can differ. This is R5's semantic lock.
CREATE TABLE account_hold (
id bigserial PRIMARY KEY,
account_id bigint NOT NULL REFERENCES ledger_account(id),
amount_minor bigint NOT NULL CHECK (amount_minor > 0),
reason text NOT NULL, -- ach_settling | risk_review | card_auth
expires_at timestamptz NOT NULL, -- a hold with no expiry is a bug
released_at timestamptz,
transaction_id bigint REFERENCES ledger_transaction(id)
);
Interviewer: Walk me through it.
You: In one sentence: accounts are the buckets, a transaction is one real-world event, entries are the movements, the balance row is a cache, and holds are why "available" differs from "current".
ledger_account — every party and every internal pot. Naming is a convention that pays for itself: customer:8842:available, clearing:ach_in, revenue:fees. owner_type and owner_id let me find all of a customer's accounts without parsing strings.
The column that does real work is normal_side. An asset increases on a debit; a liability increases on a credit. Without it, "debit ₹500" is ambiguous — it could mean the balance went up or down. With it, every entry has an unambiguous effect, and the code that applies entries needs no special cases per account type.
UNIQUE (name, currency) enforces something I'd state explicitly: an account holds exactly one currency. Mixing them in one bucket makes the balance meaningless, so a multi-currency wallet is several accounts, and an FX conversion is a transaction with legs in two of them plus an FX position account.
ledger_transaction — one row per real-world event, however many legs it has. Four columns worth defending:
idempotency_key UNIQUEis the same key the API request carried. Putting the constraint here rather than only at the API means the ledger itself refuses a double posting — the guarantee doesn't depend on the service in front behaving.reverses_idis how corrections work. You never update or delete an entry; you post a new transaction that points at the one it reverses. History gains a row rather than losing one.external_refis the join key to the outside world — the ACH trace number, the card auth code, the UETR. This is the column reconciliation matches on (R6), and without it a three-way match falls back to fuzzy amount-and-date guessing.business_dateis deliberately separate fromposted_at. A payment posted at 23:58 may belong to the next business date on that rail, and every report is cut by business date, not by timestamp (R26).
ledger_entry — the movements, and the table that is never updated and never deleted. Three decisions in it:
amount_minor bigint, always positive. Paise, not rupees, because floats lose money and mixed units cause rounding arguments. The sign is never in the amount — direction carries it, which removes a whole class of double-negative bugs when reversing.directionas a check constraint rather than a boolean, becauseis_debit = falsereads badly at 3am.currencydenormalised onto the entry even though the account has it. It's redundant on purpose: it lets a validation query catch a leg written against the wrong-currency account without a join.
The indexes follow the two questions you actually ask: (account_id, id) for "show me this account's statement in order", and (transaction_id) for "show me every leg of this posting". Partitioning by created_at keeps both inside a couple of partitions.
account_balance — a cache, and I'd call it that plainly. One row per account, which makes it the hot row for a busy account, and that contention is the thing that eventually forces batching or a queue per account (R13). version gives optimistic locking; last_entry_id records how far the cache has consumed, so a rebuild can replay from a known point and compare rather than trusting itself.
account_hold — the table of money that is present but not spendable, which is what makes "available" a different number from "current". reason matters because holds come from different places and are released by different things — an ACH settlement, a risk review, a card authorisation expiring. expires_at is mandatory, since a hold whose saga died is money locked forever.
And two things deliberately absent. There's no balance column on ledger_account, because a balance that lives next to its own definition invites someone to update it directly. And there's no status on ledger_entry — an entry is a fact that already happened, so it has no lifecycle. Payment status lives on the payment, not on the ledger.
Interviewer: What enforces that debits equal credits?
You: Three things, deliberately layered, because this is the invariant the whole design rests on.
The application posts all legs in one transaction, so a partial posting cannot commit. A deferred constraint trigger — a database check that runs at commit time rather than per row — checks the sum per transaction at commit time and aborts if it is non-zero — that is the real guard. And separately, a continuous job asserts the global invariant, SELECT SUM(CASE WHEN direction='debit' THEN amount_minor ELSE -amount_minor END) FROM ledger_entry, which must be exactly zero. If that ever returns non-zero, we stop the line.
The third one might look redundant given the second. It is not — it catches the bug in the trigger, the manual INSERT someone ran during an incident, and the restore that replayed half a partition.
Interviewer: How do you serve a balance in single-digit milliseconds?
You: From account_balance, never by summing entries — that stops working somewhere around a few million rows. The rule is that the balance row updates in the same transaction as the entries, with the version column for optimistic locking:
UPDATE account_balance
SET balance_minor = balance_minor - :amount,
version = version + 1,
last_entry_id = :entry_id
WHERE account_id = :id AND version = :seen_version;
-- zero rows → someone else moved it; re-read and retry
And because it is derived, it is rebuildable: last_entry_id lets a rebuild job replay from a known point and compare, which is the same independent-verification idea as reconciliation.
Interviewer: Available versus current balance?
You: current is the balance row. available is current minus unreleased, unexpired holds. I would expose available to the customer and keep current for accounting, which is exactly why a banking app shows you two numbers.
The two operational rules: every hold has an expiry, because a hold whose saga died is money locked forever, and a sweeper releases expired ones and alarms on anything it finds — a hold expiring unexpectedly means a saga failed silently.
Interviewer: How does this table grow, and what do you do about it?
You: ledger_entry is the biggest table you will own and it is append-only, so split the table into monthly chunks by created_at — this is partitioning — and never delete. Detach old partitions to cheaper storage rather than dropping them, because the retention requirement is regulatory rather than technical. Statement queries hit (account_id, id) and stay inside a couple of partitions.
The one thing I would resist is archiving entries out while keeping balances, because then the balance is no longer derivable from history and you have quietly lost the ability to prove it.
Interviewer: A user taps "Pay" twice because the app was slow. What happens?
You: Nothing, if I have done R3 properly. The client generates an idempotency key before the first attempt and reuses it on retry, scoped per user rather than globally — (client_id, key) — so one tenant's key can never collide with another's or return their stored response. The second request finds the key, sees the work is finished, and returns the original response byte for byte.
The detail I would add is that the key record commits in the same transaction as the ledger entries. If the key is stored separately, the key can say "done" while the payment never happened.
Interviewer: Now tell me about fraud and money laundering. Surely you don't just move the money.
You: No — and in a closed-loop wallet it matters more than on a bank rail, because the transfer is instant and irrevocable and there is no settlement window to claw back in. Every control has to run before the ledger entries. I would build three layers with quite different shapes:
| Layer | When | What it does |
|---|---|---|
| KYC / CIP | Onboarding | Identity verification, device fingerprinting, sanctions screening — plus batch re-screening whenever a watchlist changes |
| In-path risk | Synchronous, 50–100ms budget | Velocity and limits, fraud score from device, behavioural and graph features, instrument risk, watchlist check for new or external counterparties |
| AML monitoring | Asynchronous, off the event stream | Structuring detection, mule-ring network analysis, long-window aggregation, alert queue, SAR filing within deadlines |
Interviewer: How do you score a payment in 100 milliseconds if the features need 30 days of history?
You: You don't compute them in the request. The expensive work moves off the read path and onto the write path, where it is amortised — consumers of the outbox stream from R4 maintain a feature store — a small, fast store holding pre-computed numbers about each account of pre-computed aggregates, and the request does a single key-value lookup rather than a scan. It is the same read-model idea as the social feed, applied to risk.
Let me show the two paths and where the milliseconds actually go.
Three points I'd make from that. Streaming gives freshness, batch gives correctness — the nightly backfill recomputes long windows from the ledger and resets the drift, and the difference between the two is a reconciliation check in its own right (R6). Counter updates have to be idempotent, because the stream is at-least-once and a redelivered event must not increment twice (R4). And feature freshness is a risk metric, not an ops metric — if the aggregator is lagging, the fraud defences are quietly degraded, so consumer lag belongs on the risk dashboard as well as the platform one (R15).
Interviewer: What about a brand-new account with no history at all?
You: Thin file — an account with almost no history — and the mistake is to default the missing features to zero — because zero reads as "quiet, well-behaved account", which is exactly what a fraudster's fresh account looks like. Missing has to be its own modelled value, with the model trained on it. In practice a new account routes to a different policy: tighter limits, more step-up authentication, and a slower path to instant transfer. Thin-file is a risk state, not an absence of data.
Interviewer: And if a payment hits a sanctions match?
You: It parks; it does not fail. Two reasons. Legally I cannot tip off the customer, so the UI says "under review" rather than "flagged". And operationally the money needs to sit somewhere while a human reviews it — which is a hold, R5's semantic lock: visible to the rest of the system as unavailable, with an expiry and a review SLA so it cannot sit forever.
I would also fail closed. If the risk service is down, payments pend rather than flowing through. That is one of the few places where availability loses outright.
Interviewer: You mentioned retries earlier. What happens if a retried payment gets a different risk score?
You: That would be a bug, and a nasty one — the same request allowed once and declined once. So the decision is stored with the payment: the score, the feature values and the model version. A retry reuses the recorded decision rather than re-scoring. It is the same principle as returning the stored response in R3, applied to a decision rather than a result. It also gives me the audit trail a regulator will eventually ask for.
Interviewer: Let's talk about the feed. It shows a payment that the ledger says failed. How?
You: Because the feed is a read model, fed from the outbox, and it is eventually consistent by design. The rule I would state firmly is that the feed is never the source of truth for a balance or a payment status — it reads its own store, and if it disagrees with the ledger, the ledger wins and reconciliation tells me how often that happens.
If the feed is showing something the ledger never confirmed, then somebody published before committing, which is exactly the phantom case in R4. The fix is the outbox, not a patch on the feed.
Interviewer: Scale. 15 million payments a day, one celebrity user with two million followers.
You: Two separate problems. For payments, I shard by user — but a payment touches two users, so a cross-shard payment is either a saga (R5) or a ledger service that owns both entries in one transaction. I would start with the second, because a single ledger service sharded by account is simpler than distributed compensation for something this frequent.
For the feed it is the classic timeline problem: fan-out on write is fast to read and terrible for celebrities, fan-out on read is the opposite. I would start with read-side fan-out, measure, and add write-side fan-out only for the small set of accounts where it pays. Saying "I'd measure first" is the honest answer, because the crossover point is entirely dependent on the follower distribution.
Interviewer: Last question — what would you put in front of an executive?
You: Fraud and chargeback loss as a percentage of volume. Cost per transaction split by funding source, because card funding is expensive and balance funding is nearly free. And the product decision that follows from those two: whether instant transfer fees are what subsidise free ACH, because that is the economic engine of the whole thing.
Walkthrough 2 — UPI, India's real-time rails
Interviewer: You're building the payments side of a UPI app. Walk me through the design.
You: The first thing I would establish is who does what, because a lot of the design is decided for me by NPCI rather than by us.
Click a component
Every box below is a candidate microservice, a platform capability, or something you buy rather than build.
Concepts applied, and where
As a PSP we own the app, the VPA mapping for our own users, the request construction and our own ledger and reconciliation. We do not own authentication or routing — the UPI PIN is captured inside NPCI's own secure library, never by our code, and NPCI routes to the banks.
The flow is: resolve the VPA to an account, send a debit request through NPCI to the remitter bank, the payer authenticates with the PIN, the bank debits and responds, NPCI instructs the beneficiary bank to credit, and both sides return status. Settlement between banks is net and deferred — totalled up and moved later, rather than one wire per payment, in cycles through the day.
Interviewer: What are the properties of this rail that change your design?
You: Four, and I would name them explicitly:
It is irrevocable and instant — there is no chargeback, so all risk work happens before the debit. That is R25.
It is two-legged — the debit and the credit happen at different banks. The credit can fail after a successful debit, which is R0's unknown outcome at national scale.
Everything is keyed by the RRN — the reference NPCI stamps on the transaction — plus my own transaction ID, which is the end-to-end reference that makes status queries and reconciliation possible at all (R6).
And money moves in real time while banks settle net later — so my ledger position and my settlement position are two different things. Conflating them is a classic mistake.
Interviewer: Take the failure you just mentioned. Debit succeeded, credit didn't. What do you do?
You: This is the heart of it, and the wrong answers are tempting. I cannot retry blindly, because I might pay twice. I cannot tell the customer it failed, because the money has already left their account.
So: first, ask. I issue a status check to NPCI keyed by RRN — that is precisely why the reference exists. If the status comes back resolved, the ambiguity is gone and I record the outcome.
If it is still unknown, the payment is parked as indeterminate. Not success, not failure — a recorded third state. Then reconciliation settles it against NPCI's file, and if the credit genuinely did not happen, an auto-reversal returns the money to the payer within the mandated window.
Interviewer: So you're saying the customer waits?
You: The customer gets told the truth: the payment is being confirmed, and the money returns automatically if it did not go through. What I will not do is guess, because both guesses are worse — declaring success when nothing arrived, or declaring failure and letting them pay again when the first one actually landed. That is R0's "unknown is not failed" principle, and the reversal window is the regulator's way of bounding how long unknown is allowed to last.
Interviewer: How do you make sure your own retries don't double-pay?
You: Every request carries my transaction ID, generated once and reused on every attempt — R3. And my side records the attempt before sending, so a crash mid-flight leaves evidence: I know an instruction may be out there, which is what makes the status check possible rather than a guess. Without that durable marker before the call, a crash leaves me unable to tell whether I ever sent anything.
Interviewer: Give me the data model and the API.
You: The shape to hold on to: one row per payment with an explicit multi-state lifecycle, one row per attempt against NPCI, and a separate settlement position — because money moving and money settling are different facts.
- Never a boolean success flag. Every interesting failure lives between states, so the lifecycle is
initiated → debited → credited → settled, withreversedandindeterminateas first-class terminal states rather than error strings. - The attempt is a row, not a retry counter. Each request to NPCI, each status check, each reversal gets recorded — that is the durable marker R3 requires before an external call.
- RRN is the join key to the outside world, so it is indexed and it is what reconciliation matches on (R6).
- Settlement is tracked separately from the ledger, because NPCI nets and settles in cycles while the customer's money moved instantly (R26).
-- One row per payment, with a real lifecycle rather than a success flag.
CREATE TABLE upi_payment (
id bigserial PRIMARY KEY,
client_txn_id text NOT NULL, -- our end-to-end reference
idempotency_key uuid NOT NULL, -- from the client (R3)
payer_account_id bigint NOT NULL,
payer_vpa text NOT NULL,
payee_vpa text NOT NULL,
amount_minor bigint NOT NULL CHECK (amount_minor > 0),
state text NOT NULL, -- initiated | debited | credited
-- | settled | reversed | indeterminate
rrn text, -- NPCI's reference, arrives on response
npci_response text, -- raw code, kept verbatim
business_date date NOT NULL, -- R26
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL,
UNIQUE (payer_account_id, idempotency_key) -- per-client scoping (R3)
);
CREATE INDEX ON upi_payment (rrn);
CREATE INDEX ON upi_payment (state, updated_at)
WHERE state IN ('debited','indeterminate'); -- the sweeper's query
-- One row per interaction with NPCI. Written BEFORE the call goes out.
CREATE TABLE upi_attempt (
id bigserial PRIMARY KEY,
payment_id bigint NOT NULL REFERENCES upi_payment(id),
kind text NOT NULL, -- debit | status_check | reversal
request_id text NOT NULL UNIQUE, -- derived key, reused on retry
sent_at timestamptz NOT NULL,
responded_at timestamptz,
outcome text, -- success | failure | timeout | unknown
raw_response jsonb
);
-- What we owe and are owed per settlement cycle. Not the same as the ledger.
CREATE TABLE settlement_position (
business_date date NOT NULL,
cycle int NOT NULL,
counterparty text NOT NULL,
gross_out_minor bigint NOT NULL DEFAULT 0,
gross_in_minor bigint NOT NULL DEFAULT 0,
net_minor bigint NOT NULL DEFAULT 0,
state text NOT NULL, -- open | cut_off | settled | broken
PRIMARY KEY (business_date, cycle, counterparty)
);
-- VPA mappings we own, for our own users.
CREATE TABLE vpa_mapping (
vpa text PRIMARY KEY,
account_id bigint NOT NULL,
status text NOT NULL, -- active | suspended | deregistered
created_at timestamptz NOT NULL DEFAULT now()
);
Interviewer: Explain the choices.
You: upi_payment is a state machine, not a record with a flag. debited and indeterminate are the two that earn their place: debited means the payer's money is gone and the credit is unconfirmed, and indeterminate means we asked and still do not know. Both are legitimate resting states that a sweeper works through — which is why there is a partial index on exactly those two, since that is the query that runs every minute.
upi_attempt exists so a crash is interpretable. The row is written before the request leaves, which is R3's record-intent-before-the-call rule: if we die mid-flight, the row tells us an instruction may be out there and we can query by RRN rather than guess. request_id is derived deterministically, so a resumed attempt presents the same value and NPCI treats it as a replay.
settlement_position is separate on purpose. The ledger says the customer's money moved instantly; this says what we owe the counterparty at the end of a cycle. Conflating them is the classic mistake, and keeping them apart is what makes the daily reconciliation against NPCI's file meaningful (R6).
npci_response keeps the raw code verbatim rather than a mapped enum, because when a corridor starts failing at 2am you want the actual code, not our translation of it.
Interviewer: And the API?
You: Thin, and idempotent at every mutating endpoint.
| Endpoint | Purpose | Notes |
|---|---|---|
POST /v1/payments |
Initiate a payment | Idempotency-Key header required; returns 202 with a payment ID, never a fabricated success |
GET /v1/payments/{id} |
Current state | The only honest way to learn the outcome; clients poll or subscribe |
POST /v1/payments/{id}/status-check |
Force a status query to NPCI | Internal and ops-facing; rate-limited, since NPCI meters it |
POST /v1/vpa/resolve |
Resolve a VPA to a masked name | Heavily rate-limited — an open resolver is an enumeration tool |
POST /v1/collect-requests |
Payee-initiated request | Per-requester limits and reputation checks; the scam surface |
POST /v1/mandates |
AutoPay mandate | Separate consent model, pre-debit notification |
POST /webhooks/npci/callback |
Async status from NPCI | Signature-verified, idempotent on RRN — duplicates are expected (R4) |
Two API decisions I would defend. POST /payments returns 202 Accepted, not 200 OK — the payment is accepted for processing, and anything else would be claiming an outcome we do not have yet. And the callback handler is idempotent on RRN, because NPCI will redeliver, and R4's whole argument is that we accept duplicates rather than risk loss.
Interviewer: What about collect requests? People get scammed with those.
You: They do, and it is a design consequence rather than a bug. A collect request is payee-initiated — someone asks you for money and you approve with your PIN. Scammers send collect requests dressed as payouts, so NPCI and the PSPs rate-limit them, restrict who can raise them, and the apps have progressively buried the flow. From my side: strict limits per requester, reputation on the requesting VPA, and very deliberate UI wording, because the technical control cannot fix a social attack on its own.
Interviewer: Scale. You're doing a few thousand transactions a second at peak.
You: Shard by payer account, so one customer's payments and their ordering stay together. Key the event stream by account so their events stay in sequence — R9's point that the key is the ordering design. And expect hot keys: a large merchant or a popular biller takes disproportionate volume, so I would sub-key with a bounded fan-out for them rather than let one partition become the bottleneck, which is R10.
Interviewer: Anything that is unusual about operating on this rail?
You: Yes, and it is a good manager-layer point: my own technical decline rate is a regulated metric. NPCI publishes availability and decline expectations, and a PSP with poor numbers gets throttled or worse. So the observability from R15 is not just internal — success rate by bank, response times per leg, and the age of unresolved indeterminate payments are numbers I report externally, not only on a dashboard.
I would also make sure reconciliation runs against NPCI's files daily without exception, because that is the only independent check that my ledger matches what actually settled.
Interviewer: Anything else you would build in from day one?
You: Mandates for UPI AutoPay, because recurring payments need pre-debit notification and a different consent model. And I would keep the ledger multi-state from the start — initiated, debited, credited, settled, reversed — rather than a boolean success flag, because every interesting failure lives between those states.
Walkthrough 3 — FX and cross-border payments
Interviewer: Design the FX and cross-border payment capability for a corporate bank.
You: Scoping first. Corporate clients sending cross-border payments with conversion, quotes held for seconds to minutes, a few tens of currency pairs, and a treasury desk that must not end the day unintentionally exposed. That last constraint is the one that makes this different from every other payments design.
The central point: in FX, a failed payment is not free to undo, because the market moved while you were failing.
Click a component
Every box below is a candidate microservice, a platform capability, or something you buy rather than build.
Concepts applied, and where
Interviewer: Start with the concepts you think matter.
You: Six, and I would name them in this order:
- A quote, not a price. The client gets a
quote_id, a rate and an expiry — seconds for large amounts. Executing against an expired quote is refused rather than silently repriced, because silent repricing is how you end up in a dispute. - Spread and tiering. We quote the market mid plus a spread that varies by client tier, size and pair. That spread is the revenue, so mispricing it is a business incident, not a bug.
- Position keeping. Every executed trade moves our exposure in each currency, and the desk hedges against that number. It has to be real-time and correct — which makes it a ledger problem wearing different clothes: append-only, double-entry, reconciled.
- Nostro and vostro. Nostro is our account at their bank, vostro is theirs at ours. Our account at their bank and theirs at ours. Cross-border settlement is movement between these, which is why liquidity management — enough of the right currency in the right place at the right time — is a first-class concern rather than an afterthought.
- Netting and cut-offs. We do not wire each payment individually; we net per corridor and settle in cycles, against each currency's own holiday calendar and cut-off (R26).
- The message layer.
pacs.008— the ISO 20022 message for a customer credit transfer — over SWIFT for the interbank leg,camt.053— the end-of-day statement message — back for reconciliation, with a UETR threaded through for tracking (R25).
Interviewer: Show me the data model.
You: Five tables, and the interesting one is the third.
-- A quote is a promise with an expiry. It is not a trade.
CREATE TABLE fx_quote (
id uuid PRIMARY KEY,
client_id bigint NOT NULL,
sell_ccy char(3) NOT NULL,
buy_ccy char(3) NOT NULL,
sell_amount_minor bigint, -- one side is fixed, the other derived
buy_amount_minor bigint,
mid_rate numeric(18,8) NOT NULL, -- what the market showed
spread_basis points, where 100 bps is 1% int NOT NULL, -- what we added, kept separately
client_rate numeric(18,8) NOT NULL, -- what the client sees
rate_snapshot_id bigint NOT NULL, -- which tick we priced from
expires_at timestamptz NOT NULL,
state text NOT NULL, -- open | executed | expired | rejected
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON fx_quote (state, expires_at) WHERE state = 'open';
-- The rate tick we priced from, kept so any quote can be re-justified later.
CREATE TABLE fx_rate_tick (
id bigserial PRIMARY KEY,
provider text NOT NULL,
pair char(7) NOT NULL, -- 'USD/INR'
bid numeric(18,8) NOT NULL,
ask numeric(18,8) NOT NULL,
received_at timestamptz NOT NULL,
stale_after timestamptz NOT NULL -- after this, do not price from it
);
-- The trade: an executed quote. This is what moves our position.
CREATE TABLE fx_trade (
id bigserial PRIMARY KEY,
quote_id uuid NOT NULL UNIQUE REFERENCES fx_quote(id), -- executes once
payment_id bigint,
sell_ccy char(3) NOT NULL, sell_amount_minor bigint NOT NULL,
buy_ccy char(3) NOT NULL, buy_amount_minor bigint NOT NULL,
executed_rate numeric(18,8) NOT NULL,
reverses_id bigint REFERENCES fx_trade(id), -- unwind, at the NEW rate
executed_at timestamptz NOT NULL DEFAULT now()
);
-- Live exposure per currency, derived from trades, kept for the desk.
CREATE TABLE fx_position (
ccy char(3) PRIMARY KEY,
net_minor bigint NOT NULL DEFAULT 0,
version bigint NOT NULL DEFAULT 0,
last_trade_id bigint,
updated_at timestamptz NOT NULL DEFAULT now()
);
-- The payment leg, separate from the FX leg.
CREATE TABLE xborder_payment (
id bigserial PRIMARY KEY,
uetr uuid NOT NULL UNIQUE, -- the end-to-end reference (R25)
trade_id bigint REFERENCES fx_trade(id),
corridor text NOT NULL, -- 'IN->US'
state text NOT NULL, -- initiated | fx_done | submitted
-- | acknowledged | settled | returned | indeterminate
correspondent text,
value_date date NOT NULL,
business_date date NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
Interviewer: Explain those choices.
You: fx_quote separates mid_rate, spread_bps and client_rate rather than storing one number. Store only the client rate and you cannot answer "what was our margin on this corridor last month" without reverse-engineering it, and you cannot prove to a client that they were priced fairly. Three columns, one question answered forever.
rate_snapshot_id is the audit trail for the price. Months later, someone will ask why a quote was 83.42. The answer is a specific tick from a specific provider at a specific time, and fx_rate_tick.stale_after is what let us use it — freshness is a property we store, not a hope. If no fresh tick exists, we stop quoting rather than quote from a stale one, because a stale rate is worse than no rate.
fx_trade.quote_id is UNIQUE, and that is the idempotency guarantee (R3). A quote executes exactly once, enforced by the database rather than by the service — so a client retry, a network redelivery or a duplicate click cannot produce two trades at the same rate.
reverses_id is where the FX-specific pain lives. Unwinding a trade is a new trade at the current rate, not a deletion — the same append-only discipline as R6's ledger, except here the reversal has a different economic value from the original. The gap is a realised loss, and it needs an owner.
fx_position is a materialised aggregate with a version, exactly like the balance row in the Venmo model — one row per currency, so it is the hot row, updated in the same transaction as the trade so the desk's number is never behind the book.
And the payment and the trade are separate rows, joined by trade_id. That separation is what lets the payment fail after the FX succeeded, which is the situation that actually happens.
Interviewer: The API?
| Endpoint | Purpose | Notes |
|---|---|---|
POST /v1/quotes |
Request a quote | Returns quote_id, rate and expires_at. Cheap, unauthenticated-ish, high volume |
POST /v1/payments |
Execute against a quote | Body carries quote_id + Idempotency-Key. Refuses an expired quote with 409, never reprices silently |
GET /v1/payments/{uetr} |
Track the payment | UETR is the public identifier, so it is what the client and the correspondent both use |
GET /v1/rates/stream |
Indicative rates | SSE or WebSocket, explicitly labelled indicative — not executable |
GET /v1/positions |
Current exposure per currency | Internal, for the desk; read from fx_position |
POST /webhooks/swift/status |
pacs.002 status in |
Idempotent on UETR; duplicates expected |
Interviewer: Now the failure. The payment fails after the FX leg executed.
You: This is the one worth getting right, and it is R5 with money attached.
The FX trade is the pivot. Once it executes, we hold a real position — we bought dollars and sold rupees, and the market does not care that the payment later failed. So the compensation is a reverse trade at the current rate, and the difference between the two rates is a realised P&L, usually a loss.
Three consequences I would state plainly:
- Compensation is not free here. In every other saga in this pack, undoing a step restores the prior state exactly. In FX it does not, which is why I sequence the FX leg as late as possible — validation, screening, limits and the correspondent's availability all get checked before we execute.
- The loss needs an owner and a threshold. Small slippage is absorbed; past a threshold it goes to the desk as a break with the original and reversing trade attached.
- Repeated reversals are a signal, not noise. If one corridor generates them regularly, something upstream is broken and we are paying for it in market moves, so the reversal rate per corridor is a dashboard metric (R15).
Interviewer: And if the correspondent times out?
You: Indeterminate, exactly as in R0 — probe by UETR, and if the answer is still unknown, park it rather than retrying or failing it. Reconciliation against camt.053 settles it the next cycle (R6). What makes FX worse than a domestic rail is that while the payment sits unknown, the position is real and the exposure is live, so the desk needs visibility of indeterminate payments as well as settled ones.
Walkthrough 4 — A global payment processing system
Transcript style. The interviewer's questions are the ones actually asked in this round; the answers show where each part of this pack gets used.
Interviewer: Design a global payment processing system. I'm interested in how you handle network partitions, geo-redundancy, and how you'd ensure ACID compliance across your distributed databases.
You: Before I draw anything, can I check a few assumptions, because they change the design a lot? I'm assuming three regions — India, EU, US — multi-currency, 24/7 with no maintenance window, peak around 50k TPS, and data residency rules that mean Indian payment data can't leave India. And I'd want an RPO and RTO I'll be held to: zero RPO within a region, seconds across regions, RTO under a minute.
Interviewer: That's fair. Take those.
You: Then I want to challenge one phrase in the question, because it's the crux. You can't have ACID across regions for the same account, and I wouldn't try. Cross-region consensus costs a round trip per commit — 30 to 80 milliseconds between continents — and two-phase commit across regions blocks exactly when the network is worst, which is the argument in R2: a participant that has voted yes has surrendered its right to decide, so a coordinator failure leaves it holding locks.
So my principle is: keep ACID strictly local, make the unit of locality the account, and everything global is routing, replication and reconciliation.
Interviewer: So you're telling me you can't do it.
You: I'm telling you I can give you the guarantee that matters — that a debit and its ledger entries are atomic and isolated — by making sure both always happen inside one database. What I won't do is stretch a transaction across an ocean. Let me show you the shape.
Click a component
Every box below is a candidate microservice, a platform capability, or something you buy rather than build.
Concepts applied, and where
Each cell is a complete vertical stack — API, orchestrator, ledger shard, rail adapters, its own reconciliation. Every account is homed to exactly one cell, and that mapping lives in a global ownership registry, something like a DynamoDB global table. Writes for an account always go to its home cell. Reads can be served anywhere from replicas, except read-your-own-writes, which goes home.
Interviewer: Why homing rather than just replicating everything everywhere?
You: Three reasons. One writer per account means real ACID locally — one transaction, one database, no coordination. Residency falls out for free, because Indian accounts homed to the India cell means Indian data never leaves without me building a separate mechanism. And the blast radius of anything going wrong is one cell, which is also how I'd deploy, using R16's approach of shifting one slice at a time.
Interviewer: Right. Now partition me. The India cell loses its link to the rest of the world. What happens?
You: I'd answer that per operation rather than globally, because the honest answer differs:
| Operation | Behaviour | Why |
|---|---|---|
| Debit or credit on a locally homed account | Keeps working | Writer, ledger and idempotency store are all inside the cell |
| Balance read for a remote account | Stale from replica, labelled stale | Availability is fine while nobody acts on it |
| Debit on a remotely homed account | Refused | I can't take the writer role without a safe handover |
| Submitting to UPI or NEFT | Keeps working | Domestic rails are inside the partition with us |
| Cross-cell transfer | Queue the intent, complete on heal | Saga with a reservation; the pivot doesn't fire until both sides are reachable |
The point I'd make is that for the operations that matter, the partition doesn't force a CAP choice at all, because the data and its writer are on the same side of it. I only meet CAP where a payment genuinely spans cells, and there I choose consistency — I queue rather than guess.
Interviewer: You said "queue the intent". Isn't that just deferring the problem?
You: It is deferring it, deliberately. It's the saga from R5: a reservation on the debit side, which is compensatable, and the actual cross-cell movement as the pivot. Until the far cell is reachable I haven't crossed the pivot, so nothing needs undoing. If the partition lasts past my tolerance, the reservation expires and the payment fails cleanly with a recorded reason — which is better than a payment that half-happened.
Interviewer: Show me the data model that makes homing work.
You: Three pieces of state: who owns an account, what fence token that ownership is on, and how far each cell's replication has caught up. Everything else is ordinary per-cell data.
- The registry is global and tiny — one row per account range, not per account, or it becomes the bottleneck it was meant to remove.
- The fence token lives with the ownership, so promotion and fencing are a single atomic change rather than two things that can disagree (R8).
- Each cell's ledger is ordinary, exactly the Venmo model — the global design adds routing and fencing, not a new ledger.
- Replication checkpoints are data, because "how much would we lose right now" has to be a number on a dashboard rather than an estimate in a runbook.
-- GLOBAL (replicated to every cell, read on every request, written rarely)
CREATE TABLE account_ownership (
shard_key int PRIMARY KEY, -- hash(account_id) % 4096, NOT per account
home_cell text NOT NULL, -- 'ap-south-1'
fence_token bigint NOT NULL, -- bumped on every ownership change (R8)
state text NOT NULL, -- active | migrating | frozen
residency_zone text NOT NULL, -- 'IN' — constrains which cells are legal
updated_at timestamptz NOT NULL
);
-- GLOBAL: an audit of every ownership change, because failovers get investigated
CREATE TABLE ownership_event (
id bigserial PRIMARY KEY,
shard_key int NOT NULL,
from_cell text, to_cell text NOT NULL,
fence_token bigint NOT NULL,
reason text NOT NULL, -- planned | failover | rebalance
decided_by text NOT NULL, -- quorum id, or the operator
at timestamptz NOT NULL DEFAULT now()
);
-- PER CELL: the fence check lives on the data being protected
CREATE TABLE account_state (
account_id bigint PRIMARY KEY,
balance_minor bigint NOT NULL,
last_token bigint NOT NULL DEFAULT 0, -- highest fence token seen (R8)
version bigint NOT NULL DEFAULT 0,
updated_at timestamptz NOT NULL
);
-- PER CELL: how far behind our replicas are — the RPO, as a number
CREATE TABLE replication_checkpoint (
target_cell text PRIMARY KEY,
last_applied_lsn text NOT NULL, -- how far the replica has read
lag_seconds numeric NOT NULL,
measured_at timestamptz NOT NULL
);
Interviewer: Why a shard key rather than the account itself?
You: A shard key here just means hashing the account into one of 4,096 buckets and recording ownership per bucket. Because a registry with 200 million rows that every request reads is a distributed system problem of its own. 4,096 ranges gives me a table small enough to replicate everywhere and cache in memory, while still letting me move ownership at a granularity fine enough to rebalance. Moving one range is a bounded operation; moving one account at a time is an administrative nightmare, and moving a whole region is too coarse to rebalance with.
residency_zone on the same row is what makes compliance mechanical rather than procedural: a range marked IN can only be homed to an Indian cell, and that is a constraint the failover logic checks before it promotes, not a policy in a document.
And notice where last_token lives — on account_state, in the cell, next to the balance. Not in the registry. That is R8's whole argument: the resource being protected has to be the one that checks the token, because a stale writer will happily consult a stale registry and believe itself.
Interviewer: The API?
| Endpoint | Purpose | Notes |
|---|---|---|
POST /v1/payments |
Any cell accepts it | Looks up ownership; serves locally or forwards to the home cell — never serves a remote account itself |
GET /v1/accounts/{id}/balance |
Balance read | Served locally from a replica with X-Data-Freshness: stale, 4s; add ?consistent=true to force a home-cell read |
GET /v1/payments/{id} |
Status | Same rule; the payment's home is its payer's home |
POST /admin/ownership/{shard}/transfer |
Planned move or failover | Quorum-authorised, bumps the fence token, returns the new token |
GET /admin/cells/health |
Per-cell liveness and lag | Read by the external quorum, never self-reported as authoritative |
GET /admin/rpo |
Current worst-case data loss | The replication_checkpoint view, because RPO should be observable rather than assumed |
Two decisions worth defending. Stale reads are labelled rather than hidden — a header saying how old the data is, so a caller that cannot tolerate it can ask for a consistent read and pay the latency. And there is no "write anywhere" endpoint, deliberately. The moment one exists, someone will use it during an incident, and that is how split-brain gets into production.
Interviewer: Now the India cell doesn't come back. You need to fail over. Talk me through it.
You: Failover is an ownership transfer, not a DNS change, and it has to be fenced. Four steps:
First, detect from an independent quorum — never from the cell reporting on its own health. Second, fence: bump the fencing token in the ownership registry for that account range. Third, promote the standby's replica to writer and let it replay its outbox. Fourth — and this is the important one — any write still arriving from the old cell is rejected at the ledger on the token check.
That last step is R8 exactly. The old cell may be alive and simply unreachable, believing it still owns those accounts. A lock or a registry entry alone wouldn't stop it; the resource itself compares the token and refuses anything stale.
Interviewer: And what do you lose when you promote?
You: Cross-region replication is asynchronous, so I lose the last few seconds of writes. I'd quote that as the RPO honestly rather than pretending it's zero. What makes it survivable is the machinery from earlier sections: the client retries, R3's idempotency key means the retry doesn't double-pay, and R6's reconciliation finds anything that neither of those caught — the payments where we're genuinely unsure whether the rail acted.
Interviewer: Couldn't you just use Spanner and have strong consistency globally?
You: For a slice, yes, and I'd say so rather than pretend it doesn't exist. Spanner and CockroachDB run 2PC with consensus-replicated participants, so a coordinator failure doesn't block indefinitely — that's the exception R2 calls out. If I had a genuinely global invariant I couldn't shard away, like one pooled liquidity account or a global exposure limit, I'd put that there and pay the tens of milliseconds per commit.
What I wouldn't do is put the whole payment flow on it, because I'd be paying global commit latency on every domestic payment that never needed it.
Interviewer: Last one. Two payments on the same account arrive in two regions at once. How do you order them?
You: They can't arrive in two regions, by construction — that account has one home cell, so both land in the same place and the database orders them. If you're asking how I order events about that account as they flow outward: by a sequence the system issues, keyed by account, never by wall-clock timestamp. That's R0's point about logical clocks — two machines' clocks can differ by more than the interval I'm trying to order, so created_at sorting across services is a bug that looks like a design.
Interviewer: How would you run this organisationally?
You: One owning team per cell, follow-the-sun on-call, cost per transaction tracked per cell because they differ a lot by rail, and a failover drill every quarter that actually promotes a standby rather than reading a runbook. The number I'd put on a wall is value unreconciled per corridor, because that's the one that tells me whether the design is holding.
Walkthrough 5 — Migrating a monolith to microservices
Interviewer: You've inherited a legacy monolithic payments platform. Migrate it to microservices. I'd like to hear about the anti-corruption layer, how you keep data consistent during the migration, and how the services talk to each other.
You: Can I ask what's driving it? Because my first recommendation might be not to.
Interviewer: Go on.
You: If the reason is "microservices are modern", the right answer is a modular monolith — same boundaries, one deployable, and none of the distributed failure modes. Everything in sections R2 through R8 exists because we gave up a database transaction; if we don't need to give it up, we shouldn't pay for it. The reasons I'd accept are independent deployment cadence, team autonomy at scale, genuinely different scaling profiles, and blast-radius isolation.
Interviewer: Assume it's deployment cadence and team autonomy — eighty engineers on one release train.
You: Then it's justified, and I'd do it as a strangler fig rather than a big bang — named after the vine that grows around a tree until it replaces it. A facade goes in front, route one slice to the new implementation, grow it, retire the old path. That's R16's pattern, and the reason is that a big-bang cutover has no rollback that doesn't involve a bad weekend.
Click a component
Every box below is a candidate microservice, a platform capability, or something you buy rather than build.
Concepts applied, and where
Interviewer: How do you choose the first slice?
You: By risk, not by importance. High value, low write volume, few dependencies — notifications, sanctions screening, reporting. I'm proving the platform, not the architecture. The ledger and the core payment path go last, and I'd push back hard on anyone who wants to start there, because that's where a mistake is measured in money rather than in inconvenience.
Interviewer: Fine. Tell me about the anti-corruption layer.
You: An anti-corruption layer — ACL for short — is a translation layer that keeps the legacy model out of the new service. The new service speaks its own clean language, and the ACL converts, absorbing every legacy oddity.
The thing I'd be firm about is where it lives: on the new side of the boundary, owned by the new team. If it sits in the legacy codebase it becomes one more thing nobody dares change, and then it's part of the problem.
Concretely it does four things: model translation, so legacy PMT_STS_CD = 'A2' becomes PaymentStatus.SETTLED in one mapping file; vocabulary mapping, where legacy "customer" might be our "party" plus our "account"; defensive validation, because legacy data violates its own rules and I'd rather quarantine at the boundary; and idempotency, because the boundary is exactly where duplicated legacy events get deduplicated — R3's insert-and-conflict, keyed on the legacy event ID.
Interviewer: What does a bad ACL look like?
You: Three smells. One that leaks legacy field names into the new domain — then it's a pass-through, not a boundary. One that accumulates business logic until it's a second monolith. And one that's treated as permanent when it should die with the legacy system. Some ACLs do outlive the migration, and that's fine if it's a decision rather than a drift nobody noticed.
Interviewer: Now the hard part. During the migration, both systems are live. How do you keep the data consistent?
You: One rule first: exactly one system owns each piece of data at any moment. Never both writing the same field. Everything else follows from holding that line.
Ownership moves in four stages, which is R16's expand-contract applied to data rather than schema:
- Legacy owns, the new service reads via CDC or an API. Read-only, zero risk.
- Backfill history, then run a continuous comparison job. That's R6 and R7 pointed inward — an independent check that the two agree.
- Flip it: the new service owns writes, and legacy is fed from its events.
- Retire the legacy path and delete the sync.
Stage 2 is the one I wouldn't skip. It's how you earn the right to proceed, and it's the same argument as reconciliation generally: I don't want to assert the two agree, I want a number saying how many records disagree and for how long.
Interviewer: You mentioned syncing back to legacy. Isn't that bidirectional sync?
You: Only in stage 3, and only one direction is authoritative — the new service writes, legacy is a follower. True bidirectional sync is where migrations die. If I'm forced into it briefly, every change gets stamped with its origin and I drop the ones I originated, otherwise you get infinite loops and last-writer-wins corruption between two systems that both think they're right.
Interviewer: What about a use case that needs a transaction spanning the legacy system and the new service?
You: Then I'd say the seam is in the wrong place, and I'd move it rather than build a distributed transaction across a boundary I'm about to delete. If it genuinely can't move, it's a saga with compensation — R5 — with the legacy call as the pivot if it's the irreversible one. But I'd treat that as a design smell, because a transaction spanning both means I've cut through something that should have stayed whole.
Interviewer: What does the migration itself need in terms of data and interfaces?
You: The migration needs its own state: who owns each entity right now, what the comparison job found, and which flag is routing the traffic. Without those three written down, "where are we" becomes an opinion.
- Ownership is data, not a wiki page. One row per entity type saying which system writes it today, and when it changed.
- Shadow comparison results are durable and queryable, because "0.1% mismatch" is a number you have to be able to break down by field and by cause.
- Routing is a flag with a percentage and a slice key, so you can move one corridor rather than one random percent of traffic.
- CDC offsets are checkpointed, so a restart resumes rather than reprocesses a month of history.
-- Which system owns each entity type, right now. The single source of truth
-- for the rule that only one system writes a given field.
CREATE TABLE migration_ownership (
entity text PRIMARY KEY, -- 'payment', 'beneficiary', 'limits'
owner text NOT NULL, -- legacy | new
phase text NOT NULL, -- read_only | dual_write | new_owns | retired
changed_at timestamptz NOT NULL,
changed_by text NOT NULL
);
-- Every shadow comparison, kept — this is what earns the right to cut over.
CREATE TABLE shadow_comparison (
id bigserial PRIMARY KEY,
entity text NOT NULL,
external_ref text NOT NULL, -- the business key, both sides
legacy_value jsonb NOT NULL,
new_value jsonb NOT NULL,
verdict text NOT NULL, -- match | mismatch | missing_new | missing_legacy
diff_fields text[], -- which fields differed; the useful column
compared_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON shadow_comparison (entity, verdict, compared_at);
-- Routing, by slice rather than by random percentage.
CREATE TABLE migration_route (
slice_key text PRIMARY KEY, -- 'corridor:IN-US', 'client:8842'
target text NOT NULL, -- legacy | new | shadow
percentage int NOT NULL DEFAULT 100,
changed_at timestamptz NOT NULL
);
-- CDC progress, so a restart resumes instead of replaying history.
CREATE TABLE cdc_checkpoint (
source_table text PRIMARY KEY,
last_lsn text NOT NULL,
last_event_at timestamptz NOT NULL,
updated_at timestamptz NOT NULL
);
Interviewer: Why keep every comparison rather than just the failures?
You: Because the denominator matters. "Forty mismatches" means nothing without "out of how many", and the cutover decision is a rate over time rather than a count. Keeping matches also lets me prove coverage — if an entity type has 40,000 comparisons and one field has never once been compared because it is always null in the sample, I want to know that before cutover, not after.
diff_fields is the column that does the work. Mismatches cluster: forty differences in one field is one bug, while forty differences across forty fields is a mapping problem in the ACL. Being able to group by field turns an investigation into a query.
And migration_ownership is deliberately boring. One row per entity, one owner, one phase. Its value is that it makes the rule enforceable — a code review, or better a runtime assertion, can check that the new service only writes entities where it is the owner. The rule "exactly one system owns each piece of data" is only real if something can verify it.
Interviewer: And the interfaces?
| Interface | Purpose | Notes |
|---|---|---|
POST /v1/payments on the facade |
The stable public contract | Unchanged throughout; clients never learn a migration is happening |
Internal LegacyPaymentGateway port |
The ACL's outbound interface | Implemented by the adapter; the domain never sees a legacy type |
POST /internal/shadow/compare |
Feed a comparison result | Called by the shadow runner, not by business code |
GET /admin/migration/status |
Ownership, routing and mismatch rate | One page answering "where are we", from the tables above |
POST /admin/migration/route |
Move a slice | Audited; the rollback is the same endpoint with the old value |
Events: PaymentSubmitted etc. |
Published by whichever system owns it | Same schema from both, so consumers never know which side produced it |
That last row is the one that makes the migration invisible to everyone else: the event contract is defined once and honoured by both implementations, so downstream consumers — ledger, notifications, reporting — need no changes and no cutover of their own. If the event shape differs between old and new, you have not migrated a capability; you have forked it.
Interviewer: How do the new services talk to each other?
You: I decide the semantics before the transport, which is the R0 distinction. Commands where I need the answer to continue — a limit check, a screening verdict — go synchronous over HTTP or gRPC, with a timeout, a circuit breaker and a bulkhead, using the Resilience4j settings from R14. Events, meaning "this happened", go asynchronously through the outbox from R4, so the state change and the message commit together.
The thing I'd design against explicitly is the distributed monolith: services that have to deploy together, chatty synchronous chains four deep, and shared database tables. Three habits prevent it — contract tests — Pact is the usual tool — running in CI so a breaking change fails the build, schema evolution by expand-contract with a registry enforcing compatibility, and aggregation at the edge, with a backend-for-frontend assembling one screen's data, rather than chains of calls in the middle.
Interviewer: How would you know, six months in, whether this was working?
You: Four numbers. Share of traffic on the new path. Defect rate compared with legacy, because a migration that ships more bugs isn't progress. Lead time for change, since that's the thing we said we were buying. And legacy code actually deleted — that's the honest one, because a migration that retires nothing has just given you more systems to run.
Interviewer: And how would you staff it?
You: One team owning each service end to end, plus a platform team providing the paved road — a service template with observability, the outbox, idempotency and CI already wired in. Without that, every team reinvents R3 and R4 slightly differently, and then the reconciliation between their versions becomes my problem.
One standing rule I'd set on day one: the migration must never be the reason a customer-facing incident lasts longer. That means the rollback path stays tested the whole way through, not just at the start.
The rest of the bank's question bank
The ones that recur at VP and senior-manager level, with the hook that gets you started.
| Question | The angle that scores |
|---|---|
| Card authorization and settlement | Auth is a hold, not a movement; capture, partial capture, expiry, chargebacks months later. Two-phase money — TCC, as R2 shows for a card checkout — so the ledger needs pending versus posted |
| Real-time fraud and risk scoring | Sub-100ms in the payment path, feature store for aggregates, rules plus model, and the fail-open versus fail-closed decision — for money you fail closed and queue |
| Sanctions screening at scale | Fuzzy name matching, high false-positive rate, hit means park not fail, and a human queue with an SLA. Screening is a latency budget problem inside an irrevocable rail |
| Payment gateway routing to multiple PSPs | Routing rules by cost, success rate and corridor; per-PSP circuit breakers; idempotency keys per PSP; reconciliation across all of them |
| A ledger at 100k writes/sec | Double-entry, append-only, integer minor units, materialised balances with optimistic locking, shard by account, and the balance row as the hot spot (R6, R10) |
| Reconciliation platform | R6 and R7, essentially verbatim — independence, tiered matching, break lifecycle, auto-repair rules with controls |
| Bulk file ingestion (NACHA, ISO 20022, payroll) | 100k payments in one file against one debit account: validation, partial acceptance, per-file idempotency, ordering against the funding account, and progress reporting |
| Webhook / notification delivery to clients | At-least-once with signatures and retries with backoff, client-side idempotency keys, per-client DLQ, and a replay API — R4 turned outward |
| Limits and entitlements service | Read-heavy, latency-critical, per-account and per-user caps with rolling windows; the counter is a hot key, so think batching and approximate-then-verify |
| Standing instructions and scheduled payments | Warehousing (R26): validate at intake, revalidate at release, calendars and rolling conventions, and the month-end thundering herd |
| Disputes and chargebacks | A workflow with deadlines imposed by the scheme, evidence collection, provisional credit, and ledger entries for every state — a saga with a human in it |
| Data platform / reporting off payments | CDC to a lake, read models rather than reporting off the write database, and reconciliation between the lake and the ledger — they drift |
| Intraday liquidity management | Real-time position per account and currency, forecast against cut-offs, and alerting before you are short rather than after |
And the manager-layer questions that come with them, which are graded just as hard: how big a team and what is the first milestone; what you would buy rather than build; how you would migrate without a big bang; how you would know it was working; which of these decisions you would let the team make without you.
What to say in the interview
"I'd scope it first — actors, volumes, rails, what's out of scope — then do the capacity arithmetic aloud, because that decides whether this needs sharding at all. Then the happy path in one diagram, the ledger model, and after that I'd spend most of the time on failure: duplicates, the unknown outcome at a boundary I don't control, and how divergence gets detected rather than assumed. I'd finish on what it costs per transaction, what I'd measure, and what the first milestone is."