R18 — Stateful real-time servers: WhatsApp, Google Docs, stock broking, Zoom

R0 to R17 built the general tools and used them on infrastructure. R18 to R22 apply them to the product questions interviewers reach for most often. Instead of grouping those products by industry, these five sections group them by the engineering problem they share, because that is what you are actually graded on:

Each section opens with the shared problem and the toolkit for it. Then it walks through its systems in an order where each one adds a new difficulty on top of the last. Each walkthrough is a transcript condensed from a 45–60 minute mock: clarify, requirements, capacity, API, data model, high-level design, deep dives, failures, curveballs. The teaching notes sit after the exchange they belong to. Read the dialogue once for flow, then reread only the notes.

The shared concern — state that cannot leave the server

R1 gives the default rule: keep servers stateless. Put the data in a database, so any server can handle any request and a dead server loses nothing. That rule is right for most services. This section is about the three situations where it runs out:

1. The state is a live connection. A phone holding a WebSocket is connected to one specific machine. You cannot put a socket in a database. Whoever wants to reach that phone has to find that machine.
2. Many people change one object at once, and every change must be ordered. Ten people typing into one document need a single place that decides "this edit came first". Sending every keystroke through a database lock is too slow, so the live document sits in one server's memory.
3. The answer is needed in microseconds, and depends on the previous answer. A broker checking "does this customer have enough money for this order?" cannot wait for a database round trip on every order, and two orders checked at the same moment must not both spend the same rupee.

In all three, some state must live in one server's memory. That creates four problems. Every stateful design has to answer them, and each one follows from the one before.

Problem 1 — finding the server that holds the state

If the state lives on one machine, every request for it must reach that machine. There are two ways to route:

• A registry: a lookup table, such as user → gateway in Redis. It is flexible and suits things that move around, like phones that reconnect to whichever gateway the load balancer picks.
• A deterministic mapping: hash(docId) → server, through consistent hashing or a shard map (R10). Nothing to look up, and it suits things that stay put, like a document or a stock.

Problem 2 — making sure only one server holds it

Routing is only safe if there is exactly one owner. If two servers both think they own a document, each accepts edits in its own order, and the copies drift apart for good. This is split brain. The fix is R8's: ownership is a lease that expires, and every new owner gets a bigger epoch number, a fencing token. Every write carries the epoch, and anything stamped with an older one is refused. A server that froze, lost its lease and woke up can then do no harm.

Problem 3 — not losing the state when the server dies

Memory disappears in a crash, so the durable truth must live somewhere else. There are three standard moves, each stronger than the last:

• Keep the truth outside the server. The server only holds things that can be rebuilt. WhatsApp's gateways hold sockets, but messages live in a durable inbox.
• Persist before acknowledging. Write each change to a durable log first, then tell the user "saved", then update memory. After a crash, load the last snapshot and replay the log since then.
• Run a hot standby that replays the same log. If the server's logic is deterministic — the same inputs in the same order always produce the same state — a second machine reading the same log holds an identical copy and can take over in under a second. This is state machine replication.

Problem 4 — recovering without a stampede

When a server with 500,000 connections dies, 500,000 clients reconnect at the same moment. They must not all hit the next server in the same second. So clients reconnect with randomised delays (jitter), and resume from a token — the last message or revision they saw — instead of downloading everything again.

Rules of thumb for stateful servers

• Keep as little as possible in memory, and only what you can rebuild or replay.
• Route deterministically wherever the thing being routed to stays put; use a registry only for things that move.
• Give every owner a lease and an epoch, and refuse writes from old epochs.
• Write to the durable log before you say "done".
• Make reconnects cheap: jitter, resume tokens, snapshot then delta.

How the four walkthroughs build on each other

They are ordered so that each adds one difficulty:

1. WhatsApp — the simplest kind of state, a connection. The server holds sockets, but the truth stays in a durable inbox, so losing a gateway loses nothing that matters. You learn problems 1, 2 and 4.
2. Google Docs — now the state is the object itself. One server must order every edit to a document, so problem 3 becomes real: persist each edit before acknowledging, and recover from snapshot plus log.
3. Stock broking — everything at once, with money and microseconds at stake. Millions of connections receive prices, as in WhatsApp but lossy by design. Each account's risk state has one owner, as each Google Doc does. And recovery must be instant, which brings in deterministic replay on a hot standby — and, if the interviewer means the exchange, the matching engine.
4. Zoom — the state is a live media session, and for the first time late data is worthless: media is dropped, never retried. One media server owns each meeting, connections are UDP rather than a reliable socket, and recovery means reconnecting quickly rather than replaying a log.

At a glance

Walkthrough 1 — WhatsApp, real-time messaging with end-to-end encryption

Where this starts: the simplest kind of stateful server. WhatsApp's gateways hold half a billion live connections, but no data that matters — every message lives in a durable inbox. So this walkthrough is about routing to the right machine, guarding against two owners, and reconnecting millions of clients without a stampede.

Scope and requirements

Interviewer: Design WhatsApp.

You: Questions first, because each answer changes the design. 1:1 only, or groups — and how large? Which message types — text, media, calls? Is end-to-end encryption required? Does the server keep history or only relay? Multi-device? Receipts and presence? What ordering guarantee? And scale.

Interviewer: You decide.

You: 1:1 and groups up to 1,024 members. Text and media up to 100 MB; calls out of scope. E2EE required, so the server only ever routes ciphertext. One phone plus up to four linked devices. Delivery and read receipts, presence and last seen, with privacy settings. Ordered per conversation, not globally. 1B DAU sending 40 messages a day. And store-and-forward: the server holds a message only until every recipient device has it, then deletes it; undelivered messages expire after 30 days.

Interviewer: Users like old chats. Why not keep everything on the server?

You: Cost, privacy and simplicity. Five years of history is ~22 PB before replication against a few TB for a relay inbox — I'll show the math. With E2EE the server could only store ciphertext it can never search, so most of the value of server history is gone anyway. And a queue that drains is easier to run than an archive that grows forever. History lives on the device in SQLite, with optional client-encrypted backups to iCloud or Google Drive. If the product needed Slack-style history on any device without the phone, I would switch to a server-side store and pay for it.

What's being probed: whether you shape the problem before drawing boxes, and defend a scoping decision with numbers.

Concept — Store-and-forward versus system of record. A store-and-forward server is a post office: it holds a letter until it is collected, then discards its copy. A system of record is a library that keeps everything. The first needs a small, fast, constantly churning store; the second a huge, mostly cold archive with indexes. Choosing early sets the storage engine, the cost model and the privacy story.

Takeaway: Relay-only keeps the server small and private; history is the client's job.

Interviewer: Requirements?

You: Functional: send and receive 1:1 and group messages in real time; media; one tick, two ticks, blue ticks; offline delivery on reconnect, in order, with no gaps; push to wake offline apps; presence and typing; every linked device receives every message; E2EE for messages and media. Non-functional: p99 under 300 ms online-to-online in a region; 99.99% on the send path; once the server acks, the message is never lost; at-least-once on the wire, exactly-once on screen; per-conversation order; ~500M concurrent connections; minimal metadata. Out of scope: calls (a separate WebRTC plane), status, payments, channels, spam internals, contact discovery.

Two requirements drive everything: "no loss after ack" shapes the write path, and "500M open connections" shapes the gateway tier. And delivery is at-least-once on every hop with dedup by message id, so the screen shows exactly-once without a distributed transaction — the same move R3 makes at an API boundary.

Capacity

Interviewer: Size it.

You: Assumptions: 500M peak connected devices, a blended fan-out of 2 recipients per message, 300 bytes per encrypted text on the wire, 5% of messages are media at ~200 KB, 2 receipts per delivery, 5% undelivered at any moment, 500K connections per gateway host.

Now turn those rows into machines, bandwidth, disk and memory, one step at a time.

1. Receipts are the biggest load — so make them cumulative. Every delivered message produces two receipts: delivered (✓✓) and read (blue ticks).

receipts at peak = deliveries × 2 = 2.78M/s × 2 ≈ 5.56M/s
messages at peak = sends + deliveries = 1.39M/s + 2.78M/s ≈ 4.17M/s
total frames     = 1.39M + 2.78M + 5.56M ≈ 9.7M/s  → ~10M/s

Receipts, not messages, are the largest frame type, so receipts must be cumulative — "read up to seq N". Open a chat with 30 unread messages and the phone sends one receipt, not 30. If one receipt covers 5 messages on average:

receipts at peak = 5.56M/s ÷ 5 ≈ 1.1M/s
total frames     = 1.39M + 2.78M + 1.1M ≈ 5.3M/s   — roughly half the load gone

2. Gateway hosts. One host holds 500K open connections. Add 50% headroom, because when a host dies or is redeployed its users reconnect to the others, and those must have room.

hosts = 500M connections ÷ 500K per host = 1,000
      × 1.5 headroom                     ≈ 1,500 hosts

3. Memory per host. Each connection costs about 20 KB for buffers and session details.

500K connections × 20 KB = 10,000,000 KB ≈ 10 GB of RAM per host — affordable

4. Text bandwidth is small. Count every frame at the full 300 bytes, which overestimates, since receipts are far smaller:

10M frames/s × 300 B = 3 GB/s × 8 bits ≈ 24 Gbps across the fleet
24 Gbps ÷ 1,500 hosts ≈ 16 Mbps per host

Text is not the bandwidth problem. Media is (step 7).

5. The inbox: huge writes, tiny footprint. The server is a relay, not an archive. A message waits in each recipient device's inbox only until that device receives it, then the row is deleted.

rows written per day = 40B sends × 2 recipients = 80B
bytes written        = 80B × 300 B             = 24 TB a day

How much sits on disk at any moment? Little's Law: items held = arrival rate × time each one stays. Split the rows by how long they stay:

95% delivered within ~2 s (recipient online):
  76B/day ÷ 86,400 s ≈ 880K rows/s × 2 s ≈ 1.8M rows × 300 B ≈ 0.5 GB   — negligible
 5% wait ~1 day (phone off):
  4B rows/day × 1 day = 4B rows × 300 B ≈ 1.2 TB
working set ≈ 1.2 TB; × 3 replicas ≈ 3.6 TB → ~4 TB

6. What an archive would cost instead. If the server kept every message for five years, stored once per message rather than once per recipient:

40B × 300 B = 12 TB a day
× 365       ≈ 4.4 PB a year
× 5 years   ≈ 22 PB, before replication

About 22 PB against about 4 TB. Keeping history on the phone and only relaying on the server is what makes storage cheap.

7. Media is the real bytes problem. 5% of messages carry a photo or video of about 200 KB, kept 30 days so recipients have time to download it.

objects per day  = 40B × 5%            = 2B
bytes per day    = 2B × 200 KB         = 400 TB a day   (≈ 33× all the text)
stored           = 400 TB × 30 days    ≈ 12 PB
upload rate      = 400 TB ÷ 86,400 s   ≈ 4.6 GB/s ≈ 37 Gbps average, ~111 Gbps at peak
download rate    = upload × 2 recipients ≈ 220 Gbps at peak — ~9× the text traffic

So media must bypass the chat servers: the phone uploads straight to blob storage, and the chat message carries only a link and a decryption key — R20's "facts separate from bytes".

8. Caches that must sit in memory.

session registry   = 500M devices × 100 B             = 50 GB; × 3 replicas ≈ 150 GB
presence           = 500M users × ~32 B                ≈ 16 GB
group membership   = ~100M active groups × 10 members × ~32 B ≈ 32 GB

All three fit comfortably in a Redis cluster, which keeps routing and fan-out off the database.

What the numbers say: the traffic is enormous but almost none of it stays, and text needs little bandwidth. The two real costs are receipts — fixed by making them cumulative — and media — fixed by keeping it off the chat servers.

What's being probed: whether you separate throughput from footprint.

Concept — Working set versus ingest. Ingest is what you write per day; working set is what you hold at any moment. Little's Law ties them: items held = arrival rate x time each stays. 24 TB a day arrives but only the undelivered tail stays, so about 1.2 TB sits on disk — a busy airport gate that is always nearly empty. That is why a relay inbox is cheap and an archive is not.

Takeaway: Footprint = rate x dwell time; short dwell makes huge traffic cheap to store.

API

Interviewer: What does the client-server interface look like?

You: One long-lived connection — a compact binary protocol over TLS on mobile, WebSocket on web — carrying frames, not REST calls. Media and keys use HTTPS. R1.4 compares the transports; at this battery and latency budget only a persistent duplex connection fits.

{ "type": "SEND", "client_msg_id": "3f9c2a7e-1b4d-4f0a-9c2e-7d1e5a8b6c01",
  "to": { "kind": "user", "id": "u_918877" },
  "envelopes": [
    { "device_id": "u_918877:0", "ciphertext": "base64...", "type": "signal_msg" },
    { "device_id": "u_918877:2", "ciphertext": "base64...", "type": "prekey_msg" },
    { "device_id": "u_123456:1", "ciphertext": "base64...", "type": "signal_msg" } ] }

{ "type": "ACK", "client_msg_id": "3f9c2a7e-...", "conv_id": "c_123456_918877",
  "seq": 48213, "server_ts": 1790000000180, "status": "SENT" }

The sender encrypts once per recipient device and once per its own other devices. client_msg_id is the idempotency key: the server dedups on (sender_device, client_msg_id) for about 7 days and a retry after a dropped connection gets the original seq back (R3). Auth happens once at connect with a device credential. SYNC pages 500 messages per conversation with a next_seq cursor. Under overload the server sends RETRY_AFTER and the client keeps unacked messages in its outbox.

Data model

Interviewer: What do you store, where?

You:

Interviewer: You're deleting Cassandra rows seconds after writing them. Isn't that an anti-pattern?

You: Yes, and I should own it. Deletes are tombstones, and reads slow down stepping over them until compaction. Time-window compaction, a short gc_grace_seconds, reading from the cursor so tombstoned ranges below it are skipped — or better, stop deleting: keep a per-device "acked up to seq" cursor and let TTL expire rows. If it still hurts, ScyllaDB or a log store with segment-level deletion; the design doesn't change, only the engine.

What's being probed: whether you design a partition key and know the store's sharp edges.

Concept — Partition by reader, expire in bulk. The partition key decides which machine holds the rows; the clustering key decides their order inside it. "Everything pending for Bob's phone after seq 41" is one sequential read on one node because the partition is the device and the order is (conv_id, seq). In an LSM store a delete is a write of a marker, so a queue-shaped table should expire by TTL and a cursor rather than row-by-row deletes. R10 covers LSM trees.

Takeaway: Partition by who reads it, cluster by the order they read it, and let TTL do the deleting.

High-level design

Interviewer: Draw it, and walk me through Alice sending "hi" to Bob, who is online.

You: Play the 1:1 flow and the offline flow, and watch one thing: where the ack to the sender sits relative to the inbox write. It always comes after.

An L4 load balancer spreads TCP connections without terminating WebSockets. Chat gateways hold sockets, terminate TLS, register device → gateway and forward frames; they keep no durable state. WhatsApp's publicly described Erlang heritage fits this tier — one lightweight process per connection; in Java it is Netty or virtual threads. The message service, sharded by conv_id, dedups, sequences, persists, routes and handles receipts. The inbox is the durable store-and-forward queue. The group service expands groups to devices. Kafka carries push events and large-group fan-out (R9). The push notifier calls APNs/FCM. The media service issues pre-signed URLs over a blob store and CDN.

1. Alice's app mints client_msg_id, encrypts per device, writes to its local outbox, sends SEND to gateway A.
2. Gateway A forwards to the message-service shard owning conv_id(Alice, Bob).
3. Dedup check; new, so seq = last_seq + 1.
4. One inbox row per recipient device, quorum write. Only then ACK(seq) — one tick.
5. Registry lookup: Bob's phone is on gateway B.
6. DELIVER via gateway B.
7. Bob's app dedups by msg_id, decrypts, stores, replies RECEIPT(delivered, seq).
8. The inbox row is cleared or the cursor advances; the receipt reaches Alice — two ticks.

Step 4 writes the inbox even though Bob is online. Online delivery is an optimisation on top of durable storage, not a separate path.

What's being probed: whether you trace one request end to end and put the durability point in the right place.

Concept — Persist before acknowledge. An ack is a promise. If the server says "sent" before the message is on several machines, a crash in between loses a message the sender believes was delivered. So the order is always: store at quorum, then ack, then deliver — and delivery can fail and retry because the stored copy is still there. R4 makes the same argument about publishing.

Takeaway: Never say "got it" until it is written down in more than one place.

Deep dive — gateways and the session registry

Interviewer: How do you hold 500K sockets per host?

You: Event-driven I/O — Netty or virtual threads in Java, an Erlang process per connection in WhatsApp's case — so an idle socket costs kilobytes, not a thread stack (R1.1 shows why, using Tomcat's poller). Heartbeats every 30–60 s, tuned to mobile NAT timeouts; missed heartbeats mark the device offline. No durable state — a dead gateway's clients reconnect elsewhere and sync. Deploys drain gradually: stop accepting, send RECONNECT with jitter, drain over minutes. Per-connection rate and frame-size limits. WhatsApp publicly reported about 2 million connections per server in 2012; I'd treat that as context, not a capacity number.

Interviewer: Bob switches from Wi-Fi to 4G and reconnects to gateway C while gateway B still thinks it holds him.

You: The registry entry carries an epoch from a per-device counter.

on_connect(device):
    epoch = INCR epoch:{device}
    SET sess:{device} = {gw: me, epoch} EX 120
on_heartbeat(device, epoch):
    if GET sess:{device}.epoch == epoch: EXPIRE sess:{device} 120
on_disconnect(device, epoch):
    EVAL "if epoch matches then DEL" sess:{device}

Gateway C registers epoch 18. When B finally notices its dead socket and tries to delete with epoch 17, nothing happens. A delivery in flight to B gets no ack; Bob's sync on C reads it from the inbox; if both copies arrive the client dedups. Stale routing is safe because the inbox, not the registry, is the source of truth. Consistent hashing of users to gateways is the alternative; I reject it because L4 balancing and mobile reconnects make landing on a specific host hard, and losing a host moves many users.

What's being probed: whether you handle a race between an old and a new owner.

Concept — Epochs as fencing tokens. When ownership moves, give each new owner a strictly larger number and make every write or delete carry it; the store rejects anything older. That stops a slow or paused old owner undoing the new owner's work. R8 covers fencing tokens in locks; this is the same idea applied to a routing entry.

Takeaway: Stamp every ownership change with a bigger number and reject the old ones.

Deep dive — guarantees, ticks and ordering

Interviewer: Define each tick, and prove no message is lost.

You:

The proof is a chain of custody. The sender holds the message in its outbox until the ack. The server acks only after a quorum inbox write. The inbox row stays until the recipient device acks. Every hop retries and ids make retries idempotent. There is never a moment when the only copy is in flight. Read receipts are one frame — RECEIPT(read, conv_id, up_to_seq = 48213) — which turns every earlier tick blue, goes into Alice's inbox if she is offline, and also reaches Bob's other devices to clear their badges.

Interviewer: Can't you just sort by timestamp?

You: No — retries, multiple gateways and parallel workers reorder, and client clocks skew.

Per-conversation sequence. Every message for a conv_id routes to one owner shard by consistent hashing (R10), with leases so only one node owns it (R8). The owner does seq = ++last_seq[conv] and persists it with the inbox write. The recipient seeing 41 then 43 knows 42 is missing and syncs for it. The client sends one at a time per conversation, so the server sees the typed order. A new owner reads max(seq) before accepting writes. When Alice and Bob type at once, server order is canonical and clients reorder once acks land.

What's being probed: whether you know why clocks fail and scope ordering to what the product needs.

Concept — Single-writer sequencing. The simplest total order is one actor handing out numbers, like a deli ticket machine. A global machine would bottleneck, but chats are independent, so each conversation gets its own machine on whichever node owns it. Consecutive numbers also reveal gaps, which is what makes catch-up exact.

Takeaway: One numberer per conversation gives order and gap detection without a global bottleneck.

Deep dive — offline, groups, encryption, presence

Interviewer: Bob's phone is off for six hours.

You: The message is durable in Bob's inbox and Alice sees one tick. No live registry entry, so the message service publishes recipient_offline; the notifier sends a push with no plaintext — "new messages" or an encrypted blob the notification extension decrypts — collapsed per chat and rate limited. Bob's app reconnects, registers epoch 17, sends SYNC with its last acked seq per conversation, and the service streams his partition in (conv_id, seq) order. Bob acks each page; rows clear; delivered receipts flow to senders. Push rings the doorbell; sync with a cursor delivers the mail. A device offline past 30 days misses messages. A newly linked device gets history from the phone, device to device, never from the server.

Interviewer: Groups — 50 members, then 1,024 with most offline.

You: Alice encrypts once with her Signal sender key and sends one frame to g_42. The service expands membership from cache (version 12) to ~120 devices, assigns a group seq, batch-writes inbox rows, acks, then batch-MGETs the registry and delivers or pushes. Receipts aggregate on Alice's client: "read by 30".

Fan-out on write — a row per member device, rather than one shared group log that each member reads with a cursor — because groups are capped: worst case ~2,000 rows per message. Above ~100 members the message goes once to Kafka and workers do the per-member writes in parallel; the sender's ack follows the group-log write, not the fan-out. When a member leaves, everyone rotates sender keys.

Interviewer: Product wants broadcast channels with 5 million followers.

You: Then 2,000 rows becomes 10 million and write fan-out breaks. Channels above a threshold store each post once in a log partitioned by channel_id; each follower device keeps a cursor; push becomes a throttled topic notification; per-follower ticks become aggregate counts. Encryption changes too — sender keys with rotation on every leave do not scale to millions, so channels are either not E2EE the same way or use a scheme like MLS. That is a product and privacy decision, and I'd say so.

What's being probed: whether a size cap changes your answer, and whether you see the knock-on effects when a requirement breaks it.

Concept — Bounded versus unbounded audiences. Copying into every mailbox makes sending expensive and reading trivial; a shared log makes sending cheap and per-reader state harder. A hard cap like 1,024 makes copy-on-send safe. Remove the cap and you switch to a log — and that switch drags receipts and encryption with it, which is what shows senior judgement. Instagram's celebrity problem, in R21, is the same shape.

Takeaway: Bounded groups favour copy-on-send; unbounded audiences favour a shared log.

Interviewer: What does E2EE cost you as the designer?

You: WhatsApp publicly described using the Signal protocol. Each device has an identity key pair; only public keys reach the server. X3DH lets Alice derive a shared secret from Bob's prekey bundle while he is offline; the Double Ratchet gives every message a fresh key, so a leaked key unlocks one message. Media is encrypted client-side with a random AES key, uploaded as ciphertext, and {media_url, aes_key, sha256} travels inside the encrypted message. The server routes sealed envelopes; any feature that needs to read them moves to the client — so no server search, no server moderation, no server link previews. It still sees metadata: who, to whom, when, how big.

Presence is soft state, kept in Redis: set on connect, last_seen on disconnect, published only to current watchers — the people with Bob's chat open — never broadcast to all contacts. Reconnects within 10 s are debounced; typing frames are never persisted; privacy settings are enforced server-side; presence never sits on the delivery path.

Failures and scale

Interviewer: What breaks first?

You:

The failure caused by recovery

The one I worry about most is the thundering herd. A gateway crash sends 500K phones back at once; a regional failover sends tens of millions. Without jitter and admission control, the reconnect storm takes down the next gateway. R14 covers backoff with jitter and load shedding.

Interviewer: A whole region goes down.

You: Users are homed near them; their inbox and session entries live there. A cross-region message is forwarded to the recipient's home region over the backbone, and the inbox write happens there. Inboxes replicate asynchronously to a second region with LOCAL_QUORUM writes (R13). On failure, DNS steers clients to the healthy region, which promotes the replica. Replication lag can re-deliver (dedup handles it) or rarely lose the last unreplicated seconds — and the sender's outbox still holds anything unacked and resends. Synchronous cross-region writes would close the gap but add tens to hundreds of milliseconds to every send and break the 300 ms target. Each region runs N+1 to absorb a neighbour.

Curveballs

Interviewer: Why not use Kafka as the inbox?

You: Kafka is shaped for a few large streams that many consumers read in order. An inbox is 500M tiny queues, one per device, each read and expired on its own; you'd multiplex devices into partitions and a reconnecting phone would scan past everyone else's messages. Few big streams: use a log. Billions of tiny queues: use a keyed store. Kafka stays where it fits — push events and large-group fan-out.

Interviewer: Legal wants server-side search, and a regulator asks what metadata you keep.

You: Server search is not possible with E2EE; search runs on the device over its local store, and I'd push back on the request on those grounds. On metadata: we still see sender, recipient, timestamps, sizes, IPs and group membership. We minimise it by deleting after delivery, short log retention, content-free analytics, and features like sealed sender that hide the sender from the server. Encryption hides what is said, not that it was said.

Quick reference

The 60-second version

Every device keeps one persistent TLS connection to a stateless gateway; a Redis registry maps device to gateway with an epoch so reconnect races are harmless. A conversation-sharded message service dedups by client message id, assigns a per-conversation sequence, and writes one inbox row per recipient device to Cassandra at quorum before acking the sender. Online recipients get the message through their gateway and return a receipt; read receipts are cumulative. Offline recipients get a contentless push and sync everything after their cursor on reconnect. Every hop is at-least-once with idempotent ids, so the screen shows exactly-once, and the inbox — not the registry — is the source of truth. Groups fan out on write with sender keys; media is client-encrypted in a blob store behind a CDN; presence is lossy and watcher-only; regions home users and lean on the client outbox to cover async replication.

Walkthrough 2 — Google Docs, real-time collaborative editing

Building on WhatsApp: WhatsApp's servers held sockets but no data that mattered. Here the server holds the document itself, because every keystroke needs one ordering point. So each document gets exactly one fenced owner, and every edit must be durable before it is acknowledged — snapshot plus log is how the state survives a crash.

Scope

Interviewer: Design Google Docs.

You: The hard part is not the number of users, it is concurrency on one object, so my questions go there first. How many people edit one document at once, and how many watch? — that decides whether one server per document can work. Offline, and for how long? — how far two copies may drift. Latency for my keystrokes and others'? — local echo instant, remote within a few hundred ms. Then the rest: rich text only or sheets too, version history, comments and suggestions (anchors on moving text), sharing model, and data residency (where the authoritative copy lives).

Interviewer: Rich text, global, the usual features. You pick numbers.

You: 100M daily actives. Most documents have one editor at a time; a few have 50+ editors and thousands of viewers. A hard cap of about 100 simultaneous editors per document, similar to the publicly documented Google Docs limit; beyond it, users join view-only. Own keystrokes instant; remote edits under 500 ms p99 in-region. Offline for hours to days. Full history, comments, suggestions, Drive-style sharing. Each document has one home region at a time.

Functionally: create, open, edit, share; many editors at once with everyone converging; live cursors and presence; offline with merge on reconnect; version history and restore; anchored comments and suggestion mode; role-based and link sharing. Out of scope: formulas, slide layout, spelling and AI, export pipelines, billing.

The correctness property I'd name precisely is convergence plus intention preservation. Durability is a separate promise: an acknowledged edit is never lost. Availability 99.95% for editing, with reconnect in seconds if a server dies. Revoked access takes effect within seconds.

What's being probed: whether you find the hard axis, and name the correctness property precisely.

Concept — convergence versus intention preservation. Convergence: any two replicas that have applied the same set of edits show the same text, whatever order the edits arrived in. It is easy to get cheaply and wrongly — last writer wins on the whole document converges by throwing work away. Intention preservation adds that each edit still does what its author meant: an "X" typed after "the" stays after "the" even if text was inserted earlier in the line. An editor cap turns an unbounded problem into a bounded one.

Takeaway: Ask about concurrency per object first, then say "convergence plus intention preservation" out loud.

Capacity

Interviewer: Size it.

You: Assumptions: 20 minutes of editing per user a day; one batched op every 2 s while typing, because keystrokes are coalesced; 200 bytes per op on the wire, 100 stored; ten opens per user a day; 5B documents with 50 KB snapshots; peak three times average; two recipients per op.

Memory says 20 servers, connections say 200 — so connections bind. Batching keystrokes is the biggest lever on the op rate; sending each one would multiply it five to ten times. Almost all load is per document, which makes partitioning by document id natural.

API

Interviewer: What does the API look like?

You: Two surfaces. REST for metadata, open, sharing, comments and history. A WebSocket for the live session, because both sides push constantly (R1).

// client -> server
{ "type": "op", "clientId": "c_77", "seq": 42, "baseRev": 1203,
  "ops": [ { "retain": 15 }, { "insert": "hello", "attrs": { "bold": true } } ] }
// server -> author
{ "type": "ack", "seq": 42, "rev": 1206 }
// server -> everyone else
{ "type": "remote", "rev": 1206, "author": "u_19",
  "ops": [ { "retain": 18 }, { "insert": "hello", "attrs": { "bold": true } } ] }

The broadcast says retain 18, not 15: the server transformed the op against revisions 1204 and 1205, which the author had not seen. Every op carries the revision it was written against; that one number is what makes merging possible.

Interviewer: The network drops right after a client sends an op.

You: Three rules. (clientId, seq) identifies a batch, so a resend after reconnect finds the seq already applied and the server just re-sends the ack (R3). Each client has at most one unacknowledged batch in flight — the Jupiter / Google Wave style — and new edits buffer and compose into the next batch. And the WebSocket uses a signed session token issued after the ACL check on open, so the collab server does not call the ACL service per op; it re-validates on refresh and on revocation events.

What's being probed: transport choice, revision-based messages, and safe retries off the hot path.

Concept — delta ops with a base revision. The client sends "keep 15 characters, insert 'hello' in bold" plus the revision it saw. The server then knows exactly which concurrent edits the client missed. One in flight keeps client logic to "one outstanding op plus a buffer"; the idempotency key makes a resend free.

Takeaway: Retry with the same (clientId, seq); the server dedups, so retries cost nothing.

Data model

Interviewer: The data model, and where each piece lives.

You:

row key:  d_8f2a#000000001206        (docId + zero-padded rev: lexicographic = numeric order)
columns:  author=u_19, client=c_77, seq=42, ts=2026-10-02T10:15:03Z,
          ops=<compressed delta: retain 15, insert "hello" {bold}>

Open is metadata plus ACL point reads, one snapshot blob, and one range scan for ops after the snapshot's revision. Edit is one appended row per accepted batch; the owner is the only writer, and a conditional write that rev does not already exist guards against a stale owner.

Interviewer: Why not put ops in Postgres too?

You: Sixty billion rows a day is far beyond what one SQL cluster handles cheaply, and we never join ops with anything. Every read is "ops for one doc after rev N", which is exactly the native read of a wide-column store partitioned by docId. With small volume and a need for joins, distributed SQL would be fine.

What's being probed: event log plus snapshot, and a store chosen by its native read.

Concept — ledger for truth, snapshot for speed. Store every change in order, like a ledger, and periodically write the running state. The log gives history, blame and catch-up; the snapshot keeps opens fast by bounding replay. In a wide-column store the partition key decides the machine and the clustering key the order, so docId + rev makes the hottest query one sequential read (R10).

Takeaway: Pick the store whose native read is your hottest query, with a row key that sorts in the order you read.

High-level design

Interviewer: Draw it.

You: Every document has one owning collaboration server that orders all its edits. Everything else supports opening, durability and side features. Click the collab server to see what it holds in memory, then play the concurrent-edit flow and watch the second op get transformed before it is committed.

Interviewer: Open a document, end to end.

You:

1. GET /v1/docs/{docId}; the Document API asks the ACL service for the role — 404 rather than 403 if none, to avoid confirming the document exists.
2. It returns the latest snapshot (say rev 1200), the role and a signed token valid ten minutes.
3. The client renders the snapshot at once, so the page is usable before the live session connects.
4. The client opens a WebSocket; the LB finds the owner, or one takes a lease if none.
5. join {baseRev: 1200}; the collab server verifies the token, loading snapshot plus log tail if the doc is not in memory.
6. It sends ops 1201 to current, plus presence. The client is live.

What's being probed: separating the stateful real-time path from the stateless request path.

Concept — stickiness by document. Ordering every keystroke through a database is too slow, so the live document sits in one server's memory and every editor of it is routed there. The LB routes by the shared object because two users of one doc must meet on one machine. Opening is snapshot then catch-up: a fast, slightly stale picture, then "everything since 1200".

Takeaway: Route by the object being shared; show a snapshot fast, then stream the delta since its revision.

Deep dive — OT versus CRDT

Interviewer: Two people type into the same sentence at the same moment. How do they end up with the same text?

You: First, why it is a problem. Start with "abc".

• Alice: insert("X", 1) → "aXbc". Bob: insert("Y", 2) → "abYc".
• Naively, Alice applies Bob's op as is: Y at 2 in "aXbc" → "aXYbc". Bob applies Alice's: X at 1 in "abYc" → "aXbYc".

They diverge — positions refer to the document each user saw

Operational Transformation rewrites an incoming op against concurrent ops applied first, so it still means the same thing:

transform(op1 = insert(p1, s1), op2 = insert(p2, s2)):   // op1 adjusted for op2 already applied
    if p1 < p2:  return op1
    if p1 > p2:  return insert(p1 + len(s2), s1)
    // same position: deterministic tie-break, e.g. lower site id first
    return (site(op1) < site(op2)) ? op1 : insert(p1 + len(s2), s1)

transform(insert(p, s), delete(dp, n)):
    if p <= dp:      return insert(p, s)
    if p >= dp + n:  return insert(p - n, s)
    return insert(dp, s)                          // insert landed inside the deleted range

The flow on the server:

1. Doc is "abc" at rev 10. Both apply locally and send with baseRev 10.
2. The collab server handles one op at a time per document. Alice's arrives first; base equals current, no transform; persisted as rev 11.
3. After the durable write it acks Alice and broadcasts insert X @1.
4. Bob's arrives with base 10, server at 11. Alice inserted before Bob's position, so Bob's shifts 2 → 3; persisted as rev 12, insert Y @3.
5. Ack Bob; broadcast insert Y @3.
6. Bob's client received insert X @1 while its op was unacked; it transforms against its pending op (X at 1 is before Y at 2, unchanged) and gets "aXbYc".
7. Alice applies insert Y @3 to "aXbc" → "aXbYc". Converged.

With a central server defining one total order, the client transforms only against its own pending ops and the server only against ops since the client's base revision. That avoids the hard TP2 property peer-to-peer OT needs. Google Docs is publicly described as OT with a central server, descended from Jupiter and Google Wave.

Interviewer: Figma and newer editors talk about CRDTs. Why not skip writing transform functions?

You: A CRDT gives every character a permanent id and makes an insert refer to its neighbour by id, not index. "abc" has ids a=(1,s0), b=(2,s0), c=(3,s0). Alice inserts X as (4,A) "after (1,s0)"; Bob inserts Y as (4,B) "after (2,s0)". Any replica applying both, in any order, gets "aXbYc". Ties after the same character break deterministically — higher Lamport clock, then site id — and deletes leave tombstones so later ops still find their anchors. Yjs and Automerge implement this; Figma has described a CRDT-inspired design that still uses a central server as authority.

My choice stays OT with a central collab server per document. We need a server anyway for permissions, durability and history, so the main CRDT advantage matters less, and OT keeps ops small and revisions linear. I'd pick a CRDT for an offline-first or peer-to-peer product, end-to-end encrypted documents, or a team that would rather not own transform functions.

What's being probed: whether you understand why edits diverge, and can choose between two valid repairs.

Concept — OT and CRDTs. "Insert at 2" means something only against one document state; when another edit lands first, OT rewrites the position so applying both in either order gives the same result (TP1), with a deterministic tie-break. A CRDT avoids positions: permanent ids make merges commutative and idempotent, so replicas converge with no coordinator, paid for in per-character metadata and tombstone GC.

Takeaway: OT needs a server to order ops; CRDTs pay in metadata to not need one — choose by whether you already have a server.

Deep dive — exactly one owner per document

Interviewer: With hundreds of collab servers, how do you guarantee one ordering point per document?

You: If two servers accepted ops for one doc, they would assign the same revision to different ops.

Per-document ownership, with R8's machinery: consistent hashing of docId picks a default owner, and the owner takes a lease in etcd or ZooKeeper — the lease, not the hash, grants ownership. The LB looks it up and forwards, or redirects. Every op-log write is a conditional put on (docId, rev) carrying the lease epoch, so a server that lost its lease in a GC pause or partition fails its next write and drops the document. Adding servers moves about 1/N of documents; a moving doc drains — stop accepting, flush, release, clients reconnect with their base rev. One document is bounded at ~100 editors, so one owner is enough; viewers go to relays.

What's being probed: leader election per key and protection against split brain.

Concept — leases and fencing. A lease is time-limited ownership from a consistent store, and on its own it is not enough: a frozen server wakes believing it still owns the document. A fencing token — an epoch that only rises, checked by the storage layer — rejects the stale writer. R8 covers this in full; the conditional put on (docId, rev) is a second fence for free.

Takeaway: A lease says who should own it; a fenced write makes sure nobody else can act as owner.

Deep dive — op log, snapshots and acks

Interviewer: When exactly do you ack, and how do you open a doc with millions of ops?

You: The op log is the truth, one row per committed batch, written before the ack, with author and timestamp — history and blame for free. Snapshots every ~500 ops or after 30 s idle, made asynchronously from the event stream, so an open replays at most ~500 ops. Revision numbers are the vocabulary for everything: base revs, acks, catch-up, comment anchors, history.

on op(clientId, seq, baseRev, ops) for doc D:              // single-threaded per document
    if dedup[clientId] >= seq: send ack(seq, revOf(clientId, seq)); return
    for r in (baseRev+1 .. D.rev): ops = transform(ops, log[r])
    newRev = D.rev + 1
    oplog.putIfAbsent((D.id, newRev), ops, epoch)           // fenced, durable
    D.apply(ops); D.rev = newRev; dedup[clientId] = seq
    send ack(seq, newRev) to author
    broadcast remote(newRev, ops) to others; publish to event stream

On a busy doc the owner groups several ops into one storage write — group commit — and acks each only after the group is durable. After 30 days, consecutive ops by one author are compacted into coarser history entries; named versions are pinned.

Interviewer: A quorum write per keystroke batch sounds slow. Why not ack on receipt?

You: The typist never waits for the ack — local echo already shows their text. The ack only tells the client it can forget the op. Acking on receipt would shave milliseconds off something nobody sees and break "acked edits are never lost" if the server crashed before writing. An in-region quorum write is a few milliseconds.

What's being probed: durability semantics, and seeing that local echo decouples perceived latency from commit latency.

Concept — persist, then ack. An ack is a promise; keeping it means the edit is on replicated storage first. The client holds every unacked op, so a crash before persisting just means a resend. Group commit makes this cheap; snapshots bound replay; compaction bounds history growth.

Takeaway: Persist, then ack; local echo means users never feel the wait.

Deep dive — presence, offline, comments and sharing

Interviewer: Cursors without flooding the system?

You: The client throttles to one update every 100–200 ms, tagged with the revision it refers to. The server keeps it in memory, transforms the position against ops since that revision — a cursor is a zero-length insert — and broadcasts to editors, batched to relays. Clients transform remote cursors locally as ops arrive. No heartbeat for ~30 s means "user left". Viewers see "+312 viewing", not every cursor. Nothing is persisted.

Interviewer: Alice edits on a plane for three hours.

You: Her client stores the document, its last server rev (50) and its pending ops in IndexedDB, composing them where possible. On reconnect it sends join {baseRev: 50} and its pending batches with their original (clientId, seq). The server sends her ops 51–80, transforms her pending ops against them and commits them as new revisions — re-acking any seq that landed before the disconnect. Her client transforms 51–80 against its pending ops and both sides converge. It is git rebase with the live-editing transforms. For a very long gap the cost is O(n × m), so compose the pending queue first, or rebase once onto a fresh snapshot. If she lost edit rights meanwhile, the ops are rejected and the client offers to copy her text. Two people rewriting one sentence still merge mechanically; history lets them undo.

Interviewer: Comments on text that keeps moving? Suggestions, history, sharing?

You: Everything is an op.

• Comment anchors: store (start, end, anchorRev) and transform like cursors, or embed invisible markers in the model so they move with the text. Fully deleted text orphans the comment, which stays in the list.
• Suggestions: a suggested insert is an insert with a suggestion attribute; a suggested delete is a strike attribute. Accept or reject is another op, so OT handles concurrency.
• History: nearest snapshot at or before R plus replay. Restore is new ops that produce R's content, so restore itself can be undone.
• Sharing: user, group, domain and link principals; highest matching role wins, owner overrides. Checked on open and baked into the token; the collab server enforces the role per op. A revocation event makes the server downgrade or disconnect within seconds; token expiry bounds the worst case at ten minutes.

What's being probed: separating ephemeral from durable state, and reusing one mechanism for many features.

Concept — offline as a long gap, features as ops. Offline reconciliation is the same transform applied both ways over a bigger gap, and composing first keeps it cheap. Comment anchors, suggestions, restores and undo are all ops on the same log, so they inherit ordering, history and concurrency handling without new machinery. Presence is the opposite case: valuable only now, so throttled, in memory, and allowed to be lost.

Takeaway: Build features as ops on the same log; keep presence in memory and let it go.

Deep dive — one hot document

Interviewer: The CEO shares the all-hands doc. 50 editors, 20,000 viewers.

You: Viewers are the problem, not editors. Cap editors at ~100. The owner sends each committed op once to a handful of fan-out relays, each pushing to thousands of viewer sockets — a two-level tree. Presence for viewers is aggregated. Broadcasts to viewers batch every 100–250 ms. Viewers that fall behind skip ahead with a fresh snapshot. For a published doc with a huge audience, serve a periodically refreshed render through a CDN. With moderate viewer counts, a pub/sub topic per doc would do instead of relays. Keep one writer; spread reads.

Scale and failure

Interviewer: A collab server owning 15,000 documents crashes.

You: The lease expires in 5–10 s or a health checker releases it. A new owner takes the lease with a higher epoch and loads snapshot plus log tail. Clients reconnect with their base rev and resend unacked ops, idempotent by (clientId, seq). No acked op is lost, because acks follow durable writes. Users keep typing locally through the gap. Clients reconnect with jittered backoff (R14), and the new owner loads each doc once and serves every join from memory. The server's memory is a cache of durable state, so a crash costs seconds of reconnecting, not data.

Interviewer: Global users. A region goes down.

You: Each document has a home region holding its owner and authoritative log; distant users connect there and local echo hides the extra round trip. The home can move toward where most editors are, using drain-and-hand-off. The log and snapshots replicate asynchronously, so RPO is replication lag — and clients still hold unacked ops, so most of that gap is resent on reconnect. Metadata and ACLs sit in a globally consistent store, so a revocation holds everywhere. Multi-leader with a CRDT would give local commit latency on every continent, at a lot of complexity for little visible gain.

What's being probed: a complete failure story, and keeping one total order under global distribution.

Concept — reconstructible state and a home region. Make every in-memory structure rebuildable from the log plus what clients hold, and recovery is mechanical. One ordering point must live somewhere, so pick a home region; permissions are the exception that must be strongly consistent everywhere, or a revoked user keeps access in one region (R13).

Takeaway: Keep memory as a cache of durable state; one home region per document keeps ordering simple.

Curveballs

Interviewer: Why not just lock the paragraph someone is editing?

You: A lock needs a round trip before typing, which kills local echo, blocks people, and leaves "which paragraph" unclear for selections that span several. Two people rarely type into the exact same spot, so optimistic concurrency with a guaranteed repair step wins.

Interviewer: How does undo work with several editors?

You: Undo reverses my last op, not the document's last op. The client keeps the inverse of each of its own ops, transforms it past everything since, and submits it as a new op. If I typed "hello" and someone then typed before it, my undo deletes "hello" at its new position and their text stays.

Side features hang off the op stream, not the hot path: search re-indexes after a debounced burst from the latest snapshot, and mentions and audit consume the same events. A 10 MB doc loads the visible part first and snapshots more often.

What's being probed: whether the core mechanisms answer each question in one breath.

Concept — optimism, selective undo and bounded transform work. Optimistic concurrency applies now and repairs later, which is right when conflicts are rare and waiting feels broken. Undo is a new op inverting your own edit. Neither side ever transforms against the whole history — only the unseen gap.

Takeaway: Editors are optimistic; transform only against the unseen gap; undo is a new op.

Quick reference

The 60-second version

Clients hold a full local copy and apply their own edits instantly; edits travel as small ops over a WebSocket, each with the revision it was based on. Every document has exactly one owning collab server, chosen by consistent hashing and held with a fenced lease, so ops get one total order. The owner transforms each op against what the client had not seen, assigns the next revision, writes it durably, then acks and broadcasts; clients transform incoming ops against their pending ones, and everyone converges. Storage is an append-only op log in a wide-column store plus periodic snapshots; open is snapshot plus a short replay, and the log is the version history. Presence is in memory and transformed like a zero-length insert. Offline edits queue with (clientId, seq) and rebase on reconnect. A crashed owner is replaced in seconds and nothing acked is lost. CRDTs are the answer when there is no central server or offline is the main mode.

Walkthrough 3 — Stock broking: live prices, orders and risk

Building on WhatsApp and Google Docs: WhatsApp showed how to hold millions of connections; Google Docs showed how to give each shared object one owner that orders every change. A broker needs both at once — millions of screens receiving prices, and one owner per account deciding, in order, whether each new order can be afforded. It adds two things neither had: money that must never be double-spent, and recovery that has to be instant.

Scope and requirements

Interviewer: Design a retail stock broking platform — the app people use to watch prices and buy and sell shares.

You: A broker sits between customers and the stock exchange. Customers never talk to the exchange directly: the broker checks each order, sends it to the exchange, and reports back what happened. Before designing, I'd pin down a few things:

Interviewer: You're the broker. Shares and derivatives, 12 million customers, up to 2 million connected at the open. You connect to the exchanges directly.

You: Then the requirements.

Functional:

• Live prices for a watchlist, and market depth (the best five bids and offers) for one stock at a time.
• Place, modify and cancel orders: market (buy now at whatever price is available), limit (only at this price or better) and stop-loss (becomes an order once the price crosses a trigger).
• A pre-trade risk check on every order: does the customer have the money or margin for it?
• Real-time order updates: accepted, rejected, partly filled, filled, cancelled.
• Positions (today's trades), holdings (shares owned), funds, and order history.

Non-functional:

• Survive the market open. On NSE, for example, orders can be entered from 09:00 in a short pre-open session, and continuous trading starts at 09:15 — so demand arrives as a cliff, not a curve.
• Order path inside our systems: p99 under 20 ms, excluding the exchange.
• Prices on screen within about 250 ms of the exchange publishing them.
• No lost orders and no duplicate orders, ever. Every order event kept for the audit trail the regulator requires.
• 99.99% availability during market hours; maintenance only after the close.

Two very different workloads share one app: a firehose of prices where only the latest value matters, and a trickle of orders where every single one must be exact. I'll design them separately.

What's being probed: whether you clarify broker versus exchange, and spot that one product hides two workloads.

Concept — split by the cost of a wrong answer. A stale price on screen for 200 ms costs nothing; the next update fixes it. A lost or duplicated order costs real money and a regulatory report. So the price path can drop intermediate values and the order path cannot drop anything. Amazon (R19) applies the same idea to a cart versus a stock count. Saying it early lets every later choice be justified by one sentence.

Takeaway: When one product has a lossy-tolerant path and an exact path, design them as two systems.

Capacity

Interviewer: Size it.

You: Assumptions: 2M connections at the open; 20 instruments per watchlist on average; about 5,000 instruments actively trading; 15M orders a day; each order produces about 5 events (accepted, sent, acknowledged, one or two fills); continuous trading lasts 6¼ hours, about 22,500 seconds.

The price numbers need one more step. If we forwarded every exchange update to every interested screen, a popular stock changing 200 times a second, watched by 500,000 people, would alone produce 100 million messages a second. Instead, each connection gets one frame every 250 ms containing only the instruments that changed. That is 2M × 4 = 8M frames a second. At about 8 changed instruments per frame and 40 bytes each, roughly 320 bytes per frame, the total is about 2.6 GB/s — around 21 Gbps across the fleet. With 200 gateways holding 10,000 connections each, every gateway sends about 40K frames and 100 Mbps a second. That is easy.

Storage: order events 15M × 5 × 500 B ≈ 37 GB a day, kept for years. Risk state is tiny: 12M accounts × ~1 KB = 12 GB, which fits in memory across a handful of machines.

The order path is small and must be exact; the price path is huge and only needs to be fresh.

What's being probed: whether the arithmetic changes the design.

Concept — fan-out arithmetic and conflation. Multiply update rate by subscribers before deciding how to deliver. The raw product is impossible; batching per connection per time slot makes it linear in the number of clients. Conflation means "keep only the latest value per key until the next send". It is safe for prices and wrong for chat messages, where every message must arrive. R21 looks at fan-out in general.

Takeaway: Size fan-out as rate × subscribers, then conflate whatever only needs its latest value.

API

Interviewer: Show me the API.

You: Two channels. One WebSocket per app carries prices and order updates; REST carries commands.

POST /v1/orders
Idempotency-Key: 2f6c1e0a-7d4b-4a51-9b1e-5c2d9a7e3f10
{ "exchange": "NSE", "instrument": "INFY", "side": "BUY", "quantity": 10,
  "orderType": "LIMIT", "price": 152050, "product": "DELIVERY", "validity": "DAY" }

HTTP/1.1 202 Accepted
{ "orderId": "250912000184321", "status": "RECEIVED" }

• price is an integer in paise — ₹1,520.50 — never a floating-point number. It must be a multiple of the instrument's tick size, and inside the exchange's price band for the day.
• 202 Accepted, because the real answer comes from the exchange a few milliseconds later, over the WebSocket.
• The Idempotency-Key makes a double-tap or a retry return the same orderId (R3). The orderId is also what we send to the exchange as our client order id, which the exchange requires to be unique for the day — so a duplicate is refused at both ends.
• PUT /v1/orders/{id} modifies (price or quantity), DELETE /v1/orders/{id} cancels; GET for orders, trades, positions, holdings and funds.

On the WebSocket, the client subscribes by instrument and mode:

{ "action": "subscribe", "mode": "QUOTE", "instruments": [408065, 738561] }

Modes are last price only, a quote (last price, volume, day's high and low), or full depth. Frames are binary, not JSON, because at 8M frames a second every byte matters. Order updates travel on the same socket, routed to the user like WhatsApp messages.

Data model

Interviewer: Data model, and where each piece lives.

You:

Notice what is not in a database: the risk state. It lives in memory, and its durability comes from the journal it is built from — the move R18's introduction called "persist before acknowledging, then replay".

High-level design

Interviewer: Draw it.

You: Two flows share the client and the WebSocket, and nothing else. Play the order flow, then the price flow, then the failover.

An order, end to end:

1. The app sends POST /v1/orders. The gateway checks the idempotency key; the order service validates the order against the instrument master and writes RECEIVED.
2. The order service sends a PlaceOrder command to the risk shard that owns the account. The command is appended to that shard's replicated journal before anything else happens.
3. The shard checks available margin and blocks the amount the order needs, then emits Accepted — or Rejected: insufficient funds.
4. The exchange gateway sends the order on one of its exchange sessions, after writing the outbound message to its session log.
5. The exchange acknowledges with its own order number; later it sends fills. Each fill flows back to the risk shard (turning blocked margin into used margin), to the order events table, and to the user's socket.

What's being probed: separating the stateless request path from the stateful components, and naming what each stateful part holds.

Concept — stateless around a stateful core. Everything that can be stateless is: the gateway and order service scale by adding copies. Only three parts hold state, and each holds a different kind. Risk shards hold exact money state, with replay for durability. Exchange gateways hold protocol state that must survive a reconnect. Tick gateways hold disposable state that a client can rebuild by resubscribing. Saying which kind each is tells the interviewer you know how each one recovers.

Takeaway: Keep the stateful parts few, and classify each by how it recovers: replay, resync or rebuild.

Deep dive — live prices to two million screens

Interviewer: Walk me through a price, from the exchange to a phone.

You: Four hops.

1. The exchange broadcasts. Exchanges publish prices by UDP multicast — one packet sent once, received by every subscriber on the network — usually on two identical lines, A and B, for redundancy. Every packet carries a sequence number.
2. Feed handlers fix the stream. They read both lines, keep the first copy of each sequence number and drop the duplicate. If a number is missing on both lines, that is a gap: they request a retransmission, or rebuild from the exchange's periodic snapshot, and meanwhile mark the affected instruments "stale" rather than show a wrong price. Then they normalise each update into our format, keyed by instrument.
3. An in-memory bus fans out to gateways. Each tick gateway subscribes only to the instruments its own clients want. It is a fan-out tree: the feed handler sends INFY once to each interested gateway, not once per screen.
4. Tick gateways conflate and send. Each connection has a small table — the latest value of each instrument it subscribes to — and a "changed" flag per entry. Every 250 ms the gateway sends one frame per connection with only the changed entries, then clears the flags.

That last step is where the scale problem disappears. A slow phone on a weak signal never builds a backlog: if it misses a send, the next frame simply carries newer values. For prices, the latest value replaces every older one — so drop old prices, never connections.

When a client connects or subscribes, it first gets a snapshot of current values, then deltas. When a gateway dies, its clients reconnect elsewhere with jitter, resubscribe and get a fresh snapshot. Nothing is lost, because nothing on the gateway was ever the truth.

What's being probed: fan-out without unbounded queues, and recovery of disposable state.

Concept — latest-value semantics versus every-message semantics. WhatsApp must deliver every message, so a slow device needs a durable inbox. A price feed only needs the newest value, so a slow device needs nothing but one slot per instrument. Choosing the semantics first decides the whole design: inbox and acknowledgements, or conflation and snapshots. R14's back-pressure options are drop, buffer or slow the sender; for prices, dropping stale values is the right one.

Takeaway: Decide whether each message matters or only the latest value does; conflate the second kind.

Deep dive — the risk check, one owner per account

Interviewer: A customer has ₹1 lakh and sends two ₹80,000 orders a millisecond apart. Stop the second one.

You: This is the same race as two people booking the last seat (R19), and the same rule applies: the check and the update must be one step. There are two ways to make it one step.

Option 1 — a database row lock per account. UPDATE funds SET blocked = blocked + 80000 WHERE account = ? AND available - blocked >= 80000. It is correct, and right for a small broker. At the open, though, active traders send dozens of orders a second, every order pays a database round trip, and a busy account's row becomes a queue.

Option 2 — one owner per account, in memory. Accounts are hashed to risk shards. Each shard is a single thread that takes commands one at a time from its journal and owns its accounts outright. Two orders for the same account are simply processed one after the other — no locks, because no other thread ever touches that account. This is R13's single-writer principle; the LMAX exchange made the pattern well known.

// one thread per shard; the journal has already made each command durable
while (running) {
    Command c = journal.next();
    switch (c) {
        case PriceUpdate p -> prices.put(p.instrument(), p.price());   // prices enter as commands too
        case PlaceOrder o -> {
            Account a = accounts.get(o.account());
            long need = margin.required(o, prices.get(o.instrument()));
            if (a.available() < need) emit(new Rejected(o.orderId(), "INSUFFICIENT_FUNDS"));
            else { a.block(o.orderId(), need); emit(new Accepted(o.orderId())); }
        }
        case Fill f -> accounts.get(f.account()).applyFill(f);         // blocked → used
        case Closed x -> accounts.get(x.account()).release(x.orderId()); // cancel or reject: unblock
    }
}

The first order blocks ₹80,000, leaving ₹20,000 available, so the second is rejected. Blocking margin is itself a reservation: block on order (try), keep it as used margin on a fill (confirm), release it on a cancel or rejection (cancel). It is R2's try–confirm–cancel, and the whole of R19.

Interviewer: The state is in memory. The machine dies at 09:15:02.

You: That is why every command goes into a replicated journal before the shard processes it. Recovery is "load the last snapshot, replay the journal since". Replay takes seconds, which is too slow at the open. So each shard has a hot standby: a second machine reading the same journal and running the same code, holding an identical copy at all times. When the primary's lease lapses, the standby takes over with a new epoch (R8), and old-epoch outputs are refused downstream. Failover takes well under a second.

The standby only stays identical if the shard is deterministic. Same commands in the same order must give exactly the same state:

• no reading the clock inside the loop — time arrives inside the command;
• prices arrive as commands too, so a replay sees the prices the primary saw;
• no randomness, no iterating over unordered maps, and the same code version on both machines.

Deterministic logic plus a replicated input log gives you a hot copy for free — the replica does not copy state, it recomputes it.

What's being probed: single-writer concurrency, durability of in-memory state, and why determinism matters.

Concept — state machine replication. If a program is deterministic, its state is a pure function of its input history. So you replicate the inputs — a log — rather than the state, and any number of machines replaying the log hold the same state. Raft and Paxos (R8) are ways of agreeing on that log; the matching engines of exchanges, and Google Docs' op log, use the same idea. The cost is discipline: one non-deterministic line, such as reading the clock, silently makes the replicas diverge.

Takeaway: Replicate the input log, not the state, and keep the logic strictly deterministic.

Deep dive — talking to the exchange

Interviewer: How do orders reach the exchange, and what if the connection drops mid-order?

You: Each connection to the exchange is a session — FIX, the financial industry's standard message protocol, or the exchange's own binary protocol — over leased lines. A session is stateful in a specific way: every message in each direction carries a sequence number that goes up by one. Both sides track the last number they saw. Heartbeats prove the line is alive.

• If the exchange sees a gap in our numbers, or we see one in theirs, the side that noticed sends a resend request, and the other side replays the missing messages marked "possible duplicate". So every outbound message is written to the session's local log before it is sent; otherwise we could not answer a resend after a restart.
• The exchange caps how many messages a second each session may send. We run many sessions and spread accounts across them, and every message about one order — new, modify, cancel — goes on that order's session.
• At the open, orders queued overnight are released through a token bucket per session (R17's rate limiter), so we never exceed the exchange's limit and have orders rejected.

Now the drop. We send an order, and the connection dies before the acknowledgement arrives. Did the exchange receive it?

A dropped connection mid-order is an unknown outcome — never resend it as a new order

Resending blindly could buy the shares twice. Instead:

1. Reconnect and log on. The session resynchronises sequence numbers: any of our messages the exchange missed are resent, flagged as possible duplicates, and any of its messages we missed are resent to us.
2. If the order's fate is still unclear, ask: an order-status request by our client order id.
3. As a backstop, the client order id is unique for the day, so a true duplicate would be rejected by the exchange anyway.

That is R0's unknown outcome and R3's idempotency key, applied to a protocol designed decades before either name was common.

Interviewer: And fills?

You: The exchange sends an execution report for each partial or full fill. The exchange gateway turns each one into a Fill command for the account's risk shard, an order event in Postgres, and a push to the user's socket through the session registry — exactly how WhatsApp finds a device. Fills carry the exchange's trade id, so a replayed fill is recognised and applied once.

Deep dive — if they mean the exchange: the matching engine

Interviewer: Suppose I'd said "design the exchange". What is different?

You: The core becomes the matching engine, and it is the purest example of this section's idea.

For each instrument the engine keeps an order book: buy orders (bids) sorted from highest price down, sell orders (asks) sorted from lowest price up. The highest bid and lowest ask are the best prices; the gap between them is the spread. Matching follows price-time priority:

• a better price always goes first;
• at the same price, the order that arrived first goes first;
• a new order that crosses the spread trades against the resting orders, at their price, until it is filled or no price is good enough; whatever is left rests in the book.

Data structures: price levels in a sorted structure (or an array indexed by tick, since prices are discrete), a first-in-first-out queue of orders at each level, and a map from order id to its queue position, so a cancel takes constant time.

The engine runs one thread per instrument group. A sequencer stamps every incoming order with a global sequence number and journals it. The engine is deterministic, so replicas replaying the journal hold identical books — the same state machine replication as the risk shards. Market data is simply the stream of changes to the book, published by multicast.

Step through the book below: a resting order joining the queue, a sell order matching in time order, a buy that takes the better price first, cancels and modifies, a market order sweeping several levels, and a crash recovered by replay.

What's being probed: whether you know how an order book matches, and why exchanges run single-threaded deterministic engines.

Concept — the order book as a state machine. Price-time priority makes matching a pure function: given the book and the next order, the trades and the new book are fully determined. That is why one thread per instrument is not a limitation but the design — it removes locks and makes replication a matter of replaying inputs. Fairness also falls out of it: whoever the sequencer stamps first is served first.

Takeaway: Price first, then time; one thread per instrument; replicate by replaying the sequenced input.

Failures and scale

Interviewer: What breaks at the open, and how does it recover?

You:

Each stateful part recovers in its own way: risk shards by replay, exchange sessions by resync, gateways by rebuild. The mitigations table is just those three verbs applied.

Curveballs

Interviewer: The customer double-taps "Buy".

You: Both taps carry the same idempotency key, so the second returns the first orderId. Even if a bug let a second order through, it would carry the same client order id, and the exchange would refuse it.

Interviewer: Stop-loss orders — who watches the price?

You: Two ways. The exchange can hold a stop-loss order and trigger it itself, which is simplest and the default. For triggers the exchange does not support, such as a trigger that lasts for months, the broker holds them. Then it is a scheduler driven by price instead of time (R17). Triggers are indexed by instrument and price, one owner per instrument evaluates them on every price update, and a fired trigger becomes a normal order through the normal risk check, with an idempotency key derived from the trigger id so it can never fire twice.

Interviewer: The market-wide circuit breaker trips and trading halts.

You: The exchange stops matching and rejects new orders. We stop sending, keep accepting cancels, tell users clearly, and do not queue new orders silently. Re-open is a smaller version of the morning open, with the same token buckets and pre-scaling.

Interviewer: How do you know your records match the exchange's?

You: End-of-day reconciliation: compare every trade in our order events with the exchange's trade file and the clearing corporation's obligations, and investigate every break. That is R6 and R7. Real-time correctness is necessary, but the daily reconciliation is what proves it.

Quick reference

The 60-second version

A broker has two workloads. Prices are a firehose where only the latest value matters: the exchange multicasts on two lines, feed handlers keep the first copy of each sequence number and repair gaps, an in-memory bus fans out to tick gateways, and each gateway sends every connection one conflated frame per 250 ms with only the changed instruments. Gateways hold nothing that cannot be rebuilt by resubscribing. Orders are a trickle where every one must be exact: an idempotency key that doubles as the exchange's client order id, a pre-trade risk check on a single-threaded shard that owns the account in memory, margin blocked as a reservation, every command written to a replicated journal first, and a deterministic hot standby that takes over in under a second. Exchange sessions log every message before sending, resync by sequence number after a drop, and resolve unknown outcomes by asking, never by resending. Fills flow back to risk, the order history and the user's socket, and the day ends with reconciliation against the exchange. If the question is the exchange itself, the core is a price-time-priority order book on one thread per instrument, replicated by replaying its sequenced input.

Walkthrough 4 — Zoom: video calls, WebRTC and media servers

Building on WhatsApp and stock broking: WhatsApp held connections and kept the truth in a durable inbox; stock broking showed that a price feed can drop old values as long as the newest one arrives. Here the state is a live media session held by a media server — who is sending which streams, at what quality, to whom. And unlike every earlier walkthrough, late data is worthless: a message is retried until it arrives, an order is resolved until its fate is known, but a video frame that arrives after its moment has passed is simply dropped. That one fact changes the transport, the servers, the failure handling and the sizing.

Scope and requirements

Interviewer: Design a video conferencing service like Zoom.

You: I'll design with standard WebRTC — the browser's built-in way to send real-time audio and video — because it is public and well understood. A real product may use its own protocol in its native apps, but the architecture is the same. First, some questions:

Interviewer: Meetings average about 5 people, up to 1,000. Webinars up to 50,000 viewers. Users worldwide, heavily in India, Europe and the US. Screen share, chat and cloud recording, yes. End-to-end encryption as an option. All platforms. Ignore dial-in.

You: Then the requirements.

Functional:

• Create, schedule and join meetings with a link; waiting room and host controls (mute, remove, make co-host).
• Audio and video for every participant; speaker view and gallery view.
• Screen share; in-meeting chat; cloud recording.
• Webinars: panelists speak, attendees watch, and a panelist can promote an attendee.
• Optional end-to-end encryption.

Non-functional:

• Mouth-to-ear delay — from one person speaking to another hearing — under about 150 ms within a region, which is ITU-T G.114's guidance for comfortable conversation, and under about 300 ms across continents.
• Keep working on a poor network: lower the picture quality before ever freezing the audio.
• Joining a meeting takes under 2–3 seconds.
• A media server failure interrupts a meeting for a few seconds, not ends it.
• Chat messages and recordings are never lost, even though media frames can be.

Media is useful only while it is fresh: a packet that would arrive after its playout moment is worth less than nothing, so this system is built to drop data on purpose, never to queue it.

What's being probed: whether you see that real-time media has a delivery rule unlike anything earlier in the pack.

Concept — deadline-bound data. Most systems treat a lost message as a problem to fix by retrying: WhatsApp's inbox, a broker's order resend. Live media has a deadline. Audio is played every 20 ms; a packet that arrives 300 ms late cannot be used, and waiting for it would freeze everything behind it. So the system's job is to deliver as much as possible before the deadline and to give up gracefully after it. Stock broking's conflation (earlier in R18) was a softer version: there, a newer price replaced an older one; here, a late frame is simply skipped.

Takeaway: For live media, prefer losing data to delaying it.

Capacity

Interviewer: Size it.

You: These are assumptions, not any real company's numbers. Peak is a weekday top of the hour, when many meetings start at once.

First, the media traffic, assuming each camera sends all three quality layers at once (simulcast, explained in a deep dive) and each viewer in a 5-person meeting gets one large picture and three small ones.

Upload per participant  = 1.2 + 0.4 + 0.12 Mbps video + 0.04 audio   ≈ 1.8 Mbps
Download per participant (5-person meeting, speaker view):
    1 large tile at 720p         = 1.20 Mbps
    3 small tiles at 180p        = 0.36 Mbps
    4 audio streams × 0.04       = 0.16 Mbps
                                 ≈ 1.7 Mbps

Total into media servers  = 10M × 1.8 Mbps ≈ 18 Tbps
Total out of media servers = 10M × 1.7 Mbps ≈ 17 Tbps

Now, how many media servers:

Participants per server by egress = 10 Gbps ÷ 1.7 Mbps ≈ 5,900
Planned at 60%                    ≈ 3,500 participants ≈ 700 five-person meetings
Servers at peak                   = 10M ÷ 3,500 ≈ 2,900
With per-region spare capacity    ≈ 4,000 servers across regions

Packet rate per participant = (1.8 + 1.7) Mbps ÷ (1,200 B × 8 bits) ≈ 365 packets/s
Per server                  = 3,500 × 365 ≈ 1.3 million packets/s

The packet rate matters more than the megabits. Each packet is decrypted on the way in and encrypted again for each receiver on the way out, so the server's CPU cost is per packet — and it never decodes a single video frame.

Signalling — the control messages for joining, muting and so on — is tiny by comparison:

Average joins     = 10M ÷ 2,400 s ≈ 4,200 joins/s
Top of the hour   : 30% of peak users join within 3 minutes = 3M ÷ 180 s ≈ 17,000 joins/s
Messages per join ≈ 10 (join, offer, answer, candidates, subscriptions)
Peak signalling   ≈ 170,000 messages/s, over 10M WebSockets
At 100,000 sockets per gateway (the WhatsApp figure) ≈ 100 gateways

Two more numbers shape the design:

Relay (TURN) traffic: assume 5% of users are on networks that block UDP
    500,000 × (1.8 + 1.7) Mbps ≈ 1.75 Tbps through relays — about 10% of media

Recording: assume 2% of meetings record; composite at 1.5 Mbps
    2M × 2% = 40,000 recordings
    per hour = 1.5 Mbps × 3,600 s ÷ 8 ≈ 675 MB
    storage  = 40,000 × 675 MB ≈ 27 TB per peak hour

Therefore:

• Media servers forward; they do not decode. At 1.3 million packets a second per server, there is no budget for decoding and re-encoding video. That rules out mixing everything on the server for ordinary meetings.
• An ordinary meeting fits easily on one server — 700 of them do — so "one server per meeting" is natural. Only the largest meetings and webinars need several.
• Signalling is an ordinary service. It is WhatsApp-sized, not media-sized, and is scaled like WhatsApp's gateways.
• Distance is the enemy. A 150 ms budget cannot absorb a 250 ms round trip to another continent, so media servers must sit near participants.

What's being probed: whether the numbers drive the topology, and whether you size packets as well as bits.

Concept — bandwidth and packets, not compute. In video calls the expensive resources are network egress and packets per second, because a forwarding server touches every packet but never decodes one. Sizing in megabits alone hides the per-packet encryption cost; sizing in packets alone hides the network card limit. Doing both tells you how many participants a server holds, and therefore whether a meeting fits on one. R21 sizes fan-out the same way.

Takeaway: Size media servers by egress and packet rate, then check that a typical meeting fits on one.

API

Interviewer: What does the API look like?

You: Three channels. REST for meeting management and joining. A WebSocket for signalling — the control conversation. And the media itself, which does not go through any API: it flows as encrypted UDP packets directly between the app and a media server.

POST /v1/meetings/842-1173-905/join
{ "displayName": "Alice", "clientSessionId": "c7d1e9a2-4b6f-4e0b-8d3a-1f5c2b7e9a40" }

HTTP/1.1 200 OK
{ "participantId": "p_8f21",
  "signalling": "wss://sig-bom.example.com/signal",
  "joinToken": "eyJhbGciOi...",
  "iceServers": [
    { "urls": "stun:stun-bom.example.com:3478" },
    { "urls": ["turn:turn-bom.example.com:3478?transport=udp",
               "turns:turn-bom.example.com:443?transport=tcp"],
      "username": "1760004000:p_8f21", "credential": "Zm9vYmFy..." } ] }

• The relay credentials are short-lived: the username contains an expiry time and the credential is an HMAC of it with a secret the relay servers share. Nothing is stored, and a leaked credential stops working within hours.
• clientSessionId makes joining idempotent (R3). If Alice's laptop drops and rejoins, the same id replaces her old entry instead of leaving a frozen "ghost" tile beside the new one.

On the signalling socket, after joining, the server tells the client which media server to use and with which epoch — the ownership number explained later:

{ "type": "joined", "meetingId": "842-1173-905", "sfu": "sfu-bom-1.example.com",
  "epoch": 7, "participants": [ { "id": "p_3a10", "name": "Bob", "tracks": ["audio", "cam"] } ] }

Data model

Interviewer: What is stored where?

You:

Nothing on the media path is ever written down: a meeting's durable state is a few kilobytes of control data, and the megabits a second are forwarded and forgotten. That is why a media server's crash costs a few seconds of picture, not data.

High-level design

Interviewer: Draw it.

You: Two planes. The signalling plane decides who is in the meeting and where their media goes; the media plane moves the packets. Play the join first, then media with quality layers, then the bad network, the failover and the webinar.

A join, end to end:

1. Alice opens the link. POST /join checks the passcode and returns a token and relay credentials. Her app opens the signalling WebSocket to the nearest gateway.
2. The gateway sends join to the meeting controller that owns meeting 842-1173-905 (meetings are hashed to controllers). The controller looks up the registry: the meeting is on sfu-bom-1, epoch 7. If it were the first join, the allocator would pick the least-loaded SFU near Alice and create the entry.
3. Alice's app asks the STUN server for her public address, and creates an SDP offer describing what she wants to send and receive. The offer goes over signalling to the SFU, which replies with an SDP answer.
4. Alice's app and the SFU run ICE connectivity checks to find a working network path, then a DTLS handshake to agree on encryption keys.
5. Encrypted media flows over UDP: Alice sends her camera in three quality layers; the SFU forwards each other participant's best-fitting layer to her.
6. The controller tells everyone else "Alice joined, with these tracks", and their apps subscribe to her streams.

What's being probed: separating control from data, and knowing that the media server is a forwarder.

Concept — control plane and data plane. The signalling plane is low-volume, must be reliable and ordered, and runs over ordinary TCP and WebSockets. The media plane is enormous, must be fast, and tolerates loss. Splitting them lets each use the right transport and scale separately — the same split as a network router's control and forwarding planes, or the stock broker's order path and price path earlier in R18.

Takeaway: Signalling decides, media flows; never send media through the signalling path.

Deep dive — connecting: signalling, ICE, STUN, TURN and DTLS-SRTP

Interviewer: How does Alice's laptop actually get packets to the media server? And why UDP?

You: Start with why UDP. TCP guarantees every byte arrives, in order. If one packet is lost, TCP holds back everything after it until the retransmission arrives — head-of-line blocking — and it also halves its sending rate. On a call, one lost packet would freeze audio and video for a full round trip or more, and the data released afterwards would already be late. UDP just sends packets, with no ordering or retransmission, and lets the application decide what to do about loss. For media that is exactly right: skip the lost 20 ms of audio, conceal it, and carry on. Media travels as RTP packets (each with a sequence number and a timestamp), with RTCP alongside for feedback such as loss reports.

Now the connection. WebRTC deliberately does not define signalling; the application provides it. Ours is the WebSocket.

sequenceDiagram
    participant A as Alice app
    participant S as Signalling and controller
    participant T as STUN server
    participant F as SFU
    A->>S: join meeting, token
    S-->>A: joined, SFU address, epoch 7
    A->>T: binding request
    T-->>A: your public address
    A->>S: SDP offer
    S->>F: SDP offer
    F-->>S: SDP answer
    S-->>A: SDP answer
    A->>F: ICE connectivity checks
    A->>F: DTLS handshake
    A->>F: SRTP media
    F->>A: SRTP media

• SDP offer and answer. SDP, the Session Description Protocol, is a text description of a media session: which tracks, which codecs (Opus for audio; VP8, VP9, H.264 or AV1 for video), the ICE credentials, and the fingerprint of the DTLS certificate. One side offers, the other answers with what it accepts.
• ICE candidates. A candidate is one possible address to reach a client. ICE (Interactive Connectivity Establishment) gathers three kinds: host (the laptop's own address, often private, behind a home router), server-reflexive (the public address the router presents, learned by asking a STUN server) and relay (an address on a TURN server). Candidates can be sent as soon as they are found ("trickle ICE"), so checks start before gathering ends. Both sides then send STUN checks on candidate pairs and pick the best one that works.
• STUN is a tiny service that answers "what address do your packets come from?" It carries no media.
• TURN is a relay. Some corporate and hotel networks block UDP or only allow outbound connections to port 443. The client then connects to a TURN server over TCP or TLS on 443, and the relay forwards its media. It works, but TCP's head-of-line blocking returns on that leg, so quality is worse — a fallback, not the default.
• DTLS-SRTP. Once a path works, the two ends run a DTLS handshake (TLS adapted for UDP) over it, and derive keys for SRTP, the encrypted form of RTP. Each side checks that the other's certificate matches the fingerprint in the SDP. So the signalling channel must itself be secure (WSS), or an attacker could swap fingerprints.

Two things make our case easier than calls between two browsers. Our SFUs have public addresses, so most clients connect with their host or server-reflexive candidate and no relay; servers usually run a simplified "ICE-lite" mode. And because every client connects to the SFU rather than to each other, there is one connection per client, whatever the meeting size.

DTLS-SRTP encrypts each hop, not end to end: the SFU decrypts every packet and re-encrypts it for each receiver. So the operator's server could in principle access the media. That is the default in WebRTC conferencing, and the end-to-end option is a later deep dive.

What's being probed: whether you know what each WebRTC piece does, rather than reciting the acronyms.

Concept — NAT traversal. Most devices sit behind a router that shares one public address among many private ones (NAT). A server cannot simply "call" a laptop at home. ICE solves this by collecting every address a client might be reachable at and testing them all; STUN reveals the public one; TURN is the always-works relay when nothing else does. Think of trying the front door, then the side door, then calling a friend to pass messages through the window.

Takeaway: Signalling exchanges descriptions; ICE finds a path; DTLS agrees keys; SRTP carries media over UDP.

Deep dive — mesh, MCU or SFU, and simulcast

Interviewer: Why a media server at all? Browsers can connect to each other directly.

You: For two people they should. Beyond that, compare three topologies for an N-person call where each camera produces a stream of bitrate B. Take N = 10 and B = 1.2 Mbps.

• Mesh — everyone sends to everyone directly. No server cost. But each client encodes and uploads N − 1 copies.
• MCU — a Multipoint Control Unit decodes every stream, composes one picture (a grid), re-encodes it and sends each client a single stream. Clients are cheap; the server does heavy video work.
• SFU — every client uploads once; the server forwards copies, choosing for each receiver which streams and which quality, without decoding.

The arithmetic:

N = 10, B = 1.2 Mbps

Mesh
  upload per client   = (N − 1) × B = 9 × 1.2   = 10.8 Mbps   ← home uplinks are often 5–20 Mbps
  encodes per client  = up to 9, one per receiver's quality; or 1, forcing one quality on all
  server              = none

MCU
  upload per client   = 1.2 Mbps; download = 1 composite ≈ 1.5 Mbps
  server              = decode 10 streams + encode a composite per layout, every frame
  extra delay         = decode + compose + encode ≈ tens of ms or more

SFU (with three layers per camera)
  upload per client   ≈ 1.8 Mbps (720p + 360p + 180p)
  download per client = 1 × 1.2 + 8 × 0.12 (gallery thumbnails) ≈ 2.2 Mbps
  server egress       = N × 2.2 ≈ 22 Mbps; streams forwarded = N × (N − 1) = 90
  server CPU          = per-packet encryption only; no decoding

Mesh dies on upload around four or five people. An MCU's delay and per-meeting CPU are expensive at 2 million meetings, though it is useful where a client can take only one stream, such as a phone line. The SFU wins: clients upload once, the server only copies packets, and each receiver can be given different quality. An SFU turns a video call into packet routing: it never decodes, so one server forwards hundreds of meetings, and each receiver gets the quality its own network can carry.

The SFU's cost grows with N × (N − 1) forwarded streams, which is fine at 5 and painful at 1,000; the large-meeting deep dive deals with that.

Interviewer: How does the SFU give Bob, on bad Wi-Fi, a different quality from Alice on fibre — without decoding?

You: The sender encodes several versions, and the SFU picks one.

• Simulcast: the camera is encoded two or three times independently — say 720p, 360p and 180p — and all are sent. The SFU forwards one of them to each receiver. Simple, widely supported, costs the sender about 50% more upload than the top layer alone.
• SVC, Scalable Video Coding: one stream built in layers, where a base layer is watchable alone and each extra layer adds resolution (spatial) or frame rate (temporal). The SFU strips layers a receiver cannot take by reading a small descriptor in each packet. VP9 and AV1 support spatial layers in WebRTC; VP8 and H.264 are mostly used with temporal layers plus simulcast. SVC is more efficient but needs more capable encoders.

To choose, the SFU needs to know each receiver's bandwidth. That is bandwidth estimation. The receiver reports when each packet arrived (WebRTC's transport-wide congestion control feedback); the sender — here the SFU — watches whether delays are growing, which signals a filling queue somewhere on the path, and whether packets are being lost, and adjusts its estimate. A growing delay is detected before loss starts, so the estimate drops early.

Each forwarded stream then follows a simple loop:

every ~200 ms, for each receiver:
    budget = bandwidth estimate − audio − screen share
    for each video the receiver wants, most important first (active speaker, then pinned, then thumbnails):
        pick the highest layer whose bitrate fits in the remaining budget,
        but never above the size the receiver will display it at
    if the chosen layer changed:
        switch at the next keyframe of the new layer (ask the sender for one if needed)

Two details make switching work. A receiver can only start decoding a different simulcast layer at a keyframe — a frame that does not depend on earlier frames — so the SFU waits for one, or requests one from the sender. And the SFU rewrites sequence numbers and timestamps so that, to Bob's decoder, the switch looks like one continuous stream.

The sender adapts too: the SFU estimates Alice's uplink, and if it cannot carry all three layers, her browser stops sending the top one. And because nobody needs a 720p version of a thumbnail, the SFU tells senders when no one is watching a layer, so they stop encoding it — layer pausing, which saves upload and battery.

What's being probed: the topology trade-off with numbers, and how adaptation works without transcoding.

Concept — let the edge choose. An MCU adapts by doing the work centrally; an SFU adapts by having senders prepare a few versions and choosing between them per receiver. It is the same idea as Netflix's bitrate ladder in R20, but decided by the server every few hundred milliseconds instead of by the player for each segment. The price is a little extra upload from every sender.

Takeaway: Mesh for two, SFU with simulcast or SVC for everything else, MCU only where a client can take just one stream.

Deep dive — loss and jitter: when late is worthless

Interviewer: Bob's Wi-Fi loses 5% of packets and they arrive unevenly. What happens?

You: Two problems: packets arrive at uneven intervals (jitter), and some never arrive (loss).

Jitter buffer. Bob's app does not play packets the moment they arrive. It holds them briefly in a jitter buffer, reorders them by sequence number, and plays them out at a steady pace. The buffer adapts: on a smooth network it shrinks to 20–40 ms; on a jittery one it grows, perhaps to 100–200 ms. Every millisecond it holds is a millisecond of delay, so it is the main dial between smoothness and delay. A typical one-way budget within a region:

capture + encode           20–40 ms
network, client → SFU      10–40 ms
SFU forwarding              1–5  ms
network, SFU → client      10–40 ms
jitter buffer              20–80 ms
decode + render            10–20 ms
total                     ~70–225 ms   → keep the typical case under ~150 ms

Loss. Each packet has a playout deadline — the moment the jitter buffer needs it. Every tool for loss is judged against that deadline:

• NACK — negative acknowledgement. Bob's app sees a gap in sequence numbers and asks for that packet again. The SFU keeps the last second or so of each stream it forwards, so it can answer from its own memory without going back to Alice. One round trip to the SFU is usually 20–60 ms, which often fits before the deadline. Across continents it often does not, and then a NACK is pointless.
• FEC — forward error correction. The sender adds redundant data up front so the receiver can rebuild a lost packet without asking. Opus has an in-band option that carries a low-bitrate copy of the previous audio frame; video can use separate FEC packets. It costs bandwidth all the time, so it is turned up only when loss is observed.
• Concealment. For audio, the decoder stretches or smooths the surrounding sound to cover a lost 20 ms, which is usually unnoticeable.
• Keyframe requests. Video frames depend on earlier frames. If a lost packet cannot be repaired in time, the picture breaks, and every following frame is built on a broken one. Bob's app sends a PLI (Picture Loss Indication: "my picture is broken, send a keyframe"); FIR (Full Intra Request) is a stronger "send a full frame now", used for example when a new receiver starts. The SFU passes the request to Alice, ideally merging requests from many receivers so she sends one keyframe, not fifty.

Keyframes are several times larger than normal frames: a keyframe storm, after a failover or a big join, can itself cause the congestion that made you ask for it. SFUs rate-limit keyframe requests per sender.

So strictly, WebRTC does sometimes ask for a packet again — but only while the answer can still arrive in time. After the deadline the packet is abandoned, never queued. That is the rule from the start of this walkthrough, put into mechanism. Audio is always protected first: Bob can lose his video entirely and still hold the conversation, but not the other way round.

What's being probed: whether you know the loss-recovery toolbox and when each tool is worth using.

Concept — every packet has a deadline. Retransmission is useful only when a round trip is shorter than the time left before playout. FEC spends bandwidth to avoid the round trip; concealment hides what cannot be recovered; a keyframe resets a picture that broke. This is the opposite of R9's at-least-once delivery, where a message is retried until it lands: here a retry that misses its deadline is waste. R14's back-pressure choice for media is always "drop".

Takeaway: Buffer a little, repair what can arrive in time, conceal the rest, and reset video with a keyframe.

Deep dive — one owner per meeting: placement, cascading and failover

Interviewer: How do you make sure everyone in a meeting lands on the same SFU? And what happens when it dies?

You: This is R18's second problem — exactly one owner — applied to a meeting. If half the participants were sent to sfu-bom-1 and half to sfu-bom-2, each half would see only itself: a split-brain meeting. So the assignment meeting → SFU lives in the registry as a lease with an epoch (R8):

• When the first person joins, the allocator picks an SFU near them with spare capacity, and writes 842-1173-905 → sfu-bom-1, epoch 7.
• The SFU renews its lease by heartbeat every second; the lease lasts, say, 5 seconds.
• Join tokens are signed by the controller and name the SFU and the epoch. An SFU refuses a token for another server or an older epoch.
• If an SFU loses its lease — it froze, or it was cut off — it stops serving the meeting, and any messages it sends to the controller with epoch 7 are refused once epoch 8 exists.

The controller itself is the owner of the meeting's control state, also under a lease, hashed by meeting id. Two owners, nested: one for who is in the meeting, one for where its media flows.

Placement. Delay is decided by geography, so SFUs sit in many regions. A meeting is placed near its first joiner, or its expected participants for a scheduled meeting. If participants are spread — Bengaluru and Frankfurt in one call — one SFU means someone has a long path. So we cascade: each region's participants connect to a local SFU, and the SFUs exchange streams over the provider's backbone. The registry then holds one SFU per meeting per region, each with its own lease, and the Mumbai SFU sends Frankfurt one copy of each layer anyone there needs, however many people in Frankfurt watch it. The intercontinental delay is still there, but each person's weak last mile is short, and loss on it is repaired locally.

Failover. Now the SFU dies.

1. Detection. WebRTC's own consent checks take up to 30 seconds to give up — far too slow. Instead the controller sees the missed heartbeats, and clients see no packets for about 2 seconds.
2. The lease lapses after a few seconds. The allocator assigns the meeting to sfu-bom-2 with epoch 8.
3. The controller tells every participant over signalling: "move to sfu-bom-2, epoch 8". Clients open a new connection to it — new ICE checks and a DTLS handshake — and republish their tracks, with jittered start times so 700 meetings' worth of clients do not arrive in the same instant (R18's fourth problem).
4. The new SFU asks every sender for a keyframe, rate-limited, and pictures return.

What was lost, and what is rebuilt:

The SFU holds only state that can be rebuilt from the clients in seconds; anything that must survive lives with the controller or a durable store. That is WhatsApp's rule — the server holds connections, the truth lives elsewhere — applied to media.

A smaller version is far more common: Alice walks from Wi-Fi to mobile data. Her address changes, but the SFU is fine. She does an ICE restart — new ICE credentials, new candidates, new checks — on the same session, and media resumes in about a second without rejoining.

What's being probed: single ownership with fencing, and a precise account of what is lost on failover.

Concept — disposable state versus owned state. An SFU looks stateful — it knows thousands of streams — but almost all of that state is soft: the clients can recreate it by reconnecting. The lease and epoch exist so that during the switch there are never two live owners of one meeting; the controller exists so that the hard state, the roster, is somewhere else. R8 covers leases and fencing; the R18 introduction lists "keep the truth outside the server" as the first durability move.

Takeaway: One leased, epoch-fenced SFU per meeting per region; on failure, reassign, reconnect with jitter and rebuild.

Deep dive — large meetings, webinars, side paths and encryption

Interviewer: Now a 1,000-person all-hands, and a 50,000-viewer webinar.

You: Forwarding everyone to everyone breaks first.

1,000-person meeting, every video to every receiver at 180p:
    per receiver  = 999 × 0.12 Mbps ≈ 120 Mbps     ← impossible on a phone
    SFU egress    = 1,000 × 120 Mbps = 120 Gbps    ← impossible on a server

With last-N = 25 visible tiles at 180p, plus the active speaker at 720p:
    per receiver  = 25 × 0.12 + 1.2 ≈ 4.2 Mbps
    meeting egress = 1,000 × 4.2 Mbps ≈ 4.2 Gbps → spread over several cascaded SFUs

• Active speaker. Each client writes its own audio level into a small RTP header field (a standard extension), so the SFU knows who is speaking without decoding audio. It smooths over a second or so, to avoid flicking between people on every cough.
• Last-N video. Each receiver gets video only for the N participants it can see — the speaker and the current gallery page. Everyone else is a name and an avatar, and if nobody watches a participant at all, their camera layers are paused.
• Top-N audio. The SFU forwards only the three or so loudest audio streams; 997 silent microphones send nothing useful.

A webinar is the extreme: 10 panelists send, 50,000 attendees only receive.

Viewer download ≈ 1 speaker video at 720p + 3 audio ≈ 1.2 + 0.12 ≈ 1.3 Mbps
Total egress    = 50,000 × 1.3 Mbps ≈ 65 Gbps → at 6 Gbps planned per SFU ≈ 11 edge SFUs
Origin → edges  = 10 panelists × 1.8 Mbps × 11 edges ≈ 200 Mbps

So it becomes a tree: panelists connect to an origin SFU, which feeds edge SFUs in each region, which feed the attendees. Attendees send no media, so their joins only add receivers. Promoting an attendee is a signalling change: they start publishing to their edge SFU, which forwards up the tree. Beyond a few hundred thousand viewers, or if a few seconds of delay is acceptable, the better tool is to encode one composite stream and serve it as HTTP segments through a CDN, as Netflix does in R20 — interactivity traded for reach.

Screen share is another video track with different rules. Text must stay sharp, so the encoder is told the content is detail ("content hint"), keeps the resolution high and drops the frame rate to 5–15 frames a second. Its keyframes are large, because a whole screen of text compresses poorly. When bandwidth runs short, the SFU protects, in order: audio, then screen share, then the active speaker's video, then thumbnails.

Chat is not media: every message matters. It goes over the signalling socket to a chat service that stores it durably with a per-meeting sequence number before acknowledging, and a client that reconnects asks for "everything after sequence 412". It is WhatsApp inside a meeting, and it uses none of the media machinery.

Recording is another receiver. A recorder joins as a hidden participant, subscribes to every stream at the top layer, and writes the packets per track to object storage in segments of a few seconds, so a recorder crash loses seconds, not the meeting. After the meeting a batch job decodes and composites the tracks into one video. Live transcription works the same way.

Interviewer: Customers want end-to-end encryption.

You: With DTLS-SRTP alone, the SFU sees plaintext media. For end-to-end encryption, each client encrypts every encoded frame with a key the server never has, before the browser packetises it — WebRTC's insertable streams (the encoded transform API) let the app do exactly that, and SFrame is a standard format for such frame encryption. DTLS-SRTP still protects each hop on top.

The SFU can still do its job, because it never needed the picture: it routes on RTP headers and header extensions — sequence numbers, the audio level, the layer descriptor for SVC — which stay readable. That is why SVC layer information must sit outside the encrypted payload. What the server loses:

• Cloud recording, live transcription and server-side mixing — the server cannot see the content. Recording must happen on a participant's device.
• Transcoding for a client that cannot decode a codec, and gateways to phone dial-in, which cannot take part in end-to-end keys.
• Simple key management. Keys go to participants over signalling, encrypted to each one, and must change whenever someone joins or leaves, so that a person who left cannot decrypt what follows. For large meetings that needs a group key protocol such as MLS, and a large meeting with constant joins rotates keys often.

So end-to-end encryption is an option per meeting, with the host told which features turn off.

Failures and scale

Interviewer: What breaks, and how does it recover?

You:

Curveballs

Interviewer: Why not run media over WebSockets? It would be simpler.

You: WebSockets run over TCP, so one lost packet stalls everything behind it, and TCP's congestion control backs off hard. For a stream with a 150 ms budget that means freezes. It is fine for chat and signalling, which is exactly what it is used for here. Newer transports such as WebTransport over QUIC allow unreliable datagrams and may change this, but the principle — no head-of-line blocking for media — stays.

Interviewer: Two hosts both press "end meeting" while the SFU is failing over.

You: Ending is a command to the controller, which owns the meeting state and processes commands in order; the second is a no-op. The controller then tells whichever SFU holds the current epoch to drop the meeting. The SFU failover does not touch this, because the SFU never owned the meeting's life cycle.

Interviewer: A participant's video is frozen but the audio is fine.

You: Likely their uplink cannot carry video: bandwidth estimation dropped them to the lowest layer or paused video, and audio is protected first. Or a keyframe request is being rate-limited after loss. Our quality metrics — per-stream loss, jitter, round-trip time, layer switches, freeze counts — are reported by clients and SFUs, so support can see which hop failed (R15).

Interviewer: Could we put the SFU on the participants' own machines to save money?

You: For two people we already do: a one-to-one call can go peer to peer and skip the server entirely, falling back to TURN or the SFU when the direct path fails. Beyond that, a participant's home uplink becomes the meeting's bottleneck and its single point of failure, which is the mesh problem again.

Quick reference

The 60-second version

A video call has two planes. Signalling — WebSockets to gateways, routed to a meeting controller that owns each meeting's roster — exchanges SDP offers and answers and tells every client which media server to use. Media flows as RTP over UDP, because a late frame is worthless and TCP would stall on every loss; ICE finds a path using STUN, TURN relays over TLS when UDP is blocked, and DTLS-SRTP encrypts each hop. Mesh fails on upload beyond a few people and an MCU spends too much CPU, so each meeting gets one SFU that receives each stream once and forwards, without decoding, the simulcast or SVC layer each receiver's bandwidth estimate allows. Receivers smooth jitter with a buffer, repair loss with NACK and FEC only while a packet can still arrive in time, and reset broken video with keyframe requests. The meeting-to-SFU assignment is a lease with an epoch, so there is never a split-brain meeting; meetings are placed near participants and cascaded across regions, and large meetings send only what is seen — active speaker, last-N video, top-N audio — with webinars as a tree of SFUs. When an SFU dies, the lease lapses, clients reconnect with jitter to a new one and rebuild soft state in seconds; the roster, chat and recordings were never on the SFU. End-to-end encryption encrypts frames above the transport, which the SFU can still route, at the cost of server-side recording and mixing.

What to say in the interview

"These systems keep state in a server because it cannot live anywhere else — a socket is tied to one machine, a document being edited needs one ordering point, and a risk check that takes microseconds cannot wait for a database. So I answer four questions out loud. How does a request find the owner? A registry for things that move, like phones; hashing for things that stay put, like documents and accounts. How do I guarantee there is only one owner? Leases and epochs, with every write fenced. Where is the durable truth? An inbox, an op log or a replicated journal — always written before I acknowledge. And how do clients recover? Jittered reconnects, snapshot then delta. Persist before acknowledge; the inbox, not the registry, is the source of truth; one fenced owner per document orders every op; and when the logic is deterministic, a replica that replays the same log is a hot standby for free. For live media the rule flips: nothing late is worth retrying, so one SFU per meeting forwards packets over UDP, each receiver gets the layer its network can carry, and a failed server is replaced by reconnecting, not by replay."