R20 — Large content: storing, syncing and delivering bytes — Dropbox, Netflix

R18 and R19 were about small pieces of state with strict rules: a socket, a seat, an account balance. This section is about the opposite: objects so big that moving the bytes is the problem. A file can be gigabytes, a film is terabytes across all its versions, and the totals run to exabytes. The rules stay simple, but every design decision is about touching the bytes as rarely as possible.

The format is the same as before: the shared problem and its toolkit first, then two walkthroughs, the second building on the first.

The shared concern — the bytes are big, the facts about them are small

A file has two very different parts. Its metadata — name, folder, owner, size, version, which pieces it is made of — is a few hundred bytes, changes often, and needs transactions. Its content is huge, never changes once written, and needs only to be stored cheaply and delivered fast. Every large-content system starts by treating them differently. Six problems follow, each from the one before.

Problem 1 — split metadata from bytes

Put metadata in a database (R10) and bytes in an object store such as S3: cheap, durable, effectively unlimited, and built for large immutable objects. Application servers handle only the metadata. Clients upload and download bytes directly to the object store using short-lived pre-signed URLs, so a 4 GB upload never flows through your API servers.

Problem 2 — cut the bytes into chunks

One 4 GB upload that fails at 99% must not start again from zero. So files are split into chunks — Dropbox uses 4 MB blocks — that upload in parallel, retry individually and resume after a drop. When a file changes, only the changed chunks need to travel.

Problem 3 — name each chunk by its content

Name every chunk by the hash of its bytes. This is content addressing, and several things come free with it:

• Deduplication: identical chunks have identical names, so they are stored once.
• Safe retries: uploading the same chunk twice writes the same object — it is idempotent by construction (R3).
• Integrity: recompute the hash on download and you know the bytes are intact.
• Caching forever: a name can never point to different bytes, so caches never need invalidating (R12).

Problem 4 — keep copies close to the reader

Bandwidth and distance are the cost. A CDN keeps copies of popular content in data centres near users, or inside their internet providers. Immutable, content-hashed names make this easy: cache with no expiry, and publish a new name for new content.

Problem 5 — transform once, serve many times

Raw uploads are rarely what readers need. Photos need sizes for each screen; videos need versions for every device and network speed. Do the transformation asynchronously — a queue feeding workers (R9) — once per object, and store every version. Readers then pick the version that suits them.

Problem 6 — clean up safely

Chunks shared by many files, and versions kept for restore, mean you cannot delete bytes the moment a file is deleted. Garbage collection finds chunks nothing references any more and removes them after a grace period. Data that is rarely read moves to colder, cheaper storage tiers.

Rules of thumb for large content

• Metadata in a database, bytes in an object store; API servers never carry the bytes.
• Chunk everything, and upload chunks in parallel with resume.
• Name chunks by their hash: dedup, idempotent retries and integrity come free.
• Serve from the edge, under immutable names that never need invalidating.
• Transform asynchronously, once, into every version readers need.

How the two walkthroughs build on each other

They are ordered so that each adds one difficulty:

1. Dropbox — the toolkit in its purest form. Metadata in sharded MySQL with an ordered journal, bytes as content-addressed blocks, uploads that send only the chunks the server is missing, and the extra twist of two-way sync: many devices change the same tree, so the journal, not the notification, decides what happened.
2. Netflix — the same split taken to its extreme, with the reads multiplied by hundreds of millions. Writing is easy, so the problem becomes delivery: an encoding pipeline producing many versions, a CDN placed inside internet providers and filled before anyone asks, and players that switch version every few seconds as the network changes.

At a glance

Walkthrough 1 — Dropbox, file storage and sync

Where this starts: the large-content toolkit in its purest form — metadata in a transactional store with an ordered journal, bytes as content-addressed blocks in an object store — plus one extra twist: many devices change the same file tree, so sync has to decide what happened when they disagree.

Scope

Interviewer: Design Dropbox.

You: "Dropbox" could mean a web drive, a sync engine or a backup tool, and each has a different core problem, so I'd ask only questions whose answers change the architecture. Sync agent or web drive? — a sync agent drives chunking, delta sync, a local index and notification. Users, devices, file sizes? — they size the shards, the connection tier and the block size. Sharing? — it adds namespaces, ACLs and fan-out. Version retention? — storage cost and GC. Two devices editing one file? — lock, merge, or conflicted copy. End-to-end encryption? — if yes, cross-user dedup by content hash is impossible.

My assumptions, unless you'd like others: desktop and mobile clients plus web, with the desktop agent as the focus. 500M registered users, 100M daily actives, two to three devices each. Files up to 50 GB, median tens of KB, average around 500 KB. Shared folders with viewer and editor roles. Thirty days of version history on the base plan. Conflicts keep both copies rather than merging content. Server-side encryption at rest, so content-addressed dedup is allowed — scoped carefully, which I'll come back to.

Interviewer: Fine. Focus on the sync agent. What are the requirements?

You: Functionally: upload, download, edit, rename, move, delete; sync to every device within seconds; upload only changed parts and skip data the server already has; resume interrupted transfers; work offline; detect concurrent edits; keep and restore versions; share folders. Out of scope: real-time co-editing (that is R18's Google Docs walkthrough), full-text search and previews beyond async workers, billing, end-to-end vaults.

Non-functionally, the two properties that shape everything: durability is non-negotiable — eleven nines for block storage — and metadata is strongly consistent and ordered per folder, while each device is allowed to converge a few seconds later. Change visible on another online device in under five seconds at p95. Metadata availability 99.99%.

What's being probed: whether you rank properties and say which are strict and which can relax.

Concept — mixed consistency. The server-side file tree must be strongly consistent: two commits to one file are ordered, and a file never points at missing data. Devices may lag and converge later. That is eventual consistency at the edge on top of strong consistency at the core — the same split as a ledger and the balance shown in an app (R0). Saying it lets you use cheap, best-effort notification while the source of truth stays strict.

Takeaway: Strong where truth lives, eventual where copies live.

Capacity

Interviewer: Size it.

You: Assumptions: 100M DAU, ten file changes each a day, average file 500 KB, 4 MB maximum block, half the bytes saved by dedup and delta sync, peak three times average, each change reaching three devices, 2,000 files per registered user, 500 bytes of metadata per entry, and 3x physical overhead (erasure coding 1.5x in each of two regions).

Block storage is exabyte scale, metadata is petabyte scale, and 60 million idle connections need an event-driven tier. Caches: ~2 TB of hot metadata in Redis, and a ~1.25 TB Bloom filter over a trillion block hashes at 10 bits each — roughly 1% false positives, a cheap "definitely new" before the real lookup (R10).

API

Interviewer: What does the API look like?

You: HTTPS with OAuth bearer tokens per device, every call authorised against the folder's ACL. The shape follows the core mechanism: send hashes first, bytes only when the server asks.

POST /v1/files/commit
Authorization: Bearer <token>
Idempotency-Key: 7f3c-device42-op-91822

{ "ns_id": 12345, "path": "/Projects/report.docx", "parent_rev": "a1b2c3:17",
  "size": 9437184, "block_list": ["3f9a...e1", "88c0...4d", "b7e2...09"] }

{ "status": "need_blocks", "need_blocks": ["88c0...4d"] }
{ "status": "ok", "rev": "a1b2c3:18", "sjid": 90817265, "content_hash": "e4d9...77" }
{ "status": "conflict", "server_rev": "a1b2c3:18", "server_block_list": ["..."] }

The cursor is opaque — it encodes (namespace, journal id) pairs — so the server can change its format without breaking clients. Block downloads check that the hash is referenced by a file the caller can read.

Interviewer: Why commit first rather than check, upload, commit? And what about a retried commit?

You: In the common case — a rename, a restore, a re-save with no real change, a file someone already uploaded — every block exists and commit-first succeeds in one round trip. Check-upload-commit always costs an extra one; I keep /blocks/check for clients that want to pre-check large files. On retries: block uploads are idempotent by construction, same hash, same bytes. Commits carry an Idempotency-Key and a parent_rev, so a retry after a lost response returns the same rev, and a retry after someone else's change returns a conflict. A retried commit never creates a duplicate revision.

What's being probed: round-trip awareness and idempotency on an unreliable network.

Concept — two kinds of idempotent write. A content-addressed write is idempotent by nature, because the key is derived from the data. A "create revision" is not, so it needs a key the server remembers plus a precondition (parent_rev) that rejects stale retries. R3 covers the key-and-stored-response pattern in full; here the precondition does the conflict detection for free.

Takeaway: Make every retryable write idempotent — by content hash, or by key plus precondition.

Data model

Interviewer: Walk me through the data model.

You: Small, transactional metadata and large, immutable blobs have opposite access patterns, so they live in different stores.

A namespace is a self-contained file tree: your root is one, each shared folder is another. Making it the shard key means every operation on a folder is one shard, one local transaction, and one gap-free sequence number — the sjid — that clients use as a cursor. Restore and dedup become metadata-only: a restore is a new row pointing at old hashes, dedup is two rows pointing at one.

Interviewer: Why MySQL? Cassandra would scale trillions of rows more easily.

You: It would scale writes, but I need three things it makes hard: multi-row transactions (journal append, latest pointer, namespace head — atomically), strict gap-free ordering per namespace, and secondary indexes by path, file and user. Sharding by namespace keeps every transaction on one shard, so I never need distributed transactions on the hot path — R10's rule that the shard key should follow the transaction boundary. Dropbox has publicly described its metadata layer on sharded MySQL. For blocks the answer is the opposite: immutable, write-once, huge in total, so an erasure-coded object store — S3 first, a custom store at exabyte scale.

What's being probed: metadata/blob separation, and choosing a store by its guarantees.

Concept — the shard key as transaction boundary. List what the workload needs: transactions, ordering, indexes, scale. Wide-column stores trade the first two for easy horizontal scale; relational stores keep them but need manual sharding. If the key keeps every transaction inside one shard, you get both. R10 covers choosing the key; the namespace is a textbook case.

Takeaway: Small mutable index in a database, big immutable bytes in an object store, namespace as the shard key.

High-level design

Interviewer: Draw it.

You: Four core parts — sync client, metadata service, block service, notification service — plus a change stream and workers. Click through the components, then play the upload and sync flows and watch which path carries bytes and which carries only a wake-up.

What's being probed: a clean decomposition where each component scales on its own axis.

Concept — control, data and signalling planes. The metadata service decides what is true; the block service moves heavy bytes; the notification service only says "go look". Metadata scales by shards, blocks by disks and bandwidth, notifications by connection count. The thick client matters too: because the device keeps its own index and hashes its own files, the server never has to diff anything.

Takeaway: Separate who decides truth, who moves bytes, and who rings the doorbell.

Interviewer: An edit on one laptop shows up on another. Walk me through it.

You: Upload first.

1. The user saves report.docx; the watcher fires; the client debounces, then reads the file.
2. It splits into 4 MB blocks, hashes each, compares with its local index: only block 1 changed.
3. It calls commit with the full ordered hash list, no bytes.
4. The metadata service batch-checks the block index and returns need_blocks: [h1].
5. The client uploads block 1; the block service verifies the hash, stores it, records the location.
6. The client retries commit. In one transaction on the namespace's shard: check parent_rev, append a journal row with a new sjid, update the latest pointer.
7. The change event {ns_id, sjid} is published; the client updates its local index.

Then device 2:

1. Its open longpoll(cursor) — the cursor holds the last sjid per namespace — is answered changes: true when the notification service sees the event for ns_id.
2. It calls list_changes(cursor), a range scan for sjid > cursor, and gets entries plus a new cursor.
3. It already has blocks 0 and 2 by hash, so it downloads only block 1.
4. It builds the file in a temp path, verifies the content hash, then atomically renames it into place, and opens a new long-poll.

Interviewer: And sharing and restore?

You: Sharing turns /Projects into its own namespace — a migration job for big folders, but the visible switch is one metadata transaction — leaves a mount in the owner's root, writes ACL rows and an invitation, and emits an event. When Bob accepts, a mount row adds the namespace to his devices' cursors; they list from sjid 0 and fetch blocks they lack. From then on every commit there wakes every member's devices.

Restore reads the journal rows for that file, checks every block in rev 17 is still live — GC never deletes blocks referenced within retention — and appends rev 19 with rev 17's block list. No bytes are copied. Every device syncs rev 19 as an ordinary change.

What's being probed: whether sharing and versioning fall out of the model rather than needing bolt-ons.

Concept — immutability makes history cheap. With an append-only journal and immutable, content-addressed blocks, "go back to version 17" is "make a new version that points at the same blocks". Git works the same way. The client applies the same idea locally: build aside, verify, swap the pointer; on POSIX a same-volume rename is atomic. The cost is that deletion becomes its own job — garbage collection.

Takeaway: With immutable blocks and an append-only journal, restore is a pointer change.

Deep dive — chunking, content addressing and delta sync

Interviewer: Why 4 MB fixed blocks?

You: Re-uploading a 2 GB file for one changed paragraph wastes bandwidth, and many users store identical installers and PDFs. The options:

I choose fixed 4 MB blocks addressed by SHA-256, compressed on the wire. That is what Dropbox has publicly described; its API content_hash is SHA-256 over the concatenated per-block hashes. Most edits to large files — databases, VM images, media metadata — overwrite in place, where fixed blocks work well.

Content addressing gives four things at once: dedup, idempotent uploads, integrity on every read, and blocks that can be cached anywhere without invalidation.

upload(file):
  blocks = split(file, 4MB)
  hashes = [sha256(b) for b in blocks]
  resp = commit(path, parent_rev, hashes)
  while resp.status == NEED_BLOCKS:
      parallel for h in resp.need_blocks: put_block(h, compress(block(h)))
      resp = commit(path, parent_rev, hashes)
  if resp.status == CONFLICT: create_conflicted_copy(path)

Interviewer: Insert one byte at the start of a 2 GB file. Every boundary shifts. You just uploaded 2 GB.

You: Correct — that is the weakness of fixed blocks. Three responses. Content-defined chunking: cut where a rolling hash over a small window matches a pattern, say the low 13 bits are zero for ~8 KB chunks, or a wider mask for MB-sized ones; an insert changes only the chunk it lands in and later cuts re-align. rsync-style delta inside a changed block. And measure: if telemetry showed insert-heavy large files, I'd switch those types to content-defined chunking. The server does not care — it only sees a list of hashes.

What's being probed: chunking trade-offs, and knowing the limit of your own choice.

Concept — content-addressed storage and content-defined cuts. Naming data by the hash of its bytes means identical data cannot be stored twice and the name doubles as a checksum, so delta sync becomes set arithmetic: upload the hashes the server lacks. Fixed chunking cuts at positions; content-defined chunking cuts at content patterns, so an early insert moves only the cuts near it — like cutting rope at every red mark rather than every metre.

Takeaway: Name data by its hash; cut at content patterns, not offsets, when inserts are common.

Deep dive — dedup scope and privacy

Interviewer: Is cross-user dedup safe?

You: Not by default. Two risks. An existence leak: a skipped upload tells an attacker "someone already has file X". And hash as a password: if knowing a hash were enough to download a block, anyone who learned one could fetch the content. Mitigations: dedup within a user or team by default; for cross-user dedup, require proof of possession — ask for the bytes anyway or challenge random byte ranges; and never let a hash alone grant read access — check that the caller's namespaces reference the block. With end-to-end encryption, cross-user dedup is impossible anyway. Dedup is an optimisation; authorisation must never depend on knowing a hash.

Deep dive — the commit transaction and the journal

Interviewer: Show me exactly what happens in the database on a commit.

You: The namespace is the unit of sharding, ordering and consistency. Every change gets a monotonically increasing sjid assigned inside the shard's transaction.

BEGIN;
SELECT rev, sjid FROM file_latest WHERE ns_id = ? AND path = ? FOR UPDATE;
-- rev != parent_rev  ->  ROLLBACK, return CONFLICT
UPDATE namespace SET latest_sjid = latest_sjid + 1 WHERE ns_id = ?;  -- serialises commits per namespace
INSERT INTO file_journal (ns_id, sjid, path, file_id, rev, block_list, size, ...) VALUES (...);
INSERT INTO file_latest (ns_id, path, sjid, rev) VALUES (...)
  ON DUPLICATE KEY UPDATE sjid = VALUES(sjid), rev = VALUES(rev);
INSERT INTO outbox (ns_id, sjid) VALUES (?, ?);                         -- R4
COMMIT;

Three rules. Blocks before metadata: a commit succeeds only if every block exists, so a file never references missing data; the reverse — blocks with no commit — makes orphans that GC removes. The outbox: the event commits with the change and a relay publishes it (R4). And here the outbox is belt and braces, because clients read changes from the journal by cursor, not from Kafka — Kafka only wakes them up. A lost event delays sync until the next poll; it never loses a change. If a cursor is older than the retained journal, the server returns reset and the client does a full listing and reconciles against its local index.

What's being probed: transactional correctness and the dual-write problem.

Concept — the log is the truth, the event is a hint. Committing to the database and then publishing as two steps leaves a window where the change exists and nobody is told; R4 closes it with the outbox. This design goes one step further: because consumers pull from the journal by cursor, the event carries no data, so even a lost one only costs latency.

Takeaway: Never dual-write; commit the event with the data, and keep the log as the truth.

Deep dive — change notification at 60 million connections

Interviewer: How do 60 million devices learn about changes within seconds?

You:

Long polling for desktop — Dropbox's public API exposes exactly this as list_folder/longpoll plus a cursor — push for backgrounded phones, WebSocket where available. R1.4 compares these transports in full. The property that matters: the channel only says "something changed", and the device pulls the truth with its cursor.

Details: each notification server keeps ns_id → connections in memory and subscribes to the stream. On reconnect the client sends its cursor and the server compares it with the namespace's latest sjid from cache, answering immediately if behind — that closes the race between "change happened" and "connection re-established". Event-driven servers hold ~100K idle connections each. For a company-wide folder with 50K members, coalesce events per namespace for about a second and jitter the follow-up list_changes. The server can return a backoff value during incidents.

What's being probed: the cost of idle connections, and designing so a lost message is not lost data.

Concept — signal, then pull. If the pushed message is the data, losing it loses data. If it is only a doorbell, it can be lost, duplicated or merged without harm, because the pull fetches everything after the cursor. An idle held request costs a few KB on an event-driven server.

Takeaway: Push a hint, pull the truth; a lost hint costs latency, never data.

Deep dive — conflicts

Interviewer: Two people in a shared folder edit the same file at once.

You: Optimistic concurrency on parent_rev. Each commit says "I edited rev 17"; if latest is still 17 it wins and becomes 18. The namespace row lock serialises the two transactions, so exactly one wins and the other gets CONFLICT. The loser does not overwrite: it renames its version to report (Abhinav's conflicted copy 2026-10-02).docx, commits that as a new file, and downloads rev 18 under the original name.

Edge cases: edit versus delete keeps the edit; rename versus edit applies the edit to the renamed file because file_id is stable; two devices creating the same folder merge it; case clashes between macOS/Windows and Linux are enforced case-insensitively per folder and become conflicted copies.

What's being probed: a concurrency policy that fits opaque data and offline use.

Concept — optimistic concurrency control. Record the version you started from and check it at commit; if it is no longer latest, you lost and must reconcile. It suits rare conflicts and editors who may be offline for hours, where a lock would be held by a device that vanished — the same idea as an If-Match ETag or compare-and-swap. R18's Google Docs walkthrough is what you do when the content is mergeable.

Takeaway: Check the version at commit time; on conflict keep both copies rather than lose one.

Deep dive — block storage, GC and large files

Interviewer: 1.4 EB of blocks. Would you stay on S3? How do you delete anything when blocks are shared?

You: Start on S3. At exabyte scale, storage and request cost can justify a custom store — Dropbox publicly described moving most blocks to its own system, Magic Pocket. The shape of it: buffer blocks into large append-only volumes (say 1 GB) to avoid billions of tiny objects, with the block index holding (volume, offset, length); erasure code at something like 6+3, surviving any three lost pieces at 1.5x overhead versus 3x for replication; move cold versions to denser tiers.

Deletion is reference-based because blocks are shared:

• Reference counts reclaim fast but write the index on every commit and drift on failures.
• Mark-and-sweep scans live journal rows (latest plus retention), marks referenced hashes and deletes the rest. Slower, self-healing.
• My choice: hybrid. Track last-referenced time cheaply, and delete only what a verified sweep proves unreferenced, after a grace period — say seven days past retention — so in-flight uploads and restores are safe. Compaction rewrites mostly-garbage volumes.

Interviewer: A 50 GB video over a flaky hotel connection.

You: Stream the hashing so memory stays flat; upload blocks as they are hashed, four to eight in parallel; commit once with ~12,500 hashes, about 400 KB, which is fine. Resumability is free: after a crash the client asks which hashes are missing and re-sends only those. Compression on the wire — Dropbox has described Broccoli, a parallel Brotli variant. User bandwidth limits, pause on metered networks, LAN sync between devices on one network. Commit only after all blocks are stored, so no device ever sees a half-written file. On download, on-demand placeholders fetch content when opened. Many small idempotent pieces plus one final commit make any upload resumable.

Scale and failure

Interviewer: Where does it break?

You:

The one to name first: the hot namespace

Per-namespace ordering is a deliberate serialisation point. A company-wide folder with tens of thousands of writers is where I'd split into sub-namespaces.

Interviewer: A whole region goes down.

You: Each namespace has a home region with its primary shard and async replicas in a second region; failover is per shard with an RPO of seconds. Blocks are durable locally (erasure coded) before the commit, then copied cross-region in the background. Users homed in the failed region may lose the last few seconds of commits — and their devices re-upload them, because the local index still shows those edits as unsynced. A sync client covers the RPO window that a plain web upload cannot. R13 covers home regions and failover in full.

Curveballs

Interviewer: A user drags a file from their root into a shared folder on a different shard.

You: Two shards, so my single-shard transaction does not cover it. Two-phase commit is correct but fragile when a participant hangs (R2); distributed SQL works but changes the whole store. I choose a saga with a stable move id (R5): add in the target first, then delete from the source, each step idempotent by the move id, with a recovery job finishing half-done moves. Order the steps so the worst intermediate state is a duplicate, never a loss. Because blocks are content-addressed, the add copies only metadata.

Interviewer: Quick ones. A laptop back after two weeks offline? And why would anyone move off S3?

You: The laptop sends its saved cursor and pages through every journal entry since. If the journal beyond retention was trimmed, the server returns reset and the client lists the full tree and reconciles against its local index — replay the log when you can, snapshot when you cannot, as a Kafka consumer or a replica does. Its offline edits commit with their old parent_rev; collisions become conflicted copies.

On S3: at exabyte scale the provider's margin is a large absolute number, and a store tuned to one workload — immutable 4 MB blobs — can be much cheaper. Below tens of petabytes the engineering and operations cost rarely pays. So start on S3 and keep the block service storage-agnostic. Content addressing makes the migration safe: copy blocks in the background and verify every one by its hash.

What's being probed: the case your shard key does not cover, and build-versus-buy tied to scale.

Concept — sagas and storage-agnostic interfaces. A saga replaces one distributed transaction with ordered local ones, each idempotent and recoverable (R5). Build-versus-buy turns on whether the provider's margin exceeds the team it would take to replace it; hiding storage behind your own interface keeps that decision reversible.

Takeaway: Order saga steps so the worst state is a duplicate; buy until the margin outgrows the team, behind an interface you own.

Quick reference

The 60-second version

A sync client watches the folder, keeps a SQLite index, splits files into 4 MB blocks and hashes them. Blocks are content-addressed and immutable in an object store, which gives dedup, idempotent uploads, integrity checks and resumability for free. Metadata lives in MySQL sharded by namespace; a commit is a single-shard transaction that checks parent_rev and appends to an ordered journal. Uploads are commit-first: the server names the missing hashes and only those travel. Other devices hold long-polls that only say "something changed", then pull from the journal by cursor and fetch missing blocks. Concurrent edits produce a conflicted copy; restore is metadata-only. Sharing makes a folder its own namespace; GC with grace periods, erasure coding and cross-region copies give durability at exabyte scale.

Walkthrough 2 — Netflix, video on demand at 75 Tbps

Building on Dropbox: the same split of facts from bytes, but the reads are multiplied by hundreds of millions and the writes are a small studio pipeline. Storage is easy; delivery is the whole problem. So the design moves the bytes to the edge before anyone asks, and keeps that delivery working even when the services deciding what to play are down.

Scope and requirements

Interviewer: Design Netflix.

You: Netflix is three systems in one — a video factory, a delivery network and a personalised product — so I want to scope before drawing. Video on demand only, or live too? Which part do you want deep? Subscribers, daily actives, peak concurrent streams? Which devices, because that decides codecs and DRM? Do we own the CDN or rent one? Offline downloads? Profiles and concurrent-stream limits? And how fresh must "continue watching" be across devices?

Interviewer: On demand only. Go wide, then deep on encoding, CDN and playback. Netflix scale; you may own the CDN.

You: Then: ~250M accounts, ~400M profiles, ~100M daily active profiles, peak ~15M concurrent streams in local evenings. We own the video CDN — appliances inside ISPs and at internet exchanges. The control plane runs in a public cloud across three or more regions. Downloads and live are out of scope; I'll say how they would change things.

Functional: studios upload mezzanine masters plus audio, subtitles and artwork; we encode many renditions and package them for adaptive streaming; play starts fast and adapts to the network; DRM by device (Widevine, PlayReady, FairPlay); resume on any device; a personalised home page; plan rules for concurrent streams, max resolution and regional licensing. Non-functional, in user-felt terms: time-to-first-frame under ~2 s at p90, rebuffer ratio well under 1%; control plane 99.99% with fallbacks rather than errors; tens of Tbps at a low cost per GB; progress visible across devices within seconds; no single region or store that can take the service down.

Interviewer: You said the data plane should be more available than the control plane. Isn't that backwards? The control plane authorises playback.

You: It is deliberate, because the blast radii differ. If the control plane is down, new plays and home pages fail or degrade — bad, but bounded. If the data plane is down, 15M running streams freeze at once. So once a stream has started it needs nothing from the cloud: the client holds the manifest, ranked appliance URLs and a DRM licence valid for a period, and appliances validate signed URL tokens locally. A whole region can die and people already watching keep watching.

What's being probed: whether you understand control plane versus data plane, and why separating them raises availability.

Concept — Hand the data plane everything up front. The control plane decides; the data plane does. Routers compute paths with routing protocols and forward packets without asking anyone per packet. Netflix's cloud decides who can play what from where, and the appliances just serve bytes. Give the data plane tokens, URL lists and licences with lifetimes, and a control-plane outage stops new decisions without stopping work already in progress — a pilot who has the flight plan keeps flying when the airline office loses power.

Takeaway: Decide in the cloud, deliver from the edge, and make delivery independent of the deciders.

Capacity

Interviewer: What's the number that matters most?

You: Egress. Assumptions for interview math, not Netflix figures: 2 hours viewing per DAU, 5 Mbps average delivered bitrate, 3 play starts and 5 home loads per DAU, heartbeats every 30 s, a 50K-hour catalog, 100 new hours a day, ~45 GB per content hour encoded across everything, ~225 GB per hour for the master.

• Stream-hours: 100M x 2 = 200M a day, so ~8.3M concurrent on average.
• Average egress 8.3M x 5 Mbps = ~42 Tbps; peak 15M x 5 Mbps = 75 Tbps; ~450 PB a day.
• At ~100 Gbps per appliance, 750 boxes at 100%; with headroom and geography, several thousand.

75 Tbps is not something a cloud region or a single data centre can sensibly serve, so the number alone says we need a distributed CDN close to users.

What's being probed: whether you find the dominant resource and let it drive the architecture.

Concept — The dominant cost. Every system has one resource that dwarfs the others: connections for chat, index memory for search, bytes out of the door for video. The method is concurrent users times per-user rate, then divide by 8 for bytes. Once you know it, design to minimise it first — bytes close to users, better codecs — and treat the rest as ordinary engineering.

Takeaway: In video, bandwidth is the design; everything else is detail.

Interviewer: Now the control plane, storage and caches.

You: Play starts ~3.5K/s average, ~10K/s peak. Home loads ~17K/s peak, fanning out to 10–30 backend calls each, so ~0.5M internal QPS. Progress heartbeats: 8.3M / 30 s = ~280K writes/s average, 500K/s peak — the busiest write path in the system. DRM licences ~10–20K/s.

Storage: encoded catalog 50K x 45 GB = ~2.25 PB; masters ~11 PB; per day 4.5 TB encoded and 22.5 TB masters. Viewing positions: 400M profiles x ~500 titles x 150 B = ~30 TB, ~90 TB replicated. Raw heartbeats ~2.4 TB a day into Kafka and the warehouse. Precomputed home pages at ~20 KB for ~200M hot profiles = 4 TB, ~12 TB across three zones. Title metadata is 2.5 GB and fits in every service's local cache.

Two conclusions. The heaviest write is a background feature, not play starts. And the encoded catalog is only ~2.25 PB, small enough to copy most of it to many places — that is what makes proactive fill possible.

API

Interviewer: Sketch the APIs.

You: HTTPS through the gateway with an account-and-profile token; device capabilities in headers.

POST /v1/playback/start
X-Device: tv-model-x; codecs=hevc,av1; hdr=dv; drm=widevine-l1
{ "profileId": "p_81", "titleId": "t_70143836", "episodeId": "e_4021" }

{ "playbackId": "pb_9f3c",
  "manifestUrl": "https://oca-blr-07.isp.example/t_70143836/e_4021/av1/manifest.mpd?tok=...",
  "cdnUrls": [ { "host": "oca-blr-07.isp.example", "rank": 1 },
               { "host": "oca-blr-12.isp.example", "rank": 2 },
               { "host": "oca-ix-mum-03.example",  "rank": 3 } ],
  "licenceUrl": "https://api.example.com/v1/licence",
  "resumePositionSec": 1312, "maxResolution": "2160p", "streamSlotLeaseSec": 120 }

The playback API returns decisions; the CDN returns bytes. Several ranked hosts is how the client fails over without asking the cloud again, and the device header means the server returns only renditions it can play.

Interviewer: Heartbeats get retried and arrive out of order.

You: PUT /v1/progress/p_81/t_70143836 { "positionSec": 1342, "state": "PLAYING", "playbackId": "pb_9f3c", "clientTs": 1790000000123 } returns 204. It is an upsert of an absolute value keyed by (profile, title), so a repeat is harmless, and clientTs lets the store keep only the newest. Playback start returns the same playbackId for the same device within a short window, so a retry doesn't take a second stream slot. Send absolute values with timestamps, and retries and reordering stop being a problem; last-write-wins only loses data when two writers truly race, which for a resume point is rare and harmless.

Data model

Interviewer: What do you store, and where?

You:

Interviewer: Why Cassandra for progress? A sharded Postgres could handle it.

You: It could, but it fights the workload. 500K small upserts a second are cheap appends in an LSM store and random page updates in a B-tree. I want every region writing locally, and Cassandra replicates across data centres natively at LOCAL_QUORUM. And one partition per profile is a natural wide row. I give up cross-row transactions and get eventual cross-region consistency, which is fine for a position. DynamoDB global tables is the equal alternative on an AWS-native stack; relational wins if writes are low. R10 covers LSM storage and quorums. Very long histories roll old entries into a compressed summary per profile, as Netflix has publicly described, to keep partitions small.

CREATE TABLE viewing_progress (
  profile_id text, title_id text, episode_id text,
  position_sec int, duration_sec int, updated_at timestamp,
  PRIMARY KEY ((profile_id), title_id)
);

High-level design

Interviewer: Draw it.

You: Control plane in the cloud — every API. Data plane on the CDN — the video bytes. The cloud never streams video. Run the ingest, play, progress and home-page flows in turn and notice that only the client-to-appliance arrow carries real bytes; every arrow into the cloud is a small request.

The gateway does TLS, auth, routing and device shaping (Netflix publicly described Zuul). The playback API checks plan, region and maturity, takes a stream lease, picks renditions and asks steering for appliances. The CDN control plane knows which appliance holds which file and how healthy each is, ranks them per client and plans the nightly fill. Appliances (Open Connect, as publicly described) are storage-heavy servers inside ISPs. The DRM service applies policy and returns wrapped keys. Ingest validates masters and triggers encoding workers. The viewing history service writes Cassandra and publishes to Kafka. The recs pipeline writes ranked rows to EVCache.

Interviewer: End to end — studio upload to a user resuming later.

You: Four flows — and in each I look for steps that don't depend on each other, because latency is the longest dependent chain, not the sum.

• Ingest: master to S3; validate resolution, frame rate, colour space, A/V sync; complexity analysis picks this title's ladder; split at shot boundaries; thousands of chunk tasks encode every rendition in parallel; stitch, run VMAF and black-frame checks, encrypt, package segments and manifests; the files join tonight's fill plan.
• Play: authorise and lease; pick renditions; steering returns ranked appliances — inside the ISP first, then IX, then fallbacks; the client fetches the manifest and first segments while it sends the licence challenge; the CDM decrypts; ABR picks each next segment.
• Progress: every ~30 s and on pause, seek and stop, upsert at LOCAL_QUORUM, replicate async, publish to Kafka for QoE, the warehouse and nearline recs; stop releases the lease.
• Home: read precomputed rows from EVCache and continue-watching in parallel, drop titles not licensed here, pick per-profile artwork, return ~8 rows and lazy-load the rest, and serve popular-in-country rows if recs are down.

Deep dive — encoding and packaging

Interviewer: A studio delivers a 2-hour 4K film. Encode it fast and well.

You: Two problems. Encoding 100+ renditions of it on one box takes days. And one fixed ladder wastes bits on animation and starves grain and sport. The options run from whole-file encoding on one box (days, and one failure restarts everything) to chunked parallel encoding (hours, per-chunk retry, but clean boundaries and stitching), and from one fixed ladder through per-title to per-shot optimisation — each better per bit and more compute.

Chunked parallel encoding on an elastic fleet, with per-title ladders and shot-based optimisation for newer codecs — Netflix publicly described per-title encoding in 2015 and shot-based optimisation later.

for each title:
  shots  = detect_shots(master)
  ladder = choose_ladder(complexity(master))          # trial encodes, quality vs bitrate hull
  for chunk in group(shots):
    for (codec, res, bitrate) in ladder:
      enqueue(EncodeTask(chunk, codec, res, bitrate))  # idempotent by (title, chunk, rendition)
  wait_all(); stitch(); qc_vmaf(); encrypt(); package_hls_dash()

Output paths are deterministic, so retries are safe; a workflow engine tracks the DAG; launch titles get a priority queue. Why shot boundaries? Each chunk must start with a keyframe, a frame that decodes on its own. Cut mid-scene and quality dips at every chunk start, visible as a pulse. A shot change is where the encoder wants a keyframe anyway, so the cut is free.

Packaging: segments of 2–6 s with keyframes aligned across renditions, so the player can switch at any boundary. HLS and DASH manifests both reference one set of CMAF files, halving storage. Audio and subtitles are separate tracks. Segments are encrypted with CENC (cbcs for FairPlay). Four seconds is the usual middle ground between fast switching and compression.

What's being probed: whether you can turn a long serial job into a parallel one, and why one ladder doesn't fit all content.

Concept — Bitrate ladders and chunking. A ladder is the list of (resolution, bitrate) pairs a title is offered in. Per-title encoding plots quality against bitrate from trial encodes and picks rungs from the upper boundary — the convex hull — so every rung gives the most quality per bit. Chunking at shot boundaries turns days into hours, like a book translated by a hundred translators one chapter each, with only failed chapters redone. Aligned segments plus a manifest then turn video into plain cacheable HTTP files.

Takeaway: Chunk for speed, choose the ladder per title for quality per bit.

Deep dive — the CDN

Interviewer: How do you deliver 75 Tbps?

You:

Embedded appliances plus larger IX sites. The catalog is known in advance, unlike a web CDN, so fill is proactive: the control plane predicts per-region popularity and appliances pull files during each ISP's off-peak window. Fill is tiered — from peers and IX sites, not only S3. Storage-heavy boxes hold much of the catalog; flash-heavy boxes hold the hottest titles. Big releases are pre-positioned days ahead, so launch night is cache hits, not an origin storm. Fill windows are staggered per ISP so origin and IX sites aren't hit at once.

What's being probed: whether you exploit a property of the workload.

Concept — Push versus pull caching. A normal CDN is pull-through: the first request misses and fetches from origin (R12). That works when demand is unpredictable. A finite, known catalog with predictable regional demand lets you push files ahead of time, while networks are idle — the bakery bakes before the shop opens. Viewing is heavy-tailed, so rank by predicted popularity, put the head on flash and the tail on large disks, and recompute nightly.

Takeaway: When demand is predictable, pre-load the cache off-peak and serve only hits at peak.

Interviewer: Building your own CDN is a huge investment. Why not buy?

You: For a startup, I would. At tens of Tbps per-GB fees dominate, and owning hardware turns a variable cost into a mostly fixed one that is far cheaper per GB; ISPs host the boxes because it cuts their transit; and a generic CDN would take cold misses on every launch at peak. If you told me we were at 1 Tbps, I'd buy — probably multi-CDN — and revisit when the bill justified it. Build versus buy has a crossover, and naming it beats claiming one option is always right.

Deep dive — playback, steering and ABR

Interviewer: First frame under 2 s, and the right appliance.

You: Steering combines the client IP → ISP and the BGP prefixes each appliance announces, which appliances hold the file, their health and load, and historical QoE. The client gets ranked URLs and may race the first segment. Speed tricks: prefetch manifests for titles under the cursor, start at a conservative bitrate, licence in parallel, cache DRM sessions, short first segments. Appliances validate an HMAC token over (title, client, expiry) locally, so the data plane never calls the cloud per request.

Interviewer: Enforce "two concurrent streams".

You: A lease per stream: playback start acquires one per playbackId keyed by account with a 120 s TTL; heartbeats renew; stop releases; a crashed TV just stops renewing. At the limit, start returns an error listing active devices. This is the one place I want more than LWW — a conditional write in the account's home region, or atomic compare-and-set. R8 covers leases.

Interviewer: The network keeps changing. How does the player choose quality?

You: Throughput-based ABR picks ~0.8 x measured throughput and oscillates on noisy estimates; buffer-based (BBA, from Netflix-affiliated research) maps buffer level to bitrate and ramps up slowly at start; a hybrid uses throughput first and buffer once stable.

if buffer < 10s:     pick highest bitrate <= 0.5 * throughput
elif buffer < 30s:   pick highest bitrate <= 0.8 * throughput
else:                step up one rung if throughput allows; never jump more than 2 rungs
on segment timeout:  switch to next CDN URL, drop one rung

Hybrid, capped by device and plan, on the client, because only the client sees its real throughput and buffer. QoE reports flow back through Kafka to steering and A/B tests.

What's being probed: whether you understand the control loop that decides picture quality.

Concept — The buffer as shock absorber. The player keeps seconds of video downloaded ahead of the screen; while it is full a slow network costs nothing visible, and when it empties the video stalls. Throughput asks "how fast is my network?" and jumps around; buffer asks "how much cushion do I have?" and is stable but timid at start. Hysteresis — a clear margin before stepping up — stops flapping, like a thermostat.

Takeaway: Use throughput to start, buffer to steer, hysteresis to stay calm.

Deep dive — history and personalisation

Interviewer: 500K writes a second and a fast home page?

You: Coalesce heartbeats at the service — keep only the latest per key over a short window — and send raw events only to Kafka. LWW for the rare two-device race. Personalisation splits by freshness: offline batch jobs train models and rank candidates per profile; nearline consumers react to "just finished episode 3"; online work in the request is tiny — filter, merge continue-watching, pick artwork, which Netflix publicly described personalising. Full ranking per request at 17K page loads a second is too slow, so rows are precomputed into EVCache.

What's being probed: whether you know how to absorb a write firehose and when to precompute.

Concept — Offline, nearline, online. Offline jobs do the heavy work on a schedule over all data. Nearline jobs update a small part of the answer within seconds of an event. Online work inside the request must be tiny. Precomputing turns an expensive ranking into one key lookup at the cost of some staleness, which nearline narrows. Write coalescing is the same thinking applied to writes: if ten heartbeats for a key arrive in a window, only the last matters.

Takeaway: Do heavy work ahead of time, light work in the request, and write only the latest value.

Failures and scale

Interviewer: Where does it break?

You:

The home page fans out to 10–30 calls, and the slowest one sets the page latency, so every call gets a timeout, a breaker and a fallback — R14's toolkit, which Netflix popularised with Hystrix.

Interviewer: A whole cloud region dies at 9 pm.

You: The control plane is active-active in three or more regions with headroom to absorb a neighbour; traffic shifts by DNS; Cassandra and EVCache are already replicated (R13). Running streams don't notice — the data plane doesn't need the cloud and licences are still valid. Masters are replicated, and renditions can always be regenerated from them. And I would not trust any of this untested: chaos engineering — Chaos Monkey, injected latency, region evacuation drills, all publicly described — to prove the fallbacks fire.

Curveballs

Interviewer: An appliance dies mid-episode.

You: Ideally the viewer sees nothing. On a segment timeout the client switches to the next URL, drops a rung and keeps playing from tens of seconds of buffer. The fastest failover is one the client can do alone, with no round trip to a possibly unhealthy control plane.

Interviewer: Product wants live sports.

You: Almost every delivery assumption flips. Real-time encoding with a fixed ladder; very short segments or low-latency CMAF chunks; pull-through caching, because the content didn't exist an hour ago; and an origin shield that collapses a million simultaneous misses on the newest segment into a few origin fetches. Accounts and history stay as they are.

Interviewer: Offline downloads? Two devices racing on progress? A/B tests?

You: Downloads are mainly a licensing change: persistent licences with expiry and per-account download counts, size-optimised renditions, periodic check-in. The progress race is LWW by timestamp — I won't pay for distributed locking to protect a resume point. Experiments assign users to cells by a stable hash and compare QoE and engagement from the Kafka stream.

Quick reference

The 60-second version

Netflix splits into a control plane in the cloud and a data plane on its own CDN that only serves bytes. Masters land in S3; chunked parallel encoding cut at shot boundaries produces a per-title ladder, quality-checked and packaged into aligned, encrypted CMAF segments. Appliances inside ISPs are filled proactively off-peak because the catalog is known; peak egress is ~75 Tbps. On play, the API authorises, takes a stream lease and returns ranked appliance URLs; the client fetches the licence in parallel with the first segments and runs hybrid ABR, failing over between appliances on its own. Heartbeats — ~500K writes a second at peak — go to Cassandra by profile with last-write-wins and to Kafka for analytics and recommendations. The home page is precomputed rows with continue-watching merged in and popular rows as the fallback, and active-active regions, circuit breakers and chaos drills keep it standing.

What to say in the interview

"Large-content systems split the facts from the bytes. Metadata goes in a transactional database with an ordered history; bytes go into an object store, in chunks named by their hash, uploaded and downloaded directly with pre-signed URLs so my servers never carry them. Content addressing gives dedup, idempotent retries, integrity checks and caching forever. In Dropbox the per-folder journal is the truth, blocks travel only when the server asks for them, and notifications are only a doorbell. In Netflix the dominant cost is bandwidth, so I decide in the cloud, deliver from the edge, fill the edge before anyone asks, and make delivery independent of the deciders — then let the player pick the right version every few seconds."