R13 — Scale and multi-region: what breaks first and where writes are homed

R10 to R12 covered where data lives, how to make reads and writes fast, and what to cache. This section steps back to the whole system under growing load: how to reason about capacity out loud, which tier gives way first, how to autoscale the compute and the database on AWS without scaling into a new wall, and why, for money, writes are homed to one region per account rather than accepted everywhere.

First principles: what "scaling" actually means

Two ways to handle more load, and only one of them keeps working.

flowchart LR
  subgraph ORDER["1k to 100k TPS — failure order"]
    O1["Connection pools"] --> O2["Single write database"] --> O3["Synchronous downstream calls"] --> O4["Ledger hot rows"]
  end
  subgraph REG["Write homing"]
    R1["Region A<br/>WRITER for accounts 1-N"]
    R2["Region B<br/>reads only for 1-N<br/>WRITER for N+1-M"]
    R1 ---|"replication"| R2
  end
  O4 --> REG
  REG --> SB["Single writer per account<br/>prevents split-brain<br/>30s failover beats a money conflict"]

Vertical — a bigger machine. Simple, needs no code changes, and the ceiling arrives quickly and expensively. Still the right first move for a database.

Horizontal — more machines. Cheaper past a point and effectively unbounded, but it only works if the thing you are copying holds no state. A service that keeps a session, a counter or a cache in memory cannot simply be run five times; instance three will disagree with instance one. Statelessness is the price of horizontal scaling, and "push the state into the database, the cache or the token" is what most of the work actually is.

Latency and throughput are not the same thing, and confusing them is the classic junior error. Latency is how long one request takes; throughput is how many you finish per second. A queue at a bank with one slow teller and forty windows has terrible latency and fine throughput. Adding servers usually buys throughput and does nothing for latency.

Percentiles, not averages. An average hides the shape. If 99 requests take 10ms and one takes 3 seconds, the average is a comfortable 40ms and one customer is furious. Quote p50, p95, p99 — and know that at scale, p99 is the experience of your busiest customers, because the more requests someone makes, the likelier they are to hit the slow one.

Little's Law, the one piece of maths worth memorising here:

concurrency = throughput × latency

At 500 requests/sec with 200ms average latency, you have 100 requests in flight at all times. That single number tells you how many threads, how many connections, and how big the pool has to be. Halve the latency and you halve the concurrency for the same load — which is why fixing a slow dependency is often cheaper than adding machines.

Software scaling before hardware scaling

A service gets slow every month-end. The first instinct is to double the task count, and the bill doubles with it. Someone then looks at the slowest endpoint and finds a query scanning a whole table to find one customer's rows. One index later, the original task count is idle at month-end.

Vertical and horizontal are both answers to one question: which machines do I add? There is an earlier question that is cheaper to answer, and people skip it.

Software-based scaling gets more capacity out of the hardware you already have. Tune the query that does a full scan, add the missing index, put an in-process (L1) or shared (L2) cache in front of the hot read, stop serialising a payload twice. Nothing new is provisioned, and the same boxes carry more load.

Hardware-based scaling is the vertical and horizontal choice above: a bigger box, or more of them behind a load balancer.

The order matters. Adding machines to a service that does a full table scan per request multiplies the scan. Fix the obvious inefficiency first, measure, and only then buy capacity.

What scalability is for

It is worth being precise about the word, because interviewers notice when it is used loosely. A scalable system keeps its behaviour as load grows by adding resources in proportion, and three properties come with it:

• Flexibility. You can add or remove capacity when demand moves, rather than being sized once for a guess.
• Performance. Scalability helps keep latency steady under load, but it does not guarantee it. A system can scale perfectly in throughput and still be slow per request; that is the latency/throughput distinction from the first principles above.
• Reliability. More than one copy of everything means a lost instance is a capacity dip rather than an outage, provided the copies hold no state the others need.

Scalability is about throughput under growth. It does not make a slow request fast, and it does not make a stateful service safe to copy.

A worked capacity example

Interviewers ask you to do this out loud, and doing it badly is more damaging than not knowing a technology.

10 million payments a day, peak is 4× average, each payment does roughly 20ms of database work.

average   = 10,000,000 / 86,400            ≈ 116 payments/sec
peak      = 116 × 4                        ≈ 464/sec
concurrency at peak = 464 × 0.020s         ≈ 9.3 concurrent DB operations

Nine concurrent operations is nothing — one database handles that comfortably, and the honest answer is "this does not need sharding." Now change one number:

50,000/sec peak × 20ms = 1,000 concurrent DB operations

A thousand is far past any single instance, so sharding is forced, not chosen — and saying it that way shows you arrived at the design rather than reaching for it. Then size the rest from the same number: at 1,000 concurrent operations and ~50 connections per node, you need roughly 20 database nodes' worth of write capacity, and your Kafka partition count has to be at least as high as the consumer parallelism that keeps up with 50,000/sec.

Always state your assumptions as you go — peak multiplier, per-operation cost, headroom target (usually 30–40%). The interviewer is grading the method, not the arithmetic.

A second worked example: one system, three tiers

The payments example above sizes one tier, the database. Real systems have several tiers in series, and the more useful habit is to size each of them and find the one that gives way first. A deliberately simple example makes the method visible.

An internal news app for a company's employees. Every request goes through a web tier, an application tier and a database. An acquisition multiplies the headcount by five.

flowchart LR
  U[Employees] --> LB[Load balancer]
  LB --> W[Web tier<br/>250 RPS]
  W --> A[App tier<br/>500 RPS]
  A --> C[(App cache)]
  A --> D[(Database<br/>400 QPS)]

Step 1: turn people into requests

Start from the population, get to peak concurrent users, and multiply by how often each one makes a request. Here each active user makes about 2 requests a minute.

Step 2: put every tier in the same unit

Capacity figures arrive in mixed units. Convert them all to one before comparing anything, or you will compare per-second capacity with per-minute load and be off by sixty.

Step 3: the system runs at the speed of its narrowest tier

System throughput is the minimum across the tiers, not the sum and not the average. Every request must cross all three, so the web tier at 15,000/min caps the whole system, however much spare room the app tier has.

throughput = min(web, app, db) = min(15,000, 30,000, 24,000) = 15,000 req/min

Case 1 fits, but read the web row honestly: 93% is far above the 60–70% headroom band used later in this section, so even before the acquisition the web tier is one busy morning from saturating. Case 2 does not break one tier; it breaks every tier, which is what a fivefold jump usually does.

Step 4: fix the tiers in order of risk, not order of overload

The tempting move is to start with the database, because it is the scariest. The right move is to fix the cheap tiers first and see how much database work you can avoid.

• Web tier. Stateless and low risk. 80,000 / 15,000 = 5.3, so at least six servers of the current size behind the load balancer; sized for 70% utilisation, 80,000 / (15,000 × 0.7) ≈ 7.6, so eight. Bigger instances are an equally valid answer.
• App tier. Also add instances: 80,000 / 30,000 = 2.7, so three at the bare minimum, four with headroom. Then add an application-level cache for the articles everyone reads.
• Database. The cache is what saves it. If the cache absorbs about 80% of database reads, the database sees 80,000 × 0.2 = 16,000/min, which fits inside 24,000 at roughly 67% utilisation, inside the headroom band. No replication, no sharding, no migration.

The assumption to state out loud: this treats one request as one database read. Writes, and any request the cache cannot answer, pass straight through, so a write-heavy app would land on the database anyway and the conversation moves to R11's write levers.

Fix the cheap tiers first, put a cache in front of the expensive one, and prove the data tier is still the bottleneck before you touch it.

Before reading further, play with the calculator. Push the user count up, watch which tier turns red first, then raise the cache hit rate and watch the database row recover while the web and app rows do not. That asymmetry is the point: a cache protects the tier behind it, never the tiers in front.

Step 5: say what happens at the next jump

Interviewers like to push one step further: "and if it grows again?" The answer falls out of the same table.

At that point the hit rate cannot be pushed much higher, so the next moves are the data-tier ladder from the table below: a bigger instance, then read replicas behind the cache, and only for writes, partitioning. The cache bought one doubling, not every doubling, and saying so shows you know where the curve bends.

What the two examples share

The method is the same; only the formula differs. The payments example turns load into concurrency with Little's Law; this one turns load into a per-tier utilisation and takes the minimum. Both start from stated assumptions — population, peak multiplier, per-request cost, headroom target — and both end with a sentence about whether the expensive move is forced or optional.

The arithmetic, as a script you can say

Whichever example the interviewer gives you, the same six steps carry it. Say each one before you compute it.

1. Population      -> peak concurrent users         (state the peak share)
2. Users           -> requests per second at peak   (state the per-user rate and the peak multiplier)
3. Requests        -> work per tier                 (one request = how many DB queries? cache hit rate?)
4. Work per tier   -> utilisation per tier          (convert to one unit first)
5. Narrowest tier  -> system throughput             (min, never sum)
6. Headroom target -> instance counts               (size for 60-70%, not 100%)

If a step needs a number you do not have, pick a round one and say it is an assumption. The interviewer is grading the method, and a stated assumption can be corrected in one sentence; a hidden one cannot.

Which tier to scale, and how much it costs to get wrong

The tiers are not equally risky to scale. The further back you go, the more state you are carrying and the harder the change is to undo.

The rule for the data tier: prove it is the bottleneck before you change it. A database that looks overloaded is often serving a query that should have been indexed, or reads that should have been cached. Replication, partitioning and sharding (R10) are expensive and largely one-way; they are the last move, not the first.

The trap to name: scaling the tier that is shouting rather than the tier that is narrowest. In the news app, the database looks the most dangerous, but the web tier is the one that caps throughput in Case 1.

Vertical or horizontal, tier by tier

The first principles said vertical is the right first move for a database and horizontal needs statelessness. Put those two facts against the tiers and the default for each falls out.

The things that break, in order, and why

1. Connection pools. Each instance holds, say, 20 database connections. Twenty instances is 400 connections, and Postgres starts struggling in the low hundreds. This is the first wall almost everyone hits, and the fix is a connection proxy — PgBouncer, RDS Proxy — not a bigger pool. Pool size is governed by Little's Law, not optimism: more connections than the database has cores does not make it faster, it makes it thrash. R1.1 shows how a request waits for a worker thread and then for a pooled connection inside one service, and what fails first when the database slows.

2. The single write database. Reads scale out easily with replicas; writes do not, because every write must go through the primary. The order of moves is: read replicas → caching → partition the writes.

3. Synchronous call chains. Service A calls B calls C. A's latency is the sum, and A's availability is the product — three services at 99.9% give you 99.7%. Worse, when C slows down, B's threads fill waiting, then A's, and an outage travels upstream. Timeouts shorter than the caller's, circuit breakers, and bulkheads are the containment.

4. Hot rows. Everything is fine until one account, one merchant or one sequence row takes every write. Sharding does not help, because a shard key cannot split a single row. The answers are batching, a queue per key, or restructuring the write (append entries and aggregate later, rather than updating one balance in place).

Why scaling is hard: the challenges underneath

That list is the specific order a payments platform hits. The challenges underneath it are general, and every scaling decision trades one of them against another.

The part people miss is that these trade against each other. Adding read replicas relieves the bottleneck and costs you consistency. Adding regions relieves latency for distant users and costs you consistency and money. Caching relieves the bottleneck and costs you staleness. A good answer names which one it is spending.

What each move relieves, and what it spends

For containing synchronous call chains (timeouts, circuit breakers, bulkheads, retries), see R14. For read replicas, CQRS and connection pooling, see R11; for partitioning and sharding, R10.

Autoscaling ECS Fargate: what to scale on, and when

The question to answer first is not "which metric" but "what is my constraint". A service that spends its life waiting on a database is not CPU-bound, so scaling it on CPU means you add tasks long after customers started waiting — or never, because CPU stays at 15% while latency triples.

The three policy types, and when each fits:

• Target tracking — "keep requests-per-target at 300". Application Auto Scaling works out the task count. This is the default and usually the right answer.
• Step scaling — explicit thresholds and increments. Use when you want asymmetric behaviour, such as adding six tasks when badly behind but removing one at a time.
• Scheduled — set the floor before a known event: payroll on the 25th, a market open, a marketing push.

Backlog per task, the formula worth memorising for queue consumers, because "scale on queue depth" is the answer that sounds right and is not:

messages one task can clear in the target time
        = acceptable latency (s) / average processing time (s)

backlog per task = ApproximateNumberOfMessagesVisible / RunningTaskCount

target value for the scaling policy = the first number

With a 60-second acceptable latency and 200ms processing, one task clears 300 messages, so you track "backlog per task = 300". You publish that as a custom CloudWatch metric — or build it with CloudWatch metric math (m1 / m2) so you do not have to emit anything at all — and point a target-tracking policy at it.

Measuring the inputs. ApproximateNumberOfMessagesVisible and ApproximateAgeOfOldestMessage for SQS; consumer lag from MSK's SumOffsetLag or from your consumer's own Micrometer gauge; RequestCountPerTarget and TargetResponseTime from the ALB; RunningTaskCount from ECS. Age of oldest message is the better alarm even when backlog is the better scaling signal — depth tells you how much work there is, age tells you how long a customer has been waiting.

Timing, which is where people get burned:

• Scale out fast, scale in slowly. Target tracking already scales in conservatively; fight the urge to speed it up. Flapping costs more than a few idle tasks.
• Mind the cold start. A Fargate task needs image pull plus container start plus JVM warm-up plus connection pool fill — frequently 60–120 seconds for a Spring Boot service. Anything that spikes faster than that needs headroom or scheduled scaling, not faster reaction. Trim what you can: smaller images, ARM/Graviton, CDS or AOT, and a readiness probe that is honest about when the app is genuinely ready.
• Set a floor that survives one task dying in your smallest AZ, and a ceiling that protects everything downstream.
• Cooldowns and the scale-out alarm period. A 60-second alarm period plus three evaluation periods means three minutes before you even start. For spiky payment traffic, use a one-minute period and fewer evaluations for scale-out, and the opposite for scale-in.

The trap that matters most in payments: what you scale into. Ten more tasks means ten more connection pools, ten more consumers joining a group and triggering a rebalance, and ten times the pressure on any downstream service with its own limits. Autoscaling a service that talks to a database without a connection proxy is a common way to turn a slowdown into an outage. Always ask what the new tasks will exhaust.

Autoscaling Aurora Postgres

Two different mechanisms get called "Aurora autoscaling", and conflating them is a giveaway.

1. Read replica auto scaling adds and removes reader instances based on average reader CPU or average connections, between a min and max you set. Readers share the same storage volume, so a new one is available in a couple of minutes without copying data, and the reader endpoint load-balances across them.

What it does not do is help with writes. There is one writer, and it is the ceiling. When writes are the constraint the options are: a bigger writer (vertical, and quick), then splitting read traffic away properly, then sharding by account (R10), then Aurora Limitless or a different data model. Saying that ordering out loud is the answer.

2. Aurora Serverless v2 scales the instance itself, in place, in half-ACU steps, reacting to CPU, memory and connection pressure within seconds. It suits spiky and unpredictable load, and dev environments that idle. The trade-offs: a minimum capacity you pay for, a price per ACU-hour that beats provisioned only if your peaks are genuinely peaky, and the fact that it still does not give you a second writer.

Storage needs no attention — it grows automatically in 10GB increments to 128TB and you are billed for what you use.

What to watch, and what to act on:

Failover behaviour matters for payments. Writer failover promotes a reader in roughly 30 seconds, and during that window writes fail. Your application must reconnect (use the cluster endpoint, not an instance endpoint), your connection pool must evict dead connections, and in-flight transactions are simply lost — which is exactly why the outbox and idempotency machinery exists. Test it with a deliberate failover rather than assuming.

How many connections each task should hold, and why a proxy rather than a bigger pool, is R11's connection pooling section.

How to measure whether the result is good enough — SLIs, SLOs, the error budget, availability arithmetic and honest latency percentiles — is R15.

Watching a scale-out actually work

Adding tasks is easy; knowing whether it helped is the skill. Watch these together, because any one alone will mislead you:

Three patterns that tell you scaling is not the answer:

• Task count up, throughput flat. You are bottlenecked on something shared — the write database, a lock, a hot key, a third-party rate limit. More instances just means more contention.
• Lag not draining after scale-out. More consumers than partitions, so the extra ones are idle (R9). Repartitioning is the fix, and it is not instant.
• Latency worse after scaling. Almost always the database: more tasks means more connections means more contention. This is where a connection proxy earns its keep.

Capacity headroom is the number to report, not CPU. "We run at 61% of the capacity we can serve at target latency, and peak grew 4% last month" answers the question an executive is actually asking, which is when they need to spend money again.

The four dimensions an interviewer probes

"How would this scale?" sounds like one question. It is really four, and a strong answer touches each one rather than only the first.

Traffic is the dimension everyone answers, and it is covered above.

System presence is where a design changes shape. Serving static content and read-mostly data close to users is cheap: a CDN, edge caches, read replicas in other regions. Accepting writes in more than one region is not cheap, because two regions can now disagree about the same record. For most data that disagreement can be merged. For a balance it cannot, which is why the next section homes each account's writes to a single region.

Data is about volume and write rate rather than request rate. A system can serve modest traffic and still outgrow one database because it keeps every record for seven years.

Features is the one people forget. Each new feature adds a path through the system and usually a dependency, and every synchronous dependency lengthens the call chain from the list above. Scaling the feature set well means keeping new work off the critical path — asynchronously, through events (R4, R9) — rather than adding another hop to the request.

Traffic, presence, data, features: name all four, then spend your time on whichever one the design actually strains.

The multi-region ladder

"Make it multi-region" covers designs that differ by an order of magnitude in difficulty. Climb only as high as the requirement forces you.

Rung 5 is the one worth drawing, because it is the shape the payments answer takes. Every region serves reads, and each write is routed to the region that owns the record.

flowchart LR
  UA[User in region A] --> RA[Region A]
  UB[User in region B] --> RB[Region B]
  RA -- "read: serve locally" --> DA[(Replica of all records)]
  RB -- "write to an A-homed record" --> RA
  RA -- "write: home is A" --> PA[(Primary for A-homed records)]
  PA -. "replicate" .-> RB

The cost is one cross-region hop for a user writing to a record homed elsewhere. The benefit is that no record ever has two writers, so there is nothing to merge.

Rung 6 only works when conflicts can be merged. A shopping cart can take the union of two edits; a counter of likes can add. A debit cannot be merged with another debit against the same balance after both have been accepted. That is why payments stop at rung 5: the next section calls it single-writer-per-account.

The trap to name: treating "active-active" as one thing. Ask whether the interviewer means every region serving traffic (rung 5) or every region writing every record (rung 6), because the second is a conflict-resolution problem dressed as an availability feature.

Scaling a payments platform specifically

What breaks first, in order. Connection pools, then the single write database, then synchronous downstream calls, then ledger hot rows. Say the order before you say a solution; it shows you have actually watched a system scale rather than read about it.

Capacity arithmetic, done out loud. 50k TPS peak, 20ms of database work per payment, means roughly 1,000 concurrent database operations — far past any single instance, so sharding is forced, not chosen. Peak is usually 3–5× average, and you size for peak plus headroom. Doing this arithmetic aloud is worth more than naming a technology.

Sharding. Shard by account or customer, because that keeps one account's ledger entries and ordering on one node — the full treatment, including how to choose the key and what breaks afterwards, is in R10's partitioning and sharding section.

Multi-region. Active-active is genuinely hard for money because concurrent writes to the same balance in two regions cannot be merged after the fact — there is no sensible conflict resolution for a debit. The defensible design is single-writer-per-account: reads served everywhere, writes homed to one region per account, with a fast, tested failover and a stated RPO and RTO. Data residency (India, EU) often forces regional partitioning anyway, which conveniently aligns with account homing.

Split-brain. If both regions accept writes for one account during a partition, you have created money. Avoid it with a lease or a single-writer rule rather than trying to reconcile afterwards. Be willing to say: I would rather take a 30-second failover than resolve a money conflict.

Backpressure and degradation. When a rail degrades, the system must slow down rather than pile up. Queue-depth thresholds, shedding at the edge with 429s and a Retry-After, per-rail circuit breakers so one bad rail does not consume all threads, and bulkheads to isolate thread pools by rail. Timeouts everywhere, always shorter than the caller's. State the customer-visible behaviour too: a queued payment with a clear status beats a timeout the client will retry blindly.

Caching, carefully. Cache reference data — routing tables, BINs, bank directories, FX rates with an explicit staleness bound. Never cache a balance you will act on. That line is a good one to volunteer.

What to say in the interview

"I'd state the SLO and the error budget first, because that decides how much headroom is worth paying for. Then the arithmetic out loud — peak multiplier, per-request cost, concurrency from Little's Law — because that tells you whether sharding is forced or optional. Things break in a predictable order: connection pools, the single write database, synchronous call chains, then hot rows. I shard by account so one account's ordering and ledger entries stay together, home writes to one region per account rather than attempting active-active on money, and shed load at the edge rather than letting queues pile up invisibly."