R14 — Resilience and traffic control: load balancing, timeouts, circuit breakers and rate limits

R13 asked what breaks first as load grows. This section is about what happens when something does break, or slows down, while the load keeps arriving: how traffic is spread, how a failing dependency is stopped from taking everything else with it, and how you say no to work you cannot safely accept.

The story: one rail slows down and the whole platform tips over

A payment platform sends payments to three rails through one service. Tuesday morning, one correspondent bank starts answering in eight seconds instead of two hundred milliseconds. Nothing is down. Every call eventually succeeds.

Follow what happens next.

1. Threads pile up. Every call to the slow rail holds a request thread and a pooled connection for eight seconds. The pool has two hundred threads. At twenty-five calls a second to that rail, the pool is full in eight seconds.
2. The healthy rails stop too. The other two rails share the same pool. Their requests now queue behind the slow ones, so a payment to a perfectly healthy rail times out.
3. Clients retry. Merchants see timeouts and resend. The gateway retries too. Load on the service triples at exactly the moment it can handle the least.
4. The load balancer finishes the job. Health checks start timing out on the saturated tasks. The load balancer removes them, so the remaining tasks get even more traffic, and they fail their health checks next.

Ten minutes in, every rail is down and the dashboards show the payment service as the problem. The actual cause, one slow bank, is three hops away.

Nothing in that story was a bug. Each step was a reasonable default doing exactly what it was configured to do. That is why resilience is a design topic rather than a debugging one: the failure is in how the parts combine.

flowchart LR
  A[Rail slows<br/>8s per call] --> B[Threads held<br/>pool exhausted]
  B --> C[Healthy rails<br/>queue behind it]
  C --> D[Clients and gateway<br/>retry]
  D --> E[Health checks fail<br/>tasks removed]
  E --> B

Every tool in this section breaks one link in that chain.

Resilience is not making failure rarer; it is making a failure stay the size it started.

Load balancing: spreading the work, and noticing who is sick

Start with the simplest version of the problem. You have one payment API instance, and it is either too small for the traffic or it is a single point of failure. Usually both. So you run several identical instances, and now something has to decide which one each request goes to.

That something is the load balancer. It does four jobs, and the last two matter more than people think:

• Spreads traffic so no single instance is overloaded while others sit idle.
• Removes the single point of failure, because losing one instance loses only its share.
• Absorbs and filters floods, which helps (but does not by itself solve) a denial-of-service attack.
• Enables zero-downtime maintenance: drain one target, deploy, put it back. R16 builds on exactly this.

High availability and fault tolerance are different claims

Interviewers use these words to see whether you use them precisely.

Fault tolerance is the stricter claim. The load balancer's health checks are what make both possible: without them, a dead instance keeps receiving its share of traffic and the redundancy buys nothing. R15 covers how to calculate the availability number itself.

Hardware, software or managed

On AWS the question is almost always ALB or NLB, which is really the next question.

Layer 4 or layer 7

The layer is how much of the request the load balancer reads before deciding.

The default for an HTTP API is layer 7, because path-based routing, header inspection and TLS termination at the edge are worth the small latency. Layer 4 earns its place for raw TCP, very high connection counts, or when the payload must stay encrypted end to end until it reaches the service.

The algorithms: static and dynamic

A static algorithm decides without looking at the servers. A dynamic one looks at their current state.

The part people miss about IP hash in payments: a large merchant often sends all its traffic from a small number of egress IPs. Hash on client IP and that merchant's entire volume lands on one task while the others idle. If you need affinity, hash on something with real spread, or better, keep the service stateless (R1) so you do not need affinity at all.

And the trap with "fastest wins": a task that is failing fast, returning errors in two milliseconds, looks like the best server in the fleet to a latency-based algorithm. Pair dynamic algorithms with health checks and error rates, never latency alone.

Geolocation, and DNS routing versus a load balancer

For users spread across regions, the first decision is which region, before which server. Geolocation routing sends each client to the nearest healthy region (Route 53 latency or geolocation policies, Cloudflare, Akamai, AWS Global Accelerator).

These are two different decisions at two different levels, and conflating them is a common slip:

A failover that depends on DNS is only as fast as the slowest client cache honouring the TTL. R13 covers what that means for where writes are homed in a multi-region payment platform.

Health checks: the setting that turned the story into an outage

Look back at step 4 of the story. The load balancer did its job perfectly: tasks were failing health checks, so it removed them. The fault was in what the health check was checking.

A health check that includes a shared dependency turns that dependency's outage into yours.

The defensible middle: check what is specific to this instance (it can serve requests, it has a database connection, its own pools are not saturated) and leave shared downstreams to the circuit breakers below. A rail being slow is a reason to open that rail's breaker, not a reason to remove a task that can still serve the other two rails.

The defensible position on load balancing: layer 7 by default for HTTP, least connections when requests vary in length, stateless services so affinity is never needed, and health checks that report on the instance rather than on everything it depends on.

Before reading further, play with it. Send the same mixed traffic through each algorithm and watch the per-server load. Notice how round robin piles long requests onto one server while least connections keeps them even, and what IP hash does when many requests share a client.

Timeouts, circuit breakers and bulkheads

These are one idea in three parts: stop a failing dependency from consuming the resources you need for everything else. Take them in order, because each one depends on the one before it.

Start with the analogy, because it is exact

A circuit breaker in your house watches the current. When a surge happens it trips, cutting the flow so the wiring does not overheat and the house does not burn down. It is deliberately easier to reset a breaker than to rewire a building.

The software version watches calls to a dependency. If service A calls service B and B goes down, A waits on timeouts — and while it waits it is holding a thread, a connection from the pool, and the memory of a half-finished request. Enough of those and A falls over too, even though nothing is wrong with A. That is cascading failure, and the breaker's job is to convert a slow, resource-consuming failure into a fast, cheap one.

A timeout is the precondition, not an optimisation

A circuit breaker on a call with no timeout is decoration, because the call never fails — it hangs, and your threads hang with it. If no exception is ever thrown, the breaker records no failures and never trips.

By default RestTemplate uses HttpURLConnection, which has infinite timeouts. A hung downstream will hold your threads until the process dies. So there are two timeouts to set, and they catch different things. R1.1 shows the handshake and the server-side queues each one is waiting on.

@Configuration
public class RestClientConfig {

    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder builder) {
        return builder
            .setConnectTimeout(Duration.ofSeconds(2))   // fail fast if we cannot connect
            .setReadTimeout(Duration.ofSeconds(3))      // give the server 3s to respond
            .build();
    }
}

Now a frozen dependency produces a ResourceAccessException after three seconds, the breaker records a failure, and the system has something to react to. Set the timeouts first; everything below depends on them.

(RestTemplate is in maintenance mode — RestClient for blocking calls and WebClient for reactive ones are the current APIs, configured the same way. The reasoning is identical whichever you use.)

The circuit breaker itself

A state machine with three states:

• CLOSED — normal. Requests flow through, and the breaker records each outcome as success or failure.
• OPEN — the failure rate crossed the threshold. Requests are rejected immediately, with a CallNotPermittedException, without touching the downstream service at all. This is the state that protects your threads.
• HALF_OPEN — after the configured wait, a small number of probe calls are allowed through. If they succeed the breaker closes; if they fail it opens again for another wait.

How it decides: the sliding window. Resilience4j aggregates recent outcomes in one of two shapes, and you pick per instance:

• Count-based — the last N calls. slidingWindowSize: 10 means the failure rate is computed over the last ten. Predictable, and the right default for a steady call rate.
• Time-based — all calls in the last N seconds. Better when traffic is bursty, because ten calls might span an hour on a quiet endpoint and the breaker would be judging ancient history.

resilience4j:
  circuitbreaker:
    instances:
      inventoryService:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 10                    # judge on the last 10 calls
        minimumNumberOfCalls: 5                  # but not until we have seen 5
        failureRateThreshold: 50                 # >= 50% failures -> OPEN
        waitDurationInOpenState: 10s             # stay OPEN this long, then probe
        permittedNumberOfCallsInHalfOpenState: 3 # 3 probe calls decide
        slowCallDurationThreshold: 2s            # a call slower than this counts as slow
        slowCallRateThreshold: 60                # >= 60% slow -> OPEN, even with no errors

Two settings earn special mention. minimumNumberOfCalls stops the breaker judging on three samples — without it, two failures at startup can open the circuit. And slowCallRateThreshold is the one people omit and then regret: in practice a dependency gets slow long before it starts erroring, and slow is what exhausts your threads. A breaker that only counts exceptions will sit closed while your service dies of latency.

Wiring it up in Spring Boot needs the starter plus AOP, because the annotations are implemented as aspects:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

@Service
public class OrderService {

    @CircuitBreaker(name = "inventoryService", fallbackMethod = "inventoryFallback")
    public String checkInventory(String productId) {
        return restTemplate.getForObject("http://inventory-service/api/" + productId, String.class);
    }

    // runs when the call fails OR when the circuit is already OPEN
    public String inventoryFallback(String productId, Throwable t) {
        return "Inventory unavailable; defaulting to out of stock for " + productId;
    }
}

The fallback rules catch everyone once: it must live in the same class, take the same parameters as the protected method, and add a Throwable as the final argument. Miss any of those and Resilience4j cannot bind it, and you get an exception about a missing fallback rather than the graceful degradation you wrote.

And decide what a fallback should actually do, because a bad one is worse than none. Returning cached or default data is right for a product lookup. For anything that moves money, the honest fallback is to queue the work and tell the caller it is pending — never to fabricate a success, and never to silently skip a control like screening.

The transitions, drawn. The three states above form a loop, and the detail worth stating is the asymmetry in HALF_OPEN: the probes must all succeed to close the circuit, while any failure sends it straight back to OPEN and restarts the wait. Closing resets the failure counts, so the breaker judges the recovered dependency on fresh calls rather than on the outage.

flowchart LR
  C[CLOSED<br/>calls flow,<br/>outcomes counted] -->|failure or slow rate<br/>over threshold| O[OPEN<br/>fail fast,<br/>fallback]
  O -->|wait duration<br/>expires| H[HALF_OPEN<br/>a few probes]
  H -->|any probe fails| O
  H -->|all probes succeed,<br/>counts reset| C

Resilience4j is the library to name on the JVM; Hystrix is its retired predecessor and Polly is the .NET equivalent. The state machine is the same in all of them.

Before reading on, play with it. Make the dependency fail and watch the breaker count failures in CLOSED, trip to OPEN, and start failing calls instantly without touching the dependency. Then let the dependency recover and watch HALF_OPEN let a few probes through: one failure sends it back to OPEN, a clean run closes it.

TimeLimiter: the timeout the socket cannot give you

You have a read timeout, so why another one?

Because the read timeout resets every time a packet arrives. A broken server that trickles one byte every two seconds never trips a three-second read timeout — and your request stays open for minutes. The socket is technically healthy; the call is not.

A TimeLimiter puts an absolute stopwatch on the whole method. Regardless of network activity, database time or thread sleeps, if it has not finished by the deadline it aborts with a TimeoutException.

resilience4j:
  timelimiter:
    instances:
      inventoryService:
        timeoutDuration: 2s
        cancelRunningFuture: true    # interrupt the background thread on timeout

To apply it to a blocking call you must return a CompletableFuture, so the work runs on another thread and the caller can walk away when the clock runs out:

@CircuitBreaker(name = "inventoryService", fallbackMethod = "inventoryFallback")
@TimeLimiter(name = "inventoryService", fallbackMethod = "inventoryFallback")
public CompletableFuture<String> checkInventory(String productId) {
    return CompletableFuture.supplyAsync(() ->
        restTemplate.getForObject("http://inventory-service/api/" + productId, String.class));
}

public CompletableFuture<String> inventoryFallback(String productId, Throwable t) {
    return CompletableFuture.completedFuture("Inventory slow or down; defaulting to out of stock");
}

Stacking order matters, and it is the right way round by default: the TimeLimiter wraps the method, and the CircuitBreaker wraps the TimeLimiter. So a timeout becomes a TimeoutException, the breaker sees it as a failure in its sliding window, and enough of them trip the circuit. If the order were reversed, timeouts would never be counted.

The three layers together, which is worth being able to recite:

Bulkheads

A bulkhead caps how much of your capacity one dependency can consume. The name is from shipbuilding: compartments, so one breach does not sink the vessel.

• Semaphore bulkhead — at most N concurrent calls to that dependency; the rest are rejected immediately. Cheap, and it runs on the caller's thread.
• Thread-pool bulkhead — the dependency gets its own pool entirely, so even a total stall cannot consume the threads serving everything else.

resilience4j:
  bulkhead:
    instances:
      railAdapter:
        maxConcurrentCalls: 20
        maxWaitDuration: 100ms

The reason this matters in payments specifically: one rail degrading must not stop the others. Without a bulkhead, a slow correspondent bank consumes every thread in the service and every rail goes down together.

Retries, and how they amplify

Retries are necessary — R0's whole argument is that at-least-once is the only workable delivery model — but they are also how a blip becomes an outage.

• Backoff with jitter is non-negotiable. Synchronised retries arrive as a wave exactly when the dependency is weakest.
• Cap the attempts, and never retry inside an open circuit.
• Never retry a non-idempotent call without an idempotency key (R3).
• Retry at one layer only. Three attempts at the client, three at the gateway and three at the service is twenty-seven calls for one user action.

What to measure

Breaker state transitions as events (resilience4j.circuitbreaker.state), failure rate and slow-call rate separately, bulkhead saturation, retry counts, and fallback invocations. Two readings worth knowing: a breaker that opens regularly is telling you something real about a dependency, and a breaker that has never opened in a year is probably configured never to open — usually because nothing sets a timeout, so no failure is ever recorded.

The same idea at other layers

Worth naming, because interviewers like the connection: an ALB health check removing an unhealthy target is a circuit breaker for instances. ECS's deployment circuit breaker aborts a rollout whose tasks never become healthy. And SQS maxReceiveCount with a DLQ stops a poison message looping forever. Different altitudes, identical reasoning.

These last four rules come from autoscaling, which R13 covers in full. They belong here too, because scaling out is the other half of surviving load: every task you add is more pressure on the dependencies the patterns above protect.

The four rules, if you remember nothing else

• Scale on the constraint, not on CPU. Queue consumers scale on backlog per task; APIs on requests per target or concurrency.
• Scale out fast, scale in slowly. Flapping costs more than a few idle tasks, and every new task pays a cold start.
• Know what you are scaling into. More tasks means more connection pools, more consumers in the group, more pressure on every downstream limit.
• Set a floor that survives one task dying, and a ceiling that protects everything behind you.

Rate limiting: saying no before you are forced to

Everything so far protects you from a dependency that is failing. Rate limiting protects you from your callers: the merchant whose batch job sends ten thousand requests in a second, the client stuck in a retry loop, the integration that polls payment status every fifty milliseconds.

Without a limit, one caller's burst consumes the capacity everyone else is paying for. With one, that caller gets told to slow down and everyone else carries on.

There are three reasons to limit, and an interviewer will want to hear which one you are designing for:

In payments the key is the merchant or API key, not the IP address. Limits are per merchant per endpoint, and they can be weighted: submitting a bulk payout file can cost more tokens than a status lookup, because it costs the system more.

The algorithms, by mechanism

Know these by how they work, not by name. Each one is a different answer to the question "how do you count?"

The token bucket keeps no history. On each request it computes the refill from elapsed time, lazily, so there is no background timer and the state is just two numbers. That, plus burst tolerance and a natural way to compute when to retry, is why it is the industry default.

Worked numbers

These are the numbers interviewers ask you to work through on the whiteboard.

Token bucket: capacity 10, refill 5 per second. A caller who has been quiet can send 10 requests at once, the maximum burst. After that the steady rate is 5 per second. The capacity controls bursts; the refill rate controls the long-run average.

Fixed window: 100 per minute. A caller sends 100 requests at 12:00:59 and another 100 at 12:01:00. Each falls in a fresh window, so all 200 are allowed: twice the limit in two seconds.

Sliding window counter: 100 per minute. The previous minute had 80 requests, the current minute has 30 so far, and you are 25% of the way into the current minute. The estimate is 80 × (1 − 0.25) + 30 = 90, which is under 100, so the request is allowed. Same memory as a fixed window, and no doubled edge.

Hybrid: capacity 60, refill 10 per second, window limit 500 per minute. The bucket alone would allow 600 in a minute (10 per second for 60 seconds). The window caps it at 500. The bucket shapes the burst; the window holds the total.

Before reading further, play with them. Send the same burst into a token bucket and a leaky bucket and watch the difference: the token bucket lets the burst through up to its capacity and then rejects, while the leaky bucket queues it and lets requests out at a steady rate.

Then try the boundary. Fire a burst just before a fixed window resets and another just after, and watch both get through; run the same pattern against the sliding window and watch the second burst get rejected.

Which to choose:

• Token bucket as the default for API limits: constant memory, tolerates the bursts real clients send, supports weighted cost.
• Sliding window counter for long windows, "per hour" or "per day", where burst semantics would confuse customers.
• Sliding window log only for low-volume, high-stakes limits such as login or OTP attempts, where exactness matters and the memory is small.
• Leaky bucket for shaping your own outbound traffic to a partner, where a steady rate is the whole point.

Where the limiter sits

There is no single right place; each layer sees different information at a different cost per rejected request.

Enforce each limit at the earliest layer that knows enough to enforce it. A cheap IP limit before authentication protects the auth service itself; the per-merchant limit runs after it, because it needs the identity. R1 covers the gateway pipeline this sits in.

What a throttled caller sees

A rejected request returns HTTP 429 Too Many Requests, and the headers matter as much as the status:

HTTP/1.1 429 Too Many Requests
Retry-After: 1
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1790000001

A bare 429 tells the client "no" but not "when", so it retries immediately, and a thousand clients retrying at once make the overload worse. Retry-After turns the limiter into a pacing signal. With a token bucket it falls straight out of the state, (cost − tokens) / rate, and it must be rounded up: rounding down invites a retry that will fail again. Client SDKs should add jitter so retries do not all land on the same second.

When the limiter itself is down: fail open or fail closed

The limiter usually keeps its counters in a shared store such as Redis. When that store is unreachable, every request faces a choice nobody made explicitly.

For money movement, fail closed.

An unlimited burst of payment submissions while the limiter is blind is how a compromised API key, or a merchant's runaway retry loop, becomes real money out of the door. The cost is that a limiter outage now stops payments, so the limiter on that path has to be as highly available as the payment service, with a circuit breaker on its store so a Redis failure is detected in milliseconds rather than after a timeout. Say that trade-off out loud; it is what the question is grading.

R17 walks through the full distributed rate limiter design: atomic counters in Redis with Lua, hot keys, multi-region limits and rule reloading.

Backpressure: when a rail degrades

Put the pieces together for the situation the opening story described. A rail degrades, and the platform has to slow down rather than pile up.

• Per-rail circuit breakers and bulkheads, so the slow rail fails fast and cannot take threads from the healthy ones.
• Queue-depth thresholds. Accepted payments for that rail wait in a queue (R9), and when the depth crosses a threshold you stop accepting new work for that rail rather than letting the backlog grow without bound.
• Shed at the edge with 429 and Retry-After, so callers back off instead of hammering.
• Accept and report pending, rather than time out. A payment queued with a clear status beats a timeout the client will retry blindly.
• Never retry a non-idempotent payment call blindly. A timeout on a payment submission means "unknown", not "failed"; the retry goes out with the same idempotency key, and the rail call is made idempotent on your side (R3).

R13's payments-specific scaling list summarises the same behaviour from the capacity side.

Scatter-gather and tail latency

Some requests cannot be answered by one server. A merchant searches its transaction history, and the transactions are sharded across many partitions (R10). A payment initiation needs a balance check, a fraud score and a sanctions screen, owned by three different services.

The pattern is the same in both:

• Scatter: send the request to every recipient in parallel.
• Gather: combine the replies into one answer, with whatever business logic fits: merge and sort, take the top k, or take the first acceptable reply.

It is usually asynchronous, and the scatterer and the gatherer can be different components. It shows up in queries across shards, key lookups in a distributed key-value store, composing several APIs behind one endpoint, and the map and reduce steps of a batch job.

Your latency is the slowest reply

The catch is arithmetic. A scatter-gather request is as slow as its slowest recipient, so the tail latency of one server becomes the typical latency of the whole request.

Suppose each shard answers within 100 ms 99% of the time. The chance that at least one of N shards is slower than that:

At a hundred shards, a once-in-a-hundred slow reply becomes the common case. This is why R15 insists on measuring percentiles rather than averages, and why the fan-out width is a design decision, not a detail.

Designing the gather

Three decisions, and the timeout patterns above are how you implement each:

Partial results are fine for a search page and never fine for a control. A transaction search that returns nineteen of twenty shards with a "results may be incomplete" banner is a reasonable product. A payment initiation that proceeds because the sanctions screen did not reply in time is a regulatory incident. For the control path, a missing reply means the payment waits, exactly as the fallback rule above says: queue it, mark it pending, never fabricate the answer.

Before reading further, play with it. Send a query across the four shards and watch the overall response wait for the slowest one; then slow down a single shard and watch the whole request's latency follow it, however fast the other three are.

What to say in the interview

"One slow dependency becomes an outage when the parts combine badly, so I break every link in that chain: connect and read timeouts plus a TimeLimiter so nothing waits forever, a circuit breaker per rail that counts slow calls as well as errors, and a bulkhead so one rail cannot take the threads the others need. Retries happen at one layer, with backoff, jitter and an idempotency key, because a timeout on a payment means unknown, not failed. At the edge I rate limit per merchant with a token bucket and return 429 with Retry-After, and for money movement the limiter fails closed. Behind it, the load balancer routes on the instance's own health, never on a shared downstream, so a degraded rail opens a breaker instead of emptying the fleet."