R16 — Zero-downtime change: evolving schemas and systems while they keep serving traffic

Parallel change (expand, migrate, contract) is the pattern behind every safe backward-incompatible change, whether it is a method signature, an API field or a database column. Expand: add the new form alongside the old, writing both. Migrate: move consumers one at a time. Contract: delete the old form once nothing reads it. Each phase is independently deployable and independently reversible — that is the whole value.

flowchart LR
  D1["1. Add column<br/>nullable, no reads"] --> D2["2. Dual-write<br/>old and new"] --> D3["3. Backfill in batches<br/>verify as you go"] --> D4["4. Switch reads<br/>keep dual-writing"] --> D5["5. Stop writing old<br/>then drop it"]
  D1 -.->|"both versions work"| D2
  D5 --> SAFE["At every step, current and previous<br/>versions run against the same schema"]

flowchart TD
  CL["Traffic"] --> F["Facade / router"]
  F -->|"most corridors"| OLD["Legacy engine — acts"]
  F -->|"one slice, behind a flag"| NEW["New service"]
  OLD --> CMP{"Compare outputs"}
  NEW -->|"shadow: computes but does not act"| CMP
  CMP -->|"mismatch, even 0.1%"| INV["Investigate before cutover"]
  CMP -->|"clean for two weeks"| CUT["Cut over that slice<br/>rollback switch stays tested"]

Applied to a schema, it is five deploys, not one:

1. Add the new column, nullable, no reads.
2. Dual-write old and new.
3. Backfill history in batches, verifying as you go.
4. Switch reads to the new column; keep dual-writing.
5. Stop writing the old column, then drop it — usually a release later.

The discipline this enforces: at every step, the currently deployed code and the previous version both work against the same schema. That is the definition of zero-downtime, and it is why rolling deploys are the real constraint — two versions are live simultaneously whether you planned for it or not.

Deployment strategies and when each fits.

• Blue-green: instant rollback, double the infrastructure, no help with schema.
• Canary: the default for payments, but slice by account or corridor rather than randomly, or you break per-account ordering across versions.
• Rolling: cheapest, and means two versions are live at once.
• Feature flags decouple deploy from release, which is how a risky payment path ships dark and switches on per client.

Blue/green on ECS Fargate, concretely

The abstract pattern is above; this is what it looks like on the platform most teams actually deploy to.

The moving parts. An ECS service runs N tasks behind an Application Load Balancer. The ALB has two target groups — call them blue and green — and two listeners: the production listener on 443 that customers hit, and a test listener on another port that only you hit. CodeDeploy owns the switch between them.

What happens on a deploy:

1. CodeDeploy starts a replacement task set — the green tasks — running the new image, and registers them in the green target group. Blue is still serving every customer.
2. The ALB health-checks green. Tasks only become healthy once your /actuator/health/readiness endpoint says so, which is why readiness must check the things you actually need (database reachable, migrations applied) and not just return 200.
3. You can run tests against the test listener, hitting green directly while production traffic is untouched. This is where a smoke test belongs.
4. Traffic shifts. Three shapes: all-at-once, canary (10% for 5 minutes, then the rest), or linear (10% every minute). Canary is the sane default for payments.
5. A bake time follows, during which CodeDeploy watches your alarms.
6. Blue is terminated after the bake window — and until then, rollback is instant, because blue is still running.

Rollback is the feature you are paying for. Attach CloudWatch alarms to the deployment group — error rate, p99 latency, stuck payments — and CodeDeploy reverts the listener to blue the moment one fires. The shift back is a load-balancer change, so it takes seconds, and no new deployment is needed. Also enable the deployment circuit breaker, which aborts automatically when tasks fail to reach a healthy state.

Connection draining. When blue is deregistered, the ALB stops sending it new requests but lets in-flight ones finish — the deregistration delay, 30 seconds by default. Set it above your slowest request, and set the task's stopTimeout above that again, so ECS does not SIGKILL a container that is mid-payment. In Spring Boot, turn on graceful shutdown so the app itself stops accepting and drains:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=45s

What blue/green does not solve, and this is the part interviewers push on:

• The database is shared. Blue and green talk to the same schema at the same time, so every migration must be expand-contract — the new column nullable, dual-write, backfill, switch reads, drop later. A migration that breaks blue breaks your rollback path, which means you no longer have blue/green, you have a one-way door.
• Messages in flight. Green consumers may pick up events published by blue and vice versa, so message schemas need the same both-directions compatibility. During the shift both versions are live consumers in the same group.
• Long-running work. A saga started on blue may still be running when blue is gone. Its state lives in the database (R5), so green resumes it — provided green understands the version it was started with.
• Cost. You are running two full task sets during the bake. For a large service that is a real number, and it is why canary with a short bake is more common than a long parallel run.

Rolling updates, the alternative. ECS's native deployment with minimumHealthyPercent=100 and maximumPercent=200 replaces tasks gradually without CodeDeploy, target groups or a bake window. Cheaper and simpler, but the rollback is another deployment rather than a listener flip, and there is no test listener. Reasonable for internal services; for anything moving money, the instant revert is worth the setup.

Schema and message compatibility

Message schemas follow the same rule in both directions. Old consumers must tolerate new fields (never make a new field required), and new consumers must tolerate old messages (default anything absent). A schema registry with backward-compatibility enforcement turns this from a code review question into a build failure.

Strangler fig is expand-contract at system scale. Put a facade in front of the legacy system, route one slice of traffic — a payment type, a corridor, a client — to the new implementation, and grow it until the old system is unused, then retire it. Three phases: transform (build the new component), coexist (intercept and redirect at the perimeter, keeping the monolith available for rollback), eliminate (retire the old path).

Shadow running is what makes it safe for money: send the same input to both systems, let only the legacy one act, and compare outputs automatically. Investigate every mismatch rather than averaging it away — a 0.1% discrepancy on payments is not noise, it is a bug you have not found yet.

Payments-specific deployment rules. Drain gracefully: stop accepting new work, let in-flight payments finish, then terminate — never kill a pod mid-debit. Canary by account or corridor rather than randomly, or per-account ordering breaks across versions. Avoid deploying across settlement cutoffs and batch windows. And know the honest answer on rollback: if the new version has written forward-incompatible data, you roll forward with a fix, which is why expand-contract exists in the first place.

What to say in the interview

"The deploy is the easy half — schema and in-flight state decide whether zero-downtime is real. Every backward-incompatible change is expand-contract, five deploys rather than one, so the current and previous versions both work against the same schema. On ECS I'd use blue/green with a canary shift and alarms wired to the deployment group, because rollback is then a listener flip measured in seconds. And I'd drain gracefully rather than killing a task mid-payment, and avoid deploying across settlement cutoffs."