R9 — Messaging: queues, pub/sub, Kafka and SQS — keeping events ordered and processed once

R4 made sure the record of an event survives; this section is about the pipe it travels through. It is built in four parts. First the ideas every messaging system shares: why you send a message instead of making a call, the two shapes — a queue for one worker, a topic for every subscriber — and what happens to a message once it has been read. Then Kafka, from the parts of a cluster to the settings that decide whether events stay ordered and take effect once. Then Amazon SQS, given the same treatment. And finally the two side by side, with the questions that decide between them.

Messaging fundamentals

Why send a message instead of making a call

Start from the situation. A customer submits a payment. The API could do everything inline — validate, debit, post to the ledger, email the customer, update the dashboard — and make the customer wait for all of it. Or it could record the intent, reply "accepted", and let the rest happen in the background. The second is asynchronous communication, and it buys four things: throughput (the request thread is freed at once), scalability (background work scales separately from the API), fault tolerance (a slow or dead downstream no longer fails the request) and loose coupling (the API stops knowing who does the rest). That last one is inversion of control: the sender stops calling, and the receivers start listening.

There are two ways to hand the work off.

Message queue: one message, one consumer

A message queue is point-to-point. A producer puts a message on the queue; exactly one consumer takes each message. If you run five workers on the same queue, they compete, and each message goes to whichever worker is free.

That shape gives you three things:

• Decoupling. The producer knows the queue, not the worker. You can redeploy, rewrite or replace the worker without touching the producer.
• Load leveling. A burst of a thousand messages does not hit the worker as a thousand concurrent calls. It sits in the queue, and the competing consumers drain it at the rate they can sustain. Add workers to drain faster.
• Reliability. If every consumer is down, the messages wait. Nothing is lost because nobody was listening.

The payments case: the payment service needs a payment executed against a rail, so it puts ExecutePayment on a queue read by the payment mediator. One worker must do it, not three. That is a command in R0's sense — "please do this", addressed to one service, which can refuse or fail — and a queue is its natural transport.

Who owns the format. On a queue, the consumer usually defines the message format. The worker is the one with a contract to fulfil ("send me these fields and I will execute the payment"), and producers conform to it, the way callers conform to an API.

Why not just a database table? You can mimic a queue with a jobs table: insert a row, have workers poll it, mark rows done. It works, and at low volume it is fine. It is heavier than a queue: every message is a durable row with indexes and write-ahead logging, load leveling is your problem (row locking, SELECT ... FOR UPDATE SKIP LOCKED, claiming and timing out), and every poller holds database connections. The outbox in R4 is exactly a table used as a queue for one narrow purpose — surviving the gap between commit and publish — and then handing off to a real broker.

Pub/sub: one event, every subscriber

Pub/sub inverts the relationship. A publisher writes an event — a fact about a state change, "payment submitted" — to a topic. Any number of subscribers each receive their own copy and react independently. The publisher does not know or care who is listening.

The payments case: the payment service publishes PaymentSubmitted. The ledger posts entries, notifications emails the customer, the read model updates its dashboard row. Tomorrow fraud analytics subscribes, and the payment service changes nothing. That is R0's test for an event: if nobody subscribed, publishing it would still be correct, because it is already true.

The shapes this supports:

Who owns the format. Here the publisher defines the format, because it is announcing a fact about its own domain, and it cannot negotiate with subscribers it does not know exist. That is why event schemas need versioning discipline: a breaking change hits consumers you never met.

The limits of pub/sub

• Hard to test end to end. The flow is spread across subscribers that run independently; no single call stack shows the whole thing.
• Eventual consistency only. The ledger may have posted while the read model still shows the old state. Each subscriber converges on its own schedule.
• Poor subscriber visibility. The publisher cannot see who consumed, who failed, or who is lagging. You need per-subscriber monitoring to know.
• No ordering between subscribers. Notifications may email the customer before the ledger has posted. If one subscriber's work must follow another's, that is a chain of events, not a fan-out.
• No global ordering. Even within one subscriber, order is only promised per key — which is the partition story in the Kafka part below.

Before reading further, play with it. Send a few messages through the queue and watch each one land on exactly one consumer; then publish to the topic and watch every subscriber get its own copy. Notice that adding a consumer to the queue splits the work, while adding a subscriber to the topic duplicates it.

Side by side

What happens after a message is read: queues and logs

Who a message is for is the first split — the one interviewers usually ask first, because it decides the shape of the design before any broker is named. The second, just as important, is what happens to a message after it is read — and most confusion about Kafka comes from imagining it as a queue when it is really a log.

A queue (SQS, RabbitMQ) is a to-do pile. A consumer takes a message, works on it, and when it acknowledges, the message is deleted. The queue's job is handing out work exactly once-ish, and it shrinks as work gets done.

A log (Kafka) is an append-only notebook. Producers write lines at the end; the lines are never handed out and never removed on read. A consumer just remembers which line number it has read up to. Ten consumers can read the same line independently, and a consumer can go back and re-read from line 400 tomorrow.

That one difference explains almost everything else:

Concepts every broker shares

Whatever the product, the same handful of ideas decide how a messaging system behaves. Learn them once here; the Kafka and SQS parts show how each product implements them.

Dead-letter queues, briefly. Every system needs somewhere for messages that keep failing. Two rules hold whichever broker you use — the DLQ needs an alarm on depth, because a silent DLQ is just a slower way of losing data, and it needs a documented way to replay from it once the bug is fixed.

And one rule follows from at-least-once delivery on every broker: the consumer must be idempotent — deduplicate on a business event ID in the same transaction as the effect (R3, R4's inbox). The broker can reduce duplicates; it can never remove them.

Message brokers

A message broker is the server that sits in the middle and does more than store bytes: it validates, routes, stores, and gives guaranteed delivery — it holds a message until a consumer has acknowledged it.

RabbitMQ is the classic example, and its routing model is worth knowing because it shows both shapes in one product. Producers publish to an exchange, not a queue, and the exchange decides which queues get a copy:

Under the hood, pub/sub in RabbitMQ is a queue per subscriber fed by an exchange — the same shape as SNS to SQS.

A broker deliberately stays a "dumb pipe": routing by queue, exchange or key, while transformations and business decisions stay in the services. An enterprise service bus puts routing rules, format transformation and orchestration in the middleware itself. Which of the two should hold your integration logic is an architectural choice, and R1's section on architecture styles compares them.

MQTT is the lightweight end of the family: a pub/sub protocol over TCP designed for IoT devices and unreliable networks, with small headers and a broker routing messages between publishers and subscribers. Use it for sensors, parking lots, traffic lights — not for a payment backbone. One design note carries over: a single broker is a single point of failure, so production MQTT deployments distribute the broker, exactly as Kafka replicates partitions across brokers.

Decide the shape before the product: a command for one worker goes on a queue, a fact for anyone who cares goes on a topic. Then pick the broker whose delivery and retention guarantees match — which is the comparison the next sections make in detail.

Where Kafka and SQS sit. Kafka gives you pub/sub naturally, because each consumer group keeps its own offset on the same log, and gives you a work queue within a group, because each partition goes to exactly one member. SQS is a queue; to get pub/sub you put SNS in front and fan out to one SQS queue per subscriber. The rest of this section is about those two in production.

Kafka

Kafka is a distributed, replicated, append-only log. Before the tuning knobs, the parts and the mental model — because most confusion about Kafka comes from imagining it as a queue when it is really a log.

The components of a Kafka cluster

How Kafka works

Writing: keys, partitions and ordering

Ordering is per partition. Kafka guarantees offset order within one partition and nothing across partitions. A partition is consumed by exactly one consumer in a group at a time. So the key you choose is the ordering design: key by account ID and all of that account's events are ordered; key by payment ID and only that payment's events are.

A broker never chooses the partition — the producer does. The producer's partitioner hashes the key (murmur2(key) % partitions) on the client, looks up which broker leads that partition in the metadata it already holds, and sends the record straight to that leader. A record with no key goes to whichever partition the sticky partitioner is currently filling.

Why ordering is per partition, in one picture. Six messages for accounts A and B, keyed by account, two partitions:

partition 0:  A1  A2  A3        ← consumer 1 reads these in order
partition 1:  B1  B2  B3        ← consumer 2 reads these in order

A's events are strictly ordered. B's events are strictly ordered. But nothing anywhere orders A2 against B1, because they are in different logs being read by different processes at different speeds. If you need those two ordered relative to each other, they must share a key — and that means they share a partition, which means they share a consumer, which caps your throughput. Ordering and parallelism are the same dial turned in opposite directions, and picking the key is how you set it.

Replication and durability

The vocabulary becomes concrete once you see where things physically live. A cluster is a set of brokers. Creating a topic means choosing two numbers: how many partitions it has, and its replication factor, the number of copies of each partition. For every partition the cluster's controller picks one copy as the leader, which takes all writes and by default all reads, and places the followers on other brokers. It spreads leaders across brokers so that no single broker carries most of the traffic, and it never puts two copies of one partition on the same broker, because one failure would then lose both.

What happens next decides durability:

• The leader appends the record to the partition at the next offset.
• Followers pull from the leader, exactly as a consumer would. Followers that are caught up form the in-sync replica set (ISR).
• With acks=all, the leader acknowledges the producer only when every replica in the ISR has the record. The leader's high watermark marks that point, and consumers can read only up to it, so nobody ever reads a record that a leader failure could still lose.
• If the leader's broker dies, the controller promotes an in-sync follower. Every acknowledged record is already there.

Durability on the broker. acks=all is only as strong as min.insync.replicas. With replication factor 3 and min.insync.replicas=2 you survive one broker loss and still refuse writes when two are down — refusing is correct for money. With min.insync.replicas=1 you have acks=all in name only.

Reading: consumer groups and offsets

On the reading side, one rule decides everything: within a consumer group, each partition is assigned to exactly one consumer.

What "committing an offset" really means. The consumer is saying "I have finished everything up to line 812." It is a bookmark, not an acknowledgement of an individual message. Two consequences juniors get bitten by: committing before the work finishes means a crash silently skips messages, and processing messages out of order within a batch means a commit can cover work that never completed.

Consumer lag is defined per consumer group, per partition. Because every group commits its own offsets, one partition read by three groups has three independent lags:

lag(group, partition) = log-end offset of the partition − committed offset of that group on that partition

There is no such thing as "the lag of a partition" or "the lag of a topic" — only how far one group is behind on one partition. "What to monitor" below covers how it is measured and alerted on.

Step through the cluster below. The first tab builds it: five brokers, three topics of three partitions each, replication factor 2, with leaders solid and followers dashed. The second follows one producer's events from key to partition to replica, then kills a broker. The third adds two consumer groups to T1, and the fourth changes the number of consumers in a group.

Producer settings that decide correctness

enable.idempotence=true            # dedupes producer retries per partition
acks=all                           # wait for in-sync replicas
max.in.flight.requests.per.connection=5   # safe only with idempotence on
retries=Integer.MAX_VALUE
delivery.timeout.ms=120000           # the real bound on retries (default: 2 minutes)

Without idempotence, a retry after a lost ack writes the message twice and silently reorders it. This is the most common real-world source of duplicate payment events.

What producer idempotence covers, and what it does not. Producers write to the leader of a partition on the brokers; the cluster stores each partition as a replicated, append-only log; consumers read from that log and track their own offsets. Producers publish through the Kafka client library, which retries a send until the broker acknowledges it or the delivery timeout expires, at which point the error goes back to the application. With idempotence enabled, the client tags each batch with a producer ID and a per-partition sequence number, so when it retries a batch the broker already wrote, the broker drops the duplicate. That protection covers only the client's own retries. Duplicates created by the application — including a resend after a timed-out send — or by a producer restart, redeliveries after a consumer crash or rebalance, and deliberate replays all reach the consumer. So consumers deduplicate on a business event ID carried in the payload: in the same transaction as the effect when the effect is a database write, and as the idempotency key when it is a call to another system.

Delivery timeout: the bound on every retry. With retries effectively unlimited, delivery.timeout.ms is the setting that actually decides how long the producer keeps trying. It is the upper bound on the time between send() returning and the callback reporting success or failure, and it covers everything in between: waiting in the batch buffer (linger.ms), waiting for a free in-flight slot, every individual request (request.timeout.ms, 30 seconds by default), and every retry with its backoff. The client rejects a value smaller than linger.ms + request.timeout.ms.

Two neighbouring settings are easy to confuse with it:

A delivery timeout is an unknown outcome, not a failure. The callback gets a TimeoutException, but the record may well have been written: the last ack could have been lost on the way back. The application must not report the payment as failed. If it simply resends, a record that was in fact written is now in the log twice, with a new sequence number that idempotence will not catch — exactly the application-level duplicate described above. With a transactional producer, a delivery timeout aborts the current transaction.

How to size it:

Setting the delivery timeout for a payment producer

• Make it comfortably longer than a leader failover or a rolling broker restart, so routine cluster events never reach the application.
• Keep it inside the caller's own deadline, so the producer gives up before the upstream request does and nobody is left waiting on a send that will never be reported.
• Publish from an outbox (R4): when a send times out, the row simply stays unsent and the relay tries again later, so a timeout costs latency, never correctness.
• Treat a timeout as unknown: the retry carries the same business event ID, and consumers deduplicate on it.
• Alert on producer timeouts and record errors; a steady trickle usually means an unhealthy broker or ISR before anything else shows it.

Transactions: exactly-once inside Kafka

Transactions and EOS. A producer with a transactional.id can write messages to several partitions and commit consumer offsets atomically, giving exactly-once within Kafka for read-process-write: wrap consume-transform-produce in a transaction and commit the consumer offsets with sendOffsetsToTransaction, so offsets and output move together. Readers must set isolation.level=read_committed. It works well for stream processing between topics. Two limits to name: it does not extend to your database or to an external rail, and it costs latency. For a payment that leaves the building, keep at-least-once plus idempotency at the boundary and skip the complexity.

Consuming safely

// Spring Kafka — the shape of a safe consumer
@KafkaListener(topics = "payments.submitted", groupId = "ledger")
public void handle(ConsumerRecord<String, PaymentEvent> record, Acknowledgment ack) {
    // 1. dedupe + work, in ONE transaction  (R4's inbox)
    boolean fresh = inbox.recordIfNew(record.value().eventId(), "ledger");
    if (fresh) {
        ledger.post(record.value());
    }
    // 2. only now move the bookmark
    ack.acknowledge();
}

Three rules hide in those eight lines, and they are the ones worth repeating in an interview: manual acknowledgement (enable.auto.commit=false), dedupe and work in the same transaction, and acknowledge last. Everything else in this section is tuning around that shape.

Consumer settings and the commit discipline. Turn off auto-commit. Process the batch, then commit — that gives at-least-once, which is the correct default for payments because the alternative loses messages. Keep max.poll.records small enough that a batch finishes well inside max.poll.interval.ms; the usual production incident is slow processing triggering a rebalance, which triggers reprocessing, which makes processing slower.

Rebalancing

Rebalance. Triggered by a consumer joining, leaving, crashing (missed heartbeat), or exceeding max.poll.interval.ms. Under the classic eager protocol, everyone stops and all partitions are revoked; cooperative sticky assignment moves only the partitions that must move. During the pause, in-flight work may be reprocessed by the new owner — which is why consumers must be idempotent and why long processing inside poll() causes a rebalance storm.

Key configs: session.timeout.ms (failure detection), heartbeat.interval.ms (roughly a third of it), max.poll.interval.ms (the real cause of surprise rebalances when processing is slow).

A rebalance is the group agreeing on a new assignment. One broker acts as the group's coordinator: it tracks members by their heartbeats and bumps a generation number on every rebalance, so a consumer still on an old generation can no longer commit offsets. When membership changes, every member rejoins, one member is chosen as group leader and computes the assignment, and the coordinator distributes it.

Two things make this expensive:

• Under the eager protocol, every consumer revokes every partition first, so the whole group stops reading for the duration, even for partitions that end up with the same owner. The cooperative-sticky assignor revokes only the partitions that must move, and the rest of the group keeps processing.
• A crashed consumer is noticed only when its heartbeats stop, after session.timeout.ms. Until then its partitions are owned by a dead process and nobody reads them. A clean shutdown sends LeaveGroup and hands over at once.

The rebalance storm is a feedback loop. A slow dependency stretches a batch past max.poll.interval.ms, so the consumer is evicted. Its partitions move to consumers that must reprocess the uncommitted batch on top of their own work, so they miss the deadline too. Each eviction and each rejoin is another rebalance, the group spends its time rebalancing instead of reading, and lag climbs. Rolling deploys cause the same churn: every pod that leaves and every pod that joins is a rebalance.

Step through the walkthrough below. It shows a consumer joining under the eager protocol and then under the cooperative one, a graceful leave against a crash, and a storm developing from one slow batch — then what breaks it.

Static membership stops deploy-time churn. Set group.instance.id per pod: a restarting consumer rejoins with its previous assignment instead of triggering a full reshuffle, provided it returns inside session.timeout.ms. Combined with cooperative sticky assignment, this is what makes rolling deploys of a consumer fleet non-disruptive — a direct tie to the zero-downtime section.

Keeping a consumer group stable

• Size max.poll.records so a batch finishes well inside max.poll.interval.ms, or raise the interval to match real processing time.
• Keep slow work off the poll thread: hand records to a worker pool, pause() the partitions while it works, resume() when it is done, and commit only completed offsets.
• Use the cooperative-sticky assignor, so a rebalance moves only the partitions that must move.
• Set a stable group.instance.id per pod (static membership), so a restart inside session.timeout.ms returns the same partitions with no rebalance at all.
• Put timeouts and a circuit breaker on the slow dependency (R14), so it fails fast instead of stretching every batch.
• Commit after processing and commit often, so a moved partition has little to reprocess — and make the consumer idempotent for the rest (R3, R4).
• Alert on rebalance rate as well as lag; it rises first.

Partitions over time

Partition count is a one-way door. You can add partitions, never remove them, and adding breaks key-to-partition stability. Size for peak throughput and for the maximum consumer parallelism you will ever want, since consumers beyond the partition count sit idle. Err high within reason; thousands of partitions per broker have their own cost.

Repartitioning. Adding partitions changes the key-to-partition mapping for new messages while old ones stay put, so per-key ordering breaks across the boundary. The safe path is a new topic with the target partition count, dual-write or mirror, let consumers drain the old, then cut over.

Hot partition. One large corporate account saturates its partition. Options: sub-key into a bounded fan-out (account:0..n) while keeping all of one account's related events together, throttle that producer, or route whale accounts to a dedicated topic. State the trade-off plainly — any fan-out weakens ordering, so it must follow the business grain.

When a message keeps failing

Poison message. In an ordered stream you cannot skip. Park the key: stop consuming that key, move the message to a DLQ with full context, alert, and keep the rest of the partition moving only if ordering allows. Skipping silently is how payments get applied out of order.

Retries without blocking. Blocking retry inside the consumer stalls the partition. The common pattern is tiered retry topics (5s, 1m, 10m) with a final DLQ — but note that moving a message to a retry topic reorders it relative to its key. For payments, that is usually unacceptable, so prefer parking the key: record the key as blocked, stop consuming it, alert, and let the rest of the partition proceed only if your ordering requirement is per-key rather than global.

A dead-letter topic. Kafka has no built-in DLQ, so you build it: catch, publish to payments.submitted.dlq with the original headers plus the error, and move on — or, where ordering matters, park the key as above.

Resetting a consumer group's offsets

Because progress is only a committed offset per group and partition, you can move it: rewind to reprocess after fixing a bug, skip past a poison record, or start a new group from a point in time. The standard tool is kafka-consumer-groups.sh --reset-offsets.

The scope is chosen separately: --topic payments.submitted resets every partition of the topic, --topic payments.submitted:0,2 only partitions 0 and 2, and --all-topics everything the group has committed.

# 1. Preview: rewind group 'ledger' on partitions 0 and 2 to midnight UTC on 1 October
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group ledger \
  --topic payments.submitted:0,2 --reset-offsets \
  --to-datetime 2026-10-01T00:00:00.000 --dry-run

# 2. Apply it — only while the group has no running consumers
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group ledger \
  --topic payments.submitted:0,2 --reset-offsets \
  --to-datetime 2026-10-01T00:00:00.000 --execute

# Skip exactly one poison record on partition 1
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group ledger \
  --topic payments.submitted:1 --reset-offsets --shift-by 1 --execute

The group must be inactive. The tool refuses to reset a group that still has live members, because a running consumer would overwrite the new offsets with its next commit. Stop or scale the consumers to zero, reset, check with --describe, then restart them.

Two other ways to move offsets, for when a script is not enough:

• The Admin API — alterConsumerGroupOffsets in the Java AdminClient — does the same job programmatically, with the same requirement that the group be inactive.
• The consumer itself can seek(), seekToBeginning() or seekToEnd() on partitions it has been assigned (in Spring Kafka, through ConsumerSeekAware), and the new position takes effect from its next commit.

And remember the setting that resets offsets without anyone asking: auto.offset.reset decides where a group starts when it has no committed offset or its offset is out of range — earliest reprocesses everything retained, latest silently skips to new records.

Resetting offsets safely in a payments system

• Always run --dry-run first and record the old and new offsets per partition, so the change can be undone.
• Rewinding means redelivery: it is only safe because consumers deduplicate on a business event ID (R3, R4).
• Skipping forward means data the group will never process: park the skipped records in a DLQ or record their offsets, and reconcile them (R6).
• Reset only the partitions that need it, not the whole topic, and only the affected group — every other group keeps its own offsets.
• Treat a reset as a change with an owner, a ticket and a rollback plan, never as an ad hoc fix.

What to monitor

Measuring consumer lag. Lag belongs to a (consumer group, partition) pair: the partition's log-end offset minus that group's committed offset on it. Strictly, a consumer can read only up to the high watermark, and with isolation.level=read_committed only up to the last stable offset, which excludes records of open transactions; monitoring tools use whichever limit applies.

Example — partition T1·P0 holds records up to offset 1,000:

Same partition, three lags, and a slow fraud group has no effect on ledger.

Where to read it:

• kafka-consumer-groups.sh --describe --group fraud prints the current offset, log-end offset and lag for each partition of one group.
• Exporters and tools — Kafka Lag Exporter, Burrow, Datadog, Confluent Control Center — compute the same figure for every group and partition.
• The consumer client's own metric, records-lag-max, uses the consumer's in-memory position, not its committed offset, so it can read lower than broker-side lag while fetched records are still uncommitted.

Lag beyond retention is data loss for that group. If a group falls further behind than the topic's retention, the records it has not read are deleted. Its next fetch is out of range, and auto.offset.reset decides whether it jumps to the earliest remaining record or to the latest — silently skipping data either way.

Alerting on lag

• Alert per group and per partition, never on a topic-wide sum — one stuck partition in one group disappears in an aggregate.
• Prefer lag in time over lag in messages: 10,000 messages is nothing at 50,000 per second and hours at 2 per second. Convert offset lag into seconds behind using record timestamps or the produce rate, and set the payments SLO as "processed within N seconds".
• Keep alerting on groups with no live consumers: their committed offsets remain, so a group whose consumers all died shows steadily growing lag — exactly the alert you want.
• Alert well before lag in time approaches the topic's retention.

What to monitor. The thresholds below are rules of thumb to start from; tune them to your own baseline and SLOs.

Rising rebalance frequency is an early warning for almost everything else, and lag must always be read per consumer group and per partition — one lagging partition is invisible in an average.

Problems seen in production

Most Kafka incidents reported publicly are not exotic. They are the mechanisms above failing in a handful of recurring ways.

Read more: Cloudflare — automatic restarts for unhealthy Kafka consumers, Datadog — lessons learned from running Kafka, KIP-186 and KIP-211.

Amazon SQS

SQS is AWS's fully managed message queue. There is no cluster to run, size or patch: you create a queue, send to it and receive from it.

Why SQS is a queue

Everything about SQS follows from one design decision: a message is work for one consumer, and acknowledging it deletes it.

• A received message is leased, not removed: it becomes invisible to other consumers for a while. If the consumer finishes, it deletes the message and it is gone for good; if not, the message reappears for someone else.
• Consumers compete. Every worker polling a queue draws from the same pile, so adding workers splits the work — there are no consumer groups and no per-consumer positions.
• The queue shrinks as work gets done. Nothing keeps a history, so nothing can be replayed, and a new consumer sees only messages sent after it starts.

That is the to-do pile from the fundamentals, exactly. To give several services their own copy, you put SNS in front and give each its own queue.

The parts of SQS

How SQS works

A message's life is a lease, not a read:

1. Send. The producer sends; SQS stores the message redundantly across availability zones.
2. Receive. A consumer receives it; the message becomes invisible for the visibility timeout, and the receive count goes up by one.
3. Delete. The consumer finishes and deletes it with the receipt handle. The message is gone.
4. Or it comes back. If the consumer crashes, or is still working when the timeout expires, the message becomes visible again and another consumer receives it — a duplicate delivery.
5. Or it parks. Once the receive count passes maxReceiveCount, SQS moves the message to the dead-letter queue.

Step through the walkthrough below. The first tab follows messages through a standard queue — send, receive, delete — and then shows the three ways it goes wrong: a crash before delete, a worker slower than its visibility timeout, and a poison message ending in the DLQ. The second shows FIFO message groups keeping order and blocking on a failure. The third builds pub/sub out of SNS and queues, and shows why a late subscriber cannot replay.

Exactly-once effect on SQS

Start from what SQS guarantees. Standard queues are at-least-once with best-effort ordering: duplicates are not an edge case, they are a documented property. FIFO queues give ordering within a message group and deduplication on send. Neither gives you exactly-once processing, because the consumer can complete its side effect and then die before calling DeleteMessage. The redelivery will happen. Your handler must be idempotent regardless of queue type — that is the whole answer in one sentence, and it is the one interviewers are listening for.

What FIFO actually does.

• MessageGroupId is the ordering and parallelism unit, exactly like a Kafka key. Messages in one group are delivered in order and only one batch of a group is in flight at a time; different groups proceed in parallel. Use the account ID for per-account ordering.
• MessageDeduplicationId suppresses duplicate sends within a 5-minute deduplication interval. Content-based deduplication hashes the body instead, which is convenient and wrong whenever two legitimate identical payments can occur — so send an explicit ID.
• The 5-minute window is the catch. A producer retry an hour later is not deduplicated. SQS-side dedup protects against a network retry storm, not against application-level replays.
• High-throughput mode (deduplicationScope=messageGroup, fifoThroughputLimit=perMessageGroupId) raises the quota by moving dedup scope to the group. Check current published quotas before quoting numbers; the trade-off to state is that dedup is then per group rather than queue-wide.

The implementation that actually gives exactly-once effect. Pair the queue with a conditional write in DynamoDB:

PutItem(
  TableName: "idempotency",
  Item: { pk: <business key, e.g. payment_id>, state: "in_progress",
          expiry: now + 24h },
  ConditionExpression: "attribute_not_exists(pk)"
)

A ConditionalCheckFailedException means someone else has it: read the stored record, and if it is complete, return the saved response and delete the message. On success, do the work, update the item to complete with the response, then delete the message. Set a TTL well beyond your visibility timeout and retry window. Key on the business identity (payment ID), not the SQS MessageId, because a resend creates a new MessageId for the same payment. On Lambda, the Powertools idempotency utility implements this pattern rather than you hand-rolling it.

flowchart TD
  S["SendMessage<br/>MessageGroupId = account<br/>MessageDeduplicationId = explicit"] --> Q["FIFO queue<br/>5-minute dedup window on SEND"]
  Q --> RCV["Consumer receives<br/>visibility timeout starts"]
  RCV --> CW["DynamoDB PutItem<br/>ConditionExpression: attribute_not_exists(pk)<br/>pk = payment_id"]
  CW -->|"condition failed"| EX{"Stored state?"}
  EX -->|complete| DEL1["Return stored result, DeleteMessage"]
  EX -->|in_progress| WAIT["Let visibility expire, retry later"]
  CW -->|"write succeeded"| WORK["Do the work"]
  WORK --> UPD["Mark complete + store response"]
  UPD --> DEL2["DeleteMessage"]
  WORK -->|"crash before delete"| RCV

Visibility timeout is the other half. It must exceed your worst-case processing time, or SQS redelivers while you are still working and two workers race — the most common SQS bug in production. For variable work, extend the lease with ChangeMessageVisibility as you go rather than setting one huge timeout, which would delay recovery from genuine crashes. Set maxReceiveCount with a DLQ so a poison message parks after a bounded number of attempts.

Ordering behaviour worth knowing. In FIFO, a message that keeps failing blocks its own message group until it succeeds or hits the DLQ. That is parking-by-default, which is usually what payments want — but it means one bad payment stalls that account, so alert on group-level stalls.

Making the DynamoDB idempotency table CP

The pattern above leans on DynamoDB to say "this payment ID is already taken", so the table's consistency matters. DynamoDB is AP by default but can be configured as CP — CAP classifies operations, not products (R10).

MRSC has a precise price: exactly three Regions (three replicas, or two plus a non-serving witness), higher latency on every write and strongly consistent read, no transactions, and a ReplicatedWriteConflictException — to be retried — when two Regions write the same item at once, instead of a silent last-writer-wins.

Some parts never become strongly consistent. Global secondary indexes are always eventually consistent, and Streams consumers see changes after the table. Never decide "already processed" from a GSI read; use a conditional write or a strongly consistent read on the base table.

For the idempotency record the in-Region tools are what matter: a conditional PutItem on the business key is atomic regardless of read settings. MRSC earns its cost only when consumers in different Regions can process the same payment, and even then homing each account's writes to one Region usually stays the simpler design (R13).

Kafka versus SQS

Both are at-least-once, both need idempotent consumers, and both can keep per-key order. The differences that decide a design are about what happens after a message is read, and who runs the system.

A useful way to decide: is the message a fact others may care about (event), or a job for one worker (command)? Facts belong on a log, jobs belong on a queue. That is R0's distinction showing up as an infrastructure choice.

The deciding difference for payments is replay. SQS is a work queue, not a log: once acknowledged, the message is gone. If you need to rebuild state or re-run a day's events, your source of truth has to be the ledger and the outbox, not the queue — say that explicitly if an interviewer proposes SQS for an event backbone.

What to say in the interview

"Kafka is a log, not a queue, so the key I choose is the ordering design — ordering and parallelism are the same dial turned in opposite directions. I run consumers with manual acknowledgement, dedupe and work in one transaction, and commit the offset last. Exactly-once semantics inside Kafka are real but stop at its edge, so anything leaving the building gets an idempotency key at the boundary instead. On SQS the equivalent is FIFO with a message group for ordering, plus a conditional write in DynamoDB for the effect — the queue never gives you exactly-once processing on its own."