The Message Queue Decision That Determines Your Failure Modes

A queue typically guarantees a message gets processed roughly once, by roughly one consumer, then it's gone. A stream keeps events around and lets multiple independent consumers each read through them at their own pace, replaying from any point if needed. Picking based on familiarity rather than which guarantee your actual use case needs is the root cause of a lot of downstream pain: a queue used where replay is needed, or a stream used where exactly-once processing matters, both create real problems later.

Both queues and streams can, and eventually will, deliver a message more than once, whether from a retry after a timeout, a consumer crash before acknowledging, or a rebalance. Design every consumer to handle duplicate delivery safely from day one, not as a fix applied after the first incident where a duplicate charge or a duplicate email reveals the gap. A consumer that isn't idempotent is a production incident waiting for the right retry to trigger it. A simple, reliable pattern: generate a unique key for each logical operation, store it alongside the result once processed, and check that store before doing the work again on redelivery. It's a small amount of extra code per consumer, and it's far cheaper to write once upfront than to retrofit after a duplicate has already reached a customer.

Make a consumer safe against duplicate delivery like this:

1. Generate a unique key for each logical operation when it is first created.
2. Store that key alongside the result once the operation has been processed.
3. Check the store for the key before doing any work whenever a message arrives again.
4. Skip the work and acknowledge the message if the key is already there.

A message that a consumer can't process, malformed data, a downstream dependency that's down, needs somewhere to go besides silently disappearing or blocking every message behind it in the same queue. A dead letter queue with actual alerting on it, not just a place messages quietly pile up, is what turns an invisible failure into one someone notices and fixes. Test the dead letter path deliberately: send a message you know will fail and confirm it surfaces somewhere a human will see it.

Some systems guarantee order within a single partition or key but not globally, which is fine until a consumer's logic quietly assumes global order and breaks the first time two related events land on different partitions. Document explicitly what ordering your system actually guarantees, at whatever granularity it guarantees it, and audit consumer logic against that documented guarantee rather than against what engineers assumed when they wrote it.

For example, if two events about the same order land on different partitions, a consumer may see the cancellation before the creation. A useful habit is to write down, next to each consumer, the ordering assumption its logic makes and the guarantee the system actually provides at that granularity. When the two don't match, either key the events so related ones share a partition or make the consumer tolerate out of order arrival. Catching that mismatch in a design review is far cheaper than finding it after a corrupted state reaches production.

How much message backlog or reprocessing time is acceptable during an outage depends on the actual cost of delay. A 99.95% uptime target leaves roughly 4.38 hours a year of downtime budget across every incident combined¹, and a messaging outage that takes an hour to detect and another two to drain the backlog can consume most of that budget in one event. Write the acceptable backlog drain time down as an explicit target, not something discovered during the first real outage.

Say a payment confirmation consumer wasn't built to handle duplicate delivery, on the assumption the queue guaranteed exactly-once processing. During a network blip, a handful of messages got redelivered after an ack was lost in transit, and a small number of customers were charged twice. The fix, an idempotency key checked before processing, took an afternoon. Finding it took a support escalation and a manual refund process first, which is the more common order these bugs actually get discovered in.