The Data Pipeline Bug That Only Shows Up After a Retry

The instinctive fix for a flaky pipeline step is to wrap it in a retry loop, and that instinct is exactly backwards if the step itself isn't idempotent. A retry on a step that inserts rows, sends a webhook, or charges a payment will, on a partial failure, redo work that partially succeeded the first time, producing duplicate rows, duplicate webhook deliveries, or a duplicate charge. Adding retries to a non-idempotent pipeline doesn't add resilience, it adds a new, harder-to-reproduce bug that only appears under the exact failure conditions retries are meant to handle.

Say your pipeline processes a batch of orders from an upstream source once an hour. Attach a stable idempotency key to each unit of work, in this case the order ID plus the processing window, and before writing any result, check whether that key has already been recorded as processed. If a run fails halfway through a batch of 10,000 orders and restarts, it reprocesses the same batch, but every order already recorded gets skipped rather than reinserted, and only the genuinely unprocessed remainder does real work. The key insight is that the check-and-record step must be atomic with the write itself, not a separate step that can itself fail out of order.

• Webhook delivery: a retried webhook send after a timeout can double-deliver even if the first attempt actually succeeded on the receiving end; include an idempotency key the receiver can deduplicate on.
• Downstream API calls that mutate state: a payment charge, an inventory decrement, or an email send are the highest-cost places to get this wrong, since the damage is customer-visible.
• Aggregation and rollup jobs: a job that recomputes a daily total from raw events is naturally idempotent if it fully recomputes from source; one that increments a running counter is not, and a retry will double-count.
• Multi-step pipelines where only the last step has a key: if step three has an idempotency key but step one doesn't, a partial failure between one and three still causes step one's side effects to duplicate on retry.

The table or store tracking which keys have already been processed needs its own durability guarantees, since if it's lost or reset, every previously processed key looks unprocessed again and the entire safety mechanism silently stops working. Set a clear retention policy, since keeping every key forever isn't usually necessary, but expiring keys too aggressively can reopen the duplication window for a legitimately delayed retry. Back this store with the same database that has your strongest durability guarantees, not a cache that can be evicted under memory pressure.

The only reliable way to know a pipeline is actually idempotent is to deliberately kill it mid-run in a test environment and rerun it, then check for duplicates in the output. Most teams never do this and only discover a gap in production, during the exact retry event the fix was supposed to protect against. Build this kill-and-rerun test into your CI pipeline for any job that writes to a durable store, and rerun it whenever the pipeline's write logic changes.

A common mistake is testing idempotency only on the happy path of a clean rerun. For example, a team kills a job between two steps, reruns it, sees no duplicate rows in the main table, and declares the pipeline safe. But the same run also sent webhooks and updated a summary counter, and both fired twice. The fix is to widen the assertion: after the rerun, compare every output the job touches, including tables, external calls, and aggregates, against a single clean run. Record the calls made to any mocked downstream service, and fail the test if any idempotency key produces more than one side effect. That turns the kill-and-rerun test from a spot check into a guarantee about the whole write path.