Why Your Ingestion Pipeline Needs to Survive Being Rerun

Treating a pipeline rerun as a rare edge case is the root of most idempotency bugs, because in practice reruns happen constantly: a scheduler retry after a transient failure, a manual backfill after a bug fix, an at-least-once message queue redelivering a message it isn't sure was processed. Design for reruns as the normal case, not a special one, and a large category of data quality incidents simply stops happening.

Instead of blindly inserting every record a pipeline processes, write to the destination using a stable key, whether that's a natural identifier from the source system or one generated deterministically from the record's contents, and upsert rather than insert. Running the same batch twice then produces the same final state instead of duplicate rows, because the second run's writes simply overwrite the first run's writes for the same key.

A pipeline that increments a running total each time it processes a batch will double-count on a rerun, since the increment happens again. Recompute the aggregate from the underlying records instead of accumulating an increment on top of the last known value, so a rerun produces the same total regardless of how many times it's been triggered. This costs more compute than an incremental update, and that cost is almost always worth paying for the correctness guarantee.

For pipelines where recomputing from scratch on every run isn't practical, track exactly what's been processed, down to the specific batch or offset, in a durable store the pipeline checks before doing any work. On a rerun, the pipeline can then skip anything already marked complete and pick up cleanly from where it actually left off, rather than guessing based on a timestamp that might not reflect what was truly written to the destination. Mark a batch complete only after its write is confirmed durable, not before, so a crash between the write and the marker still leaves the pipeline able to safely redo that one batch.

A pipeline can get the database write exactly right, using upserts and stable keys, while still triggering a side effect like a customer notification email on every run. The database ends up correct. The customer still gets three copies of the same email because the side effect itself was never made idempotent. Any action a pipeline takes outside its own data store, not just the write to it, needs the same rerun-safe treatment, whether that means checking a sent-status flag before firing a notification or routing the notification itself through a deduplicating queue.

Make a pipeline safe to rerun with these patterns:

• Write to the destination with a stable key, natural or deterministically generated, and upsert rather than insert.
• Recompute aggregates from the underlying records instead of adding an increment onto the last known total.
• Track processed batches or offsets in a durable store that the pipeline checks before doing any work.
• Make side effects outside the data store, such as customer emails, idempotent too, not only the database writes.
• Test by running the same batch through twice and asserting the final state matches the state after one run.

Say an analytics pipeline recalculates yesterday's revenue by summing new transaction records and adding that sum onto an existing running total in a dashboard table. An engineer reruns the job to backfill a day that failed silently, and the sum gets added a second time onto a total that already included it, quietly doubling reported revenue for that day. Nobody notices until finance reconciles at month end. Recomputing the total from the full set of underlying transactions each run, rather than adding to a stored running number, would have made the exact same backfill completely safe to rerun.