Why Your Redis Lock Let Two Jobs Run at Once (and How to Fix It)

The lock's TTL was a guess at how long the critical section would take, and the critical section included a network call to a payment processor whose latency the job doesn't control. Most nights the guess held. The one night it didn't, the lock silently expired while the first worker was still mid-charge, and Redis had no way to know the job wasn't actually finished, because a TTL only tracks time, not whether the work inside it is done.

Raising the TTL to sixty seconds buys you some margin until the next slow response from the payment processor pushes past sixty too. The underlying problem is that the critical section's duration depends on an external system you don't control, so any fixed number is a bet, not a guarantee. A longer TTL also has its own cost: if a worker crashes mid-job instead of running long, that customer's lock now stays held for the full sixty seconds before anyone else can process them, which is a worse outcome for a common failure mode in exchange for protection against a rarer one.

The first is to get the slow external call out of the locked section entirely, using a reserve-then-confirm pattern: reserve the charge (write an intent record) inside the lock, release the lock, make the external call, then confirm or roll back the intent based on the result. The lock only needs to be held for the fast, local part of the work. The second is a lock with automatic lease extension: a background renewal (a watchdog) keeps extending the TTL as long as the holding worker is still alive and working, so the lock only expires early if the worker itself has actually died, not just because a call ran long.

Either approach should also carry a fencing token: a monotonically increasing number issued with the lock, checked by the downstream system (here, the charge itself) before it acts. If a worker's lock expired and a second worker took over, the first worker's fencing token is now stale, and the charge system rejects it even if that first worker wakes up (from a pause, a slow GC cycle, or a network partition) still believing it holds the lock.

For lower-stakes coordination that's already happening inside a database transaction, a Postgres advisory lock avoids the whole class of problem: it's tied to the transaction or session lifetime rather than a separately managed TTL, and there's no extra network hop to a lock service that can itself become a point of failure. The tradeoff is scope: it only coordinates work happening against that one Postgres instance, so it doesn't help when the thing you're locking spans multiple services or data stores, which is exactly the case the billing job above was in.

Even a well-tuned TTL and a careful watchdog don't fully close this gap without a fencing token, because a paused worker (a long garbage collection pause, a brief network partition) can wake up still believing it holds a lock that has since been reassigned. The fencing token is what lets the downstream system say no to a stale writer, regardless of what the worker itself believes. Skipping it is the most common reason a team's second attempt at distributed locking still has the same bug as the first, just harder to reproduce, because a shorter TTL or a watchdog narrows the window without closing it.

You can reproduce the original bug on purpose: add an artificial delay to the payment call in a test environment so it reliably exceeds the lock's TTL, then run two workers against the same customer ID and confirm only one charge goes through. That same test, run again after adding lease renewal and a fencing token, is the actual proof the fix works, rather than an absence of errors in production being read as success. A fix that isn't exercised this way tends to get 'confirmed' by a quiet week, which is exactly what the original bug also looked like right up until the payment processor had a slow night.

To reproduce the double-charge bug on purpose, follow these steps:

1. In a test environment, add an artificial delay to the payment call so it reliably runs longer than the lock's TTL.
2. Start two workers against the same customer ID, so the second can take the lock after it expires.
3. Confirm whether one charge or two goes through, and treat a second charge as a failed test.
4. Run the same test again after adding lease renewal and a fencing token, and confirm only one charge succeeds.