Picking a Distributed Lock That Won't Let Two Jobs Silently Run at Once

The simplest pattern sets a key in Redis with an expiration, and whichever worker sets it first holds the lock until it releases or the key expires. This works fine for low stakes coordination, but it has a real gap: if the lock holder pauses long enough, for a garbage collection stop or a slow network call, the key can expire while that worker still believes it holds the lock and keeps working.

That gap means a single Redis lock alone isn't safe for anything where a second worker acting concurrently causes real damage, like a financial calculation or a write that isn't idempotent. It's a good fit for lower stakes coordination, like preventing two identical cron triggers from double firing when the cost of an occasional double fire is low.

Redlock extends the single instance pattern across multiple independent Redis nodes and requires a majority to agree before granting a lock, which is meant to tolerate a single node failing. It's a meaningful improvement over one instance, but it doesn't fully solve the pause problem described above; a worker that stalls after acquiring the lock can still have its lock expire while it keeps operating, since Redlock's safety guarantee is about consensus across nodes at acquisition time, not about what happens to a slow holder afterward.

If you use Redlock, pair it with a fencing token: a monotonically increasing number issued with the lock that downstream systems check before accepting a write, so a stale lock holder's write gets rejected even if it never learned its lock had expired.

If your pipeline already writes to Postgres, an advisory lock avoids adding a new coordination dependency entirely. Session level advisory locks release automatically if the holding connection dies, which sidesteps a common failure mode where a crashed worker leaves a lock held forever. The tradeoff is that advisory locks are scoped to a single Postgres instance, so they don't help if your workers are coordinating across separate databases.

They're a strong default for a moderate scale pipeline that's already Postgres centric, since the failure handling comes largely for free from how Postgres manages connection lifecycles, without the added operational surface of running a separate coordination service.

Systems built specifically for distributed consensus, like ZooKeeper or etcd, use ephemeral nodes tied to a session and a heartbeat, so a crashed or partitioned holder loses its lock automatically without relying on a fixed expiration guess. This handles the pause and partition problems more rigorously than Redis based approaches, at the cost of running and operating another stateful service.

This is the right tool when the cost of two workers processing the same partition is genuinely severe, like a payments ledger or an inventory system where a duplicate write corrupts a balance that's expensive to reconcile after the fact. For most data pipeline coordination, it's more infrastructure than the problem calls for.

Before picking a pattern, write down what actually happens if two workers process the same partition at the same moment: does it produce a duplicate row your downstream deduplication already handles, or does it corrupt a running total that nobody notices until a customer complains. Those are very different risk profiles, and the honest answer usually points to a much cheaper solution than the one that feels safest on paper.

For a pipeline processing an event stream with idempotent writes already built in further downstream, a single Redis lock with a fencing token is often genuinely enough, and the extra operational cost of etcd buys safety the system doesn't actually need. Save the heavier tooling for the specific partitions where a duplicate write is expensive, rather than applying the strongest pattern everywhere by default.

Ask these questions before choosing a locking pattern:

• What happens if two workers process the same partition at once: a duplicate row that downstream deduplication absorbs, or a corrupted running total nobody notices?
• Can a paused worker, stalled by a garbage collection stop or a slow network call, keep writing after its lock has already expired?
• Does the pipeline already write to Postgres, so a session level advisory lock can release automatically when a crashed connection dies?
• Do you need coordination across multiple databases, or is a coordination failure severe enough to justify running ZooKeeper or etcd?
• Would a fencing token, checked by the system the lock protects, reject writes from a stale holder that never realized its lock expired?