Living With Replication Lag Instead of Pretending It Doesn't Exist

A replica applies changes from the primary's write ahead log asynchronously, and under normal healthy operation, that application happens within a small fraction of a second. Lag becomes a real problem when it grows well beyond that baseline, usually because the replica is under heavier read load than it can sustain while also keeping up with replication, or because a long running query is blocking the replay of incoming changes.

Monitor replication lag as its own explicit metric, not just replica CPU or query latency, since a replica can look perfectly healthy on every other dashboard while quietly falling further and further behind on replication specifically. A lag metric that's trending upward over days, even if it's still within an acceptable range today, is worth investigating before it becomes a customer facing problem.

Not every read has the same freshness requirement, and treating them all identically usually means either overloading the primary unnecessarily or serving stale data somewhere it actually matters. A dashboard showing historical trends can comfortably read from a replica that's a few seconds behind; a confirmation screen showing the write a user just made cannot, and that specific read needs to go to the primary or to a replica you've confirmed has caught up.

Build this routing logic explicitly into your data access layer rather than leaving it to individual engineers to remember case by case. A default that routes to the primary unless a read is explicitly marked as replica safe is a more conservative and generally safer starting point than the reverse.

For the specific case of a user reading back their own recent write, a few patterns avoid both the staleness bug and the cost of routing every read to the primary. One is tracking the primary's log sequence number at write time and having the client wait for a replica to reach at least that position before serving the read, which guarantees freshness without a blanket primary only policy. A simpler, less precise option is routing a user's own reads to the primary for a short window right after they write, then falling back to replicas afterward.

Whichever pattern you choose, test it under real replication lag, not just under the low lag conditions a local development environment usually has. A pattern that works fine when lag is a few milliseconds can fail in ways that only show up when lag climbs to a full second or more under real production load.

Common ways to avoid the read your own write bug:

• Track the primary's log sequence number at write time, and have the client wait for a replica to reach that position before the read is served.
• Route a user's own reads to the primary for a short window right after they write, then fall back to replicas.
• Send freshness sensitive reads, such as a confirmation screen, to the primary while dashboards keep reading from replicas.
• Use synchronous replication for data where staleness is unacceptable, accepting the added write latency that comes with it.

Synchronous replication, where a write isn't acknowledged until at least one replica has confirmed it, eliminates the staleness problem entirely for whatever replicas are part of that synchronous set, at the direct cost of added write latency since every write now waits on network round trips to the replica. This tradeoff is worth making for data where staleness is genuinely unacceptable, like a financial transaction record, and usually not worth it for the bulk of ordinary application data where a few hundred milliseconds of eventual consistency is a completely reasonable tradeoff.

Mixing synchronous and asynchronous replicas for different tables or workloads is a valid middle ground, rather than treating replication mode as a single all or nothing setting for the whole database. Reserve the latency cost for the specific data where it's actually earning something in return.