A read replica fell nine minutes behind primary during a bulk import, and read-after-write consistency broke for any user who used two devices in a row.

On May 8th, a subset of users reported seeing "missing" settings changes — updates that had definitely saved, according to support, but weren't showing up. The cause was a read replica that had silently fallen nine minutes behind primary during an unrelated bulk data import.

Timeline

12:00:00 — a scheduled bulk import job begins writing roughly 800,000 rows to the primary database as part of a customer data migration. 12:01:00 — write-ahead log volume from the import saturates replication bandwidth to one of two read replicas; that replica's lag begins climbing. 12:01–12:14 — lag grows from under 1 second to 9 minutes, while application code continues routing all read traffic to replicas round-robin, with no lag-awareness. 12:15:00 — a handful of users who write a settings change and immediately reload (routed to the lagging replica) see stale data; several open support tickets over the next hour. 13:20:00 — support escalates a pattern of "changes not saving" tickets to engineering. 13:35:00 — engineering identifies replica lag as the cause via HelixQL, confirms the import job as the trigger. 13:40:00 — the lagging replica is pulled from the read pool until it catches up. 13:58:00 — replica catches up and rejoins the pool.

Root cause

Our read routing had no lag-awareness — a replica 9 minutes behind was treated identically to one fully caught up. Read-after-write flows (like "save settings, then reload the settings page") are exactly the pattern that breaks under any meaningful replica lag, and we had no read-your-writes guarantee in place for them.

The fix

The lag was visible the entire time, just not connected to the symptom until we looked:

from metrics
| where metric.name == "postgres.replica.lag_seconds"
| summarize max(value) by replica_id, bin(time, 1m)
| where value > 30

We added lag-aware routing: any replica with lag over 5 seconds is automatically pulled from the read pool, and read-after-write-sensitive endpoints (settings, profile) now pin to the primary for a short window after a write from the same session.

What changed

Bulk import jobs are now rate-limited specifically to keep replication lag under a budget, verified against a live lag metric during the job rather than assumed safe from a one-time test. We also added a lag-based automatic pool eviction so this entire class of bug can't recur even if a future job saturates replication bandwidth again.

  • Make read routing lag-aware; automatically evict replicas past a lag threshold from the pool.
  • Pin read-after-write-sensitive endpoints to primary for a short post-write window.
  • Rate-limit bulk writes against a live replication-lag budget, not a fixed batch size.
  • Treat "changes not saving" support patterns as a potential consistency bug, not just a UX complaint.