The page said checkout latency had spiked to nine minutes. It hadn't. The clock had.

The page said checkout latency had spiked to nine minutes. It hadn't — the clock had. This is the story of the least satisfying kind of incident: the one where the system was fine the whole time.

A Page That Didn't Make Sense

It was 3:12am and the alert read p99(checkout.duration) > 5s for 5m. I opened the dashboard expecting a stack trace or a deploy to roll back. Instead I saw a single host reporting request durations in the nine-minute range, every request, on every route, starting at exactly 3:09am. No error rate change. No CPU spike. No memory pressure. Just duration numbers that made no physical sense for a service that answers in 80ms on a bad day.

Chasing a Nine-Minute Request That Took 340ms

I pulled the raw spans for the worst offender and joined them against the host's system logs:

from traces
| where service == "checkout-api" and host == "chk-07"
| where duration > 5s
| join logs on trace_id
| project trace_id, span_start, span_end, duration, log.message
| limit 20

The span start and end timestamps were nine minutes apart, but the log lines sandwiched between them — request in, response out — were separated by 340ms of wall-clock log timestamps. The trace's duration field was lying because the exporter on that host had NTP drift: its local clock had jumped backward eight minutes during a VM live-migration, then the span-end timestamp was recorded against the corrected clock. The math produced an apparent nine-minute span for a request that actually completed in a third of a second.

The Fix Nobody Wants to Ship

The real fix wasn't in checkout-api at all. It was two things: pin NTP sync to a tighter drift tolerance on the hypervisor's live-migration path, and add a sanity clamp in our ingest pipeline that flags any span where duration disagrees with span_end - span_start by more than a few seconds, tagging it clock_skew_suspect instead of feeding it into the alerting aggregate. It took an afternoon. The alert had cost us forty-five minutes of adrenaline for nothing.

Every observability system trusts its own clocks implicitly. The day you get burned by that is the day you start defending against it.

We now run a background HelixQL check every fifteen minutes that looks for exactly this signature across the fleet, so the next skew event gets caught before it pages anyone.

  • Duration fields computed from wall-clock timestamps are only as trustworthy as the clock that produced them.
  • Live-migration events are a real, recurring source of clock jumps — watch for them specifically.
  • Tag suspect spans instead of silently dropping them; you want the evidence trail for the next weird page.
  • A five-minute NTP drift audit is cheaper than a 3am page that goes nowhere.