An NTP sync failure on a subset of auth-service hosts drifted their clocks by four minutes, enough to make every token they issued look already expired downstream.

Starting at 06:30 on April 9th, roughly 15% of API requests began failing auth with 401 Unauthorized, even though users had valid sessions. The tokens were valid — the clock on the host that issued them wasn't.

Timeline

06:00:00 — an NTP daemon restart on a subset of auth-service hosts (7 of 40) fails silently after a config change disabled a fallback NTP pool the previous week. 06:00–06:30 — those 7 hosts' clocks drift forward by roughly 4 minutes over the following 30 minutes due to normal hardware clock drift with no correction. 06:30:00 — tokens issued by the skewed hosts carry an "issued at" (iat) timestamp 4 minutes in the future relative to the validating services' clocks. 06:31:00 — downstream services, which reject tokens with iat more than 2 minutes in the future as a replay-protection measure, start rejecting them. 06:33:00 — 401 rate crosses 5% and pages on-call. 06:41:00 — responder correlates failing tokens to their issuing host via the token's embedded host ID and finds all 7 skewed hosts implicated. 06:48:00 — skewed hosts pulled from rotation; NTP resynced. 06:55:00 — 401 rate back to baseline.

Root cause

The NTP fallback pool removal was an intentional cleanup that nobody realized also removed the only working NTP source reachable from that specific subnet after a firewall change months earlier — the primary pool had been unreachable from those hosts for a while, silently, with the fallback quietly covering for it.

The fix

We identified the specific bad hosts with:

from logs
| where message == "auth.token_rejected" and reason == "iat_future"
| summarize count() by token.issuer_host
| sort by count desc

Seven hosts accounted for 100% of the rejections. We restored NTP connectivity, and separately added clock-skew monitoring as a first-class metric: every host now reports its offset from a trusted time source, alerting at 30 seconds of drift rather than waiting for a downstream symptom like this one.

What changed

We also loosened the iat-future tolerance from 2 minutes to 5 minutes as a pragmatic buffer, while keeping the underlying clock-drift alert tight — the tolerance change reduces blast radius from future clock issues without weakening the actual anti-replay protection meaningfully.

  • Monitor clock skew directly on every host; don't wait for a downstream auth failure to reveal it.
  • Verify network reachability before removing a "redundant-looking" fallback dependency.
  • Widen tight time-based validation tolerances slightly to reduce blast radius, if security allows it.
  • Embed an issuing-host identifier in tokens — it turns a diffuse 401 wave into a 5-minute root cause.