A histogram bucket is just a count. An exemplar attaches one real trace_id to it, so 'why is p99 slow' has an answer one click away.
Metrics tell you a bucket has 340 requests over 800ms in the last minute. They don't tell you which requests, or why. Exemplars close that gap by attaching a small number of real trace_ids to each histogram bucket at write time, without turning every metric write into a trace lookup.
The write-time sampling rule
When a span closes, if it's the slowest (or one of the few sampled) requests to land in a given latency bucket within the current 15-second aggregation window, its trace_id and a handful of attributes get attached to that bucket as an exemplar. We cap exemplars at 4 per bucket per window, replacing the lowest-priority one (usually the least extreme duration) if a more interesting trace_id arrives.
from metrics
| where name == 'http.server.duration' and route == '/checkout'
| where bucket_le == '1000ms'
| project bucket_le, count, exemplars
| take 1The result carries a small array like `[{trace_id: 'a93f...', value: 1120ms, timestamp: ...}]` alongside the aggregate count, resolvable directly into a trace view.
Why we don't store all of them
An unsampled service might close 40,000 spans a minute into a single histogram. Storing every trace_id per bucket would roughly double metrics ingest volume for no proportional benefit, since a human debugging a latency spike needs a couple of representative examples, not all 40,000. Our cap of 4 per bucket per window adds about 2.3% storage overhead to a typical histogram-heavy workload — a cost we consider well worth the debugging value.
Retention mismatch and how we handle it
Exemplars point at traces, but trace retention (default 7 days) is usually shorter than metric retention (13 months at full resolution, longer downsampled). An exemplar older than the trace retention window still shows the pinned trace_id and captured attributes, but the "view trace" link degrades gracefully to a message rather than a 404, since the summary data survives even after the underlying spans are gone.
What this means for on-call
The practical effect is that "why is p99 slow" stops being a research project. A dashboard panel showing a duration histogram with exemplars turns one click into a real trace with real spans, cutting the average time-to-first-hypothesis we've measured across incident retros from about 6 minutes to under 90 seconds.