With thousands of spans closing every second, most exemplar candidates have to be rejected. The selection algorithm decides which ones are actually useful later.

Attaching a trace_id to a histogram bucket is cheap per-event, but deciding which of thousands of candidate spans deserves that slot is where the real design work is. A naive "first one wins" or "random sample" strategy tends to attach boring, unhelpful exemplars — a request that happened to close first, not one that tells you anything.

The scoring function

Each candidate span is scored on a combination of factors: how extreme its duration is relative to the bucket's recent distribution, whether its status is an error, and whether it's already the exemplar for a different, less-populated bucket (to avoid over-concentrating exemplars on one unusually chatty request). The highest-scoring candidate per bucket per 15-second window wins the slot.

scoring:
  weight_duration_percentile: 0.5
  weight_error_status: 0.4
  weight_novelty: 0.1
  replace_if_score_delta > 0.15   # avoid needless churn for marginal improvements

Why we don't always replace on a better score

Early versions replaced the exemplar on any strictly higher score, which caused a problem: exemplars would churn every few seconds as marginally-more-extreme spans arrived, making a dashboard panel's "view trace" link point somewhere different every time someone refreshed. We added a replacement threshold — a new candidate must score at least 0.15 higher (on a normalized 0-1 scale) to displace the current exemplar — which cut exemplar churn by roughly 70% while keeping genuinely more useful traces winning out.

Error-status weighting

Error spans get a strong scoring boost (weight 0.4 of the total) because an on-call engineer investigating a spike almost always wants an error example over a merely-slow one, even in a latency-focused histogram. We validated this weighting against actual incident retro data: in the 90 days before this weighting shipped, engineers manually searched for an error trace instead of using the attached exemplar in about 40% of latency-related incidents; after, that dropped to roughly 11%.

What this means for dashboard trust

The practical outcome is that clicking an exemplar link on a spike almost always lands on a trace that's actually representative of what went wrong, not an arbitrary one — which is the entire value proposition of exemplars in the first place. We keep tuning the weights quarterly against fresh incident retro data rather than treating the scoring function as fixed, since traffic patterns and error signatures shift enough across two or three quarters that a weighting tuned for one season of traffic can quietly go stale for the next.