The error message pointed at checkout. Checkout was the fifth domino, not the first. Here's the trace that walked back to the actual cause.

The error message pointed at checkout. Checkout was the fifth domino, not the first. Here's the trace that walked back to the actual cause.

Starting From the Symptom

7:20am, still on my first coffee, a page for elevated 500s on checkout. The error logs themselves were unhelpful — a generic "upstream request failed" with no further detail, the kind of error that's technically true and diagnostically empty. Rather than guess, I pulled the full trace for a sample of failing requests.

Walking the Trace Back One Hop at a Time

A single trace_id, followed end to end, showed the actual shape of the request across five services:

from traces
| where trace_id == "a13f9c2e88b4"
| project service, span.name, duration, status
| sort by span_start asc

checkout-api called pricing-svc, which called tax-calc, which called an external tax-rate provider, which called back into our own geo-lookup service for address normalization. The failure was five hops deep: geo-lookup was timing out on a specific class of malformed postal codes that a recent client-side form change had started allowing through validation. Every layer above it just saw "upstream failed" and passed that same unhelpful message up the chain, so by the time it reached checkout's logs, all the useful detail had been stripped away.

Fixing the Error, Then Fixing the Message

The immediate fix was a validation patch on the client-side form and a defensive check in geo-lookup to fail fast on malformed input instead of timing out on it. But the more durable fix was to stop every intermediate service from swallowing error context: we now require every service in this chain to propagate the originating error type and service name in the response, not just a generic upstream failure string, specifically so the next version of this incident starts with a five-hop trace already annotated instead of a blank one.

A generic "upstream failed" error is a debt every layer above the real failure inherits and passes along without paying down.
  • A single trace_id followed end to end beats guessing from a generic top-level error message every time.
  • Error-swallowing at intermediate layers turns a five-minute diagnosis into a twenty-minute one.
  • Require services to propagate origin context (type, service name) through failure chains, not just "it failed."
  • Client-side validation changes deserve the same scrutiny as backend changes — they reshape what reaches your services.