A code review missed an N+1 query hiding behind an ORM helper method. It was invisible until one customer's account, with 40,000 line items, hit it.

On April 16th, one enterprise customer's invoice-history page started timing out. Every other customer's page loaded fine. The difference was 40,000 line items versus a typical customer's few hundred — enough to turn an N+1 query from harmless to catastrophic.

Timeline

10:00:00 — invoice-service v3.12.0 ships a refactor that replaces a bulk SQL join with an ORM helper, `invoice.line_items()`, called inside a loop over each invoice. It passed all tests because test fixtures never exceeded 50 line items. 10:15:00 — a single enterprise customer with 40,000 line items across their invoice history loads the page; it issues roughly 40,000 individual queries and times out after 30 seconds. 10:16:00 — customer support escalates a "page won't load" ticket. 10:24:00 — engineering reproduces in staging by seeding a large fixture, confirms N+1 behavior via query count in the trace. 10:31:00 — a hotfix reintroducing the bulk join ships. 10:44:00 — customer's page loads in 380ms.

Root cause

The refactor was reviewed and tested, but nothing in CI measured query count per request — only correctness and response time against small fixtures. An N+1 pattern is invisible at small N and catastrophic at large N, and our test data never had a large N.

The fix

The trace made the pattern obvious the moment we looked at query count instead of just duration:

from traces
| where service == "invoice-service" and span.name == "http.request"
| extend query_count = child_span_count(span.name == "db.query")
| where query_count > 100
| summarize count() by route, bin(time, 1h)

That query, run against the prior week, would have shown this exact route occasionally spiking to hundreds of queries per request for large accounts — a pattern we'd simply never looked for.

What changed

We added a query-count budget per route as a CI gate — any route exceeding 20 queries per request fails the build, using a synthetic large fixture specifically sized to catch N+1 patterns that small fixtures miss. We also added the query_count > 100 alert above as a standing production check, independent of latency.

  • Test with fixture sizes that match your largest real customers, not your median one.
  • Add a per-request query-count budget to CI — it catches N+1 patterns correctness tests never will.
  • Alert on query count per request in production, not only on response latency.
  • Treat ORM convenience methods called inside loops as a standing code-review red flag.