A new internal analytics dashboard's nested GraphQL query fanned out into thousands of resolver calls per request, and it shared an API pool with paying customers.

On July 8th, our public GraphQL API's p99 latency spiked to 4.2 seconds for about 12 minutes. The cause was an internal analytics dashboard, three days into its rollout, whose main query resolved a deeply nested field for every item in a large list — and it ran on the same API infrastructure as external customers.

Timeline

11:00:00 — an internal analytics dashboard, recently rolled out to the ops team, is opened by several people simultaneously at the start of a planning meeting. 11:00:05 — the dashboard's main query requests `accounts { projects { services { recentIncidents { timeline } } } }` for a view spanning roughly 300 accounts, with no pagination on the nested lists. 11:00:10 — this single logical query fans out into over 40,000 individual resolver calls, because each nested field is resolved independently without batching. 11:00:15 — those resolver calls compete for the same connection pool and worker threads serving external customer API traffic. 11:01:00 — external API p99 latency crosses 2s, then continues climbing as the dashboard query (still running) keeps consuming capacity. 11:03:00 — SLO alert fires. 11:06:00 — responder identifies the query via HelixQL, traced to the analytics dashboard's service account. 11:08:00 — the dashboard's traffic is throttled at the gateway; external API latency begins recovering. 11:12:00 — full recovery.

Root cause

The dashboard's query had no depth or complexity limit applied, and dataloader-style batching wasn't implemented for the nested resolver chain, so what looked like one request to the dashboard's own logs was actually tens of thousands of individual database-touching calls under the hood — all sharing capacity with paying customers with no isolation.

The fix

We identified the query with:

from traces
| where service == "graphql-api" and span.name == "graphql.execute"
| extend resolver_calls = child_span_count(span.name == "graphql.resolve_field")
| where resolver_calls > 1000
| summarize count() by client_id, query.operation_name

One client_id and one operation_name accounted for the entire spike. We added a query complexity limit (rejecting queries estimated to resolve more than 5,000 fields) and implemented dataloader batching for the nested resolver chain, cutting the same dashboard query's actual resolver calls from 40,000 to under 400.

What changed

Internal tools now run through a separate GraphQL gateway instance with its own resource pool, isolated from external customer traffic entirely, so an internal tool's query patterns — however inefficient — can no longer affect paying customers. Query complexity limits are now enforced by default on any new client registration, external or internal.

  • Enforce query complexity/depth limits on every GraphQL client by default, internal tools included.
  • Implement dataloader-style batching for any nested resolver chain before it ships broadly.
  • Isolate internal tooling traffic from customer-facing API infrastructure — don't share pools by default.
  • Trace resolver call counts, not just request counts — a single "request" can hide a massive fan-out.