A new customer's onboarding backfill hammered a single S3 prefix hard enough to trigger request throttling that spilled over into unrelated ingest traffic.

On June 10th, unrelated customers' trace ingest started failing intermittently with S3 write errors. The cause was a single large customer's onboarding backfill, writing to a prefix structure that concentrated far more request load per second than S3's per-prefix scaling could absorb smoothly.

Timeline

19:00:00 — a scheduled backfill job for a newly onboarded customer begins writing three months of historical trace data, roughly 40 million objects, into the ingest bucket. 19:02:00 — the backfill's key naming scheme uses a shared date-based prefix that collides with live ingest traffic's own prefix pattern, concentrating request rate on a narrow key range beyond what that range had scaled to handle. 19:04:00 — S3 begins returning 503 SlowDown responses for a subset of requests in that prefix, affecting both the backfill and live ingest traffic sharing it. 19:06:00 — ingest error rate across multiple unrelated customers crosses 3%, paging on-call. 19:11:00 — responder identifies the shared prefix collision via request logs and pauses the backfill job. 19:13:00 — S3 request rate for the prefix returns to normal; live ingest recovers within 2 minutes as its own retry queue drains.

Root cause

Our object key scheme used a coarse date-based prefix (`ingest/2026-06-10/...`) shared by all customers and all job types, rather than a scheme designed to spread request load, such as a hash prefix. A single high-volume backfill was enough to exceed the request-rate scaling S3 provides for that narrow key range.

The fix

We confirmed the throttling source with:

from logs
| where message == "s3.request_throttled"
| summarize count() by key_prefix, bin(time, 1m)
| sort by count desc

One prefix accounted for effectively all throttled requests. We changed the key scheme to prepend a hash prefix distributed across a fixed number of buckets, which spreads request load evenly regardless of any single job's volume, and moved bulk backfill jobs to a rate-limited writer that paces itself against a target requests-per-second ceiling.

What changed

Bulk backfill and migration jobs now run through a shared, rate-limited ingestion path specifically so they can never exceed a ceiling that could affect live traffic, regardless of how the operator sizes the job. We also added S3 throttle-rate as a monitored metric with its own alert, rather than discovering it only through downstream ingest errors.

  • Design object key schemes for request-rate distribution, not just logical organization.
  • Route all bulk/backfill jobs through a rate-limited path that can't exceed a safe ceiling.
  • Monitor storage-layer throttle rates directly, not just the application errors they eventually cause.
  • Treat "large customer onboarding" backfills as capacity-planning events, not routine background jobs.