A JVM heap-tuning change meant to reduce memory footprint instead triggered longer, more frequent garbage-collection pauses under real production load.

On March 18th we shipped a change to trace-ingest-service's JVM flags, lowering the max heap size to reduce memory cost. In staging it looked fine. In production, under real cardinality, p99 latency doubled from 210ms to 460ms within an hour of rollout.

Timeline

13:00:00 — the new JVM flags (-Xmx reduced from 8g to 5g, intended to save cost across 40 pods) roll out fleet-wide over 20 minutes. 13:20:00 — p99 latency begins climbing steadily rather than settling; unlike a typical bad deploy, there's no sharp step change. 13:45:00 — p99 crosses 400ms, tripping the SLO burn-rate alert. 13:47:00 — on-call pulls GC metrics and sees pause frequency up 4x and average pause duration up from 12ms to 55ms. 13:52:00 — correlated with the flag change via the deploy timeline. 14:05:00 — rollback initiated. 14:22:00 — p99 back to 215ms as the old heap size pods finish replacing the new ones.

Root cause

Staging traffic has much lower cardinality than production trace-ingest workloads, so the smaller heap never hit sustained memory pressure in testing. In production, the reduced heap meant the garbage collector was running full collections roughly every 8 seconds instead of every 40, and each collection paused the world for longer because there was less headroom to collect incrementally.

The fix

We correlated GC behavior directly with request latency using:

from metrics
| where metric.name == "jvm.gc.pause_ms" and service == "trace-ingest-service"
| summarize p95_pause=p95(value), pause_count=count() by bin(time, 1m)
| join (from traces | where service == "trace-ingest-service" | summarize p99=p99(duration) by bin(time, 1m)) on time

The pause-count and p99 lines tracked almost perfectly. We reverted the heap size and instead achieved the memory savings a different way: tuning the G1GC region size and enabling string deduplication, which cut memory 15% without touching pause behavior.

What changed

We now load-test JVM flag changes against a replayed sample of real production trace cardinality, not synthetic staging traffic, specifically because this class of regression only shows up under realistic memory pressure. GC pause metrics are now a first-class panel on every JVM service's dashboard, not something we have to dig for during an incident.

  • Replay real production traffic patterns, not synthetic load, when testing memory/GC-sensitive changes.
  • Make GC pause metrics visible by default on JVM service dashboards, not just available on request.
  • Prefer collector tuning (region size, string dedup) over raw heap cuts when optimizing for memory cost.
  • A latency regression that climbs gradually rather than stepping is a strong signal to check GC first.