Three weeks of OOM restarts, always between Tuesday and Wednesday, never any other day. The pattern felt insane until we found the batch job hiding inside it.

Three weeks of OOM restarts, always between Tuesday and Wednesday, never any other day. The pattern felt insane until we found the batch job hiding inside it.

A Pattern Too Regular to Be Random

Our report-generation service had been getting OOM-killed roughly once a week for a month, and the on-call rotation had quietly started treating it as background noise — someone would just bump the pod and move on. What finally got attention was noticing all four incidents landed on a Tuesday or the small hours of Wednesday. Memory leaks don't usually know what day it is, which meant something day-specific was triggering it.

Correlating Memory Growth With a Specific Trigger

We pulled continuous profiling data alongside memory metrics for the service, filtered to the weeks with incidents:

from profiles
| where service == "report-generator"
| where profile_type == "heap"
| where timestamp >= "2026-03-10" and timestamp <= "2026-03-11"
| summarize sum(alloc_bytes) by function_name
| sort by sum(alloc_bytes) desc
| limit 10

The top allocator was a CSV export function tied to our weekly customer usage report, which runs every Tuesday at 6am and streams a full month of usage data into memory before writing it out, rather than streaming it to disk incrementally. Under normal load that function's allocations were small and short-lived. But three weeks earlier, a customer had onboarded whose usage volume was roughly forty times our median account, and their weekly export alone was enough to push the pod past its memory limit — not immediately, but over the six-to-eight hours it took the report queue to work through a backlog that had built up because the job was also slower than expected for the same reason.

Why It Took So Long to Notice

The honest answer is that a weekly-cadence problem doesn't feel like a pattern until you've seen it four times, and four weekly occurrences is a full month. We now tag every scheduled job's resource footprint with the job name in span attributes specifically so a query like this doesn't require someone to first notice the calendar coincidence by accident.

  • Day-of-week or date-specific incident patterns almost always point to a scheduled job, not a runtime leak.
  • Tag spans from batch and cron jobs with the job name so profiling queries can isolate them directly.
  • Buffer-then-write patterns for exports scale badly with your biggest customer, not your median one.
  • Don't let a recurring "just bump the pod" workaround absorb a real regression for a month before escalating it.