The crash happened at 3 a.m. The cause started on Monday. Leaks are the incidents you could have seen coming for days, if only you had been watching the slope instead of the value.
Leaks live in the derivative
A memory graph sitting at 60% tells you nothing. A memory graph that has been climbing one percent an hour for three days tells you exactly when you will fall over. The signal is not the level, it is the trend.
The usual suspects
- Memory — a cache without an eviction policy, or a slice that only ever grows.
- Goroutines — a launched worker that never exits, quietly accumulating.
- File handles and connections — a client opened in a loop and never closed.
Alert on the slope
The right alert is not a threshold on the value but a threshold on the rate of change over a long window:
metrics
| where __name__ == "goroutines_total"
| stats slope = linreg(value) over 6h by instance
| where slope > 5 # steadily climbing, never draining
| sort slope desc
Confirm with a profile
Once the slope alert fires, a heap or goroutine profile names the culprit directly — the allocation site that keeps growing, the goroutine stack that keeps multiplying. The trend tells you a leak exists; the profile tells you where. Together they turn a 3 a.m. crash into a Tuesday-afternoon fix.