Keeping every log line is expensive and mostly useless. Keeping the right one percent is cheap and mostly what you need. The trick is choosing which one percent.

The instinct to keep every log line comes from a good place and leads to a bad bill. The signal you need at 3 a.m. is a tiny fraction of what you are paying to store.

Head sampling is a blunt instrument

The simplest approach — keep one in every hundred lines — is cheap but throws away exactly the lines you care about, because errors are rare by definition. Uniform sampling optimizes for the boring case.

Sample by value, not by volume

Better sampling is value-aware. Keep everything that matters and thin out everything that does not:

  • Keep 100% of error and warning lines — they are rare and expensive to lose.
  • Keep every line that belongs to a trace you already sampled, so a request's story stays whole.
  • Aggressively sample healthy, high-volume debug chatter down to a few percent.

Express it as a rule

logs
| sample keep=1.0 where level in ("error","warn")
| sample keep=1.0 where trace_sampled == true
| sample keep=0.02 where level == "debug"

Retention is sampling in the time dimension

The same logic applies to how long you keep things. Full-fidelity logs for a week, sampled and aggregated after that. Most investigations happen within hours of the event; almost none require raw debug logs from six months ago.