Shipping every log line at high volume gets expensive fast. Head-based percentage sampling is the wrong first move, here is what to do instead.

The instinct when a log bill gets too large is to sample a flat percentage of everything. It works, and it also throws away exactly the log lines an on-call engineer needs three weeks later during an incident, because errors are rare and flat sampling treats them the same as the routine noise around them.

1. Sample by outcome, not by percentage

Configure the log processor to keep one hundred percent of error and warning level lines, and sample only the info and debug volume that dominates the bill. This single change usually cuts ingest volume by more than half while keeping every line anyone would actually search for during an incident.

# helix-collector.yaml
processors:
  log_sampler:
    rules:
      - level: error
        sample_rate: 1.0
      - level: warn
        sample_rate: 1.0
      - level: info
        sample_rate: 0.1
      - level: debug
        sample_rate: 0.01

2. Keep every line from a trace once it is flagged

A second rule matters more than the first: once a trace is marked as an error or exceeds a latency threshold, every log line associated with that trace ID should bypass sampling entirely, regardless of level. Otherwise the one info-level line that would have explained the root cause gets dropped by the same rule that correctly discarded a thousand routine ones.

rules:
  - trace_flag: error
    sample_rate: 1.0
  - trace_flag: high_latency
    sample_rate: 1.0

3. Measure what got dropped, do not just trust the config

Every sampled pipeline should emit its own metric for lines dropped per rule. Query it monthly, a sampling rule that quietly starts dropping ninety-eight percent of a service that used to be quiet but is now noisy is a config drifting out of date, not a working sampler.

from metrics
| where metric.name == "log_sampler.dropped_total"
| summarize sum(value) by rule, bin(timestamp, 1d)
  • Sample by log level and outcome, keep all errors and warnings at full rate.
  • Bypass sampling entirely for any log line attached to a flagged error or slow trace.
  • Track dropped-line metrics per rule so sampling drift shows up before an incident does.
  • Next: connect the traces that flag these logs by joining logs and traces in one query.