You do not have to send telemetry directly from every service. Running a Collector in front of Helix buys batching, retry, and a place to enforce sampling in one spot.

Every Helix SDK can export directly to the platform, and for a single small service that is often the right amount of infrastructure. Past a handful of services, a shared OpenTelemetry Collector in front of Helix earns its keep by centralizing batching, retry, and sampling policy instead of duplicating it in every application.

1. Start with the minimum viable pipeline

A collector configuration is three sections: receivers accept data in, processors transform it, exporters send it out. The smallest useful configuration for Helix needs exactly one of each.

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
    timeout: 5s
exporters:
  otlphttp/helix:
    endpoint: https://ingest.helix.dev
    headers:
      Authorization: "Bearer ${HELIX_API_KEY}"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/helix]

2. Add a memory limiter before anything else

Without a bound on buffered data, a collector under a sudden traffic spike or a temporarily unreachable Helix endpoint will happily consume memory until the process is killed by the OS, taking the telemetry pipeline for every service behind it down at once.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    timeout: 5s

3. Run it as a sidecar in low-traffic environments, a gateway in high-traffic ones

For a handful of services, a collector sidecar per pod is simple and isolates failures per service. Past a few dozen services, a dedicated collector gateway tier, deployed and scaled independently, is easier to operate and gives one place to apply account-wide policy like tail sampling.

4. Monitor the collector itself

The collector exposes its own internal metrics, queue size, dropped spans, export failures, on a standard endpoint. Wire those into Helix too, a collector silently dropping data under load is a worse failure mode than a service that never got instrumented in the first place, because the dashboards look normal right up until someone notices data is missing.

from metrics
| where metric.name == "otelcol_exporter_send_failed_spans"
| summarize sum(value) by bin(timestamp, 5m)
  • Start with a minimal receiver, batch processor, exporter pipeline before adding anything else.
  • Always set a memory limiter, unbounded buffering under load takes down the whole pipeline.
  • Choose sidecar versus gateway deployment based on service count, not habit.
  • Monitor the collector own metrics, a silently dropping collector is worse than no collector.