Rolling out profiling to an existing fleet usually means a second daemon, a second config, and a second thing to keep patched. Ours piggybacks on the agent you already run.

Every dedicated profiler we've deployed at a previous job required its own agent: its own systemd unit, its own upgrade cadence, its own set of firewall rules to punch through. Rolling it out to a few hundred services was a project in itself, separate from actually using the data it produced. We didn't want continuous profiling in Helix to repeat that.

Extending the existing agent instead of shipping a new one

The Helix agent already runs as a sidecar or host daemon for metrics and trace collection on most fleets. Continuous profiling shipped as a capability flag on that same binary, not a new deployment:

agent:
  profiling:
    enabled: true
    sample_rate_hz: 97
    memory: true
    cpu: true

Flipping that flag and restarting the existing agent is the entire rollout. No new binary to distribute, no new port to open, no new upgrade cadence to track separately from the agent you're already patching.

Why 97Hz and not something rounder

Sampling at a prime-ish frequency (97Hz rather than 100Hz) avoids lockstep aliasing with periodic application behavior — garbage collection pauses, cron-triggered work, and request-batching intervals that often land on round-number periods. It's a small detail borrowed from profiler research, but it measurably reduced systematic bias in early internal testing versus a round 100Hz default.

What adoption looked like

Because there was no new agent to roll out, one customer went from zero to profiling their entire 400-service fleet in a single config-management change, same day. Compare that to their prior profiler rollout, which took six weeks of coordinating a new daemon across every team.

The overhead budget we held ourselves to

Under 1% CPU overhead was a hard constraint from day one, not an aspiration, because a profiler that measurably slows down the thing it's profiling is worse than useless — it changes the very behavior you're trying to observe. We rejected two internal prototypes that profiled more comprehensively but crossed that budget, including one that would have captured lock-contention data we genuinely wanted, before landing on the current sampling approach that stays under budget on every workload we've tested it against.

  • Profiling ships as a flag on the existing Helix agent, not a new binary
  • One config change and restart to enable fleet-wide
  • 97Hz default sampling rate to avoid aliasing with periodic app behavior
  • Fleet-wide rollout in a single change instead of a multi-week project