Profiling has been a bolt-on product at most vendors for a decade. We shipped it as a fourth signal in the same store, joinable against the trace that triggered it.

Every profiler we evaluated as a buy decision made the same mistake: it kept CPU and memory samples in a store that had no idea what a trace or a span was. So when a service got slow, you'd find the trace, then tab over to a different tool, squint at a timestamp, and hope the flame graph you found was from the right ten seconds.

Profiling as a signal, not a separate tool

Continuous profiling in Helix runs a low-overhead sampler (under 1% CPU at our default 97Hz) alongside your existing tracing SDK. Profiles land in the same store as metrics, logs, and traces, tagged with the same service, route, and trace_id attributes. That last one is the whole point.

Jumping from a trace straight into a flame graph

Because profiles carry trace_id, you can pivot from a slow span directly to the CPU profile captured during that exact request, no time-range guessing:

from profiles
| where trace_id == "9f21c4a0e7b1"
| top 15 by self_time
| project function, self_time, cumulative_time

One design partner used this to find that a JSON serialization path was burning 34% of CPU time on a route they'd assumed was I/O-bound — a finding that took them eleven minutes instead of the two days it took the last time they chased the same symptom with a standalone profiler and a separate APM.

Memory profiles get the same treatment

The same join works for memory profiles, which matters for a different class of incident: slow leaks that only become visible over hours, not the single-request spikes traces are good at catching. Pairing a memory profile's allocation-site breakdown against a service's restart history in one query turned what used to be a multi-day bisection exercise into something closer to a five-minute lookup for one partner chasing a leak that only showed up after roughly six hours of uptime.

What this cost us in scope, and why it was worth it

Building profiling as a first-class signal instead of an integration meant six months of storage engine work before a single customer saw a flame graph — we had to extend our columnar format to hold stack traces efficiently and teach the query planner a new aggregation shape. We shipped it slower than a wrapper around an open-source profiler would have taken. We think it was the right trade: nobody has to open a third tool during an incident.

  • Continuous CPU and memory profiling, under 1% overhead
  • Profiles queryable in HelixQL alongside metrics, logs, and traces
  • Direct pivot from any trace span to the profile captured during that request
  • No separate agent, no separate retention policy to manage