Parsing every log line into fields at write time is expensive but fast to query. Parsing lazily is cheap but slow. Helix does both, depending on the field.

A log line arrives as raw text or JSON. Somewhere between ingest and query, it needs to become structured fields you can filter and aggregate on. Do that work at write time and every query is instant but ingest gets slower and storage grows. Defer it and ingest stays cheap but every query pays a parsing tax. Helix splits the difference by field, not by log line.

Eager parsing for declared fields

When a log source has a registered schema (a JSON log with declared fields, or a regex/grok pattern attached to the pipeline), the fields listed in that schema are extracted and indexed at write time — typically level, service, trace_id, and a handful of business fields like customer_id. This costs measurable CPU: eager parsing adds about 1.4ms per 1,000 lines on our reference hardware.

Lazy parsing for everything else

Free-text message bodies and undeclared fields are stored as raw text with a tokenized full-text index for substring search, but not decomposed into structured columns unless a query asks for it. When a query references a field that wasn't eagerly extracted, HelixQL falls back to a regex or JSON-path extraction applied only to the rows that survive earlier filters — not the whole scan.

from logs
| where service == 'checkout-api' and level == 'error'   # eager fields, indexed
| extend order_total = parse_json(message).amount          # lazy, applied post-filter
| where order_total > 500
| summarize count() by bin(timestamp, 5m)

Measured tradeoff

On a 90th-percentile log query in our benchmark suite, lazy-parsed field access on the post-filter set (typically a few thousand rows after `service`/`level` narrowing) adds 8-15ms. Eagerly parsing that same field for every ingested line, across a service producing 12,000 lines/sec, would have added roughly 17ms of ingest latency per batch and around 30% more storage for a field most queries never touch.

Promoting a field from lazy to eager

If a lazy field gets queried often enough — we track per-field query frequency and flag candidates automatically — it's a one-line schema change to promote it to eager extraction. Teams do this for fields like `order_total` once they realize they're filtering on it in every incident.

  • Declared schema fields are parsed and indexed at write time; everything else is lazy.
  • Lazy field access costs 8-15ms applied only to the post-filter row set, not the full scan.
  • Eager parsing of an unnecessary field would cost roughly 30% more storage and 17ms of extra ingest latency for a 12k lines/sec service.