Retention limits are a fact of any observability platform. A scheduled Parquet export keeps the history you need without keeping it all hot.

Helix keeps traces and logs hot for thirty days by default, long enough for almost every operational question, but not long enough for a year-over-year cost review or a compliance retention requirement. The export pipeline solves that without paying for hot storage on data nobody queries daily.

1. Define the export job

Exports run as a scheduled HelixQL query against a signal, writing Parquet files to an object store target on a cron schedule. Configure it once through the CLI rather than the UI, since the definition is easier to keep in version control that way.

helix export create \
  --name daily-traces-archive \
  --query 'from traces | where service.name in ("checkout-api", "payments-api")' \
  --target s3://acme-telemetry-archive/traces/ \
  --format parquet \
  --schedule "0 2 * * *"

2. Partition by date, not by service

Downstream tools, Athena, BigQuery, Spark, all expect a predictable partition layout to prune scans efficiently. Partitioning by year/month/day keeps a "give me last quarter" query fast; partitioning by service name instead makes every cross-service query a full scan.

s3://acme-telemetry-archive/traces/year=2026/month=02/day=19/part-00001.parquet

3. Query the archive without re-importing it

The point of the export is not to move data somewhere and forget about it, it is to make cold data queryable elsewhere on demand. Point a warehouse engine directly at the partitioned path rather than loading it into a separate database first.

SELECT service_name, approx_percentile(duration_ms, 0.95) AS p95
FROM traces_archive
WHERE year = 2026 AND month = 1
GROUP BY service_name;

4. Alert on export health, not just on data health

An export job that silently stops writing is easy to miss, because nothing in the live Helix dashboards changes when it fails. Set a simple freshness alert on the object store path itself, if no new partition has appeared in twenty-six hours, something upstream broke.

ALERT export_stale
WHEN age(latest_partition("s3://acme-telemetry-archive/traces/")) > 26h
NOTIFY slack:data-platform

Treat the export job itself as a small piece of production infrastructure rather than a background chore. It has an owner, a runbook entry, and a place on the on-call rotation for the team that depends on the archive, the same as any other pipeline that a compliance or finance review quietly depends on twice a year.

  • Define exports through the CLI so schedules and queries live in version control.
  • Partition by date, not by service, so downstream query engines can prune scans.
  • Query the archive in place with a warehouse engine instead of re-importing it.
  • Alert on export freshness separately, a silent export failure will not show up anywhere else.