Storage compaction would have saved customers more money on paper. We shipped tail-based sampling first anyway, and the ordering was deliberate.

Two projects competed for the same engineering quarter last year: a storage compaction rewrite that our own modeling said would cut retained-data costs by roughly 30%, and tail-based sampling, which would cut ingested volume before it was ever stored. Compaction had the bigger, cleaner number. We shipped sampling first.

The number that changed our mind

Compaction saves money on data you've already decided to keep. Sampling changes what you decide to keep in the first place. Modeling both against real customer ingest patterns, we found the median customer's real opportunity wasn't compressing 100% of their traces better — it was never storing the 96% of traces that were successful, fast, and uninteresting in the first place.

from ingest_usage
| where signal == "traces"
| summarize total_gb=sum(size_gb), error_gb=sumif(size_gb, has_error)
| project total_gb, error_gb, error_pct = error_gb / total_gb * 100.0
// typical result: error_pct around 3-5%

Sequencing matters because compaction is now easier

Shipping sampling first also made the eventual compaction project smaller: less raw data landing in storage means a simpler compaction problem to solve later, with less legacy data to migrate under the old format. We're shipping the compaction rewrite this year, on top of a storage volume that's already 40-60% smaller thanks to sampling, so its real-dollar impact is now larger relative to engineering effort than the original model predicted.

The honest tradeoff

Customers who wanted better compression for compliance-driven full-retention workloads waited longer than we'd have liked for compaction. That was a real cost of the sequencing decision, and a few of them told us so directly. We think the ordering was still right for the majority, but it wasn't free.

How we're making it up to that segment

Compliance-driven customers who waited get first access to the compaction rewrite in a private beta before general release, and we credited affected accounts a portion of their storage spend for the quarter the gap was most acute. It's a small gesture relative to the wait, but sequencing decisions have real costs for real customers and we didn't want the retrospective to read as purely a win without acknowledging who absorbed the downside of it.

  • Sampling shipped first because it reduces what's stored, not just how it's stored
  • Median trace error/interesting-traffic share: 3-5% of total volume
  • Compaction project simplified by shipping on top of a smaller data footprint
  • Full-retention, compliance-driven workloads paid the real cost of this ordering