The dashboard everyone relies on was built by someone who left last year. Nobody knows how it works, and everyone is afraid to touch it. This is the natural end state of click-built dashboards.
The case against clicking
A hand-built dashboard has no history, no review, and no reproducibility. When it breaks you cannot diff it against a working version, because there is no version. When you want the same panel on another service you rebuild it by hand and the two slowly diverge.
What "as code" buys you
- Review — a dashboard change goes through the same pull request as the code it monitors.
- Reuse — a well-defined panel becomes a template you apply across every service.
- Recovery — a broken dashboard is one
git revertaway from working again.
Keep the definition close to the service
We store each dashboard next to the code it observes, so the person changing an endpoint sees the panel that watches it. A query embedded in the repo is far more likely to be updated than one buried in a UI:
panel "checkout p95":
query: |
traces
| where service == "checkout"
| stats p95 = percentile(duration, 95) by route
viz: timeseries
alert: p95 > 800ms for 5m
Generated, not hand-tuned
The final step is templating. One definition, applied per service, produces a hundred consistent dashboards. When the standard changes, you change it once and re-render everything. Boring, reproducible, and exactly what you want from a dashboard.