Toil is easy to complain about and hard to measure. We started logging it like an incident, and the pattern that fell out surprised us.

Everyone on the rotation agreed toil was a problem. Nobody could say how much, or which task was worst, until we started logging it the same way we log incidents.

Log it like you'd log an incident

We added a lightweight toil log: task name, time spent, whether it was automatable. Engineers filled it in during the same handoff ritual where they already summarize the shift, so it added roughly 90 seconds, not a new chore. After a quarter we had enough data to rank toil the same way we rank alert noise.

from toil_log
| where logged_at > ago(90d)
| summarize total_minutes = sum(minutes), occurrences = count() by task
| where automatable == true
| order by total_minutes desc

The top offender was rarely the loudest complaint

The task everyone complained about in Slack — manually rotating a credential — turned out to be 12 minutes a month. The task nobody mentioned — re-running a stuck ETL job by hand — was 6 hours a month across the rotation. Measured data reprioritized the automation backlog completely, and the visible-complaint task stayed manual for another quarter because it genuinely wasn't worth automating yet.

Set a toil budget, not just an error budget

We now cap toil at 10% of on-call time per rotation member per month, tracked from the same log. Crossing it triggers the same kind of review an error budget breach does — not a punishment, a forcing function to pull an automation task off the backlog before the next cycle.

Keep the log honest by making it cheap

The first version of the log asked for a category, a severity, and a free-text description, and adoption was terrible within two weeks — filling it in felt like more toil on top of the toil. Cutting it down to a task name picked from a short list and a rough time estimate is what actually got people to keep using it past the novelty period.

  • Log toil with the same rigor as incidents: task, time, automatability.
  • Rank by total time spent, not by how often it's complained about.
  • Set a toil budget per rotation member and review breaches like SLO breaches.
  • Automate the highest total-time offender first, not the loudest one.
  • Keep the logging mechanism cheap or people will quietly stop doing it.