Not every toil task is worth automating. We built a simple return-on-effort filter before greenlighting any automation project.

Engineers love automating things. Not every toil task deserves it — some cost more to automate safely than they'll ever save. We needed a filter before approving the backlog.

Compute payback period before writing code

For each candidate task we estimate engineering hours to automate against monthly hours saved, from the toil log, and compute a payback period in months. Anything with a payback under six months gets greenlit by default; anything over eighteen needs an explicit justification beyond time saved, like reducing error rate or removing a single-point-of-failure person.

from toil_log
| where automatable == true
| summarize monthly_minutes = sum(minutes) by task
| extend monthly_hours = monthly_minutes / 60.0
| join (automation_estimates) on task
| extend payback_months = estimated_build_hours / monthly_hours
| order by payback_months asc

Automate the boring 80%, keep a human in the risky 20%

Full automation of a task is often far more effort than partial automation, for a fraction of the benefit. Our credential-rotation toil task didn't need a fully automated pipeline — a script that does the mechanical 80% and hands a human a two-click approval for the risky part cut the time by 90% at a fraction of the build cost of full automation.

Track whether automation actually held up

We revisit automated tasks 90 days after shipping to confirm the toil actually disappeared from the log, rather than assuming the project succeeded because it shipped. A few of our early automations quietly reintroduced manual steps within weeks because an edge case wasn't handled, and we wouldn't have caught it without checking the log again.

Let the person who does the toil pick the filter's inputs

Engineering estimates of build effort tend to run optimistic, and toil-log time estimates tend to run conservative, so we have the responder who actually performs the task review both numbers before a payback calculation gets approved. Letting the estimate come only from whoever's excited to build the automation reliably produced payback periods that looked better on paper than they turned out to be in practice.

  • Compute a payback period in months before approving any automation project.
  • Prefer partial automation with a human checkpoint over full automation of risky steps.
  • Justify automations with an 18+ month payback on non-time grounds explicitly.
  • Have the person doing the toil review both the effort and time-saved estimates.
  • Re-check the toil log 90 days post-launch to confirm the win actually held.