On-Call Has a New Job Description

Fix it, then write it down

The first step hasn’t changed. Customers are down, so you restore service and everything else waits. Nobody writes documentation in the middle of an outage.

What changes is what happens after. The old job ended when the alert cleared. Someone posted “resolved” in the channel and everybody went back to bed. What you learned during the incident stayed in the head of whoever was awake, plus a scrollback of half-explained commands. Six weeks later the same alert fires on a different engineer’s shift, and they work it out from scratch. The retro probably produced an action item to write a runbook, and that item is sitting in a backlog behind the roadmap.

So on-call needs a new definition of done. The incident closes when service is back, and resolution time keeps measuring what it should. The on-call work closes when the playbook is merged.

I wrote in Let an AI Agent Take the First Page that a runbook is a SKILL.md written for humans, and that exact commands and explicit decision points make it work for agents too. The new part is when it gets written: right after the fix, by the person who did the work, while they still remember which of the things they tried actually mattered. A useful playbook answers four questions:

• What triggers it? The alert name, plus the symptoms that separate this failure from the ones that look like it.
• What do you check? The exact queries and what each result means.
• What do you do? The exact commands, with the decision points written out.
• How do you know it worked, and when do you stop and escalate?

You don’t have to write it from scratch. The incident channel and the shell history hold most of this already, and an agent can turn them into a first draft in minutes. The person who ran the incident edits that draft, because they know which steps were dead ends. Then it goes through a pull request like any other change.

Give it to the agents

The agent can’t use a playbook in a wiki it has no access to. Put it in the repo next to the service it belongs to, in the format your agents already load, and map each alert name to its playbook so the agent finds the right one with a lookup.

A new playbook will sometimes need an action the agent isn’t allowed to take yet. That’s a change to the guardrails, and it gets reviewed like one. The reviewer can see the action, the situation it’s for, and the incident that proved it was needed. As I said about auto-merge rules, changes to the guardrails always go to a human reviewer.

Try the playbook before you trust it. Break the same thing in staging and let the agent work it, or run it in dry-run mode, where the agent posts what it would have done the next time the alert fires and a human does the real work. Either way, run it at least once before it can change anything in production.

Run the agent in a loop

An agent you start by hand still needs someone awake to start it. In a loop, the alert starts the work:

1. The alert fires and the agent picks it up.
2. It looks up the playbook. If there isn’t one, it escalates right away with what it found.
3. It runs the playbook inside its allowlist and posts each action to the incident channel.
4. It checks the signal the playbook names and waits to see that the alert stays clear.
5. It closes the incident, or escalates when its time budget runs out.

Step 4 is the one people skip. An agent reporting that it restarted the pod doesn’t tell you the incident is over. Check whether the error rate is still flat fifteen minutes later. The review loops I run in the background need a stop condition, and so does this one: a metric read from your monitoring system, separate from the agent’s own report.

Every escalation lands on a human, who fixes the incident and writes the playbook. That’s step one again, so the four steps form a loop of their own, and each pass moves one class of incident out of the human queue.

Watch what the loop hides

This loop can also hide problems. An agent that restarts a leaking worker every night at 2 a.m. keeps the dashboard green, and the leak is still there. A person paged three nights in a row would have filed a ticket. With the agent handling the page, nobody files one.

Patching symptoms is how broken projects stay broken, and an agent will repeat the same patch for as long as the alert keeps firing. Track how often each playbook runs and set a limit. If a playbook fires three times in a week, it should open a ticket for a real fix and assign it to the team that owns the service. Once the root cause is gone, delete the playbook.

What the human still does

This is the move I argued for at the merge button in Humans Out of the Loop: earn it with guardrails, then take the person out on purpose, one category at a time. Start with one alert. Run it in dry-run mode, then give it write access, and count how many runs ended in an escalation or made things worse.

The rotation doesn’t go away. Someone still takes the novel failure. As I wrote in From Doers to Directors, someone still decides whether to roll back or fix forward, and what to tell customers.

A good shift used to mean resolving incidents. Now it also means writing the playbooks, so the next person on call doesn’t get paged for the same incidents.

Start with the last incident you fixed and write its playbook this week.