Writing an Incident Runbook People Actually Follow at 2 A.M.

A stressed, tired engineer doesn't need a paragraph of background before they can act, they need to know the first concrete thing to check or do, right now. Put the first three actions at the very top, above any explanation of why the incident might be happening, and push background context and rationale further down for anyone who wants to understand the "why" once the immediate response is underway.

Resist the instinct to explain the system's architecture first out of a wish to be thorough. That instinct serves the writer's understanding of their own document, not the reader's ability to act on it in the first ninety seconds of a page going off, which is exactly the window that matters most.

"Check the database connection pool" requires the reader to remember or look up the exact command, exactly the kind of thing that's hard to recall correctly under stress. "Run this: [exact command]" removes that friction entirely. Every step that involves a command, a query, or a specific URL should be copy-paste ready, not a description the reader has to translate into the actual action themselves.

This matters even more for anyone new to the on-call rotation, who won't yet have the muscle memory of typing the same diagnostic command dozens of times. A command they can paste directly closes the gap between an experienced responder and a newer one at exactly the moment that gap would otherwise cost the most time.

The person on call during a specific incident is often not the engineer who built or best understands the affected system, especially on a team with a rotating on-call schedule. Write the runbook for that person, not for yourself. Spell out where the relevant dashboard actually lives, what a normal value looks like for whatever metric you're asking them to check, and what to do next for each likely outcome, rather than assuming judgment calls the writer would make instinctively but a less familiar responder can't.

A useful trick when drafting: imagine the newest hire on the team, three weeks into the job, having to run this document alone at three in the morning. If a step only makes sense to someone who's already lived through this exact failure before, rewrite it until it doesn't require that history.

For example, a step that reads check the queue backlog can become a step that gives the exact command, the dashboard link, what a normal value looks like, and what to do if the value is high or low. A newer responder can then act without asking anyone, and an experienced one saves time. A useful decision rule: if a step depends on knowledge that lives only in someone's head, rewrite it until a person three weeks into the job could run it alone.

The single best test of a runbook isn't a read-through by the person who wrote it, it's watching someone who's never touched this particular system try to follow it during a scheduled drill, timed, with no help or hints offered from you at all. Every single place they hesitate, ask a question, or do something different than intended is a real, fixable gap in the document itself, not a gap in their competence. Fix those gaps and re-test periodically, since a runbook that was accurate six months ago may reference a dashboard that has since moved or a command that's since changed.

• Put the first three actions at the top, before any background explanation
• Make every command and query copy-paste ready, with the exact syntax included
• Write for someone without deep context on this specific system, not for yourself
• Test with an unfamiliar responder running it cold, timed, and fix every point of hesitation