Runbooks That Actually Get Used During Incidents

What you need to know

• Runbooks must be specific, not general. A runbook for "database issues" is not a runbook; it is a troubleshooting guide. Write one runbook per specific scenario.
• Commands in runbooks must be executable as written. No placeholders, no assumptions about what the user knows to substitute. Copy-paste-ready commands.
• Runbooks decay as systems evolve. Require a runbook review in every incident postmortem and every deployment that changes the covered system.
• The worst time to write a runbook is during an incident. Write them after an incident, during the postmortem, when the incident is fresh.
• Test runbooks in staging periodically. Runbooks that have never been executed are of unknown reliability.

The core argument

The reason most runbooks are not used during incidents is that they are written for the wrong audience at the wrong time. They are written by engineers who deeply understand the system, for engineers who also deeply understand the system, at a time when no one is under pressure. The result is a document that assumes too much: that the person reading it knows which service name maps to which deployment, that they can interpret a partial command and fill in the right values, that they know which escalation path to take.

During an incident at 2am, the engineer on call does not have the mental bandwidth to interpret, clarify, and adapt. They need a procedure that is executable as written. The bar for a runbook that works under stress is higher than the bar for documentation that works under normal conditions. Every step should have an expected outcome so the responder knows if they are on the right path. Every decision point should have explicit branching criteria. Every command should be copy-paste ready.

The operational discipline that makes runbooks reliable is the postmortem. Every incident that either could not be resolved with the existing runbook or revealed a gap in the runbook's coverage results in a runbook update before the incident is closed. This creates a feedback loop: incidents improve runbooks, which improve incident response, which reduces incident severity. Teams that close incidents without updating runbooks accumulate technical debt in their incident response capability.

Common mistakes

1. Writing runbooks at too high a level of abstraction. A runbook step that says "restart the failing service" is not actionable. Which service? What is the restart command? What is the expected behavior after restart? The step should say: kubectl rollout restart deployment/api-server -n production and then "after 2-3 minutes, verify the deployment is healthy with kubectl get pods -n production - all pods should show Running status."

1. Not including expected outputs. A diagnostic command without an expected output leaves the responder unsure whether the command produced the right result. Include the expected output for each diagnostic step: "This command should show 3 running pods. If fewer than 3 are Running, proceed to step 7."

1. Writing runbooks for rarely occurring scenarios before the common ones. Runbooks for catastrophic events (data center failure, security breach) are important, but the common incidents (high memory usage, slow database queries, certificate expiration) are more likely to benefit from runbooks because they occur regularly. Write runbooks in order of incident frequency, not severity.

1. Not having a version number or last-reviewed date on runbooks. An undated runbook is of unknown reliability. Add a "last verified" date and the system version the runbook was verified against to every runbook. Any runbook older than 6 months without a review is suspect.

1. Storing runbooks in a system that is inaccessible during an incident. Runbooks stored in the same system that is experiencing an incident are inaccessible when they are most needed. Store runbooks in a system that is independent of the production infrastructure: a public GitHub repository, a PagerDuty runbook, a dedicated documentation system on a different cloud provider. The production outage should not take down the runbook library.

Where to start

1. After the next incident, write the runbook before closing the postmortem. The freshest and most accurate documentation of how to respond to an incident is written immediately after the incident, while the steps and commands are clear in the responder's mind. Make runbook creation a mandatory postmortem deliverable for incidents that did not have an existing runbook.

1. Identify the five most frequent alerts and write runbooks for each. Pull the last 30 days of PagerDuty or on-call alert history and identify the five alerts that triggered most frequently. For each, write a runbook that covers the 80% case: the most common root cause and its remediation. This provides the highest coverage for the least writing effort.

1. Run a runbook fire drill quarterly. Pick a runbook that has not been executed recently, follow the steps in a staging environment, and record any commands that fail, any steps that are unclear, or any assumptions that are not documented. Update the runbook based on the drill findings. Fire drills reveal runbook decay before an actual incident does.

Related reading

• Pager Fatigue and How to Prevent It
• Observability in 2026: Metrics, Logs, Traces
• Point-in-Time Recovery: A Founder's Insurance Policy
• Job Failure Recovery: How Good SaaS Companies Sleep at Night