Runbooks: Writing Down What to Do at 3 A.M.
Your app is live. Customers are signing up. Revenue is growing. Then one night your phone buzzes: the database is down, and you have no idea what to do. You fumble through Slack history, search old emails, try to remember what your developer said three months ago. By the time you piece it together, users have noticed—and some have already left.
This happens constantly to founders without documented procedures for handling outages. The problem is simple: when something breaks at 3 a.m., panic and sleep deprivation make even straightforward fixes feel impossible. The question becomes urgent: how do you ensure that anyone—including a groggy, half-awake version of yourself—can restore your app quickly when disaster strikes?
A runbook is a step-by-step document that tells whoever is on call exactly what to do when a specific problem occurs.
What a Runbook Is and Why You Need One
A runbook is a written set of instructions for handling a specific operational scenario. Think of it as a recipe for fixing problems. Just as a recipe tells you exactly which ingredients to use and in what order, a runbook tells whoever is responding to an incident exactly which steps to take and in what sequence. The person following the runbook doesn't need to understand why each step works—they just need to follow the instructions.
For non-technical founders, runbooks solve a critical dependency problem. Without documentation, all the knowledge about how to fix your app lives inside your developer's head. If that developer is asleep, on vacation, or has moved on to another job, you're stuck. A runbook transfers that knowledge onto paper—or into a shared document—where anyone can access it.
The financial stakes are real. According to IBM's Cost of a Data Breach 2024 report, the global average cost of a data breach reached $4.88 million, with business disruptions and customer support costs driving much of that increase. Downtime erodes customer trust, delays revenue, and forces you into reactive firefighting instead of building your business. For a startup, even a few hours of unplanned outage can mean lost sales, damaged reputation, and churned customers who never come back.
How to Write a Runbook: Four Essential Practices
Writing an effective runbook requires discipline. Your goal is to produce a document so clear that someone with no prior context can follow it successfully under pressure.
Identify Your Most Likely Failures First
Before writing anything, identify the problems most likely to occur. Ask your developer to list the five issues they've fixed most often. Common candidates include database connection failures, server memory exhaustion, third-party API outages, expired SSL certificates, and broken deployments. Those recurring problems become your first five runbooks. Start where the risk is highest.
Make Every Step a Single Concrete Action
Every instruction in a runbook should describe one action that produces one observable result. Vague instructions like "check the server" force the reader to guess. Precise instructions like "Open the terminal, type ssh user@your-server, press Enter, and confirm you see the welcome message" leave nothing to interpretation.
Include the exact commands to copy and paste. Include the exact URLs to visit. Include the exact buttons to click. Assume the reader has never seen your infrastructure before and is operating on four hours of sleep.
Add Verification After Every Action
After every action, tell the reader how to confirm it worked. If the instruction is to restart the database, the next line should explain what a successful restart looks like: "Wait 30 seconds, then run systemctl status postgresql. You should see 'active (running)' in green text." Verification steps prevent the reader from moving forward when something has gone wrong, which avoids making the problem worse.
Define When to Stop and Call for Help
Some problems can't be fixed by following a runbook. When that happens, the reader needs to know exactly whom to contact and how. Specify the conditions that warrant escalation: "If the database still shows errors after step 5, stop here and call [Developer Name] at [phone number]." Never leave the reader stranded without a next step.
The Four Components Every Runbook Must Include
A complete runbook contains four distinct sections, each serving a specific purpose during an incident.
A Descriptive Title and Trigger Condition
The title should name the specific problem the runbook addresses. "Database Connection Errors" beats "Database Issues." Below the title, describe the trigger condition—the alert, symptom, or situation that tells someone to use this runbook. For example: "Use this runbook when you receive a 'PostgreSQL connection refused' alert from the monitoring system, or when users report seeing 'Error 500' pages."
Prerequisites and Access Requirements
List everything the reader needs before they can begin. This includes login credentials, required software, VPN access, and any permissions they must have. Store sensitive credentials in a password manager and reference them by name rather than writing them directly in the runbook. For example: "You will need the 'Production Server SSH Key' from the team password vault."
Numbered Steps with Expected Outcomes
The core of the runbook is a numbered list of actions. Each step should include the action to take and the expected result. Keep steps short. If a step requires more than two or three sentences to explain, break it into multiple steps. Number every step so the reader can easily report their progress or ask for help at a specific point.
Escalation Contacts and Resolution Criteria
End every runbook with clear guidance on two scenarios: what to do if the steps don't resolve the problem, and how to confirm the incident is actually over. Include primary and backup contacts with phone numbers. Specify any follow-up actions required, such as notifying customers or filing a post-incident report.
How to Store and Maintain Your Runbooks Over Time
A runbook that no one can find—or that contains outdated commands—is worse than no runbook at all. Maintenance isn't optional.
Store Everything in One Accessible Location
Runbooks are useless if no one can find them during an emergency. Store all runbooks in one place that everyone on your team can access quickly—a shared document folder, a team wiki, or a dedicated documentation tool. The key requirements: the location is known to everyone, accessible without special permissions during an outage, and searchable. Avoid storing runbooks only on your production server; if the server is down, you can't access the instructions for bringing it back up.
Update Immediately After Every Incident
Runbooks decay like bread left on the counter. The commands that worked six months ago may not work after your developer updates the infrastructure. After every incident—whether the runbook helped or not—schedule a brief review. Ask three questions: Did a runbook exist for this problem? Did the steps work as written? What should we add or change? Update the runbook immediately while the incident is fresh in everyone's memory.
Test Before You Need Them
The worst time to discover that a runbook is incomplete is during an actual outage. Periodically, have someone unfamiliar with the system attempt to follow a runbook in a test environment. If they get stuck, the runbook needs more detail. If they complete it successfully, you have confidence that it will work when it matters.
Assign an Owner to Each Runbook
Every runbook should have an owner—a specific person responsible for keeping it accurate. This doesn't mean that person must write or execute the runbook alone. It means they're accountable for reviewing it regularly and updating it when the system changes. Without clear ownership, runbooks become orphaned documents that no one maintains.
Your Next Step
Start today by asking your developer to identify the three most common problems your app experiences. For each problem, have them write a draft runbook using the structure above: title and trigger condition, prerequisites, numbered steps with expected outcomes, and escalation contacts with resolution criteria. Store these drafts in a shared location your entire team can access.
Schedule a 30-minute meeting within the next two weeks to review the drafts together. Walk through each runbook step by step and ask whether someone unfamiliar with the system could follow it at 3 a.m. Revise any steps that assume too much knowledge.
Finally, set a recurring calendar reminder—monthly or quarterly—to review and update your runbooks. Systems change, team members change, and runbooks must change with them.
The small investment you make now will pay off the next time your phone buzzes in the middle of the night. Instead of panic, you'll feel something rare in those moments: calm. You'll know exactly what to do.