Incident-Driven Design and Runbooks

1. Writing It Before You Need It

A runbook is written at one of two moments, and the difference decides whether it is any good.

Before the incident, by the person who built the thing, while they still remember which failure modes are real and which are theoretical. It takes an hour and it is boring.

During the incident, by whoever is awake, reconstructing from a dashboard they have not looked at in six months. It takes the length of the outage and it is written in the worst available conditions.

The second is what usually happens, and the reason is structural rather than lazy