How do you build incident management (paging, rotations, escalations)?

Incident management is the set of people practices and tools that restore service fast when something breaks. In plain words it is how you detect trouble, wake up the right person, coordinate a response, and keep users informed until things are fully healthy again. The three anchors are paging to reach responders, rotations to guarantee coverage, and escalations to bring in help when the clock is ticking. Mastering this topic signals that you think like a production engineer who can defend availability at scale.

Why It Matters

Every minute of downtime hits user trust and revenue. Well built incident programs cut the time to detect and the time to recover, reduce alert fatigue, and make the next failure easier to handle. In a system design interview, strong candidates describe both the technical pipeline alerts, metrics, runbooks and the human pipeline roles, channels, communications, and learning loop. This blend of process and architecture shows you can lead during pressure.

How It Works Step by Step

Step 1. Define severity and priority Create clear and objective bands so responders instantly know the stakes. A simple map works well. SEV one means total outage or severe data loss. SEV two means a major feature is impaired for many users. SEV three means noticeable degradation. Bind each SEV to response timers, stakeholder updates, and decision rights.

Step 2. Tie alerts to SLOs Alert on user impact rather than every low level metric. Good signals are request success rate, tail latency, and error budget burn rate for the last few minutes. Suppress flapping with short rolling windows and deduplication. Correlate alerts from the same service to one incident to avoid page storms.

Step 3. Design paging and routing rules Map each service to a team and route pages to the current primary on call. Use multiple channels, for example push notification, phone call, and SMS, because phones die and apps get muted. Require acknowledgement within a set timer, for example two minutes for SEV one. If the page is not acknowledged, auto escalate to the secondary, then to the team lead, then to the group duty manager. Keep escalation chains short to reduce confusion.

Step 4. Build healthy rotations Guarantee 24 by 7 coverage with a primary and a secondary for each service. Favor follow the sun coverage across regions to protect sleep. Keep rotations fair through scheduling software, enforce quiet hours where possible, and allow easy swaps. Publish a regular handoff ritual that reviews hot issues, known risk, and pending changes.

Step 5. Create runbooks and safe automation For each alert condition, write a short runbook with exact checks and one or two safe mitigations. Include links to dashboards, logs, and feature flag consoles. Automate the safest actions, for example cache flush after a deploy or traffic shift back after a dependency recovers. Gate risky actions behind two person review.

Step 6. Use an incident command model As soon as a SEV one or SEV two starts, assign explicit roles. Incident commander leads the response and owns decisions. Operations lead drives technical diagnosis. Communications lead updates stakeholders and users. Scribe keeps a live timeline. Keep the command channel clean and push brainstorming into a separate thread.

Step 7. Standardize communication Open a single meeting room and a single chat channel per incident. Pin the tracking ticket and status template at the top. For SEV one, promise an internal update every ten to fifteen minutes and an external update every fifteen to thirty minutes. Each update states the current impact, the next action, and the next update time.

Step 8. Close with a blameless review When impact ends, capture a lean report. Include a direct user impact statement, the timeline of events, the deepest useful root cause, what worked well, and exactly which fixes you will ship with owners and deadlines. Make the write up searchable and share highlights to grow team instincts.

Step 9. Measure and improve Track detection time, acknowledgement time, diagnosis time, and full recovery time. Watch page volume and page per person as a fatigue signal. Validate that alerts are actionable. Retire any alert that did not lead to action three times in a row or that never fires during game day tests.

Real World Example

Consider a global video streaming platform during a new release evening. Error rate spikes for play events in one region. A burn rate alert triggers a SEV two, which pages the media service primary. The primary acknowledges within one minute, declares incident start, and becomes incident commander. A war room and chat channel spin up from a template. The communications lead posts that users in a single region see player failures and that the next update is in ten minutes. Metrics show downstream storage latency is high. The operations lead shifts ten percent of traffic to another region and sees instant improvement. Storage owners join, roll back a risky change, and confirm healthy write latency. Traffic is shifted back in small steps while watching error rate. The incident closes with a twenty minute timeline, a small set of fixes to add a circuit breaker and better pre release canaries, and a follow up to tighten storage SLO alerts.