# On-Call, Runbooks and Postmortems — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-incident-response

> Describe a healthy on-call process, what a runbook contains, and how blameless postmortems turn incidents into fixes.

## What happens after the page

Observability pays off during an **incident**. A typical flow: the alert pages the **on-call** engineer, who acknowledges it, judges severity and, for big incidents, declares one and opens a dedicated channel. Clear roles help: an **incident commander** coordinates and decides, others investigate, and someone handles communication with users and stakeholders.

The first goal is **mitigation**, not root cause: roll back the release, fail over, shed load, disable a feature flag. Root-cause analysis can wait until users are safe.

A **runbook** linked from each alert says what the alert means, which dashboards and queries to check first, and which safe actions to try. Common health metrics include time to detect (**MTTD**) and time to restore (**MTTR**); treat them as trends, not targets to game.

Afterwards, a **blameless postmortem** records the timeline, impact, contributing factors and follow-up actions with owners. Blameless means asking *how the system allowed the mistake*, not *who made it*; people share details honestly only when they are not punished for them.

## A fire drill, not a blame game

When a kitchen fire happens in a hostel mess, the first job is to put it out, not to find out who left the gas on. Later, a calm review asks why the gas knob was easy to leave on and why the extinguisher was hard to reach, then fixes those. That is a blameless postmortem.

## Make on-call sustainable

Healthy on-call has a rotation large enough that people get real time off, a primary and a secondary, a handover note at each shift change, and a budget for pages per shift. If one rotation pages every night, treat that as an incident in itself and fix the alerts or the system.

**Quiz:** During an outage caused by a bad deployment, what should the on-call engineer usually do first?

- [x] Mitigate impact, for example by rolling back the release
- [ ] Write the postmortem document
- [ ] Find the engineer who wrote the bad code
- [ ] Add more dashboards

*Answer:* Mitigate impact, for example by rolling back the release. Restore service first; investigation and the postmortem follow once users are no longer affected.
