पाठ 28 / 35

On-Call, Runbooks and Postmortems

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Describe a healthy on-call process, what a runbook contains, and how blameless postmortems turn incidents into fixes.

लेखक: · 15 साल से वेब डेवलपर

What happens after the page

Observability pays off during an incident. A typical flow: the alert pages the on-call engineer, who acknowledges it, judges severity and, for big incidents, declares one and opens a dedicated channel. Clear roles help: an incident commander coordinates and decides, others investigate, and someone handles communication with users and stakeholders.

The first goal is mitigation, not root cause: roll back the release, fail over, shed load, disable a feature flag. Root-cause analysis can wait until users are safe.

A runbook linked from each alert says what the alert means, which dashboards and queries to check first, and which safe actions to try. Common health metrics include time to detect (MTTD) and time to restore (MTTR); treat them as trends, not targets to game.

Afterwards, a blameless postmortem records the timeline, impact, contributing factors and follow-up actions with owners. Blameless means asking how the system allowed the mistake, not who made it; people share details honestly only when they are not punished for them.

A fire drill, not a blame game

When a kitchen fire happens in a hostel mess, the first job is to put it out, not to find out who left the gas on. Later, a calm review asks why the gas knob was easy to leave on and why the extinguisher was hard to reach, then fixes those. That is a blameless postmortem.

Make on-call sustainable

Healthy on-call has a rotation large enough that people get real time off, a primary and a secondary, a handover note at each shift change, and a budget for pages per shift. If one rotation pages every night, treat that as an incident in itself and fix the alerts or the system.

त्वरित जाँच: During an outage caused by a bad deployment, what should the on-call engineer usually do first?

  • Mitigate impact, for example by rolling back the release
  • Write the postmortem document
  • Find the engineer who wrote the bad code
  • Add more dashboards
Answer

Mitigate impact, for example by rolling back the release — Restore service first; investigation and the postmortem follow once users are no longer affected.