Create a structured incident response playbook with escalation procedures, communication templates, and post-mortem guidelines.
## CONTEXT The average cost of IT downtime is 5,600 dollars per minute for large enterprises, and the difference between a well-managed incident and a chaotic one often comes down to preparation — organizations with established incident response playbooks resolve critical incidents 60% faster than those without. Yet most teams have no documented response procedures, leading to confusion about roles, delayed communication, repeated mistakes, and post-mortems that identify the same root causes quarter after quarter. A structured incident response playbook transforms crisis management from improvised heroics into a repeatable, improvable process. ## ROLE You are an incident management specialist with 14 years of experience leading response teams through critical outages at high-traffic systems serving millions of users. You established the incident management practice at a platform with 99.99% uptime requirements where you led the response to over 200 production incidents, reducing mean time to recovery from 4 hours to 35 minutes. Your incident framework has been adopted by organizations across fintech, healthcare, and e-commerce, and your blameless post-mortem methodology has been credited with reducing recurring incidents by 70%. You understand that incident management is not just about technical troubleshooting — it is about clear communication, defined roles, and systematic learning. ## RESPONSE GUIDELINES - Define severity levels with specific, measurable criteria — not subjective judgments like "major impact" - Include communication templates that are ready to fill in and send during the stress of an active incident - Design the process for the team's actual size and structure, not an idealized large organization with dedicated SREs - Link every mitigation runbook to a specific monitoring alert so the response starts before the incident escalates - Do NOT create a playbook so complex that it cannot be followed during a high-stress incident — simplicity saves seconds that save customers - Do NOT skip the post-mortem process — the single most valuable outcome of any incident is the systemic improvement that prevents recurrence ## TASK CRITERIA 1. **Severity Classification Matrix** — Define 4 severity levels (SEV1 through SEV4) with specific, measurable criteria for each: user impact percentage, revenue impact, data integrity risk, SLA breach proximity, and reputational risk. Include decision examples that illustrate the boundary between each severity level. 2. **First 15 Minutes Checklist (SEV1)** — Create a minute-by-minute checklist for the critical first quarter hour: acknowledge the alert (minute 0-1), open the incident channel and page the team (minute 1-3), assign incident roles (minute 3-5), begin initial diagnosis with prescribed diagnostic commands (minute 5-10), and send the first stakeholder update (minute 10-15). 3. **Incident Roles and Responsibilities** — Define three core roles with clear boundaries: Incident Commander (decision authority, timeline management, escalation decisions), Communications Lead (stakeholder updates, customer communication, status page management), and Technical Lead (diagnosis, mitigation execution, change coordination). Include a role assignment protocol for different team sizes. 4. **Escalation Path Design** — Create escalation paths for each severity level: who gets notified, through which channels, at what time thresholds, and what information must be included in each escalation. Include templates for escalation messages to engineering leadership, customer success, and executive team. 5. **Internal Communication Templates** — Provide fill-in-the-blank templates for every internal communication: incident declaration message, regular status updates (every 15 minutes for SEV1, every 30 minutes for SEV2), mitigation progress updates, resolution announcement, and post-mortem scheduling notice. 6. **Customer-Facing Communication Templates** — Create templates for external communications: initial acknowledgment (we are aware and investigating), progress update (we have identified the cause and are working on a fix), resolution notice (the issue has been resolved), and follow-up message (what we are doing to prevent recurrence). Match the tone to the brand voice. 7. **Mitigation Runbooks for Common Failures** — For each common failure mode provided, create a specific runbook: symptoms that trigger this runbook, diagnostic commands to confirm the root cause, step-by-step mitigation procedure, verification steps to confirm the fix, and rollback instructions if the mitigation makes things worse. 8. **Post-Mortem Framework** — Design a blameless post-mortem template: incident timeline reconstruction, root cause analysis (using the Five Whys method), contributing factors identification, impact quantification (duration, users affected, revenue impact), action items with owners and due dates, and systemic improvement recommendations. 9. **Incident Learning Process** — Define how incident learnings feed back into the system: action item tracking and completion follow-up, monitoring improvement based on detection gaps, runbook updates based on resolution learnings, architecture changes based on recurring patterns, and quarterly incident review meetings. 10. **On-Call Handoff and Rotation** — Specify the on-call process: rotation schedule and swap procedures, handoff checklist between rotations, on-call documentation access, escalation authority limits, and guidelines for when an on-call engineer should wake up the team vs. handle alone. ## INFORMATION ABOUT ME - My system name: [INSERT SYSTEM NAME — e.g., ShopFlow platform, DataSync service, HealthPortal application] - My team structure: [INSERT TEAM STRUCTURE — e.g., 6 backend engineers, 2 SREs, 1 engineering manager, distributed across US and EU] - My communication channels: [INSERT CHANNELS — e.g., Slack for internal, PagerDuty for alerting, Statuspage.io for customers, Jira for tracking] - My common failure modes: [INSERT FAILURE MODES — e.g., database connection exhaustion, payment gateway timeout, cache invalidation storm, deployment-triggered errors] - My SLA commitments: [INSERT SLA — e.g., 99.9% availability (43 min downtime/month), p95 response under 500ms, 4-hour resolution for SEV1] - My brand communication tone: [INSERT TONE — e.g., professional and transparent, empathetic and direct, technical and factual] ## RESPONSE FORMAT - Open with the severity classification matrix as a structured table with clear criteria and decision examples for each level - Present the SEV1 first-15-minutes checklist as a step-by-step timeline with assigned roles and specific actions - Include all communication templates as fill-in-the-blank formats ready for immediate use during incidents - Provide mitigation runbooks for each common failure mode as self-contained reference documents - Include the post-mortem template with all sections and example entries - Close with implementation recommendations: how to train the team, how to practice with incident drills, and how to measure incident response maturity over time
Or press ⌘C to copy