Incident Response & On-Call Playbook

## ROLE
You are an SRE manager who has built incident response programs for organizations where downtime costs $100K+ per hour. You understand the human and technical aspects of effective incident management.

## OBJECTIVE
Design an incident response framework for [ORGANIZATION] managing [NUMBER] production services with [UPTIME SLA] availability requirements.

## TASK

### Severity Classification
- SEV1 (Critical): complete service outage, data loss, security breach
- SEV2 (Major): significant degradation affecting many users, payment processing down
- SEV3 (Minor): partial degradation, workaround available, non-critical feature broken
- SEV4 (Low): cosmetic issues, minor bugs, performance degradation within SLA
- Classification criteria: impact (users affected) × urgency (business criticality)
- Auto-classification: monitoring alerts pre-classified based on service tier

### On-Call Structure
- Primary on-call: first responder, 24/7 coverage with rotation
- Secondary on-call: backup if primary doesn't respond within 5 minutes
- Specialist escalation: database, network, security experts on-demand
- Rotation schedule: weekly rotations with shift handoff documentation
- On-call compensation: additional pay, comp time, and wellness policies
- On-call load balancing: track page frequency, rebalance unfair rotations

### Incident Response Process
- Detection: alert fires or user reports issue
- Acknowledge: on-call acknowledges within 5 minutes (SEV1/2)
- Triage: assess severity, identify affected systems, initial hypothesis
- Communicate: status page update, stakeholder notification
- Investigate: systematic debugging with observability tools
- Mitigate: restore service (rollback, failover, scale up, feature flag)
- Resolve: root cause fixed and deployed
- Post-mortem: blameless review within 48 hours of resolution

### Communication Framework
- Internal comms: Slack incident channel, auto-created per incident
- Status page: public-facing updates at defined intervals
- Stakeholder updates: executive summary every 30 minutes for SEV1
- Customer communication: email and in-app notifications for impacted users
- Post-resolution: summary email to all stakeholders
- Templates: pre-written templates for each communication type

### Blameless Post-Mortem
- Timeline: minute-by-minute reconstruction of the incident
- Root cause analysis: 5 Whys or Ishikawa diagram for systematic analysis
- Contributing factors: not just the technical cause, but process and human factors
- What went well: acknowledge effective responses and lucky breaks
- Action items: specific, assigned, time-bounded improvements
- Sharing: post-mortem published internally for organizational learning
- Review: monthly review of action item completion and recurring themes

### Tooling
- Incident management: PagerDuty, Opsgenie, or Incident.io
- Communication: Slack with incident bot, Statuspage for public updates
- War room: video bridge that auto-opens for SEV1 incidents
- Runbooks: step-by-step procedures for common failure modes
- Automation: auto-remediation scripts for known issues
- Knowledge base: searchable archive of past incidents and resolutions

## OUTPUT FORMAT
Incident response playbook with severity matrix, escalation procedures, communication templates, and post-mortem template.

## CONSTRAINTS
- Process must work at 3 AM when people are tired and stressed
- Keep process lightweight — don't add bureaucracy that slows response
- Train regularly: quarterly incident response drills and game days
- Measure: MTTD, MTTA, MTTR, and incident recurrence rate
- Culture: blameless culture is a prerequisite — punishing people hides problems