Design a sustainable on-call rotation system with fair scheduling, clear escalation paths, runbook integration, fatigue management, and continuous improvement processes for engineering teams.
## CONTEXT PagerDuty's 2024 On-Call Report reveals that 70% of engineers report on-call duties negatively impact their work-life balance, with 38% considering leaving their role due to on-call burden. Teams with poorly managed on-call programs experience 3x higher attrition rates than those with structured, well-supported programs. Conversely, organizations that invest in on-call excellence report that engineers view on-call as a valuable learning experience, with 65% saying it improves their system understanding. The difference lies in workload management, tooling quality, and organizational support structures. ## ROLE Act as a Senior Engineering Manager and SRE Lead with 12 years of experience building and managing on-call programs across engineering organizations of 50 to 500 engineers. You have designed on-call systems that reduced page volume by 80% while improving incident response times, established compensation and recognition frameworks that made on-call rotations sought-after rather than dreaded, and built escalation architectures serving 30+ service teams with clear ownership boundaries. You are recognized for creating sustainable on-call cultures that balance reliability needs with engineer wellbeing. ## RESPONSE GUIDELINES - Design the complete on-call system from rotation scheduling through incident handling to continuous improvement with specific tool configurations - Include concrete escalation policy definitions with timeout thresholds, notification methods, and fallback chains - Provide runbook templates that enable on-call engineers to handle common incidents without escalation - Address the human aspects including fatigue management, compensation models, and career development integration - Do NOT design on-call rotations where any individual is on-call for more than 7 consecutive days or more than 25% of total time - Do NOT create escalation policies without clear timeout thresholds and automated escalation to prevent single-person dependency ## TASK CRITERIA 1. **Rotation Design** — Define the rotation structure including primary and secondary on-call roles, rotation duration and handoff procedures, timezone-distributed follow-the-sun models, holiday and vacation coverage, and load balancing algorithms for fair distribution 2. **Escalation Architecture** — Design multi-tier escalation policies including automated escalation timeouts at each tier, notification methods per severity level (push notification, phone call, SMS), management escalation triggers, and cross-team escalation procedures with handoff protocols 3. **Incident Handling Workflow** — Define the on-call engineer workflow from page receipt including acknowledgment SLAs, initial triage checklist, runbook reference procedures, escalation decision criteria, and incident handoff between shifts 4. **Runbook System** — Build the runbook framework including standardized runbook template, linking runbooks to specific alerts, runbook maintenance ownership and review cadence, runbook effectiveness metrics, and automated remediation integration 5. **Tooling and Automation** — Configure on-call tooling including PagerDuty or equivalent setup, ChatOps integration for incident management, automated diagnostics that run on page, noise reduction through alert deduplication and intelligent grouping, and mobile access to dashboards 6. **Fatigue Management** — Implement fatigue prevention measures including page volume tracking per rotation, quiet hours protection with severity-gated paging, post-on-call recovery time policies, toil reduction programs, and workload equity tracking across team members 7. **Compensation and Recognition** — Design the on-call incentive structure including monetary compensation models (flat rate, per-page, hybrid), time-off-in-lieu policies, performance review integration, skill development opportunities, and public recognition for excellent incident handling 8. **Continuous Improvement** — Establish on-call program improvement processes including weekly on-call handoff reviews, monthly page volume and response time analysis, quarterly rotation satisfaction surveys, annual program structure review, and actionable improvement tracking ## INFORMATION ABOUT ME - My team size: [INSERT YOUR number of engineers available for on-call rotation] - My service count: [INSERT YOUR number of services requiring on-call coverage] - My timezone spread: [INSERT YOUR team timezone distribution e.g., single timezone, US-based, global] - My current page volume: [INSERT YOUR approximate pages per week and their severity distribution] - My alerting tools: [INSERT YOUR incident notification platform e.g., PagerDuty, Opsgenie, VictorOps] - My current on-call challenges: [INSERT YOUR biggest on-call pain points e.g., alert fatigue, unfair distribution, lack of runbooks] ## RESPONSE FORMAT - Begin with an on-call program overview showing rotation structure, escalation tiers, and feedback loops - Provide specific escalation policy configurations in the native format of the specified alerting platform - Include runbook templates with sections for alert context, diagnostic steps, remediation actions, and escalation criteria - Present a fatigue management dashboard specification tracking page volume, response times, and distribution fairness - Conclude with an on-call health scorecard measuring sustainability, effectiveness, and engineer satisfaction with quarterly targets
Or press ⌘C to copy