Build a comprehensive chaos engineering program covering experiment design, blast radius control, steady-state hypothesis formulation, automated game days, and organizational adoption for building resilient systems.
## CONTEXT Gremlin's 2024 State of Chaos Engineering Report shows that organizations practicing chaos engineering experience 60% fewer severe incidents and resolve remaining incidents 67% faster. Netflix, Amazon, and Google credit chaos engineering as a key factor in achieving their high reliability standards. However, 71% of organizations have not yet started chaos engineering, primarily due to concerns about production impact and lack of structured methodology. The chaos engineering discipline has matured significantly with tools like Chaos Mesh, Litmus, and Gremlin providing safe, controlled experiment frameworks that minimize risk while maximizing learning. ## ROLE Act as a Senior Chaos Engineering Lead with 10 years of experience in reliability engineering and 6 years specifically building chaos engineering programs. You have designed chaos engineering platforms conducting over 5,000 experiments annually across production environments serving hundreds of millions of users, built automated game day frameworks that validated disaster recovery capabilities quarterly, and established chaos engineering as a cultural practice across organizations with 400+ engineers. You are a contributor to the Chaos Engineering Principles manifesto and expert in LitmusChaos, Chaos Mesh, and Gremlin platforms. ## RESPONSE GUIDELINES - Design the complete chaos engineering program from organizational buy-in through experiment design, execution, and learning integration - Include specific experiment definitions with steady-state hypotheses, blast radius controls, and abort conditions for common failure modes - Provide tool configurations for the chosen chaos engineering platform with safety guardrails and automated rollback mechanisms - Address the cultural and organizational aspects of introducing controlled failure into production systems - Do NOT recommend chaos experiments in production without blast radius limiting, automated abort conditions, and monitoring verification - Do NOT design experiments without clear steady-state hypotheses and measurable success criteria defined before execution ## TASK CRITERIA 1. **Program Foundation** — Establish the chaos engineering program charter including business justification with incident cost data, executive sponsorship requirements, program scope definition, maturity progression stages, and success metrics for measuring program value 2. **Experiment Design Framework** — Define the experiment methodology including steady-state hypothesis formulation, independent variable selection (what to break), blast radius scoping, duration and magnitude parameters, abort conditions, and experiment documentation standards 3. **Common Experiment Library** — Create a library of standard chaos experiments including pod and container failures, network latency injection, DNS resolution failures, dependency unavailability simulation, disk pressure scenarios, CPU and memory stress, and zone or region failover exercises 4. **Safety and Blast Radius Control** — Implement safety mechanisms including percentage-based blast radius limiting, automated metric monitoring during experiments, circuit breaker abort triggers, rollback automation, customer impact detection gates, and experiment scheduling restrictions 5. **Platform and Tooling** — Configure the chaos engineering platform including tool deployment in the infrastructure, experiment definition formats, integration with existing observability for experiment monitoring, and self-service experiment creation with approval workflows 6. **Game Day Operations** — Design structured game day exercises including scenario planning templates, cross-team coordination procedures, real-time observation and documentation protocols, facilitator guides, and post-game-day analysis sessions 7. **Learning Integration** — Connect chaos engineering findings to reliability improvement including vulnerability discovery tracking, remediation priority scoring, pattern identification across experiments, integration with post-incident review findings, and reliability backlog management 8. **Organizational Adoption** — Create the adoption strategy including pilot team selection criteria, training curriculum for engineers, champion network establishment, internal communication of wins, and scaling from single-team to organization-wide practice ## INFORMATION ABOUT ME - My chaos engineering tool: [INSERT YOUR preferred tool e.g., Gremlin, Chaos Mesh, LitmusChaos, AWS FIS] - My infrastructure platform: [INSERT YOUR deployment platform e.g., Kubernetes, ECS, EC2, multi-cloud] - My service architecture: [INSERT YOUR microservices count and critical service dependencies] - My current reliability maturity: [INSERT YOUR SLO coverage, monitoring completeness, and incident frequency] - My organizational risk tolerance: [INSERT YOUR comfort level with production experiments e.g., non-production only, limited production, full production] - My team size: [INSERT YOUR engineering team size and reliability engineering staffing] ## RESPONSE FORMAT - Begin with a chaos engineering program roadmap showing maturity phases from exploratory to advanced automated chaos - Provide specific experiment definitions with YAML or JSON configurations for the chosen platform - Include game day planning templates with facilitator checklists and observation guides - Present a chaos experiment results dashboard specification tracking experiments run, vulnerabilities found, and remediation status - Conclude with a maturity assessment rubric and a 6-month adoption plan covering tooling, training, pilot experiments, and scaling
Or press ⌘C to copy