Chaos Engineering & Resilience Testing Playbook

## ROLE

You are a site reliability engineer and chaos engineering practitioner who has implemented resilience testing programs at organizations operating at massive scale. You have designed chaos experiments that uncovered critical failure modes before they caused production incidents, built game day programs that trained engineering teams to respond to outages under pressure, and established continuous chaos testing pipelines that validate resilience as a standard part of the deployment process. You have hands-on experience with Chaos Monkey, Gremlin, Litmus Chaos, Chaos Mesh, Toxiproxy, and custom fault injection frameworks. You understand the principles of chaos engineering articulated by Netflix — forming hypotheses about steady state, introducing real-world failures, measuring impact, and learning from the results — and you know how to apply these principles safely from low-blast-radius experiments to full-scale production chaos.

## OBJECTIVE

Create a chaos engineering and resilience testing playbook for [PROJECT NAME], a [SYSTEM TYPE: distributed microservices / monolith with external dependencies / serverless architecture / event-driven system / hybrid cloud application] running on [INFRASTRUCTURE: AWS / GCP / Azure / Kubernetes / bare metal / hybrid]. The system serves [USER BASE: e.g., 500K daily active users] with availability targets of [SLA: 99.9% / 99.95% / 99.99%]. The team has [CHAOS MATURITY: never done chaos engineering / run ad-hoc experiments / has a basic program / wants to formalize and scale]. The system's critical dependencies include [DEPENDENCIES: e.g., PostgreSQL database, Redis cache, Stripe payment API, SendGrid email, S3 object storage, Kafka message broker].

## TASK: COMPLETE CHAOS ENGINEERING PLAYBOOK

### Phase 1 — Resilience Assessment & Hypothesis Formation

Before injecting any failures, assess the current resilience posture. Create a dependency map documenting every internal and external dependency, its failure mode characteristics (does it fail fast or slow? does it return errors or hang?), the expected system behavior when it fails, and the actual tested behavior (if known). Build a failure mode catalog for [SYSTEM TYPE]: network partitions between services, DNS resolution failures, certificate expiration, database connection pool exhaustion, cache unavailability, message queue backpressure, third-party API degradation (slow responses vs errors vs timeouts), disk space exhaustion, CPU saturation, memory pressure, clock skew, and deployment failures (partial rollouts, configuration drift). For each failure mode, form a hypothesis using the chaos engineering scientific method: "We believe that when [FAILURE CONDITION], the system will [EXPECTED BEHAVIOR: gracefully degrade / failover to replica / serve cached responses / return a meaningful error to users / auto-recover within N seconds]. The steady-state metric we will monitor is [METRIC: error rate / latency p95 / order completion rate / user-facing availability]." Prioritize experiments by risk: start with the failures most likely to occur and most impactful to users, then progress to less likely but catastrophic scenarios.

### Phase 2 — Tool Selection & Safety Infrastructure

Select and configure the chaos engineering tooling for [INFRASTRUCTURE]. For Kubernetes environments, compare Litmus Chaos (CNCF project, broad experiment library, GitOps-friendly) vs Chaos Mesh (Kubernetes-native, fine-grained network chaos, time chaos for clock skew) vs Gremlin (SaaS, easiest to start, built-in safety controls, enterprise support). For AWS specifically, evaluate AWS Fault Injection Simulator (native integration, IAM-controlled, supports EC2/ECS/RDS/network faults). For application-level fault injection, show Toxiproxy (network-level proxy for simulating latency, bandwidth limits, and connection issues) and library-level approaches (custom middleware that randomly injects failures based on feature flags). Build the safety infrastructure before running any experiments: define blast radius controls (target specific pods/instances, not entire services), implement automatic rollback triggers (if error rate exceeds [ABORT THRESHOLD: e.g., 5%], immediately stop the experiment), set up real-time dashboards showing the steady-state metrics during experiments, establish communication protocols (who is notified during experiments, how to escalate, emergency stop procedures), and create a runbook for manually reverting any chaos experiment that gets stuck.

### Phase 3 — Starter Experiments (Low Blast Radius)

Design and document [NUMBER: 5-8] starter experiments that are safe for teams new to chaos engineering. Each experiment follows the format: Title, Hypothesis, Steady-State Metric, Method, Blast Radius, Abort Conditions, Expected Outcome, Actual Outcome, and Action Items.

Experiment 1 — Single Instance Termination: Kill one instance/pod of [SERVICE NAME] and verify that the load balancer routes traffic to healthy instances within [FAILOVER TIME: <30 seconds] with zero user-visible errors. This tests basic redundancy and health check configuration.

Experiment 2 — Dependency Latency Injection: Add [LATENCY: 500ms-2s] of latency to the connection between [SERVICE A] and [DEPENDENCY: database / cache / external API] and verify that circuit breakers trip, timeouts are appropriate, and the user experience degrades gracefully (showing cached data or a meaningful loading state) rather than cascading the latency to all upstream callers.

Experiment 3 — Cache Failure: Disable [CACHE: Redis / Memcached] completely and verify that the system falls back to database queries without errors, observe the performance impact, and validate that the system recovers automatically when the cache returns.

Experiment 4 — DNS Resolution Failure: Simulate DNS failure for a specific dependency and verify that DNS caching, retry logic, and fallback mechanisms activate correctly rather than causing widespread failures from a single DNS blip.

Experiment 5 — Certificate or Authentication Failure: Simulate expired credentials or tokens for a dependency and verify that the system handles auth failures gracefully, refreshes credentials where possible, and surfaces clear error messages rather than cryptic failures.

For each experiment, provide the exact commands or configuration to execute it with [CHOSEN TOOL], the monitoring queries to run during the experiment, and a template for documenting results.

### Phase 4 — Advanced Experiments (Higher Blast Radius)

Design advanced experiments for teams that have successfully completed starter experiments. Multi-service cascade failure: simultaneously degrade two dependencies and observe whether the failure modes compound in unexpected ways. Network partition: split the cluster into two partitions and observe whether split-brain scenarios cause data inconsistency. Availability zone failure: simulate the loss of an entire AZ and verify that the system continues operating from remaining zones without data loss. Database failover: trigger a database primary failover and measure the downtime window, connection recovery behavior, and whether any write operations are lost or duplicated during the switchover. Thundering herd after recovery: simulate a dependency going down and then coming back up, verifying that the system handles the reconnection storm without overwhelming the recovered dependency (backoff, connection pooling, queue draining). Rate limiting under load: verify that rate limiting protects the system when a single client sends [RATE: 10-100x] normal traffic, and that other clients are not impacted. For each advanced experiment, define additional safety measures: smaller blast radius percentages, shorter experiment durations, and mandatory war room staffing during execution.

### Phase 5 — Game Days & Team Readiness

Design a Game Day program that uses chaos experiments to train the engineering team on incident response. Create a Game Day runbook: select [NUMBER: 2-3] experiments to run, brief the on-call team on the rules of engagement (they should respond as if it were a real incident), inject failures without telling the team which specific failure was injected, observe and document the team's detection time (how quickly alerts fire and are noticed), diagnosis time (how quickly the team identifies the root cause), and resolution time (how quickly the team mitigates or the system self-heals). After the Game Day, run a blameless retrospective focused on: gaps in monitoring that delayed detection, runbook steps that were unclear or missing, automation opportunities that could reduce MTTR, and architecture improvements that would prevent user impact. Build a Game Day calendar: [FREQUENCY: monthly / quarterly] with rotating scenarios and rotating on-call participants to ensure the entire team builds incident response muscle memory.

### Phase 6 — Continuous Chaos & Production Readiness

Mature from ad-hoc experiments to continuous chaos testing integrated into the development lifecycle. Define the chaos testing pipeline: after every deployment to [ENVIRONMENT: staging / canary / production], automatically run the [NUMBER: 3-5] core resilience experiments and fail the deployment if any hypothesis is violated. Show the pipeline configuration for running automated chaos experiments: trigger the experiment, wait for steady-state measurement, verify the hypothesis, clean up, and report results. For production chaos (only for teams at high maturity), define the safety framework: experiments only run during business hours, only target a small percentage of traffic, have automatic abort triggers with aggressive thresholds, and are announced in advance. Build the resilience scorecard: for each service, track which chaos experiments it has passed, when they were last run, and which experiments are pending — create a dashboard showing the overall resilience posture of the system.