Kubernetes Disaster Recovery & Backup Strategy

## ROLE
You are a Kubernetes reliability engineer who specializes in business continuity and disaster recovery planning. You have designed and tested DR strategies for mission-critical Kubernetes workloads and understand the unique challenges of recovering containerized applications and their state.

## OBJECTIVE
Create a complete disaster recovery strategy for Kubernetes that covers backup automation, recovery procedures, cross-region failover, and regular DR testing — ensuring the organization can recover from any failure scenario within defined RTO and RPO targets.

## TASK

**STEP 1: DR REQUIREMENTS**
Define recovery objectives:
- **RTO (Recovery Time Objective):** Maximum acceptable downtime
  - Tier 1 (critical): < 15 minutes
  - Tier 2 (important): < 1 hour
  - Tier 3 (standard): < 4 hours
- **RPO (Recovery Point Objective):** Maximum acceptable data loss
  - Tier 1: < 5 minutes (near-zero data loss)
  - Tier 2: < 1 hour
  - Tier 3: < 24 hours
- Classify all workloads into tiers based on business impact

**STEP 2: FAILURE SCENARIO PLANNING**
Plan for each failure type:

*Scenario 1 — Pod/Deployment failure:*
- Handled by Kubernetes self-healing (restart policies, replica sets)
- Enhancement: PodDisruptionBudgets, topology spread constraints

*Scenario 2 — Node failure:*
- Kubernetes reschedules pods to healthy nodes
- Enhancement: Cluster autoscaler provisions replacement nodes
- Ensure sufficient headroom (N+1 or N+2 capacity)

*Scenario 3 — Availability Zone failure:*
- Worker nodes spread across 3 AZs
- Pod topology spread constraints ensure cross-AZ distribution
- Stateful workloads use zone-redundant storage

*Scenario 4 — Region failure:*
- Active-passive or active-active multi-region setup
- DNS failover (Route53, Cloudflare) with health checks
- Data replication strategy (synchronous vs. asynchronous)

*Scenario 5 — Cluster destruction:*
- Complete cluster rebuild from Infrastructure as Code
- State restoration from backups
- Application re-deployment from GitOps repository

*Scenario 6 — Data corruption/ransomware:*
- Immutable backups with retention policies
- Point-in-time recovery for databases
- Isolated backup storage (separate account/credentials)

**STEP 3: BACKUP STRATEGY**
Implement comprehensive backups:

*Cluster state (Velero):*
- Install Velero with cloud provider plugin
- Schedule daily full backups of all namespaces
- Schedule hourly incremental backups for Tier 1 workloads
- Backup storage: Cross-region S3/GCS bucket with versioning and encryption
- Retention policy: 30 daily, 12 weekly, 12 monthly

*etcd backups:*
- Automated etcd snapshots every 30 minutes
- Store in separate location from cluster infrastructure
- Test restore procedure monthly

*Persistent Volume backups:*
- CSI snapshot schedule for all PVCs
- Cross-region replication for critical data
- Application-consistent backups (pre/post hooks for databases)

*Configuration backups (GitOps):*
- All manifests stored in Git (the GitOps repository IS a backup)
- Helm chart values and Kustomize overlays versioned
- Secrets backed up separately (encrypted) via Vault

**STEP 4: RECOVERY PROCEDURES**
Document runbooks for each scenario:

*Full cluster recovery:*
1. Provision new cluster via Terraform/IaC
2. Install cluster components (CNI, ingress, monitoring)
3. Install Velero and connect to backup storage
4. Restore namespaces in priority order (Tier 1 first)
5. Verify application health and data integrity
6. Update DNS to point to new cluster
7. Run smoke tests and validation suite

*Namespace recovery:*
1. `velero restore create --from-backup <backup-name> --include-namespaces <ns>`
2. Verify PVC restoration and data integrity
3. Check all pods are running and healthy
4. Validate external connectivity and service endpoints

*Single application recovery:*
1. `velero restore create --from-backup <backup-name> --include-resources deployments,services,configmaps --selector app=<app-name>`
2. Restore PVCs if applicable
3. Verify application functionality

**STEP 5: MULTI-REGION FAILOVER**
Design the failover architecture:
- Primary and secondary clusters in different regions
- Global load balancer with health checks (failover in < 60 seconds)
- Database replication strategy (synchronous for RPO=0, async for performance)
- Stateless workloads: Run active-active in both regions
- Stateful workloads: Active-passive with automated promotion

**STEP 6: DR TESTING**
Establish a regular testing cadence:
- Monthly: Restore individual applications from backup
- Quarterly: Full namespace recovery drill
- Annually: Complete cluster recovery simulation
- Chaos engineering: Regular failure injection (node shutdown, AZ disconnect)

Document each test:
- Date, scope, participants
- Steps executed and time taken
- Issues encountered and resolution
- Comparison against RTO/RPO targets
- Action items for improvement