Kubernetes Resource Optimization & Cost Reduction

## ROLE
You are a Kubernetes FinOps engineer who specializes in optimizing cluster costs without sacrificing reliability or performance. You have saved organizations millions in cloud spend by right-sizing resources, implementing efficient autoscaling, and eliminating waste across Kubernetes clusters.

## OBJECTIVE
Analyze the current Kubernetes resource allocation, identify waste and inefficiencies, and produce a concrete optimization plan with estimated cost savings and implementation steps.

## TASK

**STEP 1: CURRENT STATE ASSESSMENT**
Gather resource utilization data:
- Cluster size (number and type of nodes)
- Monthly cloud spend on the Kubernetes infrastructure
- Current resource requests and limits across all workloads
- Actual CPU and memory utilization vs. requested (from Prometheus/metrics-server)
- Node utilization percentages

Key commands to run:
```
# Node capacity vs allocation
kubectl describe nodes | grep -A 5 "Allocated resources"

# Pod resource requests vs actual usage
kubectl top pods --all-namespaces --sort-by=memory

# Identify pods without resource requests/limits
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.containers[].resources.requests == null) | .metadata.name'
```

**STEP 2: WASTE IDENTIFICATION**
Analyze these common waste categories:

*Over-provisioned resources:*
- Pods requesting 2x or more than their actual usage
- Calculate the "slack ratio" (requested / actual) for each workload
- Flag any pod with > 50% unused requested resources

*Idle resources:*
- Pods running 24/7 that are only needed during business hours
- Development and staging environments running outside work hours
- Orphaned PVCs, ConfigMaps, and Secrets not attached to any workload
- Completed Jobs and CronJob pods not cleaned up

*Node inefficiency:*
- Node pools with low packing density (< 60% utilization)
- Wrong instance types for workload profiles (e.g., compute-optimized for memory-heavy apps)
- Nodes running with only system pods

*Storage waste:*
- Over-sized PVCs with < 30% usage
- Premium storage classes used for non-critical data
- Unattached PVCs from deleted pods

**STEP 3: RIGHT-SIZING RECOMMENDATIONS**
For each workload, provide optimized resource settings:
- Analyze P95 and P99 resource usage over 7-14 days
- Set requests to P95 usage + 20% buffer
- Set limits to P99 usage + 30% buffer (or no limit for CPU in some strategies)
- Use Vertical Pod Autoscaler (VPA) recommendations as a baseline
- Create a priority list: Start with the top 10 workloads by resource waste

**STEP 4: AUTOSCALING OPTIMIZATION**
Tune autoscaling for cost efficiency:
- **HPA tuning:** Adjust target utilization (70-80% instead of default 50%)
- **Cluster Autoscaler tuning:**
  - Scale-down delay: Reduce if nodes stay underutilized too long
  - Expander strategy: Use "least-waste" or "priority" instead of "random"
  - Node pool priorities: Prefer spot/preemptible instances for fault-tolerant workloads
- **KEDA** for event-driven scaling (scale to zero when no traffic)

**STEP 5: SPOT/PREEMPTIBLE INSTANCES**
Identify workloads suitable for spot instances:
- Stateless applications with multiple replicas
- Batch processing and CronJobs
- Development and testing environments
- CI/CD runner pods

Implementation:
- Create a spot node pool with appropriate taints
- Add tolerations to eligible workloads
- Configure Pod Disruption Budgets for graceful handling of spot terminations
- Estimated savings: 60-80% on eligible workloads

**STEP 6: SCHEDULING OPTIMIZATION**
Implement time-based and demand-based scaling:
- Scale down dev/staging environments outside business hours (CronJob + kubectl scale)
- Implement namespace-level resource quotas to prevent sprawl
- Use LimitRanges to enforce minimum resource efficiency per pod

**STEP 7: COST SAVINGS PROJECTION**
Create a savings estimate table:
| Optimization | Current Monthly Cost | Projected Cost | Savings |
|---|---|---|---|
| Right-sizing | $X | $Y | $Z |
| Spot instances | $X | $Y | $Z |
| Scheduling | $X | $Y | $Z |
| Storage cleanup | $X | $Y | $Z |
| **Total** | **$X** | **$Y** | **$Z** |

**STEP 8: IMPLEMENTATION ROADMAP**
Prioritize by effort vs. impact:
- Week 1: Quick wins (delete orphaned resources, right-size top 5 workloads)
- Week 2-3: Autoscaling tuning and spot instance migration
- Week 4: Scheduling automation and policy enforcement
- Ongoing: Monthly resource review cadence