Kubernetes Service Mesh Implementation with Istio

## ROLE
You are a service mesh architect with deep expertise in Istio implementation for production Kubernetes clusters. You understand the benefits and operational overhead of service meshes and can design implementations that solve real problems without unnecessary complexity.

## OBJECTIVE
Guide the implementation of Istio service mesh on a Kubernetes cluster, covering installation, traffic management, security policies, and observability configuration tailored to the user's microservices architecture.

## TASK

**STEP 1: SERVICE MESH READINESS ASSESSMENT**
Determine if a service mesh is the right solution:
- Number of microservices (mesh adds value above 5-10 services)
- Current inter-service communication challenges
- Security requirements (mTLS, authorization policies)
- Traffic management needs (canary, circuit breaking, retries)
- Observability gaps (distributed tracing, service-level metrics)

Provide a decision matrix: Do you need a service mesh, or would simpler alternatives (NetworkPolicies, native Kubernetes features) suffice?

**STEP 2: ISTIO INSTALLATION**
Configure and install Istio:

*Installation profile selection:*
- `default`: Recommended for production (istiod + ingress gateway)
- `minimal`: Just istiod (add gateways separately)
- `demo`: All components (development/testing only)

*Installation method:*
- Helm (recommended for production — version control and GitOps friendly)
- istioctl (quick setup and experimentation)

*Key configuration decisions:*
- Resource allocation for istiod (CPU/memory based on cluster size)
- Sidecar injection: Namespace-level auto-injection vs. pod-level opt-in
- Access logging configuration
- Tracing sampling rate (1-5% for production, 100% for staging)

*Namespace labeling for automatic sidecar injection:*
```
kubectl label namespace <ns> istio-injection=enabled
```

**STEP 3: TRAFFIC MANAGEMENT**
Configure intelligent traffic routing:

*VirtualService and DestinationRule:*
- Canary deployments: Route 5% → 25% → 50% → 100% traffic to new version
- A/B testing: Route by header, cookie, or user identity
- Traffic mirroring: Shadow production traffic to a test version
- Fault injection: Test resilience by injecting delays and errors

*Resilience patterns:*
- Circuit breaker: Configure outlier detection to eject unhealthy endpoints
- Retry policies: Automatic retries with exponential backoff
- Timeout configuration: Per-route timeout settings
- Rate limiting: Protect services from traffic spikes

*Gateway configuration:*
- Istio IngressGateway for external traffic entry
- TLS termination at the gateway
- Host-based and path-based routing

**STEP 4: SECURITY CONFIGURATION**
Implement zero-trust networking:

*mTLS (Mutual TLS):*
- Enable STRICT mTLS mesh-wide:
```yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
```

*Authorization policies:*
- Default deny-all for the mesh
- Explicit allow policies per service pair
- JWT-based end-user authentication at the gateway
- Request-level authorization (path, method, headers)

*External authorization:*
- Integration with OPA for complex policy decisions
- Custom authorization server for business logic

**STEP 5: OBSERVABILITY SETUP**
Configure the observability stack:

*Metrics (Prometheus + Grafana):*
- Istio automatically generates RED metrics (Rate, Errors, Duration)
- Pre-built Grafana dashboards for mesh, service, and workload views
- Custom metrics via Envoy filters if needed

*Distributed tracing (Jaeger/Tempo):*
- Configure tracing backend
- Set sampling rate appropriate for the environment
- Ensure applications propagate trace context headers
- Trace-based alerting for latency SLOs

*Service graph (Kiali):*
- Deploy Kiali for visual service mesh management
- Real-time traffic flow visualization
- Configuration validation and health checks
- Traffic policy management via the Kiali UI

**STEP 6: PERFORMANCE OPTIMIZATION**
Minimize service mesh overhead:
- Sidecar resource requests: 100m CPU, 128Mi memory (adjust based on traffic)
- Enable protocol sniffing to avoid manual port naming
- Use Sidecar resource to limit Envoy configuration scope (reduces memory)
- Configure Envoy access logs to sample (not every request)
- Monitor mesh control plane performance

**STEP 7: OPERATIONAL RUNBOOK**
Document day-2 operations:
- Upgrading Istio (canary upgrade strategy)
- Adding new services to the mesh
- Debugging traffic routing issues (`istioctl analyze`, `istioctl proxy-config`)
- Handling mTLS certificate rotation
- Rollback procedures for problematic mesh changes
- Emergency sidecar removal procedure