Kubernetes Cluster Architecture

Help me design a production Kubernetes cluster architecture with these requirements:

Managed Service: [EKS/GKE/AKS/SELF-MANAGED]
Workload Types: [MICROSERVICES/BATCH/ML/MIXED]
Expected Pod Count: [APPROXIMATE NUMBER]
Availability Requirements: [SLA TARGET]
Team Kubernetes Experience: [BEGINNER/INTERMEDIATE/ADVANCED]
Compliance Needs: [LIST REQUIREMENTS]

Design the architecture across these six areas:

1. Cluster Topology and Node Architecture
- Define the number of clusters and their purpose including production, staging, development, and shared services. Design node pool strategy with dedicated pools for system components, application workloads, and specialized needs like GPU or high-memory. Select instance types based on workload resource profiles and cost optimization. Plan node sizing considering pod density, system overhead, and resource fragmentation. Configure cluster autoscaler with appropriate scaling policies, scale-down delays, and priority expanders. Design multi-AZ node distribution for high availability. Plan for node upgrade strategies using rolling updates versus blue-green.

2. Networking Architecture
- Select the CNI plugin such as VPC-CNI, Calico, or Cilium based on requirements. Design pod and service CIDR ranges with growth capacity. Implement network policies for micro-segmentation between namespaces and workloads. Configure ingress architecture using NGINX Ingress Controller, ALB Ingress, or Istio Gateway. Plan for service mesh implementation with Istio, Linkerd, or Cilium Service Mesh. Design DNS strategy using CoreDNS configuration and external DNS for service discovery. Address load balancing patterns for internal and external traffic.

3. Security Hardening
- Implement RBAC with namespace-level isolation and least-privilege service accounts. Configure Pod Security Standards including restricted, baseline, and privileged profiles. Deploy admission controllers using OPA Gatekeeper or Kyverno for policy enforcement. Implement network policies to restrict pod-to-pod communication. Configure secrets management integration with external providers like Vault or cloud-native secrets stores using CSI driver. Set up image scanning in CI/CD pipelines and enforce signed images with admission webhooks. Plan for runtime security monitoring using Falco or similar tools.

4. Storage and State Management
- Design persistent volume strategy using CSI drivers for cloud provider storage. Configure storage classes with appropriate provisioners, reclaim policies, and volume binding modes. Plan for stateful workload management using StatefulSets with proper PDB and headless services. Implement backup solutions for persistent volumes using Velero or cloud-native snapshots. Address shared storage needs with EFS, Filestore, or Azure Files. Design ConfigMap and Secret management strategies including external configuration patterns.

5. Observability Stack
- Deploy Prometheus for metrics collection with appropriate retention and storage configuration. Set up Grafana dashboards for cluster health, node performance, and application metrics. Implement centralized logging with Fluentd or Fluent Bit shipping to Elasticsearch, Loki, or CloudWatch. Configure distributed tracing with Jaeger or Tempo integrated with application instrumentation. Design alerting rules for cluster-level events including node pressure, pod failures, and resource exhaustion. Implement cost monitoring at namespace and workload levels using Kubecost or cloud-native tools. Plan for log retention, metrics retention, and archival.

6. Deployment and Operations
- Design GitOps workflows using ArgoCD or Flux for declarative cluster management. Implement Helm chart management and versioning strategy. Define resource quotas and limit ranges per namespace to prevent noisy neighbors. Plan upgrade strategy for Kubernetes version management including control plane and node upgrades. Design disaster recovery including cluster backup, etcd backup, and cross-region failover. Implement progressive delivery using canary deployments, blue-green, or feature flags. Create operational runbooks for common scenarios including node failures, pod evictions, and certificate renewal.

For each area provide specific tool selections with versions, configuration snippets in YAML where helpful, capacity planning estimates, operational procedures, and cost implications.