GitOps Workflow Design with ArgoCD

## ROLE
You are a GitOps practitioner and platform engineer who has implemented ArgoCD-based deployment workflows for organizations running 50+ microservices across multiple Kubernetes clusters. You are an ArgoCD contributor and have deep expertise in declarative infrastructure management, Helm chart orchestration, Kustomize overlays, and the operational practices that make GitOps successful at scale. You understand both the benefits and pitfalls of GitOps — and how to design workflows that developers love while maintaining security and compliance requirements.

## OBJECTIVE
Design a complete GitOps workflow using ArgoCD for [ORGANIZATION NAME] deploying [NUMBER OF SERVICES] services to [NUMBER OF CLUSTERS] Kubernetes clusters across [ENVIRONMENTS: dev/staging/production]. The workflow must support automated deployments to lower environments, manual approval gates for production, secret management, and rapid rollback capabilities. The system should be manageable by a platform team of [TEAM SIZE] and usable by [NUMBER OF DEVELOPERS] developers.

## TASK

### Section 1: Repository Strategy & Structure
- Design the Git repository structure for GitOps:
  - Option A: Monorepo — all application manifests in a single repo with directory-per-app structure
  - Option B: Multi-repo — separate repos per application or team with a central config repo
  - Option C: Hybrid — application code repos with a dedicated GitOps config repo
  - Recommend the best approach for [ORGANIZATION NAME]'s scale and team structure
- Define the directory layout within the GitOps repo:
  - Base manifests (shared across environments)
  - Environment-specific overlays (dev, staging, production)
  - Cluster-specific configurations
  - Shared infrastructure components (ingress, monitoring, cert-manager)
- Plan the branching strategy: trunk-based with environment directories vs. branch-per-environment
- Design the pull request workflow: who reviews config changes, required approvals, automated checks

### Section 2: ArgoCD Architecture & Configuration
- Design the ArgoCD deployment:
  - Single ArgoCD instance managing multiple clusters vs. ArgoCD-per-cluster (hub-and-spoke)
  - High availability configuration: Redis HA, controller replicas, repo server scaling
  - Resource requirements and sizing based on the number of applications and sync frequency
- Configure ArgoCD Projects for multi-tenancy:
  - Project-per-team with source repository and destination cluster/namespace restrictions
  - Role-based access using ArgoCD RBAC with SSO integration ([SSO PROVIDER: Okta / Azure AD / GitHub])
  - Sync windows: restrict production deployments to business hours or maintenance windows
- Set up ArgoCD ApplicationSets for templated, scalable application management:
  - Generator strategies: Git directory, cluster, pull request, matrix
  - Automatic application creation when new services are added to the GitOps repo
- Configure ArgoCD notifications: Slack, email, or webhook notifications for sync status changes

### Section 3: Application Manifest Management
- Design the manifest strategy: Helm charts vs. Kustomize overlays vs. plain YAML — or a combination
- If using Helm:
  - Chart repository management: ChartMuseum, OCI registry, or Git-based charts
  - Values file hierarchy: default values → environment values → cluster values → application overrides
  - Helm hooks for pre/post deployment tasks (database migrations, cache warming)
- If using Kustomize:
  - Base and overlay structure for environment differentiation
  - Strategic merge patches vs. JSON patches for configuration variations
  - Component-based composition for shared configurations
- Implement config management best practices:
  - External configuration through ConfigMaps and environment variables
  - Resource limits and requests per environment (smaller in dev, production-sized in staging)
  - Health check and readiness probe configuration
  - Horizontal Pod Autoscaler and PodDisruptionBudget definitions

### Section 4: Environment Promotion & Deployment Strategy
- Design the promotion pipeline: how changes flow from dev → staging → production
  - Automated promotion: CI pipeline updates image tags in GitOps repo, ArgoCD syncs automatically
  - Manual promotion: PR-based promotion with review requirements for production
  - Hybrid: auto-promote to dev/staging, require approval for production
- Implement deployment strategies within ArgoCD:
  - Rolling updates with configurable maxSurge and maxUnavailable
  - Blue-green deployments using Argo Rollouts
  - Canary deployments with progressive traffic shifting and automated analysis
  - A/B testing with header-based routing
- Design the image update automation:
  - ArgoCD Image Updater for automatic image tag updates
  - CI-triggered GitOps repo updates via pull request
  - Image tag strategy: semantic versioning, git SHA, or environment-specific tags
- Create approval gates for production deployments: GitHub PR approvals, Slack-based approvals, or ArgoCD sync windows

### Section 5: Secret Management in GitOps
- Design the secret management strategy (secrets cannot live in Git in plaintext):
  - Option A: Sealed Secrets — encrypt secrets in Git, decrypt in cluster
  - Option B: External Secrets Operator — sync from Vault, AWS Secrets Manager, or Azure Key Vault
  - Option C: SOPS with age/GPG encryption for Git-stored secrets
  - Option D: Vault Agent Injector for runtime secret injection
- Recommend the best approach based on [ORGANIZATION NAME]'s existing infrastructure and security requirements
- Implement secret rotation workflows compatible with GitOps
- Design access controls for who can view/modify secrets in each environment
- Plan for secret emergency rotation procedures

### Section 6: Rollback, Recovery & Operations
- Design rollback procedures:
  - Git revert: revert the commit that introduced the change, ArgoCD syncs the rollback
  - ArgoCD rollback: use ArgoCD's built-in history to rollback to a previous sync
  - Argo Rollouts abort: immediately halt a canary/blue-green and revert traffic
- Implement automated rollback triggers based on health checks, error rates, or SLO violations
- Create drift detection and remediation: how ArgoCD handles out-of-band changes (self-heal vs. alert)
- Design the disaster recovery plan for ArgoCD itself: backup Application definitions, RBAC configs, and repository credentials
- Build operational dashboards: sync status across all applications, deployment frequency, failure rates
- Create troubleshooting guides for common GitOps issues: sync failures, resource conflicts, webhook timeouts

## OUTPUT FORMAT
Provide ArgoCD Application and ApplicationSet YAML definitions, repository structure examples with sample manifests, RBAC configuration, and Project definitions. Include CI pipeline examples (GitHub Actions/GitLab CI) for the image update workflow. Provide a step-by-step setup guide from ArgoCD installation to first application deployment. Include architecture diagrams described in text and operational runbooks.

## CONSTRAINTS
- Secrets must never be stored in Git in plaintext — implement encryption or external secret management
- Production deployments must require explicit approval (no fully automated production deploys without review)
- All GitOps configurations must be auditable: who changed what, when, and why (Git history)
- ArgoCD must be hardened: RBAC enforced, SSO required, API server access restricted
- Design must support multi-cluster management without requiring ArgoCD access from every cluster
- Include a migration plan for teams currently deploying via kubectl apply or CI-triggered helm install