Infrastructure Monitoring & Alerting Setup (Prometheus/Grafana)

## ROLE

You are a site reliability engineer and observability specialist with deep expertise in Prometheus, Grafana, Alertmanager, and the broader CNCF monitoring ecosystem. You have built monitoring platforms for organizations running hundreds of microservices, designed alerting strategies that reduce alert fatigue while catching real incidents, and created dashboards that enable teams to diagnose issues in minutes rather than hours. You understand cardinality management, long-term storage solutions, and the art of writing alerts that page on symptoms rather than causes.

## OBJECTIVE

Deploy and configure a comprehensive monitoring and alerting stack for [INFRASTRUCTURE: Kubernetes cluster / VM-based infrastructure / hybrid cloud / multi-cloud] running [NUMBER: 10-50 / 50-200 / 200+] services. The stack must provide infrastructure metrics, application metrics, and business metrics with appropriate dashboards and alerts. Target [MONITORING TOOL: Prometheus + Grafana + Alertmanager / Thanos + Grafana / Mimir + Grafana / Victoria Metrics + Grafana] deployed on [PLATFORM: Kubernetes via Helm / Docker Compose / standalone / managed service].

## TASK: MONITORING & ALERTING STACK DEPLOYMENT

### Architecture & Deployment

Design the monitoring architecture for [INFRASTRUCTURE]:

**Prometheus Deployment:** Deploy Prometheus using [METHOD: kube-prometheus-stack Helm chart / Prometheus Operator / standalone]. Configure:
- Retention period: [DURATION: 15d / 30d / 90d] local storage with [LONG-TERM: Thanos / Mimir / Cortex / Victoria Metrics / none] for extended retention
- Storage sizing: estimate [SERIES COUNT: based on NUMBER services × average labels] active time series, calculate storage at ~1-2 bytes per sample, ingestion rate of [INTERVAL: 15s / 30s / 60s] scrape interval
- Resource allocation: [CPU: 2-4 cores] and [MEMORY: 4-16 GB] based on series count (rough guide: 1GB per 100K active series)
- High availability: [HA MODE: dual Prometheus with Thanos dedup / Prometheus Agent mode with remote write / single with PVC for small setups]

Provide the complete Helm values or Prometheus configuration for this deployment.

**Grafana Deployment:** Deploy Grafana with: persistent storage for dashboards and settings, [AUTH: Grafana built-in / OAuth via Okta / LDAP / Azure AD SSO], organization and folder structure matching team topology, and provisioned data sources pointing to Prometheus and [LONG-TERM STORAGE: if applicable]. Configure dashboard provisioning from [SOURCE: ConfigMaps / Git repository / Grafana API] so dashboards are version-controlled and reproducible.

**Alertmanager Configuration:** Deploy Alertmanager with: [HA: clustered mode with 3 replicas / single instance], routing tree matching team ownership, and receivers for [NOTIFICATION CHANNELS: Slack / PagerDuty / Opsgenie / email / Microsoft Teams / custom webhook].

### Metric Collection Strategy

Define what to monitor at each layer:

**Infrastructure Metrics (Node Exporter + cAdvisor):**
- Node-level: CPU utilization (user, system, iowait, steal), memory (used, available, cached, swap), disk (utilization, IOPS, latency, throughput), network (bandwidth, errors, drops, connections), and filesystem usage per mount
- Container-level: CPU usage versus request and limit, memory working set versus limit (OOMKill risk), restart count, network I/O per pod, and container state (running, waiting, terminated with reason)
- Kubernetes-level: pod scheduling latency, node readiness, PV utilization, DaemonSet and Deployment rollout status, HPA current versus desired replicas, and API server latency and error rates

**Application Metrics (RED Method):**
For each service in [INFRASTRUCTURE], expose these metrics via [ENDPOINT: /metrics Prometheus endpoint / OpenTelemetry SDK / StatsD bridge]:
- **Rate:** Request throughput — http_requests_total counter labeled by method, path (cardinality-controlled), and status_code
- **Errors:** Error count — http_requests_total where status_code=~"5.." plus application-specific error counters
- **Duration:** Latency distribution — http_request_duration_seconds histogram with buckets [BUCKETS: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] customized per service SLA

Provide a sample instrumentation snippet for [LANGUAGE: Go / Java / Python / Node.js / .NET] using [LIBRARY: prometheus client / micrometer / opentelemetry-sdk].

**Business Metrics:**
Instrument [NUMBER: 3-5] business KPIs as Prometheus metrics: [EXAMPLES: orders_placed_total, payment_processed_total, user_signups_total, search_queries_total, active_sessions_gauge]. These enable the most valuable alerts — detecting business impact even when infrastructure appears healthy.

### Recording Rules & Query Optimization

Write Prometheus recording rules that pre-compute expensive queries:

```yaml
groups:
  - name: service_sli_rules
    interval: 30s
    rules:
      - record: service:http_request_rate:5m
        expr: sum(rate(http_requests_total[5m])) by (service, namespace)
      - record: service:http_error_rate:5m
        expr: sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service, namespace) / sum(rate(http_requests_total[5m])) by (service, namespace)
      - record: service:http_latency_p99:5m
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service, namespace))
      - record: service:http_latency_p95:5m
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service, namespace))
```

Add recording rules for [NUMBER: 5-10] additional scenarios specific to [INFRASTRUCTURE]: node capacity rules, PVC utilization rules, and [APPLICATION-SPECIFIC] aggregations. Recording rules reduce dashboard load time and enable alerts on pre-computed values.

### Alerting Rules (Symptom-Based)

Design alerts following these principles: alert on symptoms (user-facing impact) not causes, include runbook links in every alert annotation, set severity levels (critical = pages on-call, warning = Slack notification, info = dashboard only), and tune thresholds to minimize false positives while catching real incidents.

**Critical Alerts (Page Immediately):**
```yaml
- alert: HighErrorRate
  expr: service:http_error_rate:5m > 0.05
  for: 5m
  labels:
    severity: critical
    team: "{{ $labels.namespace }}"
  annotations:
    summary: "High error rate on {{ $labels.service }}"
    description: "Service {{ $labels.service }} in {{ $labels.namespace }} has {{ $value | humanizePercentage }} error rate (>5%) for 5 minutes."
    runbook_url: "https://[WIKI_URL]/runbooks/high-error-rate"
    dashboard_url: "https://[GRAFANA_URL]/d/service-overview?var-service={{ $labels.service }}"
```

Provide [NUMBER: 8-12] production-ready alerting rules covering: high error rate per service, latency SLA breach (p99 above [MS: threshold]), pod CrashLoopBackOff, node disk pressure (>85% utilization), node memory pressure (>90% utilization), PVC nearing capacity (>80%), certificate expiry (within 14 days), deployment stuck (progressing condition false for 15m), HPA at max replicas for extended period, and [APPLICATION-SPECIFIC: custom business metric anomalies].

**Warning Alerts (Slack/Notification):**
Provide [NUMBER: 5-8] warning-level alerts for early indicators: elevated latency (p95 above baseline), resource requests approaching node capacity, pod restart count increasing, replica count below desired, and error rate elevated but below critical threshold.

### Alertmanager Routing & Escalation

Configure the Alertmanager routing tree:

```yaml
route:
  receiver: default-slack
  group_by: ['alertname', 'namespace', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      continue: true
    - match:
        severity: critical
      receiver: slack-critical
    - match:
        severity: warning
      receiver: slack-warning
    - match_re:
        namespace: "production|staging"
        team: "[TEAM_NAME]"
      receiver: "[TEAM]-slack"
      routes:
        - match:
            severity: critical
          receiver: "[TEAM]-pagerduty"
```

Configure receivers for [NOTIFICATION CHANNELS] with templates that include: alert severity emoji/color, affected service and namespace, current metric value and threshold, direct links to relevant Grafana dashboard and runbook, and silence link for acknowledged alerts. Configure inhibition rules that suppress downstream alerts when a root cause alert is already firing (e.g., suppress pod alerts when the node is NotReady).

### Grafana Dashboard Library

Design [NUMBER: 4-6] dashboards:

1. **Cluster Overview:** Node health, resource utilization heatmaps, pod distribution, top resource consumers, and namespace cost attribution
2. **Service Health (per-service):** RED metrics, pod status, HPA behavior, resource usage versus limits, recent deployments overlay, and upstream/downstream dependency health
3. **On-Call Dashboard:** Active alerts timeline, recent incidents, error budget burn rate for [SLO: if applicable], and quick-action links
4. **[APPLICATION-SPECIFIC]:** Business metrics, user journey funnels, payment processing health, or [CUSTOM] based on [INFRASTRUCTURE]

For each dashboard, specify the panel types (time series, stat, gauge, table, heatmap, logs), queries with appropriate time ranges and aggregations, and variable templates for namespace, service, and time range filtering.