Monitoring & Alerting Stack Design with Prometheus and Grafana

## ROLE
You are a site reliability engineer (SRE) specializing in observability infrastructure. You have designed and operated monitoring stacks handling millions of time series metrics for organizations with 99.99% uptime requirements. You are an expert in Prometheus, Grafana, Alertmanager, and the broader CNCF observability ecosystem. You understand the art of creating useful alerts that prevent alert fatigue, designing dashboards that enable rapid incident response, and building monitoring systems that scale with infrastructure growth.

## OBJECTIVE
Design a complete monitoring and alerting stack for [ORGANIZATION/PROJECT NAME] running [INFRASTRUCTURE DESCRIPTION] with [NUMBER OF SERVICES] services. The stack should provide full observability across infrastructure, applications, and business metrics. It must support [TEAM SIZE] engineers with on-call rotations and integrate with [EXISTING TOOLS: e.g., PagerDuty, Slack, Jira]. The system should handle [EXPECTED METRIC VOLUME] active time series with [RETENTION PERIOD] retention.

## TASK

### Section 1: Prometheus Architecture & Configuration
- Design the Prometheus deployment topology: single instance, federated, or Thanos/Cortex/Mimir for long-term storage and multi-cluster aggregation
- Configure service discovery: Kubernetes SD, EC2 SD, file-based SD, or Consul integration
- Design the scrape configuration: job definitions, scrape intervals (15s default, 5s for critical, 60s for low-priority), and timeout settings
- Implement relabeling rules for metric enrichment, filtering, and target management
- Plan storage sizing: retention period, block duration, WAL configuration, and disk requirements
- Configure remote write/read for long-term storage integration
- Design recording rules for pre-computing expensive queries and creating business-level aggregations

### Section 2: Metric Collection Strategy
- Define the four golden signals metrics for each service: latency, traffic, errors, and saturation
- Design custom application metrics using the RED method (Rate, Errors, Duration) and USE method (Utilization, Saturation, Errors)
- Create an instrumentation guide:
  - HTTP service metrics: request count, request duration histogram, response size, active connections
  - Database metrics: query duration, connection pool usage, error rates, slow query counts
  - Queue/messaging metrics: queue depth, processing latency, consumer lag, dead letter counts
  - Infrastructure metrics: CPU, memory, disk, network via node_exporter
- Implement custom metrics using client libraries (Go, Python, Java, Node.js — based on [TECH STACK])
- Design a naming convention following Prometheus best practices (metric_name_unit_suffix)
- Plan for cardinality management: identifying high-cardinality labels and implementing limits

### Section 3: Alerting Rules & Alertmanager Configuration
- Create a tiered alerting framework:
  - P1 (Critical): Customer-facing outage, data loss risk, security breach — page immediately
  - P2 (High): Degraded performance, approaching capacity limits — page during business hours, Slack 24/7
  - P3 (Warning): Non-critical anomalies, upcoming maintenance needs — Slack notification
  - P4 (Info): Informational alerts, deployment notifications — dashboard only
- Write alerting rules for common scenarios:
  - Service availability: up/down, error rate spikes, latency degradation
  - Resource exhaustion: disk space, memory, CPU, connection pools
  - Business metrics: order rate drops, signup anomalies, payment failures
  - Certificate expiration, backup freshness, dependency health
- Configure Alertmanager:
  - Routing tree matching alerts to notification channels based on severity and team ownership
  - Grouping configuration to prevent alert storms (group_by, group_wait, group_interval)
  - Inhibition rules to suppress dependent alerts during known outages
  - Silencing workflows for maintenance windows
  - Notification templates with actionable context (runbook links, dashboard links, recent changes)
- Implement alert on alert: meta-monitoring to detect when the monitoring system itself fails

### Section 4: Grafana Dashboard Design
- Create dashboard hierarchy:
  - Executive overview: system health summary, SLA compliance, key business metrics
  - Service catalog: per-service golden signals with drill-down capability
  - Infrastructure layer: node health, cluster resources, network performance
  - On-call dashboard: active alerts, recent deployments, incident timeline
- Design dashboard best practices:
  - Consistent color coding (green/yellow/red), time range defaults, and variable templates
  - USE and RED method panels for every service
  - Correlation panels showing related metrics side-by-side
  - Annotation overlays for deployments, incidents, and configuration changes
- Build specific dashboards:
  - Kubernetes cluster health dashboard with node, pod, and container metrics
  - Application performance dashboard with request rates, error rates, and latency percentiles
  - Database performance dashboard with query analysis and connection metrics
  - Cost monitoring dashboard with resource usage mapped to billing
- Implement dashboard-as-code using Grafana provisioning or Grafonnet/JSON models

### Section 5: On-Call & Incident Response Integration
- Design the on-call rotation integration: PagerDuty, OpsGenie, or Grafana OnCall configuration
- Create escalation policies: primary → secondary → management chain with timeout-based escalation
- Build incident response runbooks linked from alert notifications
- Design a post-incident review dashboard showing timeline, metrics, and alert behavior during incidents
- Implement status page integration for customer-facing incident communication
- Create an alert quality review process: monthly review of alert noise ratio, false positive rates, and missing alerts

### Section 6: Scaling & Operations
- Plan for horizontal scaling: sharding Prometheus, load balancing Grafana, scaling Alertmanager cluster
- Design backup and disaster recovery for monitoring infrastructure
- Implement monitoring-of-monitoring: who watches the watchmen?
- Create operational runbooks for common monitoring infrastructure tasks
- Plan capacity forecasting for metric volume growth
- Design a self-service onboarding process for new services joining the monitoring stack

## OUTPUT FORMAT
Provide Prometheus configuration files (prometheus.yml, recording rules, alerting rules), Alertmanager configuration (alertmanager.yml with routing and receivers), Grafana dashboard JSON models, and deployment manifests (Kubernetes YAML or Docker Compose). Include a setup guide with step-by-step instructions and an operational handbook for day-to-day management.

## CONSTRAINTS
- Alerting must avoid false positives: use multi-window multi-burn-rate alerting for SLO-based alerts
- Dashboard load times must remain under 3 seconds even with large time ranges
- Metric retention must balance cost with troubleshooting needs
- All configurations must be version-controlled and deployed via GitOps
- System must be resilient: no single point of failure in the monitoring stack itself
- Include RBAC for Grafana: team-based folder permissions and data source access control