Monitoring & Observability Stack Design

## ROLE
You are a site reliability engineer who has built observability platforms for distributed systems with 99.99% uptime requirements. You understand the three pillars of observability and how to turn data into actionable insights.

## OBJECTIVE
Design an observability stack for [SYSTEM/APPLICATION] running [NUMBER] services handling [SCALE] requests per day.

## TASK

### Metrics (Quantitative Monitoring)
- Infrastructure metrics: CPU, memory, disk, network per host and container
- Application metrics: request rate, error rate, latency percentiles (p50, p95, p99)
- Business metrics: signups, purchases, active users, revenue per minute
- Custom metrics: application-specific measurements using StatsD, Prometheus client
- Metric storage: Prometheus, Datadog, CloudWatch, or VictoriaMetrics
- Retention: high-resolution (15s) for 2 weeks, downsampled for 13 months
- Dashboards: Grafana dashboards per service, team, and executive level

### Logging (Event-Level Detail)
- Structured logging: JSON-formatted logs with consistent field names
- Log levels: DEBUG, INFO, WARN, ERROR with appropriate usage guidelines
- Correlation IDs: trace ID and request ID in every log line
- Log aggregation: Fluentd/Fluent Bit → Elasticsearch/Loki/CloudWatch
- Log retention: hot storage (30 days), warm (90 days), cold archive (1 year)
- Search and analysis: Kibana, Grafana Loki, or CloudWatch Insights
- Sensitive data: redact PII, credentials, and tokens from logs

### Distributed Tracing
- Instrumentation: OpenTelemetry SDK for automatic and manual spans
- Trace propagation: W3C Trace Context headers across service boundaries
- Sampling strategy: head-based sampling at 10%, tail-based for errors and slow requests
- Trace storage: Jaeger, Zipkin, Tempo, or Datadog APM
- Service maps: auto-generated topology showing service dependencies
- Critical path analysis: identify which service adds the most latency
- Trace-to-log correlation: click from trace span to associated logs

### Alerting Strategy
- Alert hierarchy: page (wake someone up), ticket (fix this week), info (FYI)
- Signal vs noise: alert on symptoms, not causes — avoid alert fatigue
- Alert routing: PagerDuty/Opsgenie with on-call schedules and escalation
- Runbook links: every alert includes a link to the remediation runbook
- Alert suppression: maintenance windows, dependency-aware suppression
- Alert review: weekly review of alert quality — remove noisy, add missing
- Composite alerts: combine multiple signals to reduce false positives

### SLO-Based Reliability
- SLI definition: measurable indicators of user experience (latency, availability, correctness)
- SLO targets: 99.9% availability = 43.8 minutes downtime/month budget
- Error budget: remaining tolerance for unreliability before freezing changes
- Error budget policies: what happens when the budget is exhausted
- SLO dashboards: burn rate, remaining budget, projected budget depletion
- SLO alerts: fast-burn (immediate) and slow-burn (trending) alerts
- Reporting: weekly SLO status for engineering leadership

### Incident Management
- Detection: automated alerts trigger incident workflow
- Response: on-call engineer acknowledged, severity classified
- Communication: status page updates, stakeholder notifications
- Mitigation: focus on restoring service, not finding root cause
- Resolution: permanent fix deployed and verified
- Post-mortem: blameless retrospective with action items and timeline

## OUTPUT FORMAT
Observability platform design with architecture diagram, tool selection, alerting playbook, and SLO framework.

## CONSTRAINTS
- Observability must not significantly impact application performance (<2% overhead)
- Cost management: monitoring costs should be proportional to infrastructure costs
- Data retention must comply with regulatory requirements
- Support multi-region and multi-cloud deployments
- Self-monitoring: the observability stack must monitor itself