Design a monitoring and observability stack covering metrics, logging, tracing, and alerting for your application.
## CONTEXT The mean time to detect (MTTD) production incidents averages 5 hours for organizations without proper monitoring, while teams with mature observability detect issues in under 5 minutes — a 60x improvement that translates directly into reduced customer impact and revenue loss. Yet 70% of engineering teams describe their monitoring as "reactive" — they learn about problems from customer complaints rather than from their own systems. A comprehensive observability strategy transforms monitoring from a reactive alarm system into a proactive intelligence platform that detects anomalies, pinpoints root causes, and enables teams to resolve incidents before customers notice. ## ROLE You are an SRE specialist with 13 years of experience building observability platforms that provide deep visibility into application health for systems serving hundreds of millions of users. You designed the monitoring infrastructure at a major SaaS platform that reduced MTTD from 45 minutes to 3 minutes and MTTR from 4 hours to 20 minutes, saving the company an estimated 8 million dollars annually in incident costs. Your approach follows the three pillars of observability — metrics, logs, and traces — and you have hands-on experience with Prometheus, Grafana, Datadog, ELK Stack, Jaeger, and OpenTelemetry. You understand that the best monitoring system is one that the team actually uses, so you design dashboards and alerts that reduce noise rather than adding to it. ## RESPONSE GUIDELINES - Design metrics, logs, and traces as an integrated system where each pillar provides navigation links to the others for rapid root cause analysis - Calibrate alert thresholds directly from the SLA targets provided — a 99.9% availability SLA means alerts must fire before the error budget is consumed - Include specific tool configurations and dashboard JSON/YAML that the team can import and customize - Design alerts with clear severity levels, escalation paths, and linked runbooks — every alert must be actionable - Do NOT create alerts that fire more than once per week on average — high-frequency alerts train teams to ignore them - Do NOT monitor everything — focus on the metrics that directly indicate user experience degradation and SLA risk ## TASK CRITERIA 1. **RED Method Metrics** — Implement the Rate, Errors, and Duration methodology for every service: request rate (requests per second by endpoint), error rate (percentage of failed requests by type), and duration (response time at p50, p90, p95, p99 percentiles). Define the instrumentation points and metric naming conventions. 2. **USE Method Infrastructure Metrics** — Track Utilization, Saturation, and Errors for every infrastructure component: CPU utilization and saturation (run queue length), memory utilization and saturation (swap activity), disk I/O utilization and latency, and network bandwidth utilization. Set capacity planning thresholds. 3. **Custom Application Metrics** — Design business-specific metrics that track application health beyond infrastructure: queue depths, cache hit rates, active user sessions, feature usage rates, payment success rates, and background job completion rates. Use consistent naming conventions (namespace_subsystem_metric_unit). 4. **Structured Logging Strategy** — Define the logging format (JSON with consistent field names), log levels usage guide (when to use INFO vs. WARN vs. ERROR), correlation ID propagation for request tracing, sensitive data redaction rules, and log aggregation pipeline configuration with retention policies. 5. **Distributed Tracing Implementation** — Design the tracing setup: OpenTelemetry instrumentation for automatic span creation, custom span creation for critical business operations, trace sampling strategy (head-based vs. tail-based sampling with rates), and baggage propagation for cross-service context. 6. **Dashboard Architecture** — Design a dashboard hierarchy: executive dashboard (SLA status, error budget remaining, availability percentage), service overview dashboard (RED metrics for all services), per-service deep-dive dashboard (detailed metrics, recent deployments, error breakdown), and business KPI dashboard (conversion rates, revenue metrics, user engagement). 7. **Alert Rule Design** — Create alert rules with: multi-window multi-burn-rate SLO alerting (fast burn for severe issues, slow burn for gradual degradation), symptom-based alerts (user-facing impact) rather than cause-based alerts (CPU usage), severity classification (P1 page immediately, P2 notify in 30 minutes, P3 create ticket, P4 dashboard only), and de-duplication logic. 8. **Escalation Path Configuration** — Design the escalation workflow: P1 alerts page the on-call engineer immediately, escalate to secondary after 10 minutes with no acknowledgment, escalate to engineering manager after 30 minutes. Include communication templates for status updates and stakeholder notifications. 9. **Runbook Templates** — Create runbook templates linked to each alert rule: alert description and business impact, diagnostic steps to identify root cause, common causes with resolution procedures, escalation criteria if the runbook does not resolve the issue, and post-resolution verification steps. 10. **On-Call and Incident Process** — Define the on-call rotation structure: primary and secondary on-call schedules, rotation frequency, handoff procedures, on-call compensation guidelines, and incident severity classification criteria with response time expectations per severity. ## INFORMATION ABOUT ME - My application name and description: [INSERT APP NAME — e.g., ShopFlow e-commerce platform, DataPipe analytics service] - My architecture: [INSERT ARCHITECTURE — e.g., 12 microservices on Kubernetes, monolith on EC2, serverless on AWS Lambda] - My current monitoring tools: [INSERT TOOLS — e.g., Prometheus/Grafana, Datadog, CloudWatch, no monitoring currently] - My SLA target: [INSERT SLA — e.g., 99.9% availability, p95 response time under 500ms, error rate below 0.1%] - My team size and on-call capacity: [INSERT TEAM — e.g., 8 engineers, 2-person on-call rotation, follow-the-sun with US and EU teams] - My current pain points: [INSERT PAIN POINTS — e.g., learn about outages from customers, alert fatigue from noisy alerts, cannot trace requests across services] ## RESPONSE FORMAT - Open with an observability architecture overview describing how metrics, logs, and traces integrate for rapid root cause analysis - Present the metrics catalog as a structured table: metric name, type (counter/gauge/histogram), labels, and collection method - Include dashboard designs with described panel layouts, queries, and visualization types for each dashboard tier - Provide alert rules as a table: rule name, condition, severity, notification channel, and linked runbook - Include a runbook template filled out for the 3 most common incident scenarios - Close with an on-call rotation recommendation and a 30-60-90 day implementation roadmap
Or press ⌘C to copy