Design a comprehensive monitoring and alerting architecture that provides full-stack observability with actionable alerts, reduces noise, and enables rapid incident detection across distributed systems.
## CONTEXT According to Splunk's State of Observability 2024 report, organizations with mature observability practices resolve incidents 69% faster and experience 2.1x fewer outages. Yet the average engineering team deals with alert fatigue from over 60% of alerts being non-actionable noise, leading to critical signals being missed. A well-architected monitoring system combining metrics, logs, and traces with intelligent alerting reduces mean time to detection by 80% and mean time to resolution by 60%, directly protecting revenue and user experience. ## ROLE Act as a Principal Site Reliability Engineer with 12 years of experience designing observability platforms for high-traffic distributed systems. You have built monitoring architectures handling over 5 million metrics per second and 10 TB of logs daily for e-commerce platforms processing billions in annual revenue. You authored internal SRE handbooks adopted across multiple organizations, led alert rationalization programs that reduced paging volume by 75% while improving incident detection, and are an expert in the RED and USE monitoring methodologies. ## RESPONSE GUIDELINES - Design the complete observability stack covering metrics, logs, traces, and synthetic monitoring with specific tool recommendations and integration patterns - Include concrete alert rule examples with thresholds, evaluation windows, and escalation logic using the specified monitoring platform syntax - Define a severity classification system with clear response expectations and routing rules for each level - Provide dashboard design templates for different audiences including executive, on-call engineer, and service owner views - Do NOT create alerts without defined runbook references and clear remediation guidance - Do NOT recommend monitoring approaches that rely solely on threshold-based alerting without anomaly detection capabilities ## TASK CRITERIA 1. **Metrics Architecture** — Design the metrics collection pipeline including instrumentation standards for applications (RED method), infrastructure monitoring (USE method), business KPI tracking, custom metric naming conventions, cardinality management, and retention tier policies 2. **Log Aggregation Design** — Architect the centralized logging system including structured logging standards, collection agents, parsing pipelines, index management, retention policies, and log-to-trace correlation mechanisms 3. **Distributed Tracing Strategy** — Implement end-to-end request tracing including auto-instrumentation configuration, sampling strategies for high-throughput services, trace context propagation standards, and critical path analysis dashboards 4. **Alert Engineering** — Design the alerting framework including severity levels with SLA definitions, alert routing and escalation matrices, suppression and deduplication rules, alert-on-alert prevention, and symptom-based vs cause-based alert hierarchy 5. **Dashboard Framework** — Create a dashboard hierarchy including system health overview, service-level dashboards, infrastructure dashboards, and incident investigation dashboards with specific panel recommendations and query patterns 6. **Synthetic Monitoring** — Implement proactive monitoring including API health checks, user journey simulations, SSL certificate monitoring, DNS resolution checks, and third-party dependency monitoring with specific check intervals and failure thresholds 7. **On-Call Integration** — Design the incident notification pipeline including PagerDuty or equivalent integration, escalation policies, acknowledgment workflows, status page automation, and post-incident review data collection 8. **Cost and Performance Optimization** — Address monitoring infrastructure sizing, metric downsampling strategies, log volume reduction techniques, trace sampling optimization, and total cost of ownership projections ## INFORMATION ABOUT ME - My monitoring platform: [INSERT YOUR monitoring tools e.g., Datadog, Prometheus/Grafana, New Relic, Splunk] - My infrastructure type: [INSERT YOUR infrastructure e.g., Kubernetes, serverless, VMs, hybrid] - My service count: [INSERT YOUR number of services and their primary communication patterns] - My traffic volume: [INSERT YOUR approximate requests per second and data volume] - My current monitoring gaps: [INSERT YOUR biggest observability blind spots today] - My incident frequency: [INSERT YOUR approximate number of incidents per month and their typical causes] ## RESPONSE FORMAT - Start with an observability architecture diagram showing data flow from sources through collection, processing, storage, and visualization layers - Provide specific alert rule templates in the native query language of the specified monitoring platform - Include a dashboard wireframe description for the top three most critical dashboards - Organize sections by implementation priority with estimated effort for each component - End with an alert rationalization checklist and a 60-day rollout timeline
Or press ⌘C to copy