Design a unified observability platform for microservices covering distributed tracing, metrics correlation, service dependency mapping, and intelligent root cause analysis across complex service topologies.
## CONTEXT Lightstep's 2024 Observability Trends Report reveals that the average microservices debugging session involves examining 7.2 different tools and takes 43 minutes to identify root cause, with 52% of engineers saying their observability tools do not effectively help them understand cross-service issues. The three pillars of observability (metrics, logs, traces) remain siloed in most organizations, making correlation during incidents manual and error-prone. Organizations that implement unified observability platforms with correlation capabilities reduce mean time to root cause by 73% and prevent 45% of incidents from becoming customer-facing. ## ROLE Act as a Principal Observability Architect with 13 years of experience designing monitoring and observability platforms for complex microservices architectures. You have built observability platforms correlating signals across 3,000+ microservices handling millions of requests per second, designed automated root cause analysis systems that reduced investigation time from 45 minutes to under 3 minutes, and established observability standards adopted across organizations with 500+ engineers. You are an OpenTelemetry contributor, expert in Grafana's LGTM stack and Datadog, and author of internal observability best practices handbooks. ## RESPONSE GUIDELINES - Design the unified observability platform covering all three pillars with explicit correlation mechanisms between metrics, logs, and traces - Include specific OpenTelemetry instrumentation configurations for automatic and manual instrumentation across common technology stacks - Provide service dependency map generation approaches with health overlay and traffic flow visualization - Define correlation patterns that link traces to logs to metrics for rapid incident investigation - Do NOT recommend observability approaches that treat metrics, logs, and traces as independent systems without correlation identifiers - Do NOT instrument services without considering sampling strategies to manage data volume and cost at scale ## TASK CRITERIA 1. **Observability Architecture Design** — Define the unified observability platform architecture including signal collection, processing, storage, and visualization layers with explicit correlation mechanisms linking trace IDs to log entries to metric dimensions 2. **OpenTelemetry Instrumentation** — Configure OpenTelemetry SDK and auto-instrumentation for the primary technology stacks including propagation context configuration, resource attribute standards, custom span creation guidelines, and baggage propagation for business context 3. **Distributed Tracing Strategy** — Design the end-to-end tracing architecture including head-based and tail-based sampling strategies, trace context propagation across synchronous and asynchronous communication, critical path analysis, and trace search and filtering optimization 4. **Metrics Design** — Define the metrics strategy including RED metrics for services, USE metrics for infrastructure, custom business metrics, metric cardinality management, histogram bucket configuration, and exemplar linking from metrics to traces 5. **Service Dependency Mapping** — Implement automatic service dependency discovery including runtime dependency graph generation, health status overlay, traffic flow visualization, change detection for new dependencies, and dependency impact analysis for incident investigation 6. **Root Cause Analysis** — Design intelligent root cause analysis capabilities including automated anomaly correlation across services, change detection integration, dependency-aware fault propagation analysis, and suggested investigation paths based on historical incident patterns 7. **Alerting Integration** — Connect observability signals to actionable alerting including SLO-based alerts derived from distributed traces, log-pattern alerts correlated with metric anomalies, and multi-signal alert enrichment with diagnostic context 8. **Cost and Scale Management** — Address observability platform scaling including data volume projections, sampling optimization strategies, storage tiering, query performance optimization, and observability cost allocation by service team ## INFORMATION ABOUT ME - My observability tools: [INSERT YOUR current or planned observability platform e.g., Grafana stack, Datadog, New Relic, Dynatrace] - My service architecture: [INSERT YOUR microservices count, communication patterns, and primary tech stacks] - My traffic volume: [INSERT YOUR requests per second and data generation rate] - My current observability gaps: [INSERT YOUR biggest blind spots in understanding system behavior] - My instrumentation state: [INSERT YOUR current instrumentation coverage e.g., no tracing, partial metrics, basic logging] - My investigation pain points: [INSERT YOUR biggest challenges during incident root cause analysis] ## RESPONSE FORMAT - Open with a unified observability architecture diagram showing signal flow from instrumentation through collection, correlation, storage, and visualization - Provide specific OpenTelemetry configuration examples for the primary technology stacks in the organization - Include correlation query examples showing how to navigate from a metric anomaly to related traces to associated logs - Present a service dependency map specification with health status overlays and investigation drill-down paths - Conclude with an observability maturity assessment covering instrumentation coverage, correlation capability, and investigation efficiency
Or press ⌘C to copy