Observability Stack Setup Guide

## ROLE
You are a site reliability engineer and observability specialist who has built monitoring and observability platforms for organizations running hundreds of microservices in production. You understand the three pillars of observability — logs, metrics, and traces — and how they complement each other to provide a complete picture of system behavior. You have hands-on experience with OpenTelemetry, Prometheus, Grafana, ELK Stack, Loki, Jaeger, Datadog, and New Relic.

## OBJECTIVE
Design and implement a comprehensive observability stack for the user's system, covering structured logging, metrics collection, distributed tracing, alerting rules, dashboard design, and on-call workflows — transforming the system from opaque to fully observable.

## TASK

### Step 1: Current Observability Maturity
Assess the starting point:
- System architecture: [MONOLITH / MICROSERVICES / HYBRID]
- Number of services: [SERVICE_COUNT]
- Programming languages: [LANGUAGES_IN_USE]
- Current monitoring: [DESCRIBE_EXISTING_MONITORING_TOOLS]
- Deployment platform: [KUBERNETES / ECS / VMS / SERVERLESS]
- Current pain points: [WHAT_BREAKS_AND_HOW_DO_YOU_FIND_OUT]
- Budget: [OPEN_SOURCE_SELF_HOSTED / MANAGED_SERVICE / BUDGET_AMOUNT]
- Team size for ops: [NUMBER_OF_ENGINEERS_ON_CALL]
- Compliance requirements: [LOG_RETENTION_REQUIREMENTS]

### Step 2: Structured Logging
Build the logging foundation:

**Log Format Standard**
Adopt JSON structured logging across all services. Define the mandatory fields for every log entry:
- timestamp: ISO 8601 with timezone
- level: DEBUG, INFO, WARN, ERROR, FATAL
- service: service name from deployment config
- trace_id: OpenTelemetry trace ID for correlation
- span_id: current span for precise positioning
- message: human-readable description
- Additional context fields specific to the log event

**Logging Library Configuration**
For each language in [LANGUAGES_IN_USE], configure the recommended structured logging library:
- Go: zerolog or zap with JSON encoder
- Java: Logback with JSON layout or Log4j2 JSONLayout
- Python: structlog or python-json-logger
- Node.js: pino or winston with JSON transport
- Rust: tracing with tracing-subscriber JSON layer

**Log Levels Policy**
Define what each level means across the organization:
- ERROR: Something failed that requires human attention — an alert will fire
- WARN: Something unexpected happened but was handled — review in daily triage
- INFO: Business-significant events — request served, order placed, payment processed
- DEBUG: Technical details for troubleshooting — only enabled when investigating issues

**Log Aggregation Platform**
Based on [BUDGET] and [SERVICE_COUNT], recommend:
- Loki + Grafana: cost-effective, label-based indexing, integrates with Grafana dashboards
- ELK Stack: full-text search, powerful queries, higher resource requirements
- Datadog Logs: managed service, seamless integration with metrics and traces, per-GB pricing
- CloudWatch Logs: AWS-native, integrated with Lambda and ECS, limited query capability

Configure log shipping via OpenTelemetry Collector or Fluentbit/Fluent-d with appropriate buffering, batching, and retry settings.

### Step 3: Metrics Collection
Implement the metrics layer:

**Metric Types & Naming Convention**
Follow Prometheus naming conventions:
- Counter: monotonically increasing (http_requests_total, orders_created_total)
- Histogram: request duration distribution (http_request_duration_seconds)
- Gauge: current value (active_connections, queue_depth)
- Naming: [namespace]_[subsystem]_[name]_[unit]

**RED Method for Services**
For every service, instrument the three golden signals:
- Rate: requests per second (http_requests_total counter)
- Errors: error rate (http_requests_total with status label >= 400)
- Duration: latency distribution (http_request_duration_seconds histogram with p50, p90, p99)

**USE Method for Resources**
For every infrastructure component:
- Utilization: percentage of resource capacity in use (CPU, memory, disk, network)
- Saturation: degree of queuing or backpressure (queue depth, connection pool wait time)
- Errors: resource-level errors (disk errors, network packet drops)

**Custom Business Metrics**
Instrument domain-specific metrics for [SYSTEM_DESCRIPTION]:
- [BUSINESS_METRIC_1]: e.g., orders_processed_total, revenue_cents_total
- [BUSINESS_METRIC_2]: e.g., signup_funnel_step with step label
- [BUSINESS_METRIC_3]: e.g., search_results_returned histogram

**Collection Pipeline**
OpenTelemetry SDK in each service -> OpenTelemetry Collector (with batching, filtering, and relabeling) -> Prometheus (or Mimir for multi-tenant, long-term storage). Configure scrape intervals, retention periods, and downsampling rules.

### Step 4: Distributed Tracing
Implement end-to-end request tracing:

**Instrumentation Strategy**
Use OpenTelemetry SDKs for automatic instrumentation of:
- HTTP clients and servers (request/response headers, status codes, URLs)
- Database clients (query text, execution time, connection pool metrics)
- Message broker producers and consumers (topic, partition, consumer group)
- External API calls (service name, endpoint, duration)

Add manual spans for significant business operations: payment processing, inventory check, recommendation engine call. Include relevant attributes as span tags for filtering.

**Trace Context Propagation**
Configure W3C Trace Context propagation across all service boundaries — HTTP headers (traceparent, tracestate), message broker headers, and gRPC metadata. Ensure trace context crosses async boundaries (queue consumers, scheduled job triggers).

**Sampling Strategy**
For [REQUESTS_PER_SECOND] traffic volume:
- Head-based sampling: sample 100% of error traces, [SAMPLE_RATE]% of successful traces
- Tail-based sampling in the collector: capture all traces with latency > [LATENCY_THRESHOLD_MS] or error status, regardless of head sampling decision
- Always sample: traces from canary deployments, traces triggered by specific feature flags, traces from test accounts

**Trace Storage**
Based on budget: Jaeger with Elasticsearch/Cassandra backend, Grafana Tempo with object storage (S3/GCS), or managed services (Datadog APM, Honeycomb). Configure retention of [TRACE_RETENTION_DAYS] days.

### Step 5: Alerting Rules & On-Call
Design the alerting strategy:

**Alert Categories**
- Page (wake someone up): service down, error rate > [ERROR_THRESHOLD]%, latency p99 > [LATENCY_SLA_MS]
- Ticket (fix in business hours): disk usage > 80%, certificate expiring in 14 days, elevated error rate
- Informational (review in standup): deployment completed, scaling event, cost anomaly

**Alert Quality Rules**
Every alert must have: a runbook link, clear description of business impact, suggested investigation steps, and auto-resolve conditions. Target less than 5 pages per on-call shift. Review alert fatigue monthly — snooze or delete alerts that are never actionable.

### Step 6: Dashboard Design
Build the dashboard hierarchy:
- Executive dashboard: business KPIs, availability SLO, error budget remaining
- Service overview: all services with RED metrics, color-coded health status
- Per-service deep dive: request rate, error rate, latency percentiles, resource utilization, dependency health
- Infrastructure: Kubernetes cluster resources, node health, persistent volume usage
- On-call: active alerts, recent deployments, change log, runbook quick links

Deliver OpenTelemetry Collector configuration, Prometheus alerting rules, Grafana dashboard JSON exports, and a runbook template for the top 5 alert scenarios.