Cloud Monitoring & Observability

Help me design a cloud monitoring and observability strategy:

Cloud Provider: [AWS/GCP/AZURE]
Application Architecture: [MONOLITH/MICROSERVICES/SERVERLESS/HYBRID]
Current Monitoring Tools: [LIST EXISTING TOOLS]
Primary Concerns: [AVAILABILITY/PERFORMANCE/COST/SECURITY]
Team On-Call Structure: [DESCRIBE ON-CALL ROTATION]
Budget for Observability: [MONTHLY BUDGET]

Design the observability strategy across these six pillars:

1. Metrics Collection and Storage
- Design a metrics strategy covering infrastructure metrics from CloudWatch, Stackdriver, or Azure Monitor and application metrics using Prometheus, StatsD, or OpenTelemetry. Define key performance indicators for each service tier including latency percentiles, error rates, throughput, and saturation. Implement custom business metrics that connect technical performance to business outcomes such as orders per minute, checkout success rate, and API response times. Configure metric retention policies balancing cost with historical analysis needs. Design metric aggregation and downsampling for long-term storage. Plan cardinality management to control costs and query performance. Evaluate managed metrics solutions versus self-hosted Prometheus or InfluxDB.

2. Centralized Logging Architecture
- Design log aggregation using CloudWatch Logs, Cloud Logging, or third-party solutions like Datadog, Splunk, or Elastic. Implement structured logging standards with JSON format, correlation IDs, and consistent field naming. Configure log levels and dynamic log level adjustment for debugging. Design log routing and filtering to separate high-value logs from noise. Implement log retention policies per environment and log type. Plan for log-based alerting and metric extraction from log patterns. Address log security including sensitive data scrubbing and access controls.

3. Distributed Tracing
- Implement distributed tracing using OpenTelemetry, X-Ray, Cloud Trace, or Jaeger. Design trace propagation across service boundaries including HTTP headers, message queues, and async workflows. Configure sampling strategies balancing trace coverage with storage costs using head-based and tail-based sampling. Implement trace-to-log correlation for seamless debugging workflows. Design service maps from trace data for dependency visualization. Plan for trace analysis in common debugging scenarios such as latency spikes, error cascading, and timeout investigations. Evaluate managed tracing services versus self-hosted options.

4. Alerting and Incident Management
- Design an alerting strategy based on SLOs and error budgets rather than arbitrary thresholds. Implement multi-level alerting with warning, critical, and page severity levels. Configure alert routing to appropriate on-call teams using PagerDuty, Opsgenie, or similar. Design alert suppression and deduplication to prevent alert storms. Implement runbook links in alerts for rapid response guidance. Plan for alert tuning and noise reduction as a continuous process. Create SLO dashboards with error budget burn rate alerts.

5. Dashboards and Visualization
- Design a dashboard hierarchy from executive overview to service-level to component-level detail. Create the four golden signals dashboards covering latency, traffic, errors, and saturation for each service. Implement infrastructure dashboards for compute, database, network, and storage health. Design on-call dashboards optimized for incident triage and rapid diagnosis. Create cost dashboards integrated with performance data to identify cost-performance tradeoffs. Implement automated dashboard generation for new services using templates. Plan for dashboard access control and sharing across teams.

6. Observability Culture and Practices
- Define observability requirements for new service deployments as part of production readiness. Implement SLI and SLO definition processes for all customer-facing services. Design post-incident review processes that leverage observability data. Create observability onboarding guides for new team members. Plan for regular observability tool evaluation and optimization. Implement cost management for observability infrastructure itself. Design chaos engineering integration to validate monitoring coverage.

For each pillar provide specific tool recommendations with configurations, example queries and dashboard definitions, cost estimates, implementation priority, and maintenance requirements.