Data Observability & Monitoring Stack

## ROLE
You are a data observability engineer who builds monitoring systems that detect data issues before business users do. You've reduced data incidents by 90% through proactive observability.

## OBJECTIVE
Build a data observability stack for [ORGANIZATION]'s data platform running [NUMBER] pipelines on [TECH STACK] to achieve [TARGET] mean time to detection (MTTD) and [TARGET] mean time to resolution (MTTR).

## TASK

### Observability Pillars
- Freshness: is data arriving on time?
- Volume: is the expected amount of data present?
- Schema: has the data structure changed unexpectedly?
- Distribution: are values within expected statistical ranges?
- Lineage: where did data come from, where does it go?

### Pipeline Monitoring
- Job status: success, failure, running, queued for every pipeline run
- Duration tracking: runtime per pipeline, per task — detect slowdowns
- Resource usage: CPU, memory, I/O per job for capacity planning
- SLA monitoring: alert when pipelines risk missing delivery deadlines
- Dependency tracking: impact of upstream failures on downstream pipelines
- Cost monitoring: compute and storage costs per pipeline

### Data Quality Monitoring
- Automated checks: scheduled quality checks running after pipeline completion
- Freshness monitoring: track last update time for critical tables
- Volume monitoring: row count trends with anomaly detection
- Schema change detection: alert on column additions, removals, type changes
- Distribution monitoring: statistical profiles compared to historical baselines
- Custom business rules: domain-specific quality checks (e.g., revenue reconciliation)

### Anomaly Detection
- Statistical methods: z-score, IQR, rolling averages for numeric metrics
- Time series: seasonal decomposition, trend detection, changepoint analysis
- Machine learning: autoencoder or isolation forest for multivariate anomalies
- Threshold tuning: start conservative, tighten as you build confidence
- Contextual awareness: weekends, holidays, known events affect data patterns
- Anomaly scoring: severity ranking to prioritize investigation

### Data Lineage
- Column-level lineage: trace every field from source to final dashboard
- Automated extraction: parse SQL, dbt manifests, Spark plans for lineage
- Impact analysis: when a source changes, what downstream is affected?
- Root cause analysis: when a metric is wrong, trace back to the source of error
- Visual lineage graph: interactive visualization of data flow
- Tools: dbt lineage, OpenLineage, DataHub, Atlan, Monte Carlo

### Alerting Strategy
- Alert routing: right alert to right team via right channel
- Severity levels: P1 (business-critical, immediate), P2 (important, 4 hours), P3 (monitor, next business day)
- Alert fatigue prevention: group related alerts, suppress known issues, tune thresholds
- Escalation: automatic escalation if P1 not acknowledged within SLA
- Channels: Slack for P2/P3, PagerDuty for P1, email for summaries
- Alert context: include what's wrong, what's affected, and suggested actions

### Incident Management
- Incident workflow: detect → triage → investigate → fix → validate → postmortem
- Runbooks: documented procedures for common data incidents
- War room: communication channel for active incidents
- Status page: data health dashboard visible to data consumers
- Postmortem: blameless review of every P1 incident with action items
- Incident tracking: log all incidents for trend analysis and prevention

### Tool Stack Options
- Commercial: Monte Carlo, Anomalo, Bigeye, Metaplane, Sifflet
- Open source: Great Expectations, Elementary, re_data, dbt tests
- Custom: Grafana + custom metrics, Prometheus for pipeline metrics
- Build vs buy: commercial tools are faster to deploy, open source is more customizable
- Integration: tools must integrate with existing data stack (warehouse, orchestrator, BI)

### Observability Dashboard
- Executive view: overall data platform health score
- Domain view: quality metrics per business domain
- Pipeline view: individual pipeline health and performance
- Table view: per-table freshness, volume, quality score
- Incident view: active and recent incidents with status
- Trend view: quality metrics over time, improving or degrading

## OUTPUT FORMAT
Data observability implementation plan with tool selection, monitoring configuration, alerting rules, incident management process, and dashboard designs.

## CONSTRAINTS
- Observability must not significantly impact pipeline performance
- Start simple: basic freshness and volume checks, then add sophistication
- Alert on what matters: not every anomaly is a problem — focus on business impact
- Data governance: observability metadata is sensitive — control access
- Continuous improvement: review and tune observability regularly based on incidents