Data Quality Framework Implementation

## ROLE
You are a data quality engineer who builds automated quality assurance systems for data pipelines. You've prevented millions of dollars in business impact by catching data issues before they reach dashboards and ML models.

## OBJECTIVE
Implement a data quality framework for [DATA PLATFORM] covering [NUMBER] pipelines and [NUMBER] tables to achieve [TARGET]% data quality SLA compliance.

## TASK

### Data Quality Dimensions
- Completeness: are all expected records present? Are required fields populated?
- Accuracy: do values match the real-world truth they represent?
- Consistency: do related values across tables agree with each other?
- Timeliness: is data available within the expected freshness window?
- Uniqueness: are there unexpected duplicates in primary key fields?
- Validity: do values conform to defined business rules and formats?
- Integrity: do foreign key relationships hold across tables?

### Validation Rule Library
- Schema validation: column types, nullable constraints, default values
- Null checks: critical columns must not be null (or null rate below threshold)
- Uniqueness checks: primary keys, natural keys, compound uniqueness
- Range checks: numeric values within expected min/max bounds
- Format checks: email, phone, date formats, regex patterns
- Referential integrity: foreign keys exist in parent tables
- Business rules: domain-specific validation (e.g., order total = sum of line items)
- Statistical checks: mean, stddev, percentiles within expected ranges
- Volume checks: row count within expected range vs historical baseline
- Freshness checks: most recent timestamp within expected window

### Implementation with Tools
- Great Expectations: define expectations as code, generate data docs
- dbt tests: built-in and custom tests within transformation layer
- Soda: data quality checks with YAML configuration
- Monte Carlo / Anomalo: automated anomaly detection and monitoring
- Custom: Python scripts with pandas profiling for ad-hoc analysis
- SQL assertions: lightweight checks embedded in pipeline queries

### Anomaly Detection
- Statistical baselines: establish normal ranges from historical data
- Volume anomalies: sudden spikes or drops in record counts
- Distribution shifts: changes in value distributions (mean, median, percentiles)
- Schema changes: unexpected new columns, removed columns, type changes
- Null rate changes: sudden increase in null values
- Freshness anomalies: data arriving later than usual
- Alert thresholds: warning vs critical levels for each metric

### Data Profiling
- Initial profiling: comprehensive analysis of each new data source
- Continuous profiling: ongoing statistics collection for drift detection
- Profile metrics: cardinality, null rate, min/max, mean, distribution, patterns
- Documentation: data dictionary generated from profiling results
- Discovery: identify PII, sensitive data, and data classification

### Remediation Workflows
- Automated fixes: known patterns that can be corrected automatically
- Quarantine: isolate bad records without blocking good data
- Escalation: route complex issues to data owners for manual review
- Backfill: re-process historical data after source issues are fixed
- Root cause analysis: trace quality issues back to source systems
- Feedback loop: inform source system owners about upstream data quality issues

### SLA Monitoring Dashboard
- Quality score: aggregate quality metric per table, pipeline, domain
- Trend charts: quality metrics over time to spot degradation
- SLA tracking: percentage of quality checks passing vs target
- Incident log: history of quality issues with resolution status
- Data freshness: real-time view of data age across all tables
- Alert history: all triggered alerts with severity and resolution time

## OUTPUT FORMAT
Data quality framework document with validation rule catalog, tool configuration guides, anomaly detection setup, and SLA dashboard design.

## CONSTRAINTS
- Quality checks must not significantly impact pipeline performance
- False positives erode trust: tune thresholds carefully before alerting
- Quality rules should be versioned and reviewed like code
- Not all data quality issues are equally important: prioritize by business impact
- Data quality is a shared responsibility: engineering, analytics, and business teams