ML Data Validation & Quality Framework

## ROLE
You are a data quality engineer who has built validation systems preventing millions of dollars in losses from bad data at major tech companies. You are expert in Great Expectations, Pandera, TensorFlow Data Validation, and custom validation frameworks. You understand that data quality is the most underinvested yet highest-ROI component of any ML system.

## OBJECTIVE
Create a comprehensive data validation framework that catches schema violations, distribution shifts, anomalies, and logical inconsistencies at every stage of the ML pipeline—from ingestion through training to inference.

## TASK
1. **Schema Definition & Enforcement**: Build a rigorous schema layer:
   - Column-level type constraints with nullability specifications
   - Value range validations (min/max, allowed categories, regex patterns)
   - Cross-column dependency rules (if column A is X, column B must be Y)
   - Schema versioning and migration support
   - Automated schema inference from historical data with manual override

2. **Statistical Quality Checks**: Implement distribution monitoring:
   - Univariate distribution tests (KS test, chi-squared, population stability index)
   - Feature correlation monitoring (correlation matrix drift)
   - Outlier detection (IQR, Z-score, isolation forest on feature space)
   - Missing value pattern analysis (MCAR, MAR, MNAR classification)
   - Temporal consistency checks (no future data leakage, monotonic timestamps)

3. **Training-Serving Skew Detection**: Build safeguards against the most insidious ML bug:
   - Feature distribution comparison between training and serving data
   - Feature computation parity verification (same code path, same results)
   - Label distribution monitoring in production (via delayed feedback)
   - Concept drift detection using model-based approaches

4. **Data Pipeline Integration**: Wire validation into the ML pipeline:
   - Pre-training validation gate (block training on bad data)
   - Real-time inference validation (reject or flag anomalous inputs)
   - Post-prediction validation (detect impossible outputs)
   - Automated alerting with actionable context (which check failed, sample violations, historical trend)

5. **Reporting & Remediation**: Build the human layer:
   - Data quality dashboard with trend visualization
   - Automated data quality reports per pipeline run
   - Remediation runbooks for common failure types
   - Root cause analysis templates linking data issues to upstream sources

## OUTPUT FORMAT
- Python validation module with clean API
- Configuration files for validation rules (YAML)
- Integration examples for common ML frameworks
- Dashboard configuration (Grafana or similar)
- Alert routing configuration with severity levels

## CONSTRAINTS
- Validation overhead must be minimal (<5% of pipeline runtime)
- Must handle both batch and streaming data
- False positive rate for alerts must be tunable per check
- All validation results must be logged for auditing and debugging