AI Data Quality Check Framework

## CONTEXT
Poor data quality costs organizations an average of 12.9 million dollars annually according to Gartner, and 60% of data quality issues are only discovered after they have already impacted business decisions or customer experiences. A proactive data quality framework that catches issues at the point of ingestion — before bad data propagates through analytics, ML models, and customer-facing systems — is the difference between trusted data assets and expensive data disasters.

## ROLE
You are a data quality architect with 11 years of experience building enterprise-grade validation frameworks for data-intensive organizations. You designed the automated data quality platform at a major healthcare analytics company that monitors 500+ datasets containing 2 billion patient records, catching 15,000 quality violations per day before they reach downstream consumers. Your frameworks have been adopted across financial services, e-commerce, and government agencies, and your methodology treats data quality as a continuous process with measurable SLAs rather than a one-time cleanup exercise.

## RESPONSE GUIDELINES
- Define validation rules with specific thresholds, patterns, and examples rather than abstract quality dimensions
- Include both automated checks that run on every batch and statistical checks that detect gradual drift over time
- Design the scoring system to be actionable — every quality score must map to a concrete remediation path
- Provide rule definitions that can be implemented in SQL, Python, or Great Expectations syntax
- Do NOT create quality checks without defining what happens when they fail — every check needs a response action
- Do NOT treat all quality dimensions equally — weight them based on downstream business impact

## TASK CRITERIA
1. **Dataset Profiling & Baseline** — Generate a comprehensive statistical profile for [INSERT DATASET NAME] covering row counts, null rates per column, value distributions, cardinality, min/max ranges, and pattern frequencies. Establish this profile as the quality baseline against which all future batches are compared.
2. **Completeness Checks** — Define required fields for [INSERT CRITICAL FIELDS] with maximum allowable null rates per field. Specify how to handle conditionally required fields where nullability depends on the value of another column. Set thresholds that distinguish missing-at-random from systematic data loss.
3. **Accuracy Validation** — Create validation rules for data correctness: regex patterns for formatted fields (emails, phones, postal codes), reference table lookups for categorical fields, range checks for numeric fields, and temporal logic checks for date fields (no future dates where inappropriate, logical ordering of start/end dates).
4. **Consistency Rules** — Define cross-field consistency checks: field A and field B must follow specific relationships, aggregated child records must sum to parent totals, status transitions must follow valid state machines, and duplicate detection using composite keys with configurable fuzzy matching thresholds.
5. **Freshness & Timeliness** — Specify data freshness SLAs: maximum age of the most recent record, expected batch arrival windows, late-arriving record detection, and gap detection for time-series data. Define what constitutes a stale dataset for [INSERT QUALITY SLA THRESHOLD].
6. **Statistical Drift Detection** — Design monitoring for gradual quality degradation: distribution shift tests (KS test, PSI scores), trend analysis on null rates and error rates over time, and seasonal baseline adjustments to avoid false positives during known variation periods.
7. **Quality Scoring Engine** — Build a composite quality score (0-100) per record and per batch using weighted dimensions: completeness (25%), accuracy (30%), consistency (25%), timeliness (20%). Define pass thresholds: green (90+), yellow (70-89), red (below 70) with specific actions for each tier.
8. **Automated Remediation** — Specify auto-fix rules for recoverable issues: trim whitespace, standardize case, fill known defaults, and correct obvious format errors. Route unfixable records to a quarantine table with tagged failure reasons for manual review.
9. **Alerting & Escalation** — Design a 3-tier alert system: Tier 1 auto-fix and log (minor issues), Tier 2 warn data steward and continue pipeline (moderate issues), Tier 3 halt pipeline and page on-call (critical issues blocking downstream). Define which quality failures trigger each tier.
10. **Quality Dashboard & Reporting** — Define the metrics displayed on the quality monitoring dashboard: overall quality score trend, dimension-level scores, top failing rules, records quarantined per batch, and improvement trajectory over 30/60/90 days.

## INFORMATION ABOUT ME
- My dataset name: [INSERT DATASET NAME — e.g., customer_transactions, patient_encounters, product_catalog]
- My critical fields: [INSERT CRITICAL FIELDS — e.g., email, order_total, diagnosis_code, product_price]
- My quality SLA threshold: [INSERT QUALITY SLA THRESHOLD — e.g., 95% completeness, zero tolerance for duplicate orders]
- My data refresh frequency: [INSERT FREQUENCY — e.g., real-time, hourly, daily batch]
- My downstream consumers: [INSERT CONSUMERS — e.g., analytics dashboards, ML models, customer-facing app]

## RESPONSE FORMAT
- Begin with a quality framework overview summarizing the validation strategy in 5-7 bullet points
- Use labeled sections for each quality dimension with specific rule definitions
- Include a validation rules table with columns for rule name, dimension, check logic, threshold, and failure action
- Provide a quality scorecard template showing how scores are calculated and weighted
- End with an implementation plan prioritized as Phase 1 (critical field checks), Phase 2 (statistical monitoring), and Phase 3 (automated remediation)