Data Quality Assessment

You are a data quality specialist performing an assessment for an organization. Based on the following context, create a thorough data quality evaluation framework.

Dataset(s) Under Review: [DESCRIBE THE DATASETS - tables, sources, domains]
Data Usage: [HOW THE DATA IS USED - reporting, ML models, operational decisions]
Known Issues: [ANY DATA QUALITY PROBLEMS ALREADY IDENTIFIED]
Stakeholders: [WHO RELIES ON THIS DATA]
Regulatory Requirements: [ANY COMPLIANCE NEEDS - GDPR, HIPAA, SOX]
Current Data Infrastructure: [DATABASE/WAREHOUSE/LAKE AND TOOLS IN USE]

Perform the assessment across these six sections:

1. DATA QUALITY DIMENSIONS AND METRICS
Define the six core data quality dimensions and establish measurable metrics for each. For completeness, calculate null rates and missing value patterns at column and row levels. For accuracy, design validation rules against known reference data or business rules. For consistency, check for contradictions across related fields and across source systems. For timeliness, measure data freshness, latency from source to destination, and update frequency compliance. For uniqueness, identify duplicate records using exact and fuzzy matching. For validity, check format compliance, range adherence, and referential integrity. Provide Python code using pandas and great_expectations to compute each metric automatically.

2. PROFILING AND BASELINE ESTABLISHMENT
Conduct detailed data profiling to establish the current state of quality. Generate statistical profiles for every column including data type distribution, unique value counts, pattern analysis for strings, and value frequency distributions. Identify columns with suspicious patterns such as default values overrepresenting, dates clustering on boundaries, or numeric values showing unusual rounding. Create a quality baseline dashboard that captures the current state against which future improvements will be measured. Provide code to generate automated profiling reports using ydata-profiling or custom pandas scripts.

3. ROOT CAUSE ANALYSIS
Investigate the sources of identified data quality issues. Trace problems upstream through the data lineage to identify where corruption, loss, or inconsistency is introduced. Categorize root causes into system issues such as ETL bugs or schema mismatches, process issues such as manual entry errors or missing validation, and data source issues such as vendor data quality or API changes. For each identified issue, document the impact on downstream consumers including affected reports, models, and decisions. Create a fishbone diagram template for systematic root cause investigation.

4. DATA QUALITY RULES AND AUTOMATED TESTING
Build a comprehensive suite of automated data quality checks. Implement schema validation tests to catch structural changes. Create business rule tests such as order date must precede ship date, age must be between 0 and 120, and revenue must equal quantity times price. Build referential integrity checks across related tables. Implement statistical tests to detect anomalous distributions, sudden shifts in metrics, and outlier records. Provide a complete great_expectations setup including expectation suites, data docs generation, and checkpoint configuration. Show how to integrate these checks into the data pipeline so they run automatically on every data load.

5. DATA QUALITY SCORECARD AND REPORTING
Design a data quality scorecard that communicates quality status to both technical and business stakeholders. Create an overall quality score aggregated from dimension-level scores with configurable weighting based on business priorities. Build trend charts showing quality improvement or degradation over time. Create drill-down views from overall score to dimension to specific failing rules to affected records. Design alert thresholds that trigger notifications when quality drops below acceptable levels. Provide templates for executive summaries, detailed technical reports, and data consumer notifications.

6. REMEDIATION AND GOVERNANCE PLAN
Create a prioritized remediation roadmap based on business impact and effort. For each quality issue, define the remediation approach: fix at source, implement cleansing logic, apply business rules in the transformation layer, or accept and document the limitation. Design data quality monitoring processes for ongoing governance including ownership assignment, SLA definitions, and escalation procedures. Establish a data quality council charter with roles, meeting cadence, and decision-making authority. Create templates for data quality incident reports and resolution tracking.