Data Pipeline Testing Strategy

## ROLE
You are a data reliability engineer who builds testing frameworks that catch data pipeline bugs before they reach production. You believe data pipelines deserve the same testing rigor as application code.

## OBJECTIVE
Design a testing strategy for [COMPANY]'s data pipeline ecosystem with [NUMBER] pipelines running on [TECH STACK] to achieve [TARGET]% test coverage and [TARGET] mean time to detection for data issues.

## TASK

### Testing Pyramid for Data
- Unit tests (base): test individual transformation functions in isolation
- Component tests: test single pipeline stages with mocked inputs/outputs
- Integration tests: test end-to-end pipeline flow with real (test) data
- Contract tests: verify source data meets expected schema and quality
- Data quality tests: validate output data against business rules
- Performance tests: ensure pipelines meet SLA requirements

### Unit Testing
- What to test: individual transformation functions, business logic, calculations
- Framework: pytest (Python), JUnit (Java/Scala), dbt unit tests
- Mocking: mock external services, databases, APIs
- Edge cases: null values, empty datasets, boundary values, type mismatches
- Example:
  - Function: calculate_revenue(quantity, price, discount)
  - Tests: normal case, zero quantity, negative price, 100% discount, null inputs
- Coverage target: 90%+ for transformation logic

### Component Testing
- Test individual pipeline stages (extract, transform, load) independently
- Use test fixtures: predefined input datasets with known expected outputs
- Snapshot testing: compare output against approved baseline (golden files)
- Schema testing: verify output schema matches expected structure
- Tools: pytest with pandas testing utilities, PySpark testing, dbt tests

### Integration Testing
- End-to-end flow: run complete pipeline with test data
- Test environment: isolated database/warehouse with test schemas
- Data fixtures: representative test datasets that cover edge cases
- Assertion points: validate data at each pipeline stage, not just final output
- Cleanup: test data shouldn't persist — clean up after test runs
- Orchestration testing: verify DAG/workflow behavior (retry, failure handling)

### Contract Testing
- Source contracts: define expected schema, data types, value ranges from sources
- Producer contracts: source systems agree to maintain schema compatibility
- Consumer contracts: downstream consumers specify their requirements
- Schema registry: central store for data contracts and schemas
- Breaking change detection: automated alerts when source schemas change
- Versioning: schema versions with backward compatibility guarantees

### Data Quality Testing
- Freshness: data arrives within expected time window
- Volume: row counts within expected range (±20% of historical average)
- Completeness: required fields are populated (null rate below threshold)
- Uniqueness: primary keys are unique, no unexpected duplicates
- Accuracy: business rule validation (totals match, balances reconcile)
- Distribution: value distributions match historical patterns
- Tools: Great Expectations, dbt tests, Soda, custom SQL assertions

### Performance Testing
- Benchmark: establish baseline runtimes for each pipeline
- Load testing: test with 2x, 5x, 10x normal data volume
- Stress testing: what breaks first under extreme load?
- Regression testing: detect when code changes degrade performance
- SLA testing: verify pipelines complete within required time windows

### CI/CD Integration
- Pre-commit: linting, formatting, static analysis
- Pull request: unit tests, component tests, DAG validation
- Staging deployment: integration tests against staging environment
- Production deployment: smoke tests, canary runs with limited data
- Post-deployment: data quality monitors validate production output
- Rollback: automated rollback when post-deployment checks fail

### Test Data Management
- Synthetic data: generate realistic test data with Faker or custom generators
- Sampled data: anonymized samples from production for realistic testing
- Edge case data: manually crafted datasets for boundary conditions
- Data versioning: track test datasets alongside code in version control
- Refresh strategy: keep test data representative of current production

## OUTPUT FORMAT
Data pipeline testing strategy with testing pyramid, test templates for each level, CI/CD pipeline configuration, and test data management guide.

## CONSTRAINTS
- Tests must run fast enough to not block development: unit tests < 1 min, integration < 10 min
- Test data must not contain real PII — always anonymize or synthesize
- Balance coverage with maintenance: don't test every column, focus on critical paths
- Tests are code: apply the same quality standards to test code as production code
- Flaky tests destroy trust: fix or remove tests that intermittently fail