Python Data Pipeline (Pandas)

Design a Python data pipeline using Pandas for the following scenario:

Data Sources: [CSV FILES/DATABASE TABLES/API RESPONSES/EXCEL SHEETS]
Data Volume: [SMALL <1GB/MEDIUM 1-10GB/LARGE >10GB]
Transformation Complexity: [SIMPLE CLEANING/JOINS AND AGGREGATIONS/COMPLEX FEATURE ENGINEERING]
Output Destination: [DATABASE/DATA WAREHOUSE/FILE SYSTEM/DASHBOARD]
Schedule: [ONE-TIME/DAILY/HOURLY/REAL-TIME]
Data Quality Requirements: [BASIC/STRICT WITH VALIDATION]

Build the complete pipeline covering:
1. Data ingestion layer reading from multiple source types with chunked reading for large files, connection pooling for databases, pagination for APIs, and schema detection with automatic dtype optimization to reduce memory footprint
2. Data validation stage using pandera or custom validators to enforce column types, value ranges, uniqueness constraints, null thresholds, and referential integrity checks with clear error reporting for rejected records
3. Transformation chain implementing method chaining patterns for readability, including column renaming, type casting, string normalization, date parsing, categorical encoding, outlier handling, and derived column calculations
4. Join and aggregation logic combining multiple DataFrames with merge strategies for different relationship types, handling duplicate keys, aggregation functions with groupby operations, pivot tables, and window functions for running calculations
5. Memory optimization techniques including category dtypes for low-cardinality strings, downcast numeric types, sparse arrays for mostly-null columns, and chunked processing patterns for datasets exceeding available RAM
6. Output and monitoring with atomic writes to prevent partial data loads, checksum verification, row count reconciliation between source and destination, performance timing for each pipeline stage, and alerting on anomalous record counts

Include logging, configuration files, and a CLI interface for running pipeline stages independently.