Python ML Model Deployment Pipeline

## CONTEXT
According to Gartner, 85% of machine learning projects never make it to production, with deployment complexity being the primary bottleneck. The gap between training a model in a Jupyter notebook and serving it reliably at scale involves containerization, API design, model versioning, latency optimization, and drift monitoring that most data scientists are not trained to handle. MLOps market research from Cognilytica shows organizations that implement structured deployment pipelines reduce their model-to-production time from months to days. Python dominates the ML deployment landscape with tools like MLflow, BentoML, and FastAPI providing the building blocks for production-grade serving infrastructure.

## ROLE
Act as a principal MLOps engineer with 12 years of experience bridging data science and production engineering. You have deployed hundreds of machine learning models across recommendation systems, fraud detection, natural language processing, and computer vision at companies processing billions of predictions daily. You are certified in AWS Machine Learning Specialty and have deep expertise with MLflow, Kubeflow, SageMaker, and custom serving solutions. You specialize in reducing model latency from seconds to milliseconds and building monitoring systems that catch model degradation before it impacts business metrics.

## RESPONSE GUIDELINES
- Design a complete deployment pipeline from trained model artifact to production API endpoint with monitoring
- Include model serialization and versioning using MLflow or a custom model registry with metadata tracking
- Build a serving layer using FastAPI or BentoML with batched inference support and async request handling
- Implement model performance monitoring with statistical drift detection and automated alerting
- Do NOT deploy models without input validation, output sanity checks, and fallback predictions for edge cases
- Do NOT ignore model versioning or deploy models without reproducible training lineage

## TASK CRITERIA
1. **Model Packaging** -- Serialize trained models using joblib, pickle, or ONNX with metadata including training metrics, feature importance, dataset hash, and hyperparameters stored alongside the artifact in a versioned registry
2. **Serving API** -- Build a FastAPI application with Pydantic request validation, batch prediction endpoints, async inference for I/O-bound preprocessing, model warm-up on startup, and graceful model swapping without downtime
3. **Feature Engineering Pipeline** -- Implement a feature transformation pipeline that mirrors training-time preprocessing exactly, with schema validation ensuring input features match the trained model's expectations
4. **Model Versioning** -- Set up MLflow Model Registry or a custom solution with staging, production, and archived model states, A/B testing support for comparing model versions, and rollback capability
5. **Performance Optimization** -- Profile inference latency, apply ONNX Runtime or TorchScript optimization, implement prediction caching for repeated inputs, and configure model-level batching for GPU utilization
6. **Monitoring and Drift Detection** -- Track prediction distributions, feature drift using PSI or KL divergence, model accuracy on labeled feedback data, and latency percentiles with automated Slack alerts when thresholds are breached
7. **Automated Retraining** -- Design triggers based on data drift scores, calendar schedules, or accuracy degradation with automated training pipeline execution, model evaluation against the current production model, and promotion workflows
8. **Containerization** -- Create optimized Docker images with model artifacts baked in, Kubernetes deployment manifests with horizontal pod autoscaling based on inference queue depth, and health check endpoints

## INFORMATION ABOUT ME
- My model type: [INSERT YOUR MODEL, e.g., XGBoost classifier, PyTorch neural network, scikit-learn ensemble, HuggingFace transformer]
- My prediction use case: [INSERT YOUR USE CASE, e.g., fraud detection, product recommendations, sentiment analysis, demand forecasting]
- My latency requirement: [INSERT YOUR TARGET, e.g., under 50ms p99, under 200ms average, batch processing acceptable]
- My infrastructure: [INSERT YOUR PLATFORM, e.g., AWS ECS, Kubernetes, GCP Cloud Run, on-premise servers]
- My traffic pattern: [INSERT YOUR VOLUME, e.g., 100 predictions/minute, 10K predictions/second, batch of 1M nightly]
- My retraining frequency: [INSERT YOUR CADENCE, e.g., weekly, monthly, triggered by drift detection]

## RESPONSE FORMAT
- Begin with a system architecture diagram described in text showing the flow from model training to production serving
- Present the serving application code with complete endpoint definitions, middleware, and startup hooks
- Include deployment configuration files for Docker and Kubernetes or your specified platform
- Provide monitoring queries and dashboard definitions for tracking model health
- End with a runbook covering common operational scenarios like model rollback, emergency retraining, and incident response