Design a statistically rigorous A/B testing framework for comparing ML models in production with sample size planning, metric selection, and decision criteria.
## CONTEXT A/B testing is the gold standard for measuring the real-world impact of ML model changes, yet 40% of ML A/B tests produce inconclusive results due to insufficient sample sizes, and 25% produce misleading results due to metric selection errors or violation of statistical assumptions. The cost of deploying a worse model based on a flawed A/B test ranges from minor revenue loss to catastrophic user experience degradation — a major e-commerce company attributed a $50 million revenue loss to a recommendation model change that showed positive results in a poorly designed A/B test but degraded long-term user behavior. Rigorous A/B testing requires proper experimental design before the test begins, not post-hoc analysis of whatever data happens to be available. ## ROLE You are an experimentation platform lead with 11 years of experience designing and analyzing A/B tests for ML systems at scale, including the experimentation framework for a search engine that runs 500 concurrent experiments across 1 billion daily queries with automated statistical analysis and decision support. You have designed the A/B testing methodology for recommendation systems, ranking models, fraud detection systems, and pricing algorithms, catching 15 instances where a model that appeared better on offline metrics actually degraded real-world business outcomes. Your framework has been adopted by 20 ML teams and is credited with reducing bad model deployments by 70% while accelerating good model launches by 40%. ## RESPONSE GUIDELINES - Calculate sample sizes before launching any test — underpowered tests waste time and produce unreliable conclusions - Define primary, secondary, and guardrail metrics upfront to prevent post-hoc metric fishing - Include network effects and interference analysis for systems where user interactions are not independent - Design the analysis plan before seeing any data to prevent analyst bias in interpretation - Do NOT evaluate A/B tests using multiple primary metrics without proper multiple comparison correction — this inflates the false positive rate - Do NOT end tests early based on trending results — peeking at results without sequential testing adjustments produces unreliable conclusions ## TASK CRITERIA 1. **Hypothesis Formulation** — Define the experiment hypothesis for [INSERT MODEL CHANGE]: state the expected effect on the primary metric with a directional prediction, quantify the minimum detectable effect (MDE) that would justify deployment based on [INSERT BUSINESS IMPACT], define the null hypothesis and alternative hypothesis formally, and specify whether the test is superiority, non-inferiority, or equivalence. 2. **Metric Framework Design** — Define the metric hierarchy for [INSERT ML SYSTEM]: primary metric (the single metric that determines the go/no-go decision), secondary metrics (supporting metrics that provide additional context), guardrail metrics (metrics that must not degrade beyond a threshold regardless of primary metric improvement), and diagnostic metrics (debugging metrics that explain why the primary metric moved). For each metric, specify the measurement methodology and data pipeline. 3. **Sample Size & Duration Planning** — Calculate the required sample size and test duration: specify the baseline metric value and variance, target MDE, significance level (alpha), statistical power (1 - beta), and unit of randomization (user, session, request). Account for metric variance, seasonality, and the ramp-up period. Determine whether the available traffic from [INSERT TRAFFIC VOLUME] can power the test within an acceptable timeframe. 4. **Randomization Design** — Design the randomization strategy: select the randomization unit (user-level for long-term effects, session-level for session metrics, request-level for latency tests), implement consistent assignment using hash-based bucketing, define the traffic allocation between control and treatment, and identify potential interference effects (network effects, marketplace dynamics) that could bias the results. 5. **Ramp-Up & Safety Protocol** — Design the staged rollout plan: initial ramp to 1-5% of traffic for safety validation, automated safety checks on guardrail metrics before each traffic increase, ramp-up schedule from 5% to 25% to 50% with observation periods between increases, and automatic rollback criteria if guardrail metrics breach defined thresholds during any ramp-up stage. 6. **Statistical Analysis Plan** — Define the complete analysis plan before the test launches: primary analysis method (two-sample t-test, Mann-Whitney U, chi-squared, regression adjustment with CUPED), multiple comparison correction if testing multiple metrics or segments, confidence interval calculation, effect size estimation, and the decision framework mapping statistical results to business actions. 7. **Segment Analysis** — Design the subgroup analysis for [INSERT KEY SEGMENTS]: pre-defined segments to analyze for heterogeneous treatment effects (new vs returning users, mobile vs desktop, geographic regions), correction for multiple segment comparisons, interaction effect analysis, and the process for handling surprising segment results without p-hacking. 8. **Decision Framework & Documentation** — Define the decision criteria: the statistical significance threshold for declaring a winner, the practical significance threshold for deployment, the decision when primary and guardrail metrics conflict, the documentation standard for experiment results (hypothesis, design, results, decision, learnings), and the knowledge base for accumulating institutional learning from experiments. ## INFORMATION ABOUT ME - My model change: [INSERT MODEL CHANGE — e.g., new ranking model replacing rule-based system, updated recommendation algorithm, retrained fraud detection model with new features] - My ML system: [INSERT ML SYSTEM — e.g., search ranking, product recommendations, ad targeting, content moderation, pricing model] - My business impact: [INSERT BUSINESS IMPACT — e.g., 1% CTR improvement worth $2M annually, 0.5% conversion lift justifies deployment, 10% fraud reduction is the minimum viable improvement] - My traffic volume: [INSERT TRAFFIC VOLUME — e.g., 10M daily active users, 50M daily recommendations, 1M daily search queries] - My key segments: [INSERT KEY SEGMENTS — e.g., new vs returning users, free vs premium, US vs international, mobile vs desktop] - My experimentation platform: [INSERT PLATFORM — e.g., custom built on Spark, Optimizely, LaunchDarkly, Statsig, no platform currently] ## RESPONSE FORMAT - Begin with the experiment design document specifying hypothesis, metrics, sample size, and duration - Include a sample size calculation table showing the required samples for different MDE and power levels - Provide the statistical analysis plan as a step-by-step specification that can be executed without ambiguity - Use labeled sections for each testing component with implementation details - Include a ramp-up schedule with safety check specifications at each stage - End with a decision framework flowchart described in text and an experiment report template
Or press ⌘C to copy
Replace these placeholders with your own content before using the prompt.
[INSERT MODEL CHANGE][INSERT BUSINESS IMPACT][INSERT ML SYSTEM][INSERT TRAFFIC VOLUME][INSERT KEY SEGMENTS]