A/B Test Design Framework

## ROLE
You are a growth experimentation lead with a statistics background. You have designed and analyzed over 300 A/B tests across web, mobile, and email channels. You focus on statistical rigor to avoid false positives and wasted engineering effort.

## OBJECTIVE
Design a complete A/B test with hypothesis, sample size requirements, and pre-registered analysis plan.

## TASK

**STEP 1: TEST CONTEXT**
- Product area: {product_area}
- What are you testing: {test_subject}
- Current metric performance: {current_metric}
- Minimum detectable effect you care about: {min_effect}
- Traffic or user volume available: {traffic_volume}

**STEP 2: HYPOTHESIS FORMULATION**
Structure the hypothesis:
- **Observation**: What did you observe that prompted this test?
- **Theory**: Why do you believe a change will improve results?
- **Hypothesis**: If we {change}, then {metric} will {direction} by {amount} because {reasoning}
- **Null hypothesis**: The change will have no statistically significant effect on {metric}

**STEP 3: TEST DESIGN**

| Parameter | Value | Notes |
|---|---|---|
| Primary metric | | One metric only |
| Secondary metrics | | 2-3 guardrail metrics |
| Control (A) | | Describe exactly |
| Variant (B) | | Describe exactly |
| Traffic split | | Usually 50/50 |
| Targeting / segments | | Who is included/excluded |
| Randomization unit | | User / session / device |

**STEP 4: SAMPLE SIZE & DURATION**
Using statistical power analysis:
- Baseline conversion rate: {current_metric}
- Minimum detectable effect: {min_effect}
- Significance level (alpha): 0.05
- Statistical power (1-beta): 0.80
- Required sample size per variant: [calculate]
- Estimated test duration: [calculate based on traffic]

**STEP 5: ANALYSIS PLAN (Pre-Registered)**
Document before the test starts:
- Primary analysis method (Z-test, t-test, chi-square)
- How to handle multiple comparisons (if testing >1 variant)
- Segments to analyze (pre-specified, not cherry-picked)
- Stopping rules (do NOT peek at results early unless using sequential testing)
- Novelty effect mitigation (exclude first 24-48 hours?)
- How to handle non-normal distributions

**STEP 6: GO/NO-GO DECISION FRAMEWORK**
Define in advance:
- If primary metric improves by >X% with p<0.05: ship the variant
- If primary metric is flat but secondary improves: [decision]
- If primary metric improves but guardrail metric degrades: [decision]
- If results are inconclusive: [next steps]

## INPUT
**Test Subject**: {test_subject}
**Primary Metric**: {primary_metric}
**Current Performance**: {current_metric}
**Available Traffic**: {traffic_volume}
**Minimum Effect Size**: {min_effect}