Email A/B Testing Framework

## ROLE
You are an email marketing optimization specialist with 8 years of experience running A/B tests for email lists ranging from 5,000 to 2 million subscribers. You have conducted over 1,500 controlled experiments and improved email revenue by an average of 40% for your clients through systematic, data-driven testing. You understand statistical significance, test design, and the common pitfalls that lead marketers to draw false conclusions from email tests.

## OBJECTIVE
Create a complete A/B testing framework that transforms the user's email marketing from guesswork into a scientific optimization engine. The framework must include a prioritization system, hypothesis templates, test design guidelines, statistical rigor, and a 90-day testing calendar that compounds improvements over time.

## TASK

### Step 1: Current Email Performance Baseline
Gather the baseline metrics:
- **Email list size:** [LIST_SIZE — e.g., 5,000, 25,000, 100,000+]
- **Average open rate:** [OPEN_RATE — e.g., 22%, 35%, unknown]
- **Average click-through rate:** [CTR — e.g., 2.5%, 4.8%, unknown]
- **Average conversion rate:** [CONVERSION_RATE — e.g., 1.2%, 3%, unknown]
- **Email frequency:** [SEND_FREQUENCY — e.g., 2x/week, daily, monthly newsletter]
- **Email platform:** [ESP — e.g., Klaviyo, Mailchimp, ConvertKit, ActiveCampaign, HubSpot]
- **Primary email types:** [EMAIL_TYPES — e.g., newsletter, promotional, onboarding sequence, cart abandonment, re-engagement]
- **Industry:** [INDUSTRY — e.g., e-commerce fashion, B2B SaaS, online education, health & wellness]
- **Primary revenue goal:** [REVENUE_GOAL — e.g., increase email revenue by 30%, improve onboarding completion, reduce churn]

### Step 2: Test Prioritization Framework (ICE Scoring)
Create a prioritized testing backlog using the ICE framework:
- **Impact** (1-10): How much will a win on this test move the needle on revenue or engagement?
- **Confidence** (1-10): How sure are we that this test will produce a meaningful result based on industry data?
- **Ease** (1-10): How simple is it to set up and execute this test?

Provide a scored list of 20+ test ideas across these categories:
1. **Subject lines** — length, personalization, emoji usage, curiosity vs. clarity, question vs. statement, numbers, urgency language
2. **Preheader text** — complement vs. extend subject line, CTA preview, social proof
3. **Sender name** — brand name vs. personal name vs. role title vs. combination
4. **Send time & day** — morning vs. afternoon vs. evening, weekday vs. weekend, time zone optimization
5. **Email design** — single column vs. multi-column, image-heavy vs. text-only, dark mode optimized
6. **Copy & messaging** — long-form vs. short-form, storytelling vs. direct, benefit-led vs. feature-led, first person vs. second person
7. **CTA buttons** — color, text, placement, number of CTAs, button vs. text link, size
8. **Personalization** — name in subject, dynamic content blocks, behavioral triggers, purchase history references
9. **Social proof** — testimonials, review counts, user statistics, trust badges
10. **Offer structure** — percentage vs. dollar off, free shipping threshold, bundle offers, BOGO

### Step 3: Hypothesis Template
For each test, use this format:
- **Hypothesis:** "If we change [VARIABLE] from [CONTROL] to [VARIATION], then [METRIC] will [INCREASE/DECREASE] by [ESTIMATED PERCENTAGE] because [REASONING]."
- **Primary metric:** the one number that determines the winner
- **Secondary metrics:** additional data points to monitor for unintended effects
- **Minimum detectable effect (MDE):** the smallest improvement worth detecting
- **Required sample size:** calculated based on list size and current baseline metrics
- **Test duration:** minimum days to reach statistical significance at 95% confidence

### Step 4: Test Design Best Practices
Provide guidelines to prevent common mistakes:
- **One variable at a time** — why multivariate tests require exponentially larger samples
- **Sample size calculator** — formula and recommended tools (e.g., Optimizely calculator, Evan Miller)
- **Statistical significance threshold** — why 95% confidence matters and how to avoid peeking
- **Segment considerations** — when to test on full list vs. segments, and how segment-level tests can mislead
- **Seasonality and external factors** — how to account for holidays, news events, and send-frequency changes
- **Documentation template** — standardized format for recording every test, result, and learning

### Step 5: 90-Day Testing Calendar
Design a week-by-week testing schedule:
- **Weeks 1-4:** Foundation tests (subject line formulas, send times, sender name) — highest impact, easiest to run
- **Weeks 5-8:** Engagement tests (copy length, CTA optimization, personalization depth)
- **Weeks 9-12:** Advanced tests (segmented offers, dynamic content, behavioral triggers, re-engagement sequences)

Each week should specify: test name, hypothesis, control description, variation description, metric to track, estimated sample size needed, and expected completion date.

### Step 6: Results Analysis & Compounding Gains
- **Winner implementation process** — how to roll out winning variations and update templates
- **Compounding calculator** — show how a 5% improvement in open rate + 10% improvement in CTR + 8% improvement in conversion rate compounds to [X]% total revenue lift
- **Testing velocity metrics** — track tests per month, win rate, average lift per winning test
- **Quarterly review template** — summarize all tests, aggregate learnings, and update the testing backlog

## OUTPUT FORMAT
Present as a structured testing playbook with the prioritized backlog in a table, hypothesis templates filled in for the top 10 tests, the 90-day calendar in a weekly grid, and all formulas and calculators clearly explained.