A/B Testing for Email Marketing
Most email marketers send one version of an email and hope for the best. A/B testing (also called split testing) changes that: you send two versions of the same email to different halves of your list, measure which one wins, and send the winner to everyone else. Done consistently, A/B testing is the single most reliable way to improve your email performance over time. According to Mailchimp's 2024 internal data, brands running regular A/B tests achieve 37% higher click rates than brands that never test.
Quick Summary
- A/B testing means sending two versions of an email to a split audience to find which one performs better.
- Test only ONE element at a time: subject line, sender name, send time, CTA (call to action), or body copy.
- You need at least 1,000 subscribers per variant to get results you can trust, and ideally 10,000+ for high-confidence decisions.
- Wait a minimum of 4 hours (ideally 48 hours) before declaring a winner, early results mislead.
- Only 1 in 8 A/B tests produces a significant improvement, so consistent testing over many sends is what builds real advantage.
Why A/B Testing Beats Guessing
Intuition is wrong more often than marketers admit. The subject line you are "sure" will win often loses. The CTA button color you think does not matter often changes everything. A/B testing replaces opinion with evidence.
The stakes are real. For a list of 50,000 subscribers with an average order value of $60 and a 2% conversion rate, a 10% lift in click-through rate is worth roughly $6,000 in additional revenue per email send. Across 24 sends a year, that is $144,000 from one testing habit.
How common is A/B testing? 93% of US companies run A/B tests on their email marketing campaigns (Marketing Land, 2024). The gap between brands that test and brands that do not is growing. If your competitors are testing and you are not, they are learning while you stay still.
Obama for America campaign (2008, still the gold-standard case study): The Obama presidential campaign A/B tested photos, button text, and layouts on their email sign-up pages. Testing button text alone ("Learn More" vs. "Sign Up Now") produced roughly 4 million additional email sign-ups. Across all tests, the campaign generated an estimated $75 million in additional fundraising. The lesson: even small copy changes at scale produce outsized financial results. This principle applies to every email list, not just political campaigns.
How an Email A/B Test Works
The 45/45/10 split is the most common approach. Some platforms use 50/50 with no held-back group, which is fine for lists under 10,000 where the winner-send step is less meaningful. The key is that the split is random, the platform handles this automatically.
What to Test, and in What Order
Not all tests are equal. Some elements have 10x more impact than others. Test in order of impact:
1. Subject Line (Highest Impact)
The subject line determines whether your email gets opened at all. It affects every downstream metric. Always test this first.
What to compare:
- Question vs. statement: "Want better open rates?" vs. "How to get better open rates"
- Long vs. short: under 6 words vs. 10+ words (Litmus' 2024 State of Email Report found 41-50 character subject lines perform best on mobile)
- Personalized vs. generic: "Surya, your report is ready" vs. "Your report is ready"
- Urgency vs. benefit: "Last 24 hours" vs. "Get 30% off this week"
Emerson's subject line test (2024): Emerson (a manufacturing and technology company) tested two subject lines for the same white paper offer. Version A referenced the white paper directly in the subject line. Version B used a longer, feature-focused approach. Version A won by 23% more opens. The lesson: specificity beats cleverness. Tell people exactly what is inside the email.
2. Sender Name (High Impact)
Many subscribers open based on who sent the email, not what the subject says. Campaign Monitor's research found emails from a real person's name generate 0.53% more opens and 0.23% higher click-through rates versus a company name. That sounds small, but on a list of 100,000 that is 530 additional opens per send.
Test these combinations:
- Company name: "Marketing Academy"
- Person name: "Surya"
- Person plus company: "Surya from Marketing Academy"
3. Send Time and Day (Medium Impact)
Tuesday and Thursday mornings are conventional wisdom, but your audience may differ significantly. Test:
- Morning (7-9 AM) vs. afternoon (1-3 PM) vs. evening (7-9 PM)
- Weekday vs. weekend
- Different days within the same week
For international lists: test in subscribers' local time zones rather than your own.
4. CTA (Call to Action) Button (Medium Impact)
The CTA is what drives conversions. Indeed (the jobs platform) ran a CTA test in 2024 and found that changing form copy to include the word "activate" instead of more generic alternatives increased email sign-ups by 12%. The winning variant was the one their team least expected to win, which is exactly why you test.
Test:
- Button text: "Get started" vs. "Start free trial" vs. "Try it now"
- Button color: within your brand palette
- Button placement: above the fold vs. below vs. both
5. Email Body and Copy (Lower Impact, Harder to Measure)
Body copy changes are harder to isolate because they involve more variables. Only test these after you have exhausted subject line and sender name tests:
- Long copy vs. short copy
- Plain text vs. HTML formatted email
- One image vs. no image
Never test two elements at the same time. If you change both the subject line and the CTA in the same test, you cannot know which change caused the result. The entire value of A/B testing is isolating one variable. This is the most common beginner mistake, and it produces data that is completely useless for future decisions. One test, one variable. No exceptions.
Sample Size: The Rule That Kills Most Tests
The biggest mistake in email A/B testing is drawing conclusions from too few opens. If Version A gets 50 opens and Version B gets 60 opens, that 20% difference could easily be random chance.
You need statistical significance (the probability that the result is real, not luck) before acting on a result.
Practical minimums:
- Minimum viable test: 1,000 subscribers per variant, with at least 200-300 opens per variant
- Reliable test: 10,000+ subscribers per variant (this is what Litmus recommends for 95% confidence)
- High-stakes test (revenue decisions): 95% confidence level required before acting
For a 95% confidence level (the standard used in serious testing), use the free calculator at abtestguide.com. Enter your open counts for each variant and it tells you whether the result is statistically significant.
Do not stop a test early. If Version B is ahead after 1 hour, the temptation is to call it the winner and move on. Early results are almost always misleading. Open rates spike in the first hour (immediate openers) and then trickle in over 24 to 48 hours (delayed openers). Litmus recommends waiting a minimum of 48-72 hours before declaring a winner. Set your test window before you send, and stick to it.
What Metric to Optimize For
Choose your success metric BEFORE you run the test. The three most common are:
- Open rate: the percentage of delivered emails that were opened. Use this when testing subject lines and sender names. Industry average in 2025: 43.46% (MailerLite benchmark data).
- Click-through rate (CTR): the percentage of delivered emails where someone clicked a link. Use this when testing body copy, CTA, and layout. Industry average in 2025: 2.09%.
- Conversion rate: the percentage of recipients who completed a purchase or sign-up. Use this for revenue-focused tests, but requires longer test windows and larger lists.
Optimizing for open rate when your goal is revenue can lead you astray. A clickbait subject line may lift opens but destroy clicks because the email body does not deliver on the promise.
Match the metric to the element. Subject line test? Measure open rate. CTA test? Measure CTR. Offer test? Measure conversion rate. Using the wrong success metric is a silent killer: your test "wins" but your revenue does not move.
Building a Test Log
One test is interesting. Ten tests is a pattern. Fifty tests is a competitive advantage. The brands that win at email marketing over time are the ones that record every test result and build a library of what works for their specific audience.
A simple test log needs six columns:
| Date | Element Tested | Version A | Version B | Winner | Lift |
|---|---|---|---|---|---|
| 2025-01-10 | Subject line | Question format | Statement format | B (statement) | +6.2% open rate |
| 2025-01-24 | Sender name | Brand name | First name | A (brand) | +3.1% open rate |
| 2025-02-07 | Send time | 8 AM Tue | 1 PM Thu | A (8 AM Tue) | +4.8% open rate |
After 10 to 15 tests, patterns emerge. You might discover your audience consistently prefers short subject lines, or that Tuesday 8 AM beats Thursday 2 PM by 12% every time you test it. These are insights your competitors who do not test will never have.
A/B Testing Process Checklist
Platform-Specific Notes
Every major email platform supports A/B testing, but the setup differs:
- Mailchimp: Built-in A/B test wizard. Free plan supports subject line testing only. Paid plans add send time, sender name, and content tests. Automatic winner selection available.
- Klaviyo: Strong A/B testing with automatic winner selection. Best for e-commerce lists with revenue tracking.
- HubSpot: Supports A/B testing on emails and landing pages, with CRM integration so you can track conversion all the way through.
- ConvertKit: Simpler A/B testing focused on subject lines. Good for creator businesses with smaller lists.
- ActiveCampaign: Supports multivariate testing (more than two variants at once) on higher-tier plans.
- Brevo (formerly Sendinblue): Solid A/B testing on free and paid plans. Good option for smaller budgets.
The One-Line Takeaway
Run one test per send, log every result, and within 20 sends you will know your audience better than any competitor who is still guessing.
