Call the Winner: Reading a Real Subject Line Test

Marketing Academy · Field Work●Email & Lifecycle

MiniAudit· 20 minutes

Zendesk

Objective: Given a real subject-line A/B test results table, apply the lesson's sample-size and confidence rules to decide whether the result is trustworthy enough to act on.

You run lifecycle email at Zendesk. A subject-line test on last week's product-update send just finished, and your manager wants a one-line verdict before the next campaign, not a guess.

Apply the lesson's practical minimums, not intuition, and say whether this result clears the bar to act on.

Before you start

What you'll need

Free path (everything below is enough to finish)

Google Sheets

FreeRecord the sent/opens/open-rate numbers and the elapsed-time check in one row

Free, fast enough to hand the one-line verdict back the same day

The process

1 step

Step 01 of 01

Sample Size and Statistical Significance

The lesson's practical minimums: at least 1,000 subscribers per variant with 200-300 opens per variant to be a minimum viable test, 10,000+ per variant for a reliable, high-confidence test. Wait a minimum of 48-72 hours before declaring a winner (Litmus recommendation cited in the lesson).

Version A: 5,200 sent, 1,340 opens (25.8%). Version B: 5,150 sent, 1,410 opens (27.4%), test ran for 30 hours. Does this clear the bar to declare a winner today?

Mailchimp— Mailchimp's A/B test results dashboard, showing sends, opens, and open rate per variant

Procedure

Check opens per variant against the 200-300 minimum: both variants have well over 1,300 opens each
Check subscribers per variant against the 1,000 minimum-viable and 10,000 reliable-test thresholds
Check elapsed time against the lesson's 48-72 hour minimum wait

Sample output

Version A: 5,200 sent / 1,340 opens (25.8%)
Version B: 5,150 sent / 1,410 opens (27.4%)
Elapsed: 30 hours

Healthy

Opens per variant clear the 200-300 minimum by a wide margin, and per-variant volume clears the 1,000-subscriber minimum-viable threshold

Unhealthy

Elapsed time (30 hours) falls short of the lesson's 48-72 hour minimum wait before declaring a winner

What this means

The opens and volume are large enough to trust the numbers, but the test hasn't run long enough yet, delayed openers over the next 18-40+ hours could still close or widen the gap.

So what do I do about it?

Symptom	Action	Effort
Manager wants a winner declared at the 30-hour mark	Hold the decision until the 48-hour minimum passes, then re-check the same numbers before sending to the held-back group	5 min

YouYou can do this yourself, no engineering access required.

Final deliverable

A one-line verdict: act now, wait, or re-test, with the specific number that drove the call.

See a reference example

Sample output

Bonobos test-verdict note (reference)

"Opens (1,340 / 1,410) clear the minimum, but elapsed time (30h) is under the 48h floor. Hold the call until tomorrow's 48h mark, re-check before sending to the held-back 10%."

Success criteria

You're done when you can:

Correctly identifies that both variants clear the opens and volume minimums
Correctly flags the elapsed-time shortfall as the actual blocker
Does not declare a winner before the lesson's minimum wait window