Building a Complete CRO Test Brief: Hypothesis, Sample Size, and a Paired Tool Stack
Objective: Given Utkarsh Small Finance Bank's account-opening funnel drop-off data, build a full test brief: a hypothesis written with a metric, a sample-size calculation, and a paired experimentation-plus-behavioral tool recommendation that triangulates the quantitative result.
You're the CRO analyst at Utkarsh Small Finance Bank, the Varanasi-founded small finance bank that listed in July 2023 and closed its debut day 92% above its ₹25 IPO price. The digital account-opening funnel loses 62% of users at the KYC-document upload step, and you've been asked to write the test brief before any test is built.
Write a hypothesis with a stated metric, calculate the sample size needed, and name the paired tools (experimentation platform plus behavioral analytics tool) that will run and validate the test.
Before you start
What you'll need
Free path (everything below is enough to finish)
Free tier covers Utkarsh's account-opening funnel traffic and includes the sample-size calculator
Free Basic plan captures a working sample of daily sessions, enough to triangulate a test result
Paid upgrades (optional, faster/deeper)
Starts at $29/month, useful once the free Hotjar session cap is regularly exceeded
The process
2 steps
Step 01 of 02
The lesson's playbook format is: 'Because we saw X, we believe changing Y for Z users will lift conversion by N%,' followed by calculating sample size up front using the platform's built-in calculator and refusing to peek early.
62% of users drop at KYC-document upload, and session recordings show repeated failed re-uploads on mobile. What's the hypothesis, and how many users does the test need before you can call a winner?
Procedure
- State the observed problem in one sentence citing the drop-off rate
- Write the hypothesis in the lesson's format: because/believe/for/by
- Enter baseline conversion rate and minimum detectable effect into VWO's calculator
- Record the required sample size per variant and the estimated days to reach it at current traffic
- Set a rule to not check results before the sample size is reached
HYPOTHESIS: Because 62% of mobile users abandon at KYC-document upload after a failed re-upload attempt, we believe adding inline file-size and format guidance before the upload button will lift step completion by 8% for mobile applicants. SAMPLE SIZE: baseline completion 38%, MDE 5pp, 95% confidence, 80% power → 3,900 users per variant (7,800 total). At current traffic of ~1,100 daily mobile applicants, that's roughly 8 days minimum runtime.
Healthy
Hypothesis names the observed behavior, the specific change, the audience, and a numeric target; sample size is calculated before the test launches.
Unhealthy
Test launches with a vague hypothesis ('improve the upload step') and no sample-size number, so the team calls a winner after 3 days on gut feel.
What this means
A hypothesis without a number is an opinion, and a test without a sample size is a coin flip dressed up as data.
So what do I do about it?
| Symptom | Action | Effort |
|---|---|---|
| Team calls a winner after 3 days at 95% confidence | Calculate sample size before launch and set a minimum runtime rule the team commits to in writing | 30 min |
Step 02 of 02
The lesson's final playbook step is to 'triangulate,' pairing the quantitative result with replays and survey responses before shipping a winning variant.
The A/B test shows the inline guidance variant winning by 6%. What behavioral evidence closes the loop before you ship it to 100% of traffic?
Procedure
- Filter 20-30 session recordings to users who saw the winning variant
- Confirm re-upload attempts dropped compared to the control recordings
- Check Hotjar's post-upload survey responses for mentions of the new guidance text
- Only ship to 100% of traffic once both the number and the behavioral evidence agree
BEHAVIORAL CHECK — winning variant, 25 recordings reviewed Re-upload attempts per session: 0.4 avg (down from 1.7 in control recordings) Survey mentions of the new file-format guidance: 9 of 14 respondents referenced it positively DECISION: Ship to 100% of traffic. Quantitative lift and behavioral evidence agree.
Healthy
Winning variant is shipped only after recordings and survey data confirm the same story as the test's numeric result.
Unhealthy
Winning variant ships purely on the 6% lift number with no recordings reviewed, and it turns out the lift was driven by a tracking bug on one device type.
What this means
A number that agrees with what you see in a recording is trustworthy; a number that stands alone is not.
So what do I do about it?
| Symptom | Action | Effort |
|---|---|---|
| A winning variant ships and the lift disappears a month later | Review 20-30 recordings of the winning variant before rolling out to 100% of traffic | 30 min |
Final deliverable
A complete test brief document: hypothesis, sample-size calculation with runtime estimate, and the paired experimentation-plus-behavioral tool stack.
See a reference example
TEST BRIEF — RateGain Travel Technologies, hotel-partner signup form HYPOTHESIS: Because 44% of partner-hotel signups abandon at the pricing-tier selection step, we believe replacing the three-tier comparison table with a single recommended-tier default will lift signup completion by 6% for first-time visitors. SAMPLE SIZE: baseline completion 51%, MDE 4pp, 95% confidence, 80% power → 2,600 users per variant. At current traffic, estimated runtime is 11 days. BEHAVIORAL PAIRING: VWO for the test, Hotjar recordings on the winning variant filtered to first-time visitors before full rollout.
Success criteria
You're done when you can:
- Hypothesis follows the because/believe/for/by format with a stated numeric target
- Sample size is calculated before the test launches, with a runtime estimate
- A behavioral tool and check step are named before the winning variant ships to 100%