The Research Plan Audit: Catching Bad Methodology Before It Ships

Marketing Academy · Field Work●Conversion Rate Optimization

All projects·From: Customer Research for CRO

MiniAudit· 25 minutes

Grab Holdings

Objective: Given a draft CRO research plan for a real product page, identify methodology flaws against the lesson's four-stage cycle and qualitative-tool best practices.

You're the CRO researcher at Grab. A junior analyst has drafted a research plan for the ride-booking checkout page and wants a sign-off before running it.

Check the plan against the lesson's minimum thresholds and Common Mistakes, then flag every methodology gap.

Before you start

What you'll need

Free path (everything below is enough to finish)

Google Sheets

FreeReview the traffic, timeline, and hypothesis tabs of the draft plan

Free, sufficient for a methodology audit

Google Forms(optional)

FreeCheck survey question design against the exit/post-purchase format

Free tool, sufficient to confirm single-question, open-text survey format

The process

4 steps

Step 01 of 04

Setting a minimum traffic threshold before starting qualitative research

Pages with fewer than 1,000 monthly sessions will not yield reliable heatmap data; reliable A/B test results need at least 25,000 visitors.

The draft plan proposes running heatmaps on the ride-booking checkout page, which gets 420 monthly sessions. Is that enough?

Google Sheets— Traffic tab of research-plan-draft.xlsx

Procedure

Pull the page's monthly session count from the traffic tab
Compare it against the 1,000-session minimum for reliable heatmap data
If below threshold, flag the page or recommend a longer collection window

Sample output

Checkout page: 420 sessions/month
Threshold: 1,000 sessions/month minimum
Status: Below threshold -> heatmap data will not be reliable at this volume

Healthy

The plan is revised to either extend the collection window to 3+ months or pick a higher-traffic page.

Unhealthy

Heatmap research runs anyway on a 3-week window and gets treated as reliable.

What this means

Below-threshold traffic produces noisy, unreliable heatmap patterns, the qualitative-research equivalent of an underpowered A/B test.

So what do I do about it?

Symptom	Action	Effort
A low-traffic page is queued for heatmap research on a short window	Extend the collection window until session count clears 1,000, or reprioritize a higher-traffic page	5 min

YouYou can do this yourself, no engineering access required.

Step 02 of 04

Applying qualitative tools in sequence: heatmaps, then recordings, then surveys

Stage 2 sequences the tools deliberately: start broad (heatmaps), go deep (recordings), then ask directly (surveys), each layer builds on the previous one.

The plan schedules an exit survey to launch on day 1, before any heatmap or recording review. Is that the right order?

Google Sheets— Timeline tab of research-plan-draft.xlsx

Procedure

Check the scheduled order of research methods against the heatmap-recordings-surveys sequence
Flag any survey questions written before heatmap/recording review, since they won't be grounded in observed behavior
Recommend reordering so surveys are written after the first two layers

Sample output

Draft schedule: Day 1 exit survey, Day 8 heatmap install, Day 15 recordings review
Correct order: Day 1 heatmap install, Day 8 recordings review, Day 15 survey (informed by findings)

Healthy

Survey questions reference specific friction points already observed in heatmaps and recordings.

Unhealthy

Survey questions are generic ('How was your experience?') because they were written before any behavioral data existed.

What this means

Writing surveys before heatmaps and recordings means the questions can't target the actual friction points, defeating the point of layering the tools.

So what do I do about it?

Symptom	Action	Effort
Survey questions are generic and not tied to an observed friction point	Reorder the plan so heatmaps and recordings run first, then write survey questions from what they show	30 min

YouYou can do this yourself, no engineering access required.

Step 03 of 04

Balancing exit surveys with post-purchase surveys

Surveying only non-converters is systematically skewed; balancing every exit survey with a post-purchase survey reveals the real decision threshold.

The plan only includes an exit-intent survey. What's missing?

Google Forms— Survey tab of research-plan-draft.xlsx

Procedure

Check whether the plan includes both an exit survey and a post-purchase survey
If only exit surveys are planned, add a one-question post-purchase survey to the order-confirmation screen
Confirm both surveys ask a single open-text question, not multiple choice

Sample output

Planned: Exit survey only ('What stopped you from booking today?')
Missing: Post-purchase survey on the confirmation screen ('Was there anything that almost stopped you from booking?')

Healthy

Both surveys run in parallel, and their contrast is analyzed together.

Unhealthy

Only non-converters are surveyed, so the research never learns what almost stopped people who did convert.

What this means

Exit-only surveys are skewed toward people who were never going to convert regardless of any fix.

So what do I do about it?

Symptom	Action	Effort
Only an exit survey is planned	Add a one-question post-purchase survey and plan to compare both response sets	5 min

YouYou can do this yourself, no engineering access required.

Step 04 of 04

Writing a hypothesis with a 'because' clause

A valid hypothesis follows: 'Because [research finding], we believe [change] will [outcome] for [segment].' Without the because clause it's a guess, not a hypothesis.

The plan's draft test idea reads: 'We should make the Book Now button bigger.' Is this a valid hypothesis?

Google Sheets— Hypothesis backlog tab of research-plan-draft.xlsx

Procedure

Check each backlog row for a 'because [finding]' clause tied to a specific research source
Reject any row that states a change with no cited finding
Send it back for a specific finding before it enters the backlog

Sample output

Rejected: 'Make the Book Now button bigger' (no finding cited)
Revised: 'Because heatmap data shows 60% of sessions never scroll to the current Book Now button position, we believe moving it above the fold will increase checkout starts for mobile users'

Healthy

Every backlog row traces to a specific heatmap, recording, or survey finding.

Unhealthy

Backlog rows are design preferences with no evidence attached.

What this means

A hypothesis without a because clause is opinion-driven regardless of how many research tools were used elsewhere in the plan.

So what do I do about it?

Symptom	Action	Effort
A backlog row has no cited finding	Reject it and require a specific research citation before it's scheduled for testing	5 min

YouYou can do this yourself, no engineering access required.

Final deliverable

A written audit memo listing every methodology gap in the draft plan, with the fix for each.

See a reference example

Sample output

Nubank checkout research-plan audit (excerpt)

1. Traffic check: Page clears 1,000 sessions/month, heatmap data reliable - PASS
2. Sequence check: Surveys scheduled before heatmap review - FAIL, reorder
3. Survey balance: Post-purchase survey missing - FAIL, add
4. Hypothesis check: 2 of 5 backlog rows have no cited finding - FAIL, reject and revise

Success criteria

You're done when you can:

Correctly flags all methodology gaps present in the draft plan
Cites the correct lesson concept or threshold for each flag
Provides a specific fix, not just a criticism