CRO Interview Questions

An A/B test compares two versions of a single element (page A vs. page B, headline A vs. headline B) with all other variables held constant. Multivariate testing (MVT) tests multiple elements changing simultaneously; for example, two headline variants crossed with two CTA variants, producing four combinations. A/B testing requires less traffic to reach statistical significance and produces clearer, more actionable results because the winning variable is isolated. MVT is theoretically more efficient for finding the optimal combination of changes, but it requires much more traffic to power each cell and risks interaction effects where changes interfere with each other.

In practice, most growth teams use A/B tests because they have sufficient traffic for a two-cell test but not enough for six or eight cells. MVT becomes viable on very high-traffic pages (millions of sessions per month) or inside platforms like VWO where the tooling manages traffic allocation automatically.

The most widely used prioritization framework is PIE: Potential (how much improvement could this change drive?), Importance (how much traffic or revenue does this page or element touch?), and Ease (how quickly can this be implemented and tested?). You score each hypothesis 1-10 on each dimension and prioritize by average score.

The limitation of PIE is that it is highly subjective. Alternatives include the ICE framework (Impact, Confidence, Ease) where Confidence replaces Potential and requires evidence rather than intuition, and PXL, developed by CXL, which breaks Importance into more granular questions about above-the-fold placement and traffic share.

In practice, the most important filter before any prioritization framework is a funnel analysis: identify which step in your conversion funnel has the steepest drop-off, because that is where improving conversion yields the highest leverage regardless of how any individual hypothesis scores.

Minimum detectable effect is the smallest true difference between control and variant that your test can reliably detect with a given sample size and statistical confidence level. If you run a test with a sample too small to detect your MDE at 80-90% statistical power, you will likely end up with a false negative: a real improvement that looks like no effect because the test was underpowered. Calculating MDE before the test tells you whether your planned sample size and run duration are sufficient to give a trustworthy result.

For example, if your baseline conversion rate is 3% and you expect the variant to lift it to 3.6%, your MDE is 0.6 percentage points. A sample calculator tells you you need approximately 25,000 visitors per variant at 80% power. If you only get 5,000 per week, running the test for less than five weeks is likely to produce a false result. Teams that skip MDE calculations are the ones who ship 'winning' tests that revert or produce inconsistent results at scale.

A heatmap is a visual representation of where users click, move, or scroll on a page, aggregated across thousands of sessions into a color-coded overlay, with hot colors indicating high interaction density and cool colors indicating low interaction. Click maps show where visitors are clicking, including elements that are not links, which reveals false affordances (images that look clickable but are not) or strong interest in elements that are not yet hyperlinked. Scroll maps show what percentage of visitors reach each section of the page, which is essential for deciding where to place CTAs. Move maps approximate gaze paths on desktop, showing where users focus attention before clicking.

The key limitation is that heatmaps show what users do, not why; they surface friction points that session recordings, user interviews, and A/B tests must then explain and resolve.

In 2026, tools like Hotjar, Microsoft Clarity (free), and Lucky Orange provide heatmaps with automatic rage-click and dead-click detection.

CRO programs that rely exclusively on quantitative data can tell you where users are dropping off but not why.

Qualitative methods add the 'why' layer and are essential for generating high-quality hypotheses before testing.

The three fastest, most actionable qualitative methods are: on-exit surveys (a single question popping when a user is about to leave a page, asking why they are leaving), user interviews (30-minute structured sessions with 5-8 users using a think-aloud protocol), and usability tests (asking a user to complete a specific task on your site while narrating their experience aloud).

In practice, five user interviews consistently surface 80% of the friction points that require testing, making this the highest-ROI qualitative investment before scaling a CRO program. These methods are especially valuable for diagnosing form abandonment and checkout failures, where analytics can identify the drop-off point but only qualitative research reveals whether the cause is trust, clarity, or friction.

Problem: A 65% checkout abandonment is above the industry average of 55-60%, which suggests a specific friction point rather than normal pre-purchase hesitation.

Approach: Start with GA4 to run a step-by-step funnel breakdown within the checkout flow, isolating which specific step has the steepest single drop-off. Then pull Hotjar session recordings filtered to 'checkout page - abandoned' and watch 20-30 sessions, specifically looking for rage clicks, dead clicks, and hesitation pauses. Common findings are: a required field that is confusing, a hidden shipping cost that appears on the final step, a limited payment options display (no PayPal or Apple Pay for mobile users), or a security badge that is missing or broken. Supplement with the exit survey data: if users are citing unexpected costs or trust concerns, those directly point to specific fixes.

Result: in most checkout audits, 65% abandonment reduces to two or three root causes. The highest-impact single fix is almost always unexpected cost disclosure; moving shipping cost earlier in the funnel reduces abandonment by 15-25% in most A/B tests.

Problem: A 10-day test that shows 20% lift and 95% confidence can still be a false positive if the test violated standard experimental hygiene.

Approach: Ask four diagnostic questions before recommending shipping. First: is the test past its minimum required runtime? A/B tests should run for a minimum of two full business cycles (typically two weeks) to account for day-of-week variation in traffic quality and behavior. Second: is there a sample ratio mismatch? Verify that traffic split between control and variant was exactly 50/50; a mismatch indicates a technical problem that invalidates the result. Third: are the secondary metrics healthy? A 20% lift in primary conversion is suspicious if average order value or return rate moved in the opposite direction, which would indicate the variant is accelerating low-quality actions. Fourth: was the test designed with pre-specified statistical power, or did it run until it hit significance? Peeking inflates false positive rate significantly.

Result: if all four checks pass, ship with confidence. If any fail, extend the test or accept that the result is inconclusive.

Problem: A 40% lift is a strong result, but shipping the variant for all traffic means non-paid-search visitors see a headline optimized for a different acquisition context, which may hurt conversion for organic, direct, or social visitors who arrived with different intent.

Approach: First, check whether the headline test result holds or reverses for other traffic segments by querying the test data broken down by acquisition channel. If organic visitors perform better on the control and paid search visitors perform better on the variant, this is a personalization opportunity, not a universal change. Recommend implementing dynamic headline substitution using a tool like Mutiny or a simple UTM-conditional render in the page component: if utm_medium=cpc, show variant; otherwise show control. Set up a post-ship measurement window of four weeks to confirm the personalized version maintains the lift in production traffic.

Result: personalization infrastructure adds engineering complexity, but leaving a 40% lift unshipped because of implementation reluctance is a concrete revenue decision that should be made explicitly, not by default.