The Holdout Readout: Auditing Zomato's Geo-Lift Numbers
Objective: Given a supplied 8-city-pair geo holdout dataset for Zomato's food delivery ads (test spend, test conversions, control conversions), calculate iROAS per pair and flag any pair that looks contaminated or underpowered.
You're the growth analyst at Zomato reviewing a just-completed 5-week geo holdout test across 8 matched city pairs before the quarterly channel review.
Apply the lesson's iROAS formula and contamination checks to the raw city-pair numbers, don't just average the headline lift.
Before you start
What you'll need
Free path (everything below is enough to finish)
Free, handles an 8-row city-pair table and simple line charts without any statistics software
The process
2 steps
Step 01 of 02
iROAS = (Revenue in test group minus Revenue in control group) / Ad spend in test group. Platform ROAS is attribution; iROAS is causation.
Pair 3 (Pune test / Nashik control): test group had 620 conversions at ₹450 AOV and ₹95,000 spend, control had 540 conversions. What's the iROAS?
Procedure
- Enter test and control conversions and spend for each of the 8 pairs
- Compute incremental conversions = test conversions minus control conversions
- Compute iROAS = (incremental conversions x AOV) / spend for each pair
Pair TestConv ControlConv IncrConv AOV Spend iROAS Pune/Nashik 620 540 80 450 95000 0.38x Surat/Vadodara 710 410 300 450 88000 1.53x
Healthy
Pairs like Surat/Vadodara (1.53x) clear the bar to keep or scale; Pune/Nashik (0.38x) does not.
Unhealthy
Averaging all 8 pairs into one iROAS number and missing that one pair is dragging the average down for a fixable reason.
What this means
A per-pair iROAS calculation surfaces which specific markets are working, a single blended number hides it.
So what do I do about it?
| Symptom | Action | Effort |
|---|---|---|
| One city pair's iROAS is far below the others | Check that pair for spillover or a demand mismatch before pooling it into the headline number | 30 min |
Step 02 of 02
The lesson's Common Mistakes: if a paused market shares a media market with its test counterpart, the control is contaminated.
Pune and Nashik are 210km apart with separate media markets, but pair 3's control-region conversions rose 18% mid-test with no local promotion. What do you check first?
Procedure
- Plot weekly control-region conversions for the flagged pair
- Cross-check for any city-wide event, competitor promo, or shared ad exposure mid-test
- Mark the pair as contaminated if no local explanation exists and note it separately from the clean pairs
Week Pune(test)conv Nashik(control)conv 1 140 95 2 155 98 3 150 142 <- jump, no known cause 4 175 138
Healthy
The contaminated pair is excluded from the headline iROAS and reported separately with a note, not silently blended in.
Unhealthy
Averaging the contaminated pair's low iROAS into the topline number and concluding the channel underperforms.
What this means
An unexplained control-region jump means the control isn't clean, don't treat its iROAS as a real read on the channel.
So what do I do about it?
| Symptom | Action | Effort |
|---|---|---|
| A control region's conversions move without a known local cause mid-test | Exclude that pair from the headline calculation and investigate separately | 30 min |
Final deliverable
A per-city-pair iROAS table with contaminated pairs flagged and excluded from the headline number.
See a reference example
Grab food delivery geo test, city-pair iROAS (excerpt) Pair iROAS Status Cebu/Davao 2.1x Clean, scale Manila/QC 0.6x Clean, hold Iloilo/Bacolod n/a Contaminated, excluded (shared media market)
Success criteria
You're done when you can:
- Computes iROAS correctly per city pair using the test/control gap formula
- Correctly identifies and excludes the contaminated pair from the headline number