The Leaky Bucket or the Loyal Core? Auditing a Cohort Retention Table
Objective: Given a real 4-cohort retention table (monthly signups, weeks 0-12), diagnose whether retention is improving, flat, or decaying, and catch the seasonality trap before calling it a win.
You're the growth analyst at Nykaa reviewing Q1 cohort retention after a redesigned onboarding flow shipped in February.
Read the curve shape across four monthly cohorts, decide whether the February onboarding change actually worked, and flag any cohort whose result looks inflated by seasonality.
Before you start
What you'll need
Free path (everything below is enough to finish)
Free, handles a 4x5 table and line charts with no setup
The process
2 steps
Step 01 of 02
The lesson's Step 4 says a healthy retention curve drops steeply in the first few periods, then flattens. The flat tail is the loyal core; a curve that never stops falling is a leaky bucket.
Four monthly cohorts are given below. Which one shows a genuine flat tail by week 12, and which is still falling with no sign of leveling off?
Procedure
- Import retention-cohorts.csv (4 rows: Nov, Dec, Jan, Feb cohorts; columns Week 0/1/4/8/12)
- Plot each row as a line series on one chart
- Compute the week 8-to-week 12 drop for each cohort (should shrink if the curve is flattening)
- Rank the four cohorts from most flattened to still-falling
Cohort W0 W1 W4 W8 W12 W8→W12 drop Nov 100% 44% 26% 19% 14% -5.0 pts Dec 100% 47% 29% 22% 18% -4.0 pts Jan 100% 50% 33% 28% 25% -3.0 pts Feb 100% 56% 39% 34% 32% -2.0 pts
Healthy
The Feb cohort's week 8-to-12 drop (-2.0 pts) is the smallest of the four, and shrinking with each newer cohort, that is a flattening curve.
Unhealthy
The Nov cohort is still losing 5 points between week 8 and 12, with no sign the drop is slowing, that is a leaky bucket, not a loyal core.
What this means
A cohort's story is in its slope near the end of the table, not its raw week-12 number.
So what do I do about it?
| Symptom | Action | Effort |
|---|---|---|
| A cohort's week 8-to-12 drop is roughly the same size as its week 4-to-8 drop | Treat that cohort as still decaying; wait for more weeks before calling it flat | 5 min |
Step 02 of 02
The lesson's Step 6 says the entire point of a cohort table is the vertical comparison: if a later cohort beats an earlier one at the same week number, the change that shipped between them worked.
Onboarding changed in February. Comparing the Jan and Feb rows at week 4 (33% vs 39%), is that a real onboarding win, or could November-to-January's dip actually be a seasonality artifact worth checking first?
Procedure
- Isolate the Jan and Feb rows
- Compare week 1, week 4, and week 8 values side by side
- Note that the Nov cohort overlaps India's festive shopping season, then flag it as a possible outlier rather than a baseline
Week 4 comparison Jan (pre-onboarding): 33% Feb (post-onboarding): 39% (+6 pts) Note: Nov cohort signed up during Diwali sale traffic, its W1 (44%) may be inflated by one-time deal-seekers, not representative of a normal month.
Healthy
Feb beats Jan by 6 points at week 4, and Nov is flagged separately instead of being averaged into the baseline.
Unhealthy
Blending the Nov festive-season cohort into an 'average pre-onboarding retention' number and comparing Feb against that blend.
What this means
Vertical, same-week comparisons are the whole point of the table; seasonality is the one confound that can fake a vertical win.
So what do I do about it?
| Symptom | Action | Effort |
|---|---|---|
| One cohort's signup window overlaps a major sale or holiday period | Footnote that cohort as non-baseline instead of folding it into a trend line | 5 min |
Final deliverable
A one-paragraph verdict memo: is post-onboarding retention improving, which cohort proves it, and which cohort should be excluded as a seasonality outlier.
See a reference example
Lenskart, Q1 cohort verdict (excerpt) VERDICT: IMPROVING. Feb cohort (post-onboarding) beats Jan at every checkpoint (W4: +6pts, W8: +6pts). The Nov cohort is excluded from the baseline trend, its W1 spike coincides with the festive sale and does not reflect normal-month behavior.
Success criteria
You're done when you can:
- Correctly identifies which cohort has the flattest tail using the W8-to-W12 drop, not the raw W12 number
- Makes a same-week vertical comparison between Jan and Feb
- Flags the Nov cohort's seasonality risk instead of folding it into the trend