Find the Magic Number: Cohort Analysis on a Real Retention Split

Marketing Academy · Field Work●Growth Marketing

All projects·From: Activation & Aha Moments

MiniAudit· 25 minutes

Glossybox

Objective: Given cohort data comparing retained vs. churned users by an early behavior, find the specific frequency and time window where the two curves diverge.

You're the lifecycle analyst at Glossybox. Product suspects 'rated at least 2 products in the first box' predicts subscription renewal, but nobody has actually tested it against the numbers.

Compare Month 3 retention for users above and below the candidate threshold. Confirm the divergence is real before it becomes an onboarding goal.

Before you start

What you'll need

Free path (everything below is enough to finish)

Google Sheets

FreeSplit cohorts by threshold and compare retention percentages

Filtering and a COUNTIF-style split handle this comparison without any analytics tool setup

Paid upgrades (optional, faster/deeper)

Amplitude(optional)

FreemiumRun this same cohort comparison at full scale across many candidate behaviors at once

Built-in cohort comparison views designed for exactly this analysis, per the lesson

The process

1 step

Step 01 of 01

Finding the magic number with cohort analysis

The lesson defines the magic number as a behavior, frequency, and time window combination where retention curves for retained vs. churned users visibly diverge, found only by comparing real cohorts, not by guessing.

Users who rated 2+ products in their first box show 61% Month 3 retention. Users who rated 0-1 products show 24%. Is 'rate 2+ products in box 1' a real magic number, and what should the onboarding goal become?

Google Sheets— Import cohort-retention.csv, split rows by the rating-count threshold, and compare the Month 3 retention percentage for each group side by side.

Procedure

Import cohort-retention.csv with columns: user_id, products_rated_box1, retained_month3
Split users into two groups: rated 2+ products vs. rated 0-1 products
Calculate Month 3 retention percentage for each group separately
Confirm the two percentages are far enough apart to count as a real divergence, not noise

Sample output

Glossybox Box 1 cohort split (n=800)
  Rated 2+ products (n=310): 61% retained at Month 3
  Rated 0-1 products (n=490): 24% retained at Month 3
  Gap: 37 percentage points -> genuine divergence, this is a real magic number candidate

Healthy

A 37-point retention gap between the two groups is wide enough to confidently set 'rate 2+ products in box 1' as the activation event and design onboarding around it.

Unhealthy

Treating a 3-4 point gap as a magic number and rebuilding onboarding around noise that won't replicate next quarter.

What this means

A real magic number produces a gap wide enough that nobody on the team would argue it's coincidence. Small gaps mean keep looking.

So what do I do about it?

Symptom	Action	Effort
Onboarding has no clear activation goal beyond 'complete signup'	Add a first-box rating prompt and track the 2+ threshold as the new activation metric on the weekly dashboard	30 min

YouYou can do this yourself, no engineering access required.

Final deliverable

A cohort comparison table showing Month 3 retention for both groups, plus a one-line verdict on whether the candidate behavior is a real magic number.

See a reference example

Sample output

Lenskart first-purchase cohort split (n=1,200)
  Completed a virtual try-on before purchase (n=430): 58% retained at Month 3
  Purchased without virtual try-on (n=770): 31% retained at Month 3
  Gap: 27 percentage points -> real divergence; virtual try-on is a strong activation-event candidate

Success criteria

You're done when you can:

Splits the cohort by the candidate threshold correctly, not by an unrelated variable
Reports Month 3 retention as a percentage for each group separately
Gives a clear real-vs-noise verdict based on the size of the gap, not just a description of the numbers