The Vanity Metrics Deck: Spot What's Actually Growth
Objective: Given a synthetic-realistic monthly growth update modeled on an early-stage language-learning app, correctly separate the metrics that prove real value delivery from the ones that just look good in a board deck, and defend each call against the lesson's own vanity-vs-outcome distinction.
You've joined as a fractional growth advisor for an early-stage language-learning startup that is explicitly copying Duolingo's playbook, streaks, mascot, and all. The founder just forwarded you this month's growth update before the board meeting and wants your read within the hour.
One specimen, one prompt: which of these six numbers is actual growth, and which is just activity that looks like growth? Flag every metric that fails the lesson's outcome-vs-output test, defend the ones you leave alone, and don't flag a metric just because 'more is good' feels automatic.
Before you start
What you'll need
Free path (everything below is enough to finish)
Free for a single user, enough to produce the one-page deliverable this project asks for.
Free tier covers cohort and retention analysis for a startup this size, this exercise is complete without ever opening it.
Paid upgrades (optional, faster/deeper)
The free path (Notion for the write-up, Mixpanel free tier for the real cohort check) is complete on its own. Amplitude is only worth paying for once volume outgrows Mixpanel's free tier.
An upgrade once the team has enough volume that Mixpanel's free tier gets limiting, never required to finish this teardown.
The process
Specimens to review
Read this monthly growth update exactly as the board will. For each numbered metric, decide: is this proof of real growth (a customer receiving value), or a vanity metric dressed up as one? Write your call and one sentence of why for each.
MONTHLY GROWTH UPDATE, LinguaLeap (Month 4) Prepared for the board 1. Total app downloads: 340,000 this month (+180% MoM) 2. Instagram followers: 89,400 (+12,100 this month) 3. Total signups (account created): 61,800 (+94% MoM) 4. Learners completing at least 1 lesson every day this week: 8,900 (14.4% of Month-1 signup cohort) 5. Average app store rating: 4.8 stars (up from 4.3 last quarter) 6. 7-day retention rate (returned and completed a lesson): 19.2%, down from 22.1% last month Founder's note: "Downloads and followers both nearly doubled. We're clearly winning."
Specimen: synthetic, realistic
Final deliverable
A one-page call sheet: for each of the six metrics, real growth or vanity, with a one-sentence justification tied to the lesson's outcome/output distinction, plus a one-line recommendation for what LinguaLeap's actual North Star Metric candidate should be.
See a reference example
Flagged: downloads (1), followers (2), and the 'we're clearly winning' framing of signups (3), they measure reach and account creation, not product value delivered. Left alone: daily lesson completion (4) and 7-day retention (6), the closest thing in the deck to a real value-delivery signal, the same category Slack was tracking when it watched teams that had sent 2,000+ messages rather than just counting logins. Recommendation: LinguaLeap's North Star candidate should be built from metric 4, not metric 1 or 3, and the board should hear that retention fell 2.9 points before it hears that followers grew 12%.
Success criteria
You're done when you can:
- Correctly flags downloads and followers as vanity/output metrics, not proof of growth
- Correctly leaves daily-completion and 7-day-retention metrics unflagged, recognizing them as legitimate outcome signals
- Explains the AARRR-stage gap between signups (metric 3) and the founder's 'winning' claim
- Names a specific North Star Metric candidate for LinguaLeap grounded in metric 4, not in downloads or followers