Verified against ChatGPT · 2026-08-14
Design a cohort retention analysis that answers the actual question instead of just producing a heatmap
Designs a cohort analysis structure matched to a specific retention question, then reads a described cohort result for the pattern that actually matters, rather than just describing the shape of the heatmap.
The prompt
Ready to copy — highlighted parts are example details you can swap.
Help me with a cohort retention analysis in two parts: first designing it correctly for my actual question, then reading the results once I have them.
PART 1 - DESIGN
RETENTION QUESTION
Did the new onboarding flow we launched 10 weeks ago improve 30-day retention compared to before?
AVAILABLE DATA
Signup timestamp, weekly login events, and plan tier for every user since 6 months before the onboarding change.
What I need: the right cohort definition (grouped by signup week, by acquisition channel, by initial plan tier — whichever actually matches the question), the right retention event definition (what counts as "still retained" — a login, a specific action, a paid renewal), and the right time window granularity (day 1/7/30 vs week-by-week vs month-by-month), each justified against the retention question specifically rather than picked by default convention.
PART 2 - READ THE RESULT
COHORT DATA
Week 0 cohorts at 100% by definition; by week 4, pre-change cohorts average 22% retained, post-change cohorts average 31% retained; both curves still declining slightly through week 10.
Once you've confirmed the design, read this actual cohort data for:
1. Whether retention curves are converging to a stable long-run plateau or continuing to decay — this distinguishes healthy products with a stable core user base from ones quietly losing everyone eventually, and it's the single most important read of any retention curve.
2. Which specific cohort(s) deviate meaningfully from the others, and whether that lines up with anything that changed around when that cohort joined (an onboarding change, an acquisition channel shift, a pricing change).
3. Whether the earliest cohorts (most time to mature) are systematically different from recent ones in a way that suggests survivorship bias in how the data is being read, rather than an actual trend.
WHAT NOT TO DO
Do not just describe the shape of the retention curve ("retention declines over time, as expected") — every cohort analysis declines over time by definition; the useful read is where it plateaus, which cohorts deviate, and why.
OUTPUT FORMAT
1. Recommended cohort definition, retention event definition, and time granularity, each with a one-line justification
2. Plateau read: does this cohort set show a stable long-run floor, and at roughly what retention level
3. Deviating cohort(s) and the most plausible explanation, flagged as hypothesis if not confirmed
4. Survivorship-bias check on early-vs-recent cohort comparison
5. One paragraph: the single most important takeaway for whoever owns retentionCustomize
Optional — swap in your own details for the highlighted parts above.
Why this works
Splitting this into a design phase before a read phase matters because cohort analysis has more failure modes in its setup than in its interpretation — the wrong cohort grouping (by signup week when the real driver is acquisition channel) or the wrong retention event (counting any login when the real question is about paid renewal) will produce a technically correct-looking heatmap that answers a different question than the one being asked, and a model asked only to "analyze this cohort data" has no way to catch a mis-specified design after the fact. Requiring the plateau read specifically, rather than accepting a generic "retention declines over time" observation, targets the actual information content in a retention curve: every cohort curve declines by construction, so noting the decline is content-free, while where it plateaus (or whether it plateaus at all) is the one number that distinguishes a healthy sticky product from one slowly bleeding its entire user base, and a model not explicitly told this will default to describing the curve's shape rather than extracting its single most decision-relevant feature. Explicitly requiring the deviating-cohort check to be tied to something that actually changed around that cohort's join date, flagged as hypothesis rather than fact, prevents the model from noting a deviation and stopping there — an unexplained deviation is much less useful than one connected to a plausible cause, even a tentative one, since it gives whoever owns retention somewhere concrete to look next. The survivorship-bias check exists because comparing an old cohort that's had a year to mature against a three-week-old cohort is a very common way to draw a false conclusion about improvement or decline — older cohorts always look more "resolved" than younger ones purely due to more elapsed time, and a model not instructed to check for this will sometimes read a maturity artifact as a real trend.
What you get back
Cohort definition: group by signup week, retention event defined as any login within the 30-day window (matches your stated question about the onboarding flow specifically). Plateau read: neither curve has clearly plateaued through week 10 yet, both are still declining slightly, so it's premature to call a stable floor for either group. Deviating cohorts: post-change cohorts sit consistently above pre-change ones (31% vs 22% at week 4), but the 3-week pricing promotion overlapping the same new-signup window is a plausible confound — recommend isolating cohorts that signed up after the promotion ended before crediting the full lift to onboarding alone.
Verified against
ChatGPT GPT-5.1 · 2026-08-14
Changelog
- 2026-08-14 — Initial publish, verified against ChatGPT GPT-5.1.
Need this built into your business?
If a prompt isn't enough — what Scult builds, built and maintained for you — that's Scult's day job.
EXPLORE WHAT SCULT BUILDS
