Verified against ChatGPT · 2026-08-10
Get a straight verdict on an A/B test instead of a hedge that avoids saying whether it won
Runs your test results through a significance and practical-impact check and forces a plain ship/hold/kill call, instead of the noncommittal "trending positive" summary that leaves the actual decision to you.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are reading out an A/B test result the way a senior data analyst would to a product team that needs an actual decision, not a hedge. TEST SETUP Checkout page: control (single-page checkout) vs. variant (3-step checkout with progress bar) RESULTS Control: 4.1% conversion, 12,400 sessions. Variant: 4.6% conversion, 12,600 sessions. SAMPLE SIZE AND DURATION Ran for 9 days, roughly 25,000 total sessions split evenly DECISION AT STAKE Whether to roll the 3-step checkout out to 100% of traffic next sprint What I need from you: 1. State the statistical significance of the result plainly — the p-value or confidence interval if given, and whether it crosses the standard 95% threshold. If significance can't actually be calculated from what I've given you (missing sample size, no variance data), say that explicitly instead of eyeballing a verdict from the topline numbers alone. 2. Separate statistical significance from practical significance. A result can be statistically significant and still too small to justify the engineering or design cost of shipping it — state both dimensions and don't let a small but "significant" lift get inflated into a slam-dunk recommendation. 3. Check whether the test ran long enough to cover a full business cycle (e.g., at least one full week, ideally two, to avoid day-of-week bias) given the duration provided, and flag it if it didn't. 4. Give one of exactly three verdicts: SHIP, HOLD FOR MORE DATA, or KILL — with the one sentence of reasoning that would change if a manager pushed back on it. 5. Name the single biggest risk in trusting this result as-is (novelty effect, seasonality, a segment that skews the topline number, sample ratio mismatch) if one is plausible given what's described. WHAT NOT TO DO Do not use words like "promising," "trending well," or "worth considering" as a substitute for one of the three verdicts — pick one. Do not treat a large percentage lift on a small sample as equivalent in confidence to the same lift on a large sample; call out the difference. OUTPUT FORMAT 1. Verdict: SHIP / HOLD FOR MORE DATA / KILL, in bold, first line. 2. Statistical significance readout (one line). 3. Practical significance readout (one line). 4. Duration/cycle-coverage check (one line). 5. Single biggest risk to trusting this result. 6. Two-sentence rationale a skeptical stakeholder could push back on.
Customize
Optional — swap in your own details for the highlighted parts above.
Why this works
Asked to summarize an A/B test without a forced-choice constraint, GPT-5.1 tends to produce hedged, non-committal language because a confident wrong call feels riskier to generate than a vague one that can't be pinned down later — restricting the output to exactly three named verdicts removes that escape hatch and forces the model to actually commit to a position based on the numbers given, the same discipline a rigorous human analyst would apply. Separating statistical from practical significance targets a specific and common misreading of experiment results: a genuinely significant p-value on a 0.5-percentage-point lift can still not be worth the engineering cost to ship, and collapsing both into one "it worked" statement is exactly the kind of oversimplification that gets a team to ship low-value changes because the test was "significant." The explicit duration/cycle-coverage check exists because a 9-day test spans less than two full weekly cycles, which is a known source of misleading results if weekday and weekend behavior differ, and a model summarizing only the topline percentages has no structural reason to flag that unless explicitly told to check for it. Requiring the single biggest trust risk to be named, rather than leaving the analysis as a clean number, mirrors what a skeptical stakeholder would ask in the room anyway — surfacing it in advance means the person presenting this readout isn't caught flat-footed defending a result they haven't actually stress-tested themselves.
What you get back
Verdict: HOLD FOR MORE DATA Statistical significance: p ≈ 0.09, does not clear the 95% threshold. Practical significance: +0.5pp lift would be worth shipping if confirmed, but isn't confirmed yet. Duration check: 9 days covers barely one full cycle — recommend extending to 14. Biggest risk: sample size is borderline for a lift this small; could be noise.
Verified against
ChatGPT GPT-5.1 · 2026-08-10
Changelog
- 2026-08-10 — Initial publish, verified against ChatGPT GPT-5.1.
Need this built into your business?
If a prompt isn't enough — what Scult builds, built and maintained for you — that's Scult's day job.
EXPLORE WHAT SCULT BUILDS
