Verified against Claude Code · 2026-08-03
Diagnose a flaky test's real cause before Claude Code touches it
A diagnosis-first prompt that stops Claude Code from silently wrapping a flaky test in a retry or a longer timeout, and instead forces it to isolate the actual source of nondeterminism through repeated real test runs and git history.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are diagnosing a flaky test, not fixing it. A flaky test passes and fails on the same code with no relevant change between runs — the job here is finding the actual mechanism, not making the failure stop by whatever means is fastest. TEST tests/checkout/discount.spec.ts, 'applies the loyalty discount when cart total exceeds threshold' WHAT HAS ACTUALLY BEEN OBSERVED Fails roughly 1 in 15 runs on CI, has never failed locally in about 50 manual runs; failure is always a timeout waiting for a mocked API response. ENVIRONMENT DIFFERENCES BETWEEN WHERE IT PASSES AND WHERE IT FAILS CI runs the suite in parallel with 4 workers; locally the suite always runs single-threaded. SUSPECTED CAUSE, IF ANY Might be related to another test in the same file mutating a shared mock client instance. DIAGNOSTIC PROCESS — DO NOT SKIP OR REORDER STEPS 1. Run the test in isolation, alone, at least ten times in a row. Report the exact pass and fail count. 2. Run the full suite it normally runs inside, at least ten times in a row, with this test included. Report the exact pass and fail count. 3. If isolated runs are stable but full-suite runs are not, this is order-dependence or leaked shared state — identify the specific test that runs immediately before it and what state that test leaves behind: a shared database row, a module-level variable, a mock that was never reset. 4. If both are unstable independently, look for genuine nondeterminism: unseeded randomness, a fixed sleep or timeout racing real async work, a system clock or timezone dependency, or a network or filesystem call that is not actually mocked. 5. Check git log for the test file and the code it exercises. Was this test reliable before a specific recent commit? A regression introduced by a real code change is a lead, not a coincidence, and should outrank a theory about inherent randomness. 6. State the diagnosis as one sentence naming the actual mechanism, backed by the evidence from steps one through five. 'It is flaky' or 'there might be a race condition' is not an acceptable diagnosis on its own — name the specific line, shared resource, or timing assumption responsible. NOT ACCEPTABLE AS A SUBSTITUTE FOR DIAGNOSIS Do not propose adding a retry, increasing a timeout, or marking the test skipped or quarantined as your finding. Any of these might be a legitimate part of an eventual fix once the mechanism above is named and confirmed, but proposing one before that point is a failed diagnosis, not a shortcut to one. IF THE ENVIRONMENT DIFFERENCE ITSELF LOOKS LIKE THE CAUSE If CI runs the suite in parallel with 4 workers; locally the suite always runs single-threaded. points at parallelism, worker count, or resource limits as the likely trigger, still complete steps one through five before concluding that — an environment difference is a plausible contributing factor, not by itself a mechanism, and the specific shared resource or timing assumption it exposes still needs to be named. Once the diagnosis is confirmed and approved, propose the actual fix in a separate message, not this one.
Customize
Optional — swap in your own details for the highlighted parts above.
Why this works
Asked to fix a flaky test with no other constraint, a model's highest-probability response is exactly the three moves that make CI green without making anything true: add a retry wrapper, bump a timeout, or quarantine the test — each is a legitimate tool in a human engineer's kit but a worthless diagnosis on its own, since it treats the symptom as the problem. The 'not acceptable as a substitute for diagnosis' clause exists because a model under pressure to produce a working answer quickly will reach for the cheapest fix that makes the failure disappear, and naming that exact move as unacceptable up front closes off the shortcut before it is taken. Steps one and two — run the test alone repeatedly, then run it inside the full suite repeatedly — reproduce the standard engineering technique for distinguishing order-dependence and shared-state leakage from genuine nondeterminism inside the test itself; a model that only reasons about the code as text without actually executing it this way is guessing, whereas Claude Code's ability to run a test command in a loop through its own Bash tool turns this from a thought experiment into an empirical check with real pass and fail counts a chat-only assistant reading the same code could never produce. The git log step matters because flaky tests very often are not actually random at all — they became unreliable the moment a specific commit changed a timing assumption or a shared fixture, and a model that skips version history will hunt for randomness in code that was reliable for months before one dependency bump. The instruction to still complete every step even when the environment difference looks like an obvious explanation exists because 'CI runs in parallel and local does not' describes a condition that makes flakiness possible, not the specific defect that makes it happen under that condition — accepting the environment difference itself as the diagnosis leaves the actual shared-state bug in place, ready to resurface the next time parallelism changes for an unrelated infrastructure reason.
What you get back
Step 1, isolated, 10 runs: 10 passed. Step 2, full suite, 10 runs: 7 passed, 3 failed. Since isolated is stable but suite is not, this points to order-dependence, not timing nondeterminism. Step 3: the test immediately before it uses a once-only mock resolve that, on an early return, never gets consumed — this test's own mock setup is then silently skipped and it hits the real, unmocked path, producing the timeout. Diagnosis: shared mock client not reset between tests in this file, missing a clearAllMocks call in an afterEach — not a race condition. Not proposing a fix yet, awaiting approval to proceed.
Verified against
Claude Code Sonnet 4.6 · 2026-08-03
Changelog
- 2026-08-03 — Initial publish, verified against Claude Code (Sonnet 4.6).
Building this for real?
This is a free starting point. If you'd rather have custom software built and running for your business, that's Scult's day job.
EXPLORE CUSTOM SOFTWARE
