Verified against Claude Code · 2026-08-06
Diagnose a flaky test's real cause before Claude Code touches it
A diagnosis-first prompt that stops Claude Code from silently wrapping a flaky test in a retry or a longer timeout, and instead forces it to isolate the actual source of nondeterminism by re-running the test under different conditions.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are diagnosing a flaky test, not fixing it yet. A flaky test is one that passes and fails on the same code with no relevant change — your job is to find out why, not to make it stop failing by any means available. TEST tests/checkout/discount.spec.ts → 'applies the loyalty discount when cart total exceeds threshold' WHAT HAS BEEN OBSERVED Fails roughly 1 in 15 runs on CI, has never failed locally in about 50 manual runs; failure is always a timeout waiting for a mocked API response. SUSPECTED CAUSE Might be related to another test in the same file mutating the shared mockApiClient instance. DIAGNOSTIC PROCESS — DO NOT SKIP STEPS 1. Run the test in isolation, alone, at least 10 times in a row. Report the pass and fail count. 2. Run the full suite the test normally runs in, at least 10 times in a row, with the test included. Report the pass and fail count. 3. If step 1 is stable but step 2 is not, this is order-dependence or shared state leaking from another test — identify which other test in the suite runs immediately before it and what state it leaves behind, such as a shared database row, a module-level variable, or a mock that was not reset. 4. If both steps are unstable independently, look for genuine nondeterminism inside the test or the code under test: unseeded randomness, a real timing assumption such as a fixed sleep or setTimeout racing real async work, system clock or timezone dependence, or a network or filesystem call that is not mocked. 5. Check git log for the test file and the code it exercises — was this test reliable before a specific recent change? If so, that commit is a strong lead, not a coincidence. 6. State your diagnosis as one sentence naming the actual mechanism, with the evidence from steps 1 through 5 that supports it. \"It's flaky\" or \"there might be a race condition\" is not an acceptable diagnosis — name the specific line, resource, or assumption. FORBIDDEN AS A DIAGNOSIS SUBSTITUTE Do not propose adding a retry, increasing a timeout, marking the test as skipped or quarantined, or wrapping an assertion in a wait-for-condition helper as your finding. Any of those may be a legitimate part of an eventual fix, but only after the mechanism above is identified and named — proposing one now, before diagnosis, is treated as a failed task. Once the diagnosis is confirmed and I approve it, propose the fix in a separate message.
Customize the highlighted detailsoptional — the prompt above already works
Why this works
Asked to fix a flaky test with no other constraint, a model's highest-probability response is exactly the three moves that make CI green without making anything true: add a retry wrapper, bump a timeout, or quarantine the test — each is a legitimate tool in a human engineer's kit but a worthless diagnosis, since it treats the symptom as the problem. The forbidden-as-diagnosis-substitute clause exists because a model under pressure to produce a working answer quickly will reach for the cheapest fix that makes the failure disappear, and naming that exact move as unacceptable up front closes off the shortcut before it's taken. Steps 1 and 2 — run the test alone repeatedly, then run it inside the full suite repeatedly — reproduce the standard engineering technique for distinguishing order-dependence and shared-state leakage from genuine nondeterminism inside the test itself; a model that only reasons about the code without actually executing it this way is guessing, whereas Claude Code's ability to run a test command in a loop via its Bash tool turns this from a thought experiment into an empirical check with real pass and fail counts. The git log step matters because flaky tests very often are not actually random — they became unreliable the moment a specific commit changed a timing assumption or shared fixture, and a model that skips version history will hunt for randomness in code that was reliable for months before one dependency bump.
What you get back
Step 1 — isolated, 10 runs: 10 passed, 0 failed. Step 2 — full suite, 10 runs: 7 passed, 3 failed. Since isolated runs are stable but suite runs are not, this points to order-dependence or leaked state, not genuine timing nondeterminism. Step 3 — the test immediately before it, 'rejects a loyalty discount below threshold,' calls mockApiClient.mockResolvedValueOnce(), but the discount test calls mockApiClient.mockResolvedValue() unconditionally afterward; when the once-mock from the prior test hasn't been consumed due to an early return, this test's own mock setup is silently skipped and it hits the real, unmocked rejection path — that's what produces the timeout. Diagnosis: shared mockApiClient instance not reset between tests in this file, missing a clearAllMocks() call in an afterEach — not a race condition. Not proposing a fix yet, awaiting approval to proceed.
Verified against
Claude Code Sonnet 4.6 · 2026-08-06
Changelog
- 2026-08-06 — Initial publish, verified against Claude Code (Sonnet 4.6).
Building this for real?
This is a free starting point. If you'd rather have custom software built and running for your business, that's Scult's day job.
EXPLORE CUSTOM SOFTWARE
