regression-suite

---
name: regression-suite
description: Map test coverage to GDD critical paths, identify fixed bugs without regression tests, flag coverage drift from new features, and maintain tests/regression-suite.md. Run after implementing a bug fix or before a release gate. ShiningPlague-adopted: adapts to the project's headless harness pattern ({{TEST_HARNESS}}, e.g. tools/<step>_<feature>_check.gd) + scene parse-checks + deferred test-framework decision.
license: MIT
---

# Regression Suite

> 🌱 **ShiningPlague-adopted.** Originally Donchitos; battle-tested on a shipped Godot project. Upstream pattern preserved (capture-as-test + coverage drift detection + manifest at `tests/regression-suite.md`); project test types added (headless script harness, Python validators, scene parse-checks).

**Core principle:** every fixed bug should have a regression test that would have caught it. A regression suite is not a new test category — it is a **curated list of tests already in `tools/` and `tests/`** that collectively cover critical paths and known failure points. This skill maintains that list.

**Output:** `tests/regression-suite.md`

---

> **If an artifact named here is absent:** say so plainly in one line, skip that step, and
> continue. Never invent the file to satisfy a checklist, and never fail a close because an
> optional artifact was never created.

## When to fire

- 🟢 **After every bug fix lands** — capture-as-test pattern. The bug becomes a test that would have caught it pre-fix.
- 🟢 **After each step ships end-to-end** — coverage gap pass. Identify which step features lack coverage.
- 🟢 **On-demand** — designer asks *"are we covered for X?"* / *"do we have a regression test for the slot drift bug?"*
- 🟢 **Before a major refactor** — confirm the existing test coverage will catch breakage.
- 🟢 **Before a release gate** — `/gate-check polish` requires regression suite exists.
- 🟢 **As part of sprint close** — detect coverage drift.

---

## Arguments

- `/regression-suite update` — scan new bug fixes this sprint, check for regression test presence, add new tests to manifest
- `/regression-suite audit` — full audit of GDD critical paths vs existing coverage; flag paths with no test
- `/regression-suite report` — read-only status report (no writes), suitable for sprint reviews
- No argument — run `update` if sprint active, else `audit`

---

## In-scope test types (project pattern — {{TEST_HARNESS}})

| Type | Where it lives | Example |
|---|---|---|
| Headless script harness | `{{TEST_HARNESS}}` (e.g. `tools/<step>_<feature>_check.gd`) | e.g. `tools/step3_your_system_check.gd` |
| Headless data validation | Python script in `tools/` | `tools/consistency_check.py` (cross-doc checks) |
| Scene parse-check | `godot --headless scene.tscn` parse output | entry scene loads with no SCRIPT ERROR |
| Cross-step regression runner | `tools/regression_check.gd` (if built — check registry) | runs all step harnesses in sequence |
| Manual smoke test | run in editor + designer eyeball | in-engine smoke test at step ship |

A full test-framework adoption (e.g. GUT for Godot) may be a deferred decision per the project's CLAUDE.md — until then, scene-based + headless-script verification.

---

## Procedure: capture-as-test mode (fires after a bug fix)

1. **Identify the bug.** What was the symptom? What was the root cause? What commit fixed it?
2. **Identify the test layer.** Where would the test live?
   - Pure data layer (JSON validity) → Python script
   - Engine logic (apply/tick/expire) → headless GDScript harness
   - Scene loading → headless scene parse check
   - User-facing render → manual smoke test (no automation possible without engine running)
3. **Write the failing test.** Reproduce the bug as a test case that would have failed PRE-fix. Verify it passes POST-fix.
4. **Add to the relevant harness file** (or create a new harness if none exists for that domain).
5. **Append entry to `tests/regression-suite.md`** with:
   - Bug description + commit hash
   - Test location (file:line)
   - Critical path coverage tag (combat / map / data / etc.)
6. **Verify the harness runs cleanly:** `godot --headless --path . --script "res://tools/<harness>.gd"` shows N+1/N+1 PASS (the +1 is the new test).
7. **Commit + push:** `test(<step>): add regression for <bug short> (closes commit <hash>)`.

## Procedure: coverage-gap mode (fires after a step ships)

1. **List every system/feature shipped in the step.** From `system_registry.json → recently_changed_*` + the step's spec.
2. **For each, ask: is there a test that would catch a regression?**
3. **Classify gaps:**
   - 🔴 No test exists for a critical path (blocks player from playing)
   - 🟠 No test exists for a feature explicitly marked in spec
   - 🟡 No test for an edge case or polish detail
4. **Write tests for the 🔴 cases inline.** Propose `/writing-plans` for 🟠+ batch if 3+ items.
5. **Update `tests/regression-suite.md`** with new coverage rows + flag critical gaps in `system_registry.json → flagged_for_designer_review`.

## Procedure: on-demand audit (designer asks "are we covered for X?")

1. **Find the relevant tests.** Grep `tools/` for keywords related to X.
2. **Read each test's assertions.** Are they actually covering the behaviour, or just sanity-checking?
3. **Report coverage assessment:** strong / partial / none, with file:line references.
4. **Propose new tests if gaps exist.**

## Procedure: full audit mode (Donchitos pattern, adapted)

For `audit` mode:

### Step A — Load existing manifest
Read `tests/regression-suite.md` if it has been written. Extract total registered tests, last updated date, any flagged as STALE/QUARANTINED. The installer seeds this file, so existence proves nothing — a copy still carrying the `scaffold-seed: unwritten` marker near the top is the blank skeleton. If it is blank or absent: note "No regression suite registered yet — filling the seeded skeleton." Fill the seeded file in place and delete its marker line; never write a second copy beside it.

### Step B — Load test inventory
Glob all test files (`tools/*_check.gd`, `tools/*.py`, `tests/`). For each file, note system + filename.

### Step C — Load GDD critical paths
Read `{{GDD_PATH}}` or per-system GDDs. For each MVP-tier system, extract Acceptance Criteria, Formulas, Edge Cases. These define critical paths.

### Step D — Load closed bugs
From `system_registry.json → flagged_for_designer_review[]` with `resolved_in_commit`. For each, check if a test references the bug or its scenario.

### Step E — Map coverage
For each critical path/bug, assign:

| Status | Meaning |
|--------|---------|
| **COVERED** | Test file exists targeting this criterion's logic |
| **PARTIAL** | Test exists but doesn't cover all cases (happy path only) |
| **MISSING** | No test found for this critical path |
| **EXEMPT** | Visual/Feel/UI criterion — not automatable by design |

Elevate MISSING items that correspond to formulas/state machines to **HIGH PRIORITY**.

### Step F — Detect coverage drift
- Stories completed this sprint with no test files
- New systems in registry since last regression-suite update
- GDD sections added/revised since manifest last updated
- Manifest `Last Updated` date — if gap > 2 sprints, flag stale

---

## Output Report Format

```
## Regression Suite Status

**Mode**: [update | audit | report]
**Existing registered tests**: [N]
**Test files scanned**: [N]

### Critical Path Coverage (audit mode only)
| System | Total ACs | Covered | Partial | Missing | Exempt |
|--------|-----------|---------|---------|---------|--------|
| [name] | [N] | [N] | [N] | [N] | [N] |

**Coverage rate (non-exempt)**: [N]%

### Bug Regression Coverage
| Bug ID | System | Severity | Has Regression Test? |
|--------|--------|----------|----------------------|
| BUG-NNN | [system] | S[N] | YES / NO ⚠ |

**Bugs without regression tests**: [N]

### Coverage Drift Indicators
[List new systems or stories with no test coverage, or "None detected."]

### Recommended New Regression Tests
| Priority | System | Suggested Test File | Covers |
|----------|--------|---------------------|--------|
| HIGH | [system] | `tools/<step>_<bug>_check.gd` | BUG-NNN / AC-[N] |
| MEDIUM | [system] | `tools/<step>_<feature>_check.gd` | [criterion] |
```

---

## `tests/regression-suite.md` schema

```markdown
# Regression Suite Manifest

> Last Updated: [date]
> Total registered tests: [N]
> Coverage: [N]% of GDD critical paths

## How to run

godot --headless --path . --script "res://tools/<harness>.gd"
python tools/consistency_check.py

## Registered Regression Tests

### [System Name]

| Test File | Test Function | Covers | Added |
|-----------|---------------|--------|-------|
| `tools/<harness>.gd:line` | `test_<scenario>` | AC-N / BUG-NNN | [date] |

## Known Gaps

| Priority | System | Suggested Path | Covers | Reason Not Yet Written |
|----------|--------|----------------|--------|------------------------|
| HIGH | [system] | `tools/<path>` | BUG-NNN | Bug fixed without test |

## Quarantined Tests

| Test File | Function | Reason | Quarantined Since |
|-----------|----------|--------|-------------------|
```

---

## Known regression test gaps (seed entries)

These are bugs that shipped without regression tests — every one is a future-firing of this skill:

_None currently logged._ Append new gap entries here as bugs ship without regression tests.

**Template for new entries:**

- **<bug name>** (commit `<hash>`). <where the test should live + what it should assert>.

---

## Write Output

Ask: "May I write/update `tests/regression-suite.md` with the current regression suite manifest?"

For `update` mode: append new entries; never remove existing entries (use `Edit` with targeted insertions).
For `audit` mode: rewrite the full manifest with updated coverage data.
For `report` mode: do not write anything.

After writing (if approved):

- For each HIGH priority gap: "Consider creating the missing regression test before the next sprint."
- If bug regression gaps > 0: "These bugs can silently return without regression tests. The next sprint should include a story to write the missing tests."
- If coverage drift detected: "Regression suite may be drifting. Consider running `/regression-suite audit` at the next sprint boundary."

Verdict: **COMPLETE** — regression suite updated. (If user declined write: Verdict: **BLOCKED**.)

---

## Collaborative Protocol

- **Never remove existing regression tests from the manifest** without explicit user approval
- **Gaps are advisory, not blocking** — surface them clearly but don't prevent other work (except at release gate)
- **Quarantine is not deletion** — flaky tests should be quarantined (noted in manifest), not removed; they should be fixed, and a bug filed with `/bug-report` if the cause is not obvious
- **Ask before writing** — always confirm before creating or updating the manifest

---

## What this skill does NOT do

- Replace `verification-before-completion` (which verifies CURRENT change, not builds regression suite for FUTURE changes)
- Auto-write tests without designer review on test approach (especially 🟠+ cases)
- Touch the GUT framework adoption decision — that's a future technical-director call

---

## Cross-link

If a cross-step regression runner is parked in `system_registry.json → next_session_priorities` as a deferred item, note that when it lands it provides the cross-step runner this skill writes tests for.