optimize-runbook

---
name: optimize-runbook
description: Analyze previous Jetty workflow runs and propose targeted improvements to your runbook. Use when the user wants to optimize, improve, or debug a runbook based on past execution results — including 'optimize runbook', 'improve runbook', 'why is my runbook failing', 'analyze my runs', 'runbook not working well', 'make my runbook better', 'debug runbook performance', or 'learn from past runs'. Also trigger when the user mentions trajectory analysis, run patterns, or evaluation score improvements.
license: MIT
---

# Optimize Runbook

You are analyzing previous Jetty workflow runs to identify patterns and propose targeted improvements to a local runbook. The goal is to produce specific, evidence-backed changes — not generic advice.

## Cross-Agent Compatibility

This skill uses `AskUserQuestion` for interactive choices. If running in an environment where `AskUserQuestion` is not available, replace each call with a direct question in your text output.

---

## Mode dispatch — READ THIS FIRST

Before Step 1, check whether you are in **headless mode**:

```bash
echo "${JETTY_OPTIMIZE:-0}"
```

- If the value is `1`, follow **ONLY** the "Headless Mode" section below. Do not execute Steps 1–7.
- Otherwise, follow Steps 1–7 as normal (interactive flow with the user).

Headless mode runs inside a Jetty Optimize sandbox where the inputs have been pre-loaded to known file paths and there is no user on the other end to answer `AskUserQuestion` prompts mid-skill. The user's interaction (if any) comes **after** you finish the initial pass, in the form of follow-up turns.

---

## Headless Mode (when `JETTY_OPTIMIZE=1`)

### H1. Verify pre-loaded inputs

Confirm the three inputs are on disk. The trajectory metadata is at `/app/trajectory.json` (Mise also symlinks it under `/app/trajectory/trajectory.json` and the canonical `/app/trajectory/{trajectory_id}.json` — any of these works).

```bash
ls -la /app/RUNBOOK.md /app/baseline-RUNBOOK.md /app/trajectory.json
```

If any of those files is missing, **emit the H3 JSON payload with `patterns:["precheck failed: <which file>"]` and `proposed_changes:[]`, then stop.** Never exit silently — Spot relies on the JSON to surface the failure to the user.

Read the runbook and the trajectory:

- `/app/RUNBOOK.md` — the runbook to edit (your working copy)
- `/app/baseline-RUNBOOK.md` — read-only snapshot for comparison
- `/app/trajectory.json` — the one trajectory you are analyzing
- `/app/trajectory/` — step output assets (images, documents, uploaded inputs)

Do **NOT** call the Jetty trajectory list/fetch API. Do **NOT** use `AskUserQuestion` anywhere in headless mode.

### H2. Analyze

From `trajectory.json`, extract and hold in memory:

- `trajectory_id`, `status`, total duration, iteration counts per step
- Each step's inputs, outputs, errors, duration
- Any labels already applied

**Stay-alive cue.** Mise's stream-tail heartbeat keys off log writes from your CLI. Long quiet stretches (e.g. >20s of pure thinking after a batch of `Read`s) used to starve the heartbeat and silently kill the activity mid-turn — your edits would still land on disk but Spot would lose the trail. The newer mise build heartbeats from a separate timer so this is no longer load-bearing, but **prefer printing a one-line status before transitioning between phases** (e.g. `"Reads complete; analyzing patterns now."`) regardless. It costs nothing, keeps the SSE stream fresh, and it gives the user a visible breadcrumb when the inter-tool gap is long.

**Special case — degenerate baseline.** If the trajectory's top-level `status` is anything other than `completed`, OR every step has `status` of `null` / `failed` / no `outputs`, the baseline never produced real evidence (e.g. the original agent failed at startup with `Not logged in`, an authentication error, or a quota issue). In that case, **skip the six pattern lenses** and go straight to H3 with `patterns:["baseline trajectory produced no usable evidence: <one-line cause>"]` and `proposed_changes:[]`. Do not fabricate patterns from log noise.

Otherwise, apply the same six pattern lenses as Step 4 of the interactive flow, against this single trajectory:

1. Consistent failures (evaluation criteria below threshold)
2. Iteration waste (retry-heavy steps)
3. Timeout / long-duration bottlenecks
4. Divergent agent behavior vs. the runbook's stated expectations
5. Missing guardrails (errors that the runbook's evaluation section doesn't catch)
6. Score plateaus (rubric-only — iterations that didn't improve)

For each pattern you find, plan one concrete edit to `/app/RUNBOOK.md`. Edits must be specific (exact before/after text) and cite the trajectory step(s) that motivated them.

### H3. Emit the structured analysis payload (REQUIRED — always)

**Before editing any file, and before any other terminal action**, print a single-line JSON object on its own line with this exact shape (Spot parses this for the diff-gutter citations):

```json
{"type":"optimize_analysis","trajectory_id":"<id>","patterns":["<short description>","..."],"proposed_changes":[{"section":"<runbook section name>","before":"<exact current text>","after":"<exact replacement>","citations":["<step_name>","..."]},"..."]}
```

Rules:

- One JSON object, one line, no pretty-printing.
- `citations` must reference concrete step names or event ids from the trajectory. An empty citations array is a bug — don't emit one.
- **Always emit this payload, even on the no-evidence path or precheck failure.** When there's nothing to propose, emit `"proposed_changes": []` and put the reason in `"patterns"` (one short string, e.g. `"baseline trajectory failed at startup: Not logged in"`). Stopping without emitting the payload leaves the user staring at a dead UI.

After the JSON line, print a short human-readable summary in prose (Spot will render it in the chat pane alongside the diff). Even when there's nothing to apply, the prose paragraph tells the user what you saw and what to do next (e.g. "Re-run the baseline with credentials configured, then start a new optimize session.").

### H4. Apply each change

For every item in `proposed_changes`, use the `Edit` tool against `/app/RUNBOOK.md` to replace the `before` text with the `after` text. Then bump the runbook version in the frontmatter (patch increment — e.g. `1.0.0` → `1.0.1`). If no version field exists, add `version: "1.0.1"`.

Do **not** edit `/app/baseline-RUNBOOK.md` — it stays pristine for the diff view.

### H5. Stop and wait

Emit a one-paragraph summary (counts of patterns and edits applied). Then stop. The user may send follow-up turns asking for refinements (e.g. "also tighten step 3's iteration budget"). Apply those incrementally via `Edit` and wait again. Do **not** call the skill's subsequent steps; the interactive flow's Step 6–7 "apply changes" prompts are not applicable in headless mode.

### H6. What's different from the interactive flow

| | Interactive | Headless |
|---|---|---|
| Runbook location | Discovered via `ls RUNBOOK*.md` | Always `/app/RUNBOOK.md` |
| Trajectory source | Jetty API (list + fetch) | `/app/trajectory.json` |
| Trajectory count | N (user-chosen) | Always 1 |
| `AskUserQuestion` | Used | **Never used** |
| Apply-changes gate | User picks "Apply all" / "Let me choose" / "Save as report" | Auto-apply all |
| Version bump | After apply | After apply |
| Output artifact | Modified `RUNBOOK.md` + optional `./runbook-optimization-report.md` | Modified `/app/RUNBOOK.md` only |
| Analysis JSON emission | Not emitted | **Required** (§H3) |

---

## Step 1: Identify the Runbook

1. Look for runbook files in the current directory:

```bash
ls -la RUNBOOK*.md 2>/dev/null
```

2. If multiple runbooks are found, use AskUserQuestion:
   - Header: "Runbook"
   - Question: "Multiple runbooks found. Which one do you want to optimize?"
   - Options: list each filename

3. Read the chosen runbook with the Read tool. Extract from frontmatter:
   - `version`, `evaluation` (programmatic or rubric)
   - `agent`, `model`, `model_provider`, `snapshot` (if present)

4. Parse the evaluation section:
   - **Programmatic**: extract the PASS/PARTIAL/FAIL criteria table
   - **Rubric**: extract the rubric table (criteria, score descriptions, pass threshold)

5. Identify the collection and task name. Check the skill argument first, then look in the runbook for Jetty API references. If not found, use AskUserQuestion:
   - Header: "Collection/Task"
   - Question: "Which Jetty collection and task does this runbook run as? (format: collection/task_name)"
   - Options:
     - "I'll type it" / "Let me enter the collection and task name"

---

## Step 2: Fetch Trajectories

Parse the skill argument for trajectory IDs or `--last N`. If not provided:

Use AskUserQuestion:
- Header: "Trajectories"
- Question: "How many recent runs should I analyze?"
- Options:
  - "Last 5 runs" / "Analyze the 5 most recent trajectories"
  - "Last 10 runs" / "Analyze the 10 most recent trajectories"
  - "Specific IDs" / "I'll paste trajectory IDs"

Fetch the trajectory list:

```bash
TOKEN="$(cat ~/.config/jetty/token)"
COLLECTION="the-collection"
TASK="the-task"
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://flows-api.jetty.io/api/v1/db/trajectories/$COLLECTION/$TASK?limit=$LIMIT"
```

Parse the response — format is `{"trajectories": [...], "total": N}`.

For each trajectory, fetch full details:

```bash
TOKEN="$(cat ~/.config/jetty/token)"
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://flows-api.jetty.io/api/v1/db/trajectory/$COLLECTION/$TASK/$TRAJECTORY_ID"
```

Extract and record for each:
- **Status**: completed / failed / timed_out
- **Duration**: total execution time
- **Step outputs**: iterate over `.steps` object keys
- **Errors**: any error messages in failed steps
- **Labels**: any quality labels applied

Download output files where available (validation_report.json, summary.md):

```bash
TOKEN="$(cat ~/.config/jetty/token)"
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://flows-api.jetty.io/api/v1/file/$FILE_PATH"
```

---

## Step 3: Build Analysis Summary

Create and display a summary table:

```markdown
| # | Trajectory ID | Status | Duration | Iterations | Score/Result | Key Issue |
|---|---------------|--------|----------|------------|-------------|-----------|
```

Fill in from trajectory data. Present to the user.

---

## Step 4: Pattern Analysis

Analyze trajectories against the runbook for these patterns:

### 4a: Consistent Failures
Evaluation criteria scoring below threshold across multiple runs.
- Rubric: criteria scoring < 4 in more than half of runs
- Programmatic: stages showing FAIL/PARTIAL across runs

### 4b: Iteration Waste
Steps that consistently need 2-3 retry rounds. Predictable first-attempt failures that could be prevented with better instructions or templates.

### 4c: Timeout Patterns
Runs that timed out or took disproportionately long. Which steps are the bottlenecks?

### 4d: Divergent Agent Behavior
Cases where the agent interpreted instructions differently across runs. Structurally different outputs suggesting ambiguous instructions.

### 4e: Missing Guardrails
Errors not caught by evaluation criteria. Environment setup issues (wrong versions, missing tools).

### 4f: Score Plateaus (rubric only)
Criteria that iterate but don't improve — suggesting the Common Fixes table lacks actionable guidance.

Present each pattern found with supporting evidence (trajectory IDs, scores, error messages).

---

## Step 5: Generate Proposed Changes

For each pattern, propose a specific change to the RUNBOOK.md:

```markdown
## Proposed Changes

### Change 1: {Brief title} (addresses: {pattern})

**Section:** {Which runbook section}
**Current:**
> {Exact current text}

**Proposed:**
> {Replacement text}

**Evidence:** {Trajectory IDs, scores, errors that support this change}
```

Guidelines:
- Changes must be **specific** — quote exact sections, provide exact replacements
- Changes must be **evidence-backed** — cite trajectories, scores, or errors
- Prefer **additive** changes (add a Common Fix, add a tip, strengthen descriptions)
- If frontmatter fields are missing (`agent`, `model`, `model_provider`, `snapshot`), propose adding them. Default to `agent: claude-code`, `model: anthropic/claude-sonnet-4.6`, `model_provider: openrouter` unless the trajectories show the runbook is already pinned to a different agent or routing.
- Don't fabricate evidence — only cite patterns actually observed

---

## Step 6: Apply Changes

Use AskUserQuestion:
- Header: "Apply Changes"
- Question: "I found {N} proposed improvements. Which should I apply?"
- Options:
  - "Apply all" / "Apply all {N} changes to the runbook"
  - "Let me choose" / "I'll approve each change individually"
  - "Save as report" / "Don't modify the runbook — save analysis to a file"

**If "Apply all"**: Apply each change using Edit. Bump the version (patch increment).

**If "Let me choose"**: For each change, ask approve/skip/modify.

**If "Save as report"**: Write to `./runbook-optimization-report.md`.

---

## Step 7: Summary

```markdown
## Optimization Summary

- **Runbook**: {filename}
- **Trajectories analyzed**: {count}
- **Patterns identified**: {count}
- **Changes applied**: {count} / {total}
- **Version**: {old} → {new}

### Recommended next steps
- Run the updated runbook 2-3 times to verify improvements
- Run `/jetty optimize-runbook` again after new runs to measure progress
```

---

## Important Notes

- **Read the token from file**: `TOKEN="$(cat ~/.config/jetty/token)"` at the start of each bash block.
- **URL**: Use `flows-api.jetty.io` for API calls. Never `jetty.io`.
- **Trajectories shape**: `{"trajectories": [...]}` — access via `.trajectories[]`.
- **Steps are objects**: keyed by name, not indexed.
- **Minimum trajectories**: Works with 1+, but 3+ gives better patterns.
- **Don't fabricate**: Only report patterns actually observed in the data.