spark-history-cli

---
name: spark-history-cli
description: Query a running Apache Spark History Server from Copilot CLI. Use this whenever the user wants to inspect SHS applications, jobs, stages, executors, SQL executions, environment details, or event logs, especially when they mention Spark History Server, SHS, event log history, benchmark runs, or application IDs.
license: Apache-2.0
---

# spark-history-cli

Use this skill when the task is about exploring or debugging data exposed by a running Apache Spark History Server.

## Installation

```bash
pip install spark-history-cli
```

Or if not on PATH after install:

```bash
python -m spark_history_cli --json apps
```

## Why use this skill

- It gives you a purpose-built CLI instead of scraping the Spark History Server web UI.
- It wraps the REST API cleanly and already handles attempt-ID resolution for multi-attempt apps.
- It supports `--json`, which makes downstream reasoning and comparisons much easier.

## Workflow

1. Prefer the CLI over raw REST calls.
2. Prefer `--json` unless the user explicitly wants a human-formatted table.
3. Use `--server <url>` or `SPARK_HISTORY_SERVER` to point at the right SHS. If the user does not specify one, assume `http://localhost:18080`.
4. Start broad, then drill down:
   - list applications
   - choose the relevant app
   - inspect jobs, stages, executors, SQL executions, environment, or logs
5. If the user says "latest app", "recent run", or similar, list apps first and choose the most relevant recent application before continuing.
6. If the CLI is unavailable, install it with `python -m pip install spark-history-cli` if tool permissions allow it.

## Command patterns

```bash
spark-history-cli --json --server http://localhost:18080 apps
spark-history-cli --json --server http://localhost:18080 app <app-id>
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> jobs
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> stages
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> executors --all
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> sql
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> sql-plan <exec-id> --view final
spark-history-cli --server http://localhost:18080 --app-id <app-id> sql-plan <exec-id> --dot -o plan.dot
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> sql-jobs <exec-id>
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> summary
spark-history-cli --json --server http://localhost:18080 --app-id <app-id> env
spark-history-cli --server http://localhost:18080 --app-id <app-id> logs output.zip
```

If `spark-history-cli` is not on `PATH`, use:

```bash
python -m spark_history_cli --json apps
```

## What to reach for

- `apps` for recent runs, durations, status, and picking candidates
- `app <id>` for high-level details about one run
- `attempts` for multi-attempt apps (list or show specific attempt details)
- `jobs`, `job <id>` for job-level failures or progress
- `job-stages <id>` for stages belonging to a job
- `stages`, `stage <id>` for task/stage bottlenecks
- `stage-summary <id>` for task metric quantiles (p5/p25/p50/p75/p95) — duration, GC, memory, shuffle, I/O
- `stage-tasks <id>` for individual task details — sorted by runtime to find stragglers
- `executors --all` for executor churn or skew investigations
- `sql` for SQL execution history and plan graph data
- `sql-plan <id>` for SQL plan extraction:
  - `--view full` (default): full plan text
  - `--view initial`: only the Initial Plan (pre-AQE)
  - `--view final`: only the Final Plan (post-AQE)
  - `--dot`: Graphviz DOT output for visualizing the plan DAG
  - `--json` + `--view`: structured JSON with `isAdaptive`, `sectionCount`, `plan`, and `sections`
  - `-o <file>`: write output to file instead of stdout
- `sql-jobs <id>` for jobs associated with a SQL execution (fetches all linked jobs by ID)
- `summary` for a concise application overview: app info, resource config (driver/executor/shuffle), and workload stats (jobs/stages/tasks/SQL)
- `env` for Spark config/runtime context
- `logs` only when the user explicitly wants the event log archive saved locally

## Practical guidance

- Preserve the user's server URL if they gave one explicitly.
- Summarize findings after retrieving JSON; do not dump raw JSON unless the user asked for it.
- Treat event logs and benchmark history as potentially sensitive. Download them only when necessary and keep them local.
- This CLI needs a running Spark History Server. It does not replace SHS and it does not parse raw event logs directly.

## Troubleshooting

| Issue | Solution |
|-------|----------|
| `Connection refused` | SHS not running — start with `$SPARK_HOME/sbin/start-history-server.sh` |
| `404 Not Found` on app | App ID may include attempt suffix — use `apps` to list valid IDs |
| No apps listed | Check `spark.history.fs.logDirectory` points to the right event log path |
| `ModuleNotFoundError` | CLI not installed — run `pip install spark-history-cli` |
| Wrong server | Set `SPARK_HISTORY_SERVER` env var or use `--server <url>` |
| Timeout on large apps | SHS may be parsing event logs — wait and retry, or check SHS logs |