troubleshoot-spark-yarn

---
name: troubleshoot-spark-yarn
description: Diagnose and triage Spark applications running on Hadoop YARN using bundled local YARN documentation and an operational routing guide. Use when Codex needs grounded help with Spark jobs stuck in SUBMITTED or ACCEPTED, queue starvation, ApplicationMaster or executor loss, container OOM or exit 137, NodeManager unhealthy nodes, cgroups or container runtime issues, ResourceManager HA or failover symptoms, preemption, placement constraints, log collection, or separating Spark-engine issues from YARN infrastructure problems.
license: MIT
---

# Troubleshoot Spark YARN

## Objetivo

Use this skill to turn Spark-on-YARN symptoms into a focused diagnosis backed by the bundled YARN scrape. Start with the incident routing file, then load only the YARN sections needed to confirm or reject the current hypothesis.

## Triage Minimo

Collect this before proposing fixes:

- Spark application ID and user.
- Queue, deploy mode, submission time, and current YARN state.
- Whether the failure is on submit, AM startup, executor runtime, or control plane.
- Number of attempts, lost executors or containers, and the node names involved.
- Requested Spark resources that matter to YARN: AM or driver memory, executor memory, executor cores, and container overhead if known.
- Whether there was RM failover, NM restart, decommission, or node health degradation during the incident.

## Workflow

1. Classify the symptom:
   - submission or scheduling
   - AM startup
   - executor instability
   - node-scoped or container runtime
   - control-plane or HA
2. Read `references/spark-incident-routing.md` for the symptom-specific path.
3. Use `references/navigation.md` to jump into the exact YARN chapter.
4. Search `references/yarn_references.md` with exact headings or narrow keywords before opening long ranges.
5. Confirm whether the bottleneck is YARN-side or Spark-engine-side.
6. Return an evidence-backed diagnosis with immediate actions first.

## Evidence Sequence

Start with the smallest command set that can anchor the diagnosis:

```bash
yarn application -status <appId>
yarn applicationattempt -list <appId>
yarn logs -applicationId <appId> -show_application_log_info
yarn logs -applicationId <appId> -am ALL
yarn queue -status <queueName>
```

If the incident is node- or container-scoped, expand with:

```bash
yarn container -list <applicationAttemptId>
yarn container -status <containerId>
yarn node -status <nodeId>
```

If HA or failover is suspected, expand with:

```bash
yarn rmadmin -getServiceState <rmId>
```

Use the documented YARN flags and semantics from the bundled scrape when interpreting these commands.

## Decision Rules

- If the app is stuck in `SUBMITTED` or `ACCEPTED`, inspect queue capacity, user limits, maximum allocation, node labels or attributes, and placement constraints before tuning Spark code.
- If the AM starts but executors never stabilize, inspect `NodeManager`, `Using CGroups`, secure containers, and container runtime sections before changing Spark retries.
- If failures cluster on specific hosts, prioritize `yarn node -status`, node health, disk checks, decommission, and local or log directory pressure.
- If symptoms correlate with RM transitions or sudden app-attempt churn, inspect `ResourceManager HA` and `ResourceManager Restart`.
- If YARN allocations look healthy but the job is slow due to skew, shuffle explosion, UDFs, or poor physical plans, switch to or combine with `ds-shopper-spark`.
- State explicitly when the local YARN material is insufficient instead of inferring unsupported behavior.

## Reference Files

- `references/spark-incident-routing.md`: symptom-driven routing for Spark on YARN incidents.
- `references/navigation.md`: compact YARN topic map and line anchors.
- `references/yarn_references.md`: full scraped documentation from the Hadoop YARN site.

## Response Contract

Respond with this structure:

```markdown
## Diagnostico
- Sintoma y alcance:
- Evidencia YARN:
- Evidencia Spark:
- Hipotesis principal:

## Acciones
- P0:
- P1:
- P2:

## Validacion
- Comandos o metricas a revisar:
- Resultado esperado:
- Criterio para descartar la hipotesis:

## Trazabilidad
- Referencias usadas:
- Lineas o secciones consultadas:
```