profile-model

---
name: profile-model
description: >
license: Apache-2.0
---

# Profile a model on MAX

This skill answers three questions about any model you can run on MAX, whether
it's a built-in architecture or a custom one loaded with
`--custom-architectures`:

1. **Is my GPU actually being used?** (utilization, memory, clocks)
2. **Where does my model spend the most time?** (which kernels / phases)
3. **Why is *this* kernel slow?** (occupancy, stalls, roofline)

It works from a `pip` or `pixi` install of MAX, driving the public `max` CLI or
a Python script.

## How to decide what to run

Each question needs a deeper, more expensive capture than the last. Work from
the cheapest check toward the most invasive, and let each result decide whether
going deeper is even worth it — don't capture more than the question needs.

1. **Start with the utilization check**. If the GPU is idle or lightly loaded
   while the workload runs, the bottleneck is host-side (CPU, transfers, small
   batch) — stop and report that. A kernel trace won't tell you anything a
   busy GPU wouldn't.
2. **If the GPU is busy, capture a kernel breakdown**. This is the common case
   and usually the final answer: it shows which kernels dominate GPU time.
3. **Only if one kernel dominates and you need to know *why*, do a single-kernel
   deep dive** on that one kernel. Skip this unless a breakdown has already
   pointed at a specific kernel — it replays the kernel many times and is slow.

| What you're answering    | Tool                                                        | Cost                        |
|--------------------------|-------------------------------------------------------------|-----------------------------|
| Is the GPU busy?         | `max.profiler.gpu` (pure Python, NVIDIA + AMD)              | seconds, no extra installs  |
| Where does time go?      | `max ... --profile`, or `nsys` (NVIDIA) / `rocprofv3` (AMD) | a couple of profiled runs   |
| Why is one kernel slow?  | Nsight Compute (`ncu`), NVIDIA only                         | one slow capture per kernel |

Read the reference for the step you're on rather than loading all of them:

- **Utilization check** →
  [`references/utilization-api.md`](references/utilization-api.md) (and the
  bundled `scripts/gpu_snapshot.py`)
- **Kernel breakdown, NVIDIA** → [`references/nsys.md`](references/nsys.md)
- **Kernel breakdown, AMD** →
  [`references/rocprofv3.md`](references/rocprofv3.md)
- **Single-kernel deep dive (NVIDIA)** →
  [`references/ncu.md`](references/ncu.md)

### Models loaded with `--custom-architectures`

Profiling itself works the same for a custom architecture as for a built-in
one. What differs is loading: a custom architecture has extra prerequisites and
failure modes — import compatibility with your installed MAX version,
`PYTHONPATH` traps, architecture-name collisions, base models with no chat
template, and a `max serve` limitation for custom embeddings models. Those stop
the model from loading *before* any profiling can run. If you're profiling a
model passed via `--custom-architectures`, read
[`references/custom-architectures.md`](references/custom-architectures.md)
**first**.

## Reading the result (what the kernel mix tells you)

Profiling answers two things: *is the GPU saturated* (the utilization check) and
*where does time go* (the kernel breakdown). Map what you see back to a
diagnosis:

- **Decode dominated by `gemv_split_k_*` GEMVs, with `*_mha_*` attention and
  `rms_norm` below** — this is the normal, healthy shape for a *dense* LLM token
  generation (decode is memory-bound GEMV). Nothing to chase.
- **Sparse-MoE decode dominated by grouped / block-scaled matmul plus routing
  kernels** — the per-expert grouped matmul (`*grouped*` / `block_scaled_*`
  matmul) is the dominant kernel instead of a dense GEMM, alongside top-k
  routing (`topk_*`, `moe_create_indices`) and expert gather/scatter; on
  multi-GPU, EP `dispatch`/`combine` collectives appear too. This is the healthy
  shape for a Mixture-of-Experts model — routing and gather overhead is
  expected, and only worth chasing if it rivals the matmul itself.
- **Low-bit weights (FP8 / FP4 / NVFP4) add quant-prep kernels** — alongside the
  matmul you'll see dynamic activation-quantization and scale-layout kernels
  (`*quantize*`, `*block_scales*`) and block-scaled matmul variants rather than
  plain `gemv`/`gemm`. Time spent in quant/dequant prep is normal for a low-bit
  model; flag it only if it dwarfs the matmul it feeds. Kernel *prefixes* are
  GPU-arch-specific (`sm100_*` on Blackwell, different on Hopper / MI) — match
  on the kernel *family*, not the exact name.
- **A custom architecture *not* hitting the expected fused kernel** (for example
  attention showing as generic `elementwise`/`matmul` instead of a `*_mha_*`
  kernel, or norms/RoPE unfused) — a wiring signal: the graph may not be built
  the way you think, even if logits pass. Worth a look during bring-up.
- **Pathologically low decode tok/s + an outsized prefill-shaped kernel mix
  repeating every step** — classic O(n²) re-prefill (no/broken KV cache). This
  is a correctness-adjacent bug, not just slowness.
- **Low `gpu_usage_percent` while a workload runs** — host-bound, launch/sync
  overhead, or batch too small; the GPU isn't the bottleneck.
- **Throttle reasons present** — the GPU is clock-limited; numbers aren't
  representative until you address thermal/power.

## The durable algorithm

Tooling and MAX packaging change often; this shape is the part worth
preserving. Whichever capture you run, follow it:

1. **Detect the environment before committing to a tool**. Run `nvidia-smi` or
   `rocm-smi` to learn the vendor — the kernel-breakdown and deep-dive tools
   differ by vendor. Confirm the profiler is installed (`which nsys` /
   `which rocprofv3` / `which ncu`); if it's missing, tell the user the exact
   install line (see the reference) rather than failing midway.
2. **Confirm before any long-lived run**. Profiling `max serve` warms a
   compile cache (cold compile can take minutes), launches a server, and runs a
   benchmark. Before doing that, show the user the model, the flags, and the
   planned commands, and wait for confirmation. A one-shot
   `max generate --profile` on a tiny model is cheap enough to skip this.
3. **Warm caches first, profile second**. The first run of a model pays
   one-time compile and weight-load costs that drown out the real kernel time.
   Do an unprofiled warm-up run, *then* the profiled run, so the capture
   reflects steady-state inference — not compilation.
4. **Keep the capture window small**. A 10-second window at concurrency 1 is
   enough to see the kernel mix. Long captures produce huge trace files that
   are slow to open and no more informative.
5. **Fail fast**. After launching a server, check within a few seconds that the
   process is alive and the log has no early error (model not found, OOM, bad
   flag) before you start polling for `/health`. Don't wait out a full timeout
   on a server that already died.
6. **Verify the artifact, then report**. Confirm the output file exists and is
   non-trivial in size before claiming success. Report the artifact path, how
   to open it, and the headline numbers (top kernels, or peak GPU utilization).
7. **Clean up only what you started**. Kill the server / benchmark process
   group you launched. Avoid broad `pkill -f max` on a shared box — you may
   stop someone else's run.

## Nightly vs stable

Profiling features land in nightly before stable, so this skill targets the
nightly `max-serve` build. Install it with pixi:

```bash
pixi init quickstart -c https://conda.modular.com/max-nightly/ -c conda-forge
cd quickstart && pixi add max-serve
pixi run max --version          # expect a *.dev build
```

The utilization API (`GPUDiagContext`, `BackgroundRecorder`) ships in the
**conda** `max` package, so install via pixi or conda rather than a plain
`pip install max` wheel, which may not expose `max.profiler.gpu`. If a
command below 404s or an import fails, confirm you're on a recent nightly
build.

## Install notes

- MAX itself: a project with the nightly `max-serve` package installed via pixi
  (see [Nightly vs stable](#nightly-vs-stable) above). All
  `max` CLI commands below assume you can run `pixi run max ...` in that
  project. The `max-serve` package includes extra dependencies required to
  serve a model with MAX, which aren't included with the main `max` package.
- `nsys` and `ncu` ship with the CUDA Toolkit. If `which nsys` fails but CUDA is
  present, `export PATH=/usr/local/cuda/bin:$PATH`. Otherwise install Nsight
  Systems / Nsight Compute from NVIDIA, or
  `sudo apt-get install nsight-systems`.
- `rocprofv3` ships with ROCm (`/opt/rocm/bin`). Add it to PATH if needed.
- The utilization API needs no extra tooling beyond `max` — it talks to
  NVML / ROCm SMI directly through `max.profiler.gpu`.

## Fast paths

The lowest-friction commands, each expanded in its reference file:

- **"Is my GPU being used?"** → run `scripts/gpu_snapshot.py` alongside a short
  benchmark. Pure Python, works on NVIDIA and AMD.
- **"Give me a kernel breakdown, fast."** → `pixi run max generate --model <m>
  --prompt "hello" --num-warmups 1 --profile`. Prints a ranked top-N GPU-kernel
  table
  and writes a `.nsys-rep` (NVIDIA). Falls back to a CPU summary without a GPU.
- **"Profile my serving benchmark."** → the serving-endpoint flow with
  `MODULAR_ENABLE_PROFILING=detailed` + `nsys launch` / `rocprofv3`.