signoz-explaining-alerts

---
name: "signoz-explaining-alerts"
description: "Decode an existing SigNoz alert's configuration into a plain-language explanation."
license: "MIT"
---

# Alert Explain

Decode an existing SigNoz alert's configuration into a plain-language
explanation. The skill is read-only and stays focused on the rule
itself: what it watches, when it fires, where it notifies. A single
line of fire-frequency data is included to ground the explanation, but
this skill does **not** investigate any specific fire; that is
`signoz-investigating-alerts`'s job.

## Prerequisites

This skill calls SigNoz MCP server tools (`signoz_get_alert`,
`signoz_list_alert_rules`, `signoz_get_alert_history`). Before running
the workflow, confirm the `signoz_*` tools are available. If they are
not, run `signoz-mcp-setup` first to initialize or repair the MCP connection.
Do not guess at alert configuration from the rule name alone.

## When to use

Use this skill when the user wants to:
- Understand or interpret an existing alert rule.
- Confirm what signal an alert watches and at what threshold.
- Audit whether an alert is reasonably configured.
- Translate raw alert JSON into operational language.

Do NOT use when the user wants to:
- Create a new alert → `signoz-creating-alerts`.
- Diagnose why an alert fired or correlate signals around a fire window
  → `signoz-investigating-alerts`.
- Modify an existing alert → call `signoz_update_alert` directly.

## Required inputs

| Input | Required | Source if missing |
|---|---|---|
| Alert identifier (rule ID or name) | yes | `$ARGUMENTS`, recent context, or fuzzy match |

If the input is missing or ambiguous, this skill is **best-effort** (not
strict; read-only operations are cheap to recover from):

1. Call `signoz_list_alert_rules`, paginate through every page, and find
   the closest name match.
2. State the interpretation in the response:
   "Interpreting your request as alert 'High Error Rate - Checkout' (id 42).
   If you meant a different one, tell me the name or id."
3. Proceed with the explanation. The user can correct after.

## Workflow

### Step 1: Resolve the alert

If the user provided a rule ID (UUID or legacy numeric ID), skip to Step 2.
Otherwise:

1. Call `signoz_list_alert_rules` and **paginate every page**:
   `pagination.hasMore` is true until the full list is walked.
2. Match by name (case-insensitive substring). If multiple match,
   present the candidates and ask which one (interactive) or pick the
   closest and flag the assumption (autonomous).

### Step 2: Fetch the full configuration

Call `signoz_get_alert` with the rule id. This is **mandatory**: the
list response does not include the full condition / thresholds /
notification settings, and explanations based on the name alone are
guesses.

### Step 3: Pull a one-line fire-frequency summary

First call `signoz_get_alert_history` with `timeRange: "7d"`,
`state: "firing"`, `order: "desc"`, and a large limit. Continue only when
`data.nextCursor` exists (the completeness note also reports `hasMore: true`):
pass that value as `cursor`, replace `timeRange` with the note's resolved
absolute `start` and `end`, and preserve the same state, filter, and order.
Stop when `nextCursor` is absent / the note reports `hasMore: false`. Never use
`offset` or infer completeness from page fullness.
History `state` accepts `inactive`, `pending`, `recovering`, `firing`, `nodata`,
or `disabled`. For "resolved" / "recovered", use `state: "inactive"`;
`resolved` is not a value; `recovering` is a transient keep-firing state, not resolution.
Derive a single rule-level line from the complete set. History rows are emitted
per label-group `fingerprint`, so never equate row count with alert-fire count.
Count distinct rule transitions where `overallStateChanged: true` and
`overallState: "firing"` (deduplicate rows from the same transition timestamp):

> Fired N times in the last 7d (last fire: <relative-time>).

If the user asks about a particular service/host/group instead, partition by
`fingerprint` and count that series' rows where `stateChanged: true` and
`state: "firing"`; include its labels and call the result a **series fire**, not
a rule-wide fire. A grouped rule can have many series transitions while the
overall rule remains continuously firing.

If the alert never fired in the window, say so explicitly:
"Has not fired in the last 7d." If the alert is disabled, mention that
separately; disabled rules can still have earlier fires in the lookback.

This single line grounds the explanation. Do **not** drill into specific
fires here; that's `signoz-investigating-alerts`.

### Step 4: Build the explanation

The single most useful thing for the user is a tight summary. Lead
with a **TL;DR that directly answers the question they asked**, not a
generic alert summary. The TL;DR is the only thing some users will
read; burying their answer under a fixed template forces them to
scroll for what they wanted in the first place.

Match the TL;DR shape to the user's question:

- **"What does this alert do?" / "Explain X"**: describe what fires:
  > **TL;DR**: Fires when `<condition>` for `<scope>`, notifies
  > `<channel>`. `<fire-frequency line>`.

- **"Is it configured correctly?" / "Audit this" / "Anything I should
  change?"**: lead with the **verdict and the top 1–3 changes**, not
  the description of what fires:
  > **TL;DR**: Mostly well-configured, but recommend: (1) add
  > `alertOnAbsent` (currently a crashed service stays silent); (2)
  > fix annotation template `{{$topic}}` → `{{$labels.topic}}` (won't
  > interpolate); (3) split critical to PagerDuty (both tiers
  > currently route to Slack). `<fire-frequency line>`.

- **"How does X work?" / "Explain the count guard"**: answer the
  mechanism in 1–2 sentences before any framing:
  > **TL;DR**: The count guard is a `having: count() > 50` clause on
  > query A: any 1-minute bucket with ≤50 spans is dropped before
  > evaluation, so low-traffic minutes can't fire the alert.

- **"What's the threshold?" / focused config question**: state the
  exact thing they asked about:
  > **TL;DR**: Threshold is **3 standard deviations** (z-score), not
  > a raw rate value. Daily seasonality means the model compares
  > each hour against historical norms for that hour.

Always include the fire-frequency line and `disabled` status if
non-default; those ground every kind of TL;DR. But put the answer to
the user's specific question first.

After the TL;DR, write the explanation in prose, organized into the
four sections below. **Skip any section that has nothing meaningful to
add**: empty severity labels, default notification settings, vanilla
annotations don't deserve a header. Short and skimmable beats
perfunctorily complete; the user is not reading a checklist.

**1. What it watches**: one short paragraph. Combine signal type
(metrics / logs / traces / exceptions), what the query measures, and
scope. Translate the query to operational language; for formulas, name
each sub-query (A, B, …) and state what F1 (or whichever
`selectedQueryName` triggers) computes, e.g. "F1 = A × 100 / B → error
percentage". Decode filter operators (`=` equals, `!=` not equals,
`IN` / `NOT IN`, `LIKE` / `ILIKE`, `CONTAINS`, `REGEXP`, `EXISTS` /
`NOT EXISTS`); enumerate `IN` / `NOT IN` value lists so the user can
verify them. Name each `groupBy` dimension and its practical effect
("fires separately per service" for `service.name`).

For **anomaly rules** (`ruleType: anomaly_rule`), explicitly state that
the threshold is in **standard deviations from the learned pattern, not
the raw value**; this is the most common point of confusion. Include
`algorithm` (`standard`, which is z-score based), `seasonality` (hourly /
daily / weekly), and how
lower/higher targets shift sensitivity (lower → more noise, higher →
only extreme deviations).

**2. When it fires**: one paragraph covering threshold + timing.
Decode the threshold spec into plain English using these mappings:

- `op` codes: `1` above, `2` below, `3` equal, `4` not equal, `5`
  above-or-equal, `6` below-or-equal, `7` outside-bounds
  (`outside_bounds` in payloads).
- `matchType` codes: `1` at_least_once (any point in window), `2`
  all_the_times (entire window), `3` on_average (window average), `4`
  in_total (window sum), `5` last (most recent point).

State each threshold tier's `name`, `target`, `targetUnit`, and
attached channels. **Always state the threshold in `targetUnit`, not
the native query unit** (e.g. "fires when p99 exceeds 500 ms", not
"…exceeds 500 000 000 ns"). Note `recoveryTarget` if set (hysteresis);
if absent, mention flap risk when the value hovers near the boundary.
Describe timing as "checks every `<frequency>` over the last
`<evalWindow>`", and mention that with `at_least_once` a single-point
breach triggers, while `all_the_times` requires the full window.

**3. Where it notifies**: decode effective routing, not just fields in
isolation. If `notificationSettings.usePolicy` is true, say that org policy
matches the rule/static/dynamic labels and that threshold channels are ignored.
Otherwise, a v2 `threshold_rule` or `promql_rule` routes only through each
threshold tier's `channels`. If a tier has no channels, call it unrouted or
misconfigured; never substitute top-level `preferredChannels` as a per-tier
fallback. A v1 `anomaly_rule` has no thresholds or notification settings, so
its direct routing comes from `preferredChannels`. Treat that field as routing
only when the fetched rule schema defines it, not as a cross-schema fallback.
Resolve names from the fully paginated `signoz_list_notification_channels`
result if needed. Also explain `notificationSettings.groupBy` (bundling) and
`renotify` (interval + states). Skip this section only when routing is already
unambiguous in the TL;DR.

**4. Notable concerns**: flag *only* what's non-default and worth the
user's attention. Don't list every absent field; focus on the
high-impact ones:

- **`alertOnAbsent` missing** when the signal is critical: silent data
  loss (crashed service, broken instrumentation) won't trigger the
  alert. Always call this out for production-tier rules.
- **`alertOnAbsent: true` but `nodata` not in `renotify.alertStates`**:
  the absent-data fire pages once and then goes silent (easy to miss).
- **Template variable bugs**: `{{$topic}}` won't interpolate; the
  correct form is `{{$labels.topic}}`. Dots in label keys become
  underscores (`service.name` → `{{$labels.service_name}}`).
- **Multiple severity tiers but `labels.severity` missing on the rule**:
  breaks label-based routing policies. Common gap.
- **All tiers route to the same channel**: defeats the point of
  graduated thresholds.
- **High-cardinality `groupBy`** (e.g. `pod.name` × `partition`) →
  notification-storm risk during cluster-wide events.
- **Annotation/description text contradicts `matchType`** (e.g.
  description says "for over 5 minutes" but `matchType=at_least_once`
  fires on first breach within the window).
- **Alert name doesn't match the filter target** (e.g. name says
  "checkout" but filter targets `payments`); call this out.

If none of these apply, omit the section rather than pad it.

If the user asked only "what does this alert do", stop here. The audit
(Step 5) is for "is it configured correctly" / "audit this" /
"anything I should change" requests.

### Step 5: Assess the configuration (only if asked)

The user may ask "is this alert reasonable" alongside the explanation.
Only assess when asked or when the request implies it (audit, review,
"is this configured correctly"). Keep assessment grounded in what's
actually in the config:

- **Threshold calibration**: appropriate for the signal? Consider
  service criticality and traffic.
- **matchType fit**: `at_least_once` is sensitive (catches transients);
  `all_the_times` is conservative; `on_average` smooths noise.
- **Window vs frequency**: short window + `at_least_once` can be noisy.
  Long window can delay detection.
- **Multi-severity**: alerts with both warning and critical thresholds
  enable graduated response. Single-severity alerts miss this.
- **Notification routing**: critical → high-urgency channels (PagerDuty);
  warning → low-urgency (Slack).
- **Missing runbook / description**: if `annotations` are empty or
  default, suggest adding context.
- **Absent-data monitoring**: for critical signals, recommend
  `alertOnAbsent: true` if it isn't set.
- **GroupBy cardinality**: high-cardinality groupBy fields can produce
  many independent alert series; flag potential notification storms.
- **Filter completeness**: for `IN` / `NOT IN` filters with explicit
  value lists, flag values that look out of place or missing values
  that seem expected.
- **Fire frequency vs threshold**: if Step 3 shows the alert fires
  many times a day (>10/day in the 7d window), the threshold is likely
  too tight; if it never fires and the user is asking because they
  expected it to, the threshold may be too loose or the query may be
  wrong.

### Step 6: Offer next steps

Surface up to 3 follow-up intents based on what the explanation
revealed: things like investigating a recent fire, running the
underlying query to see current values, adjusting a threshold, or
creating a related alert for a coverage gap. Use your judgment; do
not pad to 3.

Skip follow-ups entirely when the user is purely inspecting ("what
does this alert do?") and signals no further intent. Offering no
follow-ups is better than offering wrong ones.

## Guardrails

- **Fetch before explaining.** Always call `signoz_get_alert`. Do not
  base explanations on the rule name or list response alone.
- **Always pull fire history.** The one-line frequency summary grounds the
  explanation, including for disabled alerts: disabled rules can still have
  earlier fires in the lookback.
- **Decode, don't dump.** Translate `op`, `matchType`, filter
  expressions, and query JSON into operational language. Show raw JSON
  only if the user asks.
- **Decode every sub-query.** For formula alerts, explain each builder
  query and the formula. Don't gloss over a formula as "error rate";
  show the user what A, B, and F1 each compute.
- **Anchor to the actual config.** Don't speculate based on the alert
  name. If the name says "checkout" but the filter targets `payments`,
  call that out.
- **No live queries by default.** Do not run the alert's query unless
  the user asks. Explain is about understanding the rule, not
  inspecting current data.
- **Paginate alert listing.** Before concluding an alert does not
  exist, walk every page of `signoz_list_alert_rules`.
- **SigNoz evaluation model.** SigNoz uses `evalWindow` + `frequency`
  + `matchType`, not Prometheus-style `for` durations. Don't describe
  the evaluation model in Prometheus terms.
- **Stay in scope.** Investigation of a specific fire window belongs
  to `signoz-investigating-alerts`. Modifications belong to
  `signoz_update_alert`.

## Examples

**User:** "Explain my checkout error rate alert"

**Agent:**
1. `signoz_list_alert_rules` (paginated) → finds
   "High Error Rate - Checkout" id 42.
2. `signoz_get_alert id=42` → traces formula (A errored spans / B
   total spans × 100), single critical threshold at 5%, `op=1`,
   `matchType=1`, channel `pagerduty-oncall`.
3. `signoz_get_alert_history id=42 timeRange=7d state=firing order=desc`
   → fired 3 times, last fire 2h ago.
4. Replies:

   > **TL;DR**: Fires when checkout error rate (errored spans / total
   > spans) exceeds 5% at any point in the last 5 min, pages
   > `pagerduty-oncall`. Fired 3 times in the last 7d (last 2h ago).
   >
   > **What it watches**: traces from `service.name = 'checkout'`.
   > Query A counts spans with `has_error = true`, query B counts all
   > spans, F1 = A × 100 / B is the error percentage; the alert
   > triggers on F1.
   >
   > **When it fires**: `op=above`, `matchType=at_least_once`, target
   > 5%. Checks every 1 min over the last 5 min; a single elevated
   > minute fires it.
   >
   > **Notable**: no `recoveryTarget` (could flap near 5%); no
   > `alertOnAbsent` (a checkout outage that drops traffic to zero
   > won't trigger this rule).

5. Offers: "Want me to investigate the fire from 2h ago?"

---

**User:** "Explain alert rule 17 - is it well configured?"

**Agent:**
1. `signoz_get_alert id=17` → metrics, `system.cpu.utilization`,
   two thresholds (warning 80%, critical 90%), `matchType=3` on_average,
   filter `deployment.environment.name = 'production'`, groupBy
   `host.name`.
2. `signoz_get_alert_history id=17` → has not fired in last 7d.
3. The user asked "is it well configured?"; the TL;DR leads with the
   **verdict**, not a generic description. Then explanation + full
   audit:

   > **TL;DR**: Mostly well-configured (dual-threshold + `on_average`
   > is the right call for CPU), but two gaps to fix: (1) no
   > `alertOnAbsent` (a crashed agent silently stops reporting and
   > this rule won't catch it); (2) annotations are default (a
   > runbook URL would help on-call). Has not fired in the last 7d
   > (could mean production is healthy or thresholds too loose).
   >
   > **What it watches**: `system.cpu.utilization` on
   > `deployment.environment.name = 'production'`, grouped by
   > `host.name` (fires per host).
   >
   > **When it fires**: `op=above`, `matchType=on_average`, two tiers
   > (warning 80%, critical 90%) over a 5-min window. Short spikes
   > don't fire; smoothing is correct for CPU.
   >
   > **Notable concerns**: same as TL;DR fixes above; nothing else
   > non-default.

4. Offers next steps.

---

**User:** "Help me understand this anomaly detection alert" (alert id 88)

**Agent:**
1. `signoz_get_alert id=88` → `ruleType: anomaly_rule`,
   `algorithm=standard` (z-score based), `seasonality=daily`, target 3, metric
   `http.server.request.duration`, scope `service.name = 'api-gateway'`.
2. History: fired 1 time in last 7d.
3. Replies:

   > **TL;DR**: Fires when api-gateway request latency deviates by more
   > than **3 standard deviations** (not raw latency, not a fixed value)
   > from its learned daily pattern. Fired once in the last 7d.
   >
   > **What it watches**: `http.server.request.duration` for
   > `service.name = 'api-gateway'`, evaluated as a Z-score anomaly
   > with **daily seasonality**: the model learns the typical pattern
   > for each hour of day, so peak-hour latency won't false-trigger if
   > it matches the historical norm for that hour.
   >
   > **When it fires**: when |Z-score| > 3, i.e. the value is more
   > than 3 standard deviations away from the expected pattern. Lower
   > target → more sensitive (more noise); higher → only extreme
   > deviations. The threshold is **not** in seconds or milliseconds.

4. Offers to investigate the recent fire.