Define what your agent needs to log before you can debug it in production

You are designing the observability and tracing schema for an agent pipeline, deciding exactly what gets logged at each stage before an incident forces you to add logging retroactively to a system nobody can currently explain the behavior of.

PIPELINE STAGES
query rewrite -> hybrid retrieval -> reranking -> generation -> citation mapping

CURRENT LOGGING GAPS
Currently only end-to-end latency and a final success/failure boolean are logged; when generation returns a wrong answer there's no way to tell if retrieval or the prompt was at fault — start from what's actually missing today, not a theoretical complete observability stack, since maximal logging that never gets used is its own cost.

SPAN SCHEMA REQUIREMENTS
For each stage, define a span with:
- Inputs actually received — not "the prompt" as a blob, but the specific variables/parameters that went in, so a bad output can be traced back to a specific bad input.
- The exact model/tool version invoked, since "the LLM said X" is useless for debugging a regression if you can't tell whether it was the old model version or the new one that said it.
- Latency for that stage specifically, not just end-to-end — an end-to-end latency spike could be retrieval, generation, or a downstream tool call, and only per-stage timing tells you which.
- Token counts, input and output, per LLM call, for cost attribution per stage, not just a total per request.
- A stage-specific outcome field, such as chunks returned plus scores for retrieval or success/failure plus error for a tool call — not a generic "success: true/false" that can't distinguish a clean success from one that technically finished but returned something wrong.

CORRELATION
Every span for a single end-to-end request must share a UUID generated at the API gateway, propagated as a header through every internal service call, so a slow or wrong response can be reconstructed as one connected trace across every stage it touched, not scattered log lines with no way to associate them.

PII AND RETENTION
user question text and retrieved document excerpts are logged in full to a restricted, access-controlled trace store; the same fields are redacted to a hash in the general-purpose metrics store used for dashboards — state explicitly what gets redacted before it's written to logs versus what's needed in full for debugging and therefore has to live in a more restricted store with its own retention and access policy, rather than either logging everything raw or redacting so aggressively that debugging becomes impossible.

SAMPLING
10% of successful requests sampled for full tracing; 100% of requests where generation's finish_reason is not 'stop', or where a stage returned an error — full tracing on every request if volume allows it; if not, state the sampling rate and, critically, whether failed or flagged requests are always fully traced regardless of the sampling rate. They should be — sampling should never be the reason a real incident has no trace.

OUTPUT FORMAT
1. The span schema per stage, as field names and types.
2. The trace correlation strategy.
3. The PII redaction rule per field category.
4. The sampling rule, stated as a specific percentage plus the always-trace exceptions.