Build an automated eval harness for your RAG pipeline instead of eyeballing answers

You are designing an automated evaluation harness for a RAG pipeline, meant to run on every meaningful change to chunking, retrieval, or the prompt template — not a one-time manual review of a handful of answers that looked fine and then got shipped.

PIPELINE UNDER TEST
Internal support RAG bot: Qdrant dense retrieval top-8, no reranker, GPT-5.1 generation with a grounded-answer system prompt

GOLDEN SET
40 hand-written questions from real support tickets, each labeled with the 1-3 chunk IDs that should be retrieved and a reference answer written by a subject-matter expert

METRICS TO IMPLEMENT
Implement all of these, don't substitute a single overall "looks good" score:
- Retrieval precision@k and recall@k against the golden set's labeled relevant chunks — did the right source actually make it into the top-k candidates, separate from whether the final answer was good.
- Faithfulness/groundedness — for each generated answer, can every claim be traced to something actually present in the retrieved context, or did the model add something not supported by the sources it was given.
- Answer relevance — does the answer actually address the question asked, independent of faithfulness, since a perfectly grounded answer to the wrong question still fails the user.
- Latency, measured separately for retrieval and generation, at p50 and p95, not just an average that hides tail latency.

REGRESSION RULES
- Define a numeric threshold for each metric below which a pipeline change is blocked, e.g. "recall@5 must not drop more than 5 percentage points versus the last approved baseline." A metric with no threshold is a chart nobody acts on, not a gate.
- Run the full golden set on every candidate change, not a sample — a chunking change that regresses a handful of golden questions out of forty is real signal a five-question spot check would miss entirely.
- When faithfulness drops, require the harness to surface the specific unsupported claim and the retrieved context it should have been grounded in, not just a pass/fail score — a raw score change with no example is not actionable.

JUDGE MODEL DISCIPLINE
Claude Sonnet 4.6 as the faithfulness/relevance judge, calibrated against 15 questions double-scored by a human reviewer before trusting it on the remaining 25
State explicitly how the LLM-judge's own accuracy was checked against a human-labeled subset before trusting its faithfulness and relevance scores at scale — an unvalidated judge model can systematically miss the same failure category the pipeline itself is prone to, silently agreeing with wrong answers it should be catching.

OUTPUT FORMAT
1. The full metric list with formulas/definitions as they'll actually be computed, not just names.
2. The regression thresholds per metric, with a one-line justification for each number.
3. A worked example: one golden-set question scored end to end (retrieval hit/miss, faithfulness verdict with the specific claim checked, relevance verdict, latency).
4. How often this harness should run — on every PR, nightly, or only before a release — and why, tied to how expensive a full run is.