evaluate

---
name: evaluate
description: Evaluate your RAG pipeline with metrics
license: MIT
---

# Evaluate: Measure Your RAG Pipeline's Quality

> **Scope:** This skill focuses on **answer quality metrics** (faithfulness, relevancy, correctness) against a labeled test set. For operational performance metrics (latency, throughput, cost), use `/benchmark`.

Run a structured evaluation of the learner's RAG pipeline using established metrics. This skill teaches evaluation methodology while generating actionable results.

## Step 1: Identify the Pipeline

Welcome! Let's see how your RAG pipeline is performing.

First, check whether the learner has existing work to evaluate:
- Look for a learner profile at `progress/learner-profile.md` and for code in `src/` and `projects/`.
- If pipeline code exists in `projects/`, great — proceed. If multiple pipelines exist, ask which one to evaluate.
- If **no pipeline or RAG code exists** anywhere in `projects/` or `src/`, guide them warmly:
  > "It looks like you haven't built a pipeline yet — that's totally fine! Let's get you set up first. Run `/build` to create your first RAG pipeline, and then come back here to see how it scores. It only takes a few minutes to get something running!"

  Stop here — do not continue to Step 2.
- If a pipeline is found, verify it has the minimum components: a retriever and a generator.

## Step 2: Explain the Evaluation Framework

Before running metrics, teach the learner what they are measuring and why:

### Core RAG Metrics
- **Faithfulness**: Does the generated answer stick to the retrieved context? (measures hallucination)
- **Answer Relevancy**: Is the answer actually relevant to the question asked?
- **Context Precision**: Are the retrieved documents relevant to the question?
- **Context Recall**: Did the retriever find all the relevant information?

### Additional Metrics (if applicable)
- **Answer Correctness**: How close is the answer to a ground truth answer?
- **Latency**: How long does each pipeline stage take?
- **Token Usage**: How many tokens are consumed per query?

Explain each metric with a simple analogy so the learner builds intuition.

## Step 3: Prepare Test Data

Help the learner create or use evaluation data:

1. **Generate test queries**: Create 10-20 representative questions for their corpus.
2. **Create ground truth** (if needed): For each question, write the ideal answer and list the relevant source documents.
3. **Save the test set** to `projects/[pipeline-name]/eval/test-set.json`.

If they already have a test set, load and validate it.

## Step 4: Run the Evaluation

Walk through running the evaluation step by step:

1. Set up the evaluation framework (RAGAS, custom, or both).
2. Run each test query through the pipeline, capturing: the query, retrieved contexts, generated answer, and latency.
3. Compute metrics for each query.
4. Aggregate results.

Provide code snippets the learner can run. Explain what each part does.

## Step 5: Present Results

Display results in a clear, readable format:

```
RAG Pipeline Evaluation Report
==============================
Pipeline: [name]
Test queries: [N]
Date: [today]

Overall Scores:
  Faithfulness:      [score] / 1.0
  Answer Relevancy:  [score] / 1.0
  Context Precision:  [score] / 1.0
  Context Recall:     [score] / 1.0

Avg Latency: [X]ms (retrieval: [Y]ms, generation: [Z]ms)
```

Highlight the weakest metric and explain what it means practically.

## Step 6: Improvement Recommendations

Based on the results, provide specific, actionable suggestions:

- Low faithfulness -> check prompt template, add grounding instructions, consider smaller chunks
- Low context precision -> revisit chunking strategy, try re-ranking, tune embedding model
- Low context recall -> check chunk coverage, try hybrid search, increase top-k
- High latency -> profile each stage, consider caching, batch embeddings

Prioritize the top 2-3 improvements that would have the biggest impact.

## Step 7: Save and Track

Save the evaluation report to `projects/[pipeline-name]/eval/reports/report-[date].md`. Update `progress/module-tracker.md` to note the evaluation was completed. Encourage the learner to re-evaluate after making improvements to see their progress.

Suggest 2-3 relevant next steps using slash commands:

- `/benchmark` — measure operational performance (latency, throughput, cost) alongside quality
- `/debug-rag` — diagnose and fix the weakest areas identified in your evaluation
- `/build` — implement the improvements suggested above and re-evaluate