Evaluating LLM accuracy with RAGAS: a practical walkthrough

The evaluation problem in RAG systems

Shipping a RAG system without systematic evaluation is flying blind. You can manually inspect a handful of responses and feel confident, then discover in production that the system consistently fails on a category of queries you never thought to test.

The challenge is that RAG evaluation has multiple dimensions. A response can be fluent and confident while being completely unfaithful to the retrieved context—a hallucination dressed up in authoritative prose. It can faithfully summarise retrieved documents that were entirely irrelevant to the question. It can answer a different question than the one asked. Each failure mode requires a different measurement.

RAGAS (Retrieval Augmented Generation Assessment) addresses this by defining four independent metrics that together characterise the quality of a RAG pipeline: faithfulness, answer relevancy, context precision, and context recall. Each metric uses LLM-as-judge to produce a score between 0 and 1, requiring no human-labelled reference answers for three of the four metrics.

This walkthrough covers a real evaluation pipeline from dataset construction through CI integration, with the kind of failure modes you will encounter and how to diagnose them.

Installing and configuring RAGAS

pip install ragas langchain-anthropic datasets

RAGAS uses LangChain's LLM interface for its judge model. Configure it with Claude:

from langchain_anthropic import ChatAnthropic
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

# The judge model — use a capable model for reliable scoring
judge_llm = ChatAnthropic(
    model="claude-opus-4-5",
    temperature=0,  # deterministic scoring
)

# Configure RAGAS metrics to use your judge model
from ragas.llms import LangchainLLMWrapper

judge = LangchainLLMWrapper(judge_llm)
faithfulness.llm = judge
answer_relevancy.llm = judge
context_precision.llm = judge
context_recall.llm = judge

Constructing the evaluation dataset

RAGAS expects an EvaluationDataset (or HuggingFace Dataset) with specific columns. The required fields depend on which metrics you run:

The contexts field is a list of strings—each retrieved document chunk passed to the LLM. The ground_truth field is the expected answer, needed only for context precision and recall.

from datasets import Dataset

# Build your evaluation samples
eval_samples = [
    {
        "question": "What is the maximum file size for Firebase Storage uploads?",
        "answer": "Firebase Storage supports uploads up to 5 TB per file.",
        "contexts": [
            "Firebase Storage allows you to upload files of any size. The maximum object size is 5 TB.",
            "For resumable uploads, files larger than 5 MB should use the resumable upload API to handle network interruptions.",
        ],
        "ground_truth": "The maximum file size for Firebase Storage uploads is 5 TB per object.",
    },
    {
        "question": "How do Firestore security rules handle nested collections?",
        "answer": "Firestore security rules apply recursively to all subcollections by default.",
        "contexts": [
            "Firestore security rules are NOT recursive. A rule written for /users/{userId} does not automatically apply to /users/{userId}/posts/{postId}. You must write explicit rules for each collection path.",
        ],
        "ground_truth": "Firestore security rules do not apply recursively. Each subcollection requires explicit rules.",
    },
    # Add more samples...
]

dataset = Dataset.from_list(eval_samples)

Note the second sample: the answer ("applies recursively by default") directly contradicts the retrieved context. This is exactly the kind of hallucination faithfulness scoring is designed to catch.

Running the evaluation

from ragas import evaluate

results = evaluate(
    dataset=dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ],
)

print(results)
# Output:
# {'faithfulness': 0.5, 'answer_relevancy': 0.78, 'context_precision': 0.83, 'context_recall': 0.91}

# Access per-sample scores
df = results.to_pandas()
print(df[["question", "faithfulness", "answer_relevancy"]])

The aggregate scores hide important information. Always inspect per-sample scores to understand which questions your system handles poorly.

Understanding each metric

Faithfulness

Faithfulness measures whether every claim in the answer can be attributed to the retrieved context. The RAGAS implementation decomposes the answer into atomic statements, then asks the judge model whether each statement is entailed by the context.

faithfulness = (number of statements supported by context) / (total statements in answer)

A score of 1.0 means every claim in the answer is grounded in the retrieved documents. A score of 0.5 means half the claims are hallucinated.

The second sample in our dataset should score close to 0.0 on faithfulness—the answer asserts recursive application, but the context explicitly states the opposite.

Faithfulness failures fall into three patterns:

1. Outright hallucination: the model asserts facts not present in any retrieved document.
2. Overgeneralisation: the model extends a specific fact to a broader claim not supported by the context.
3. Context contradiction: the model asserts the opposite of what the context states, often because the model's parametric knowledge conflicts with the retrieved information.

Answer Relevancy

Answer relevancy measures whether the answer actually addresses the question asked. The RAGAS implementation generates alternative questions from the answer and measures their similarity to the original question.

A low relevancy score means the answer drifts from the question—it may be accurate about something, just not the thing that was asked. This often occurs when:

• The retrieved context is topically adjacent but not directly relevant
• The model hedges so extensively that the actual answer is buried
• The system prompt pushes the model toward a different communication goal than answering directly

Context Precision

Context precision measures whether the retrieved contexts are actually useful for answering the question, weighted by rank position. A high-precision retrieval puts the most useful chunks at the top.

context_precision@k = Σ (precision@k * relevance_k) / number_of_relevant_chunks

A low context precision score with high context recall indicates that your retrieval is finding relevant documents but burying them in a pile of irrelevant ones. The LLM has to process more context to find the answer, which increases latency and can degrade answer quality.

Context Recall

Context recall measures whether the retrieved context contains all the information needed to answer the question. It requires a ground_truth field because it measures coverage of the expected answer.

A low recall score means your retrieval is missing relevant documents. This is the most common failure mode in production RAG systems and usually indicates one of:

• Chunk size is too large (relevant sentences diluted by irrelevant ones, hurting similarity scores)
• Chunk size is too small (relevant information split across chunks, neither half scores high individually)
• The embedding model is poorly suited to the domain
• The corpus lacks the information needed to answer the question at all

Building a synthetic evaluation dataset with LLMs

Collecting hundreds of human-labelled (question, ground_truth) pairs is time-consuming. RAGAS provides a TestsetGenerator that creates synthetic evaluation datasets from your document corpus:

from ragas.testset import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_community.document_loaders import DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load your documents
loader = DirectoryLoader("./docs", glob="**/*.md")
documents = loader.load()

# Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(documents)

# Generate synthetic test questions
generator = TestsetGenerator.with_anthropic(
    generator_llm="claude-opus-4-5",
    critic_llm="claude-opus-4-5",
    embeddings_model="voyage-3",
)

testset = generator.generate_with_langchain_docs(
    chunks,
    test_size=50,
    distributions={
        simple: 0.5,        # straightforward factual questions
        reasoning: 0.25,    # questions requiring inference
        multi_context: 0.25, # questions requiring synthesis across chunks
    },
)

testset_df = testset.to_pandas()
testset_df.to_csv("eval_dataset.csv", index=False)

The generator creates questions at different complexity levels: simple factual recall, questions requiring reasoning over a single chunk, and questions that require synthesising information from multiple documents. This distribution catches different failure modes.

Validate a sample of the generated questions before using them as your evaluation set. LLMs sometimes generate ambiguous, unanswerable, or misleading questions that will produce noisy scores.

Diagnosing low scores

When RAGAS returns a low score, the next step is understanding why. The per-sample dataframe is your primary diagnostic tool:

import pandas as pd

df = results.to_pandas()

# Find the worst-performing samples by faithfulness
low_faithfulness = df.nsmallest(5, "faithfulness")[
    ["question", "answer", "contexts", "faithfulness"]
]

for _, row in low_faithfulness.iterrows():
    print(f"\nQuestion: {row['question']}")
    print(f"Score: {row['faithfulness']:.2f}")
    print(f"Answer: {row['answer']}")
    print(f"Context: {row['contexts'][0][:200]}...")
    print("-" * 60)

Patterns to look for:

• Faithfulness < 0.3 on factual queries: the model is ignoring the retrieved context entirely. Check your system prompt—it may not be instructing the model clearly to use only the provided context.
• Answer relevancy < 0.5: look at the answer text. Does it answer a related but different question? This often indicates retrieval failure—the model is answering based on parametric knowledge because the retrieved documents are not relevant.
• Context precision < 0.4 with high recall: your retrieval returns too many documents. Reduce top_k or add a reranker.
• Context recall < 0.5: the information is not in the corpus, or chunking is fragmenting relevant content. Experiment with chunk sizes and overlap.

Comparing retrieval strategies with RAGAS

RAGAS becomes especially powerful when comparing retrieval strategies. Run the same set of questions through multiple pipeline configurations and compare scores:

from typing import Callable

def run_pipeline(
    questions: list[str],
    retriever: Callable[[str], list[str]],
    generator: Callable[[str, list[str]], str],
) -> Dataset:
    samples = []
    for question in questions:
        contexts = retriever(question)
        answer = generator(question, contexts)
        samples.append({
            "question": question,
            "answer": answer,
            "contexts": contexts,
        })
    return Dataset.from_list(samples)

# Evaluate three retrieval configurations
configurations = {
    "bm25_only": (bm25_retriever, llm_generator),
    "vector_only": (vector_retriever, llm_generator),
    "hybrid_rrf": (hybrid_retriever, llm_generator),
}

comparison_results = {}
for name, (retriever, generator) in configurations.items():
    dataset = run_pipeline(eval_questions, retriever, generator)
    result = evaluate(dataset, metrics=[faithfulness, context_recall])
    comparison_results[name] = result

# Print comparison table
print(f"{'Config':<20} {'Faithfulness':<15} {'Context Recall':<15}")
for name, result in comparison_results.items():
    print(f"{name:<20} {result['faithfulness']:<15.3f} {result['context_recall']:<15.3f}")

This kind of structured comparison is how you build evidence for architectural decisions. "Hybrid search improved context recall by 0.12 on our evaluation set" is a data-backed argument; "hybrid search feels better" is not.

Integrating RAGAS into CI

Evaluation should run automatically on every change to your RAG pipeline—prompt changes, embedding model upgrades, chunk size adjustments, retriever parameter tuning. A regression in faithfulness or recall should block the merge, just as a failing unit test would.

# .github/workflows/ragas-eval.yml
name: RAG Evaluation

on:
  pull_request:
    paths:
      - "src/rag/**"
      - "prompts/**"
      - "config/retrieval.yaml"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install ragas langchain-anthropic datasets

      - name: Run RAGAS evaluation
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python scripts/evaluate.py --threshold-faithfulness 0.75 --threshold-recall 0.80

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: ragas-results
          path: eval_results.json

# scripts/evaluate.py
import argparse
import json
import sys
from datasets import load_from_disk

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--threshold-faithfulness", type=float, default=0.75)
    parser.add_argument("--threshold-recall", type=float, default=0.80)
    args = parser.parse_args()

    # Load your fixed evaluation dataset
    eval_dataset = load_from_disk("data/eval_dataset")

    # Run your current pipeline to generate answers
    # (import your pipeline here)
    dataset_with_answers = generate_answers(eval_dataset)

    results = evaluate(
        dataset_with_answers,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )

    # Save results for the artifact
    with open("eval_results.json", "w") as f:
        json.dump(dict(results), f, indent=2)

    print(f"Faithfulness: {results['faithfulness']:.3f} (threshold: {args.threshold_faithfulness})")
    print(f"Context Recall: {results['context_recall']:.3f} (threshold: {args.threshold_recall})")

    failed = False
    if results["faithfulness"] < args.threshold_faithfulness:
        print(f"FAIL: faithfulness below threshold")
        failed = True
    if results["context_recall"] < args.threshold_recall:
        print(f"FAIL: context_recall below threshold")
        failed = True

    sys.exit(1 if failed else 0)

if __name__ == "__main__":
    main()

One important design decision: use a fixed evaluation dataset in CI, not a freshly generated one. Generating the dataset with an LLM introduces variability—the same corpus produces slightly different questions each run. A fixed dataset makes regressions reproducible and comparable across PRs.

Limitations of RAGAS

RAGAS is not a perfect oracle. Understanding its limitations prevents you from over-trusting scores.

LLM judge variability: scores from different judge models differ. Claude and GPT-4 often disagree on borderline faithfulness cases. Use the same judge model consistently across your evaluation history. If you upgrade the judge model, re-score your historical baseline to make scores comparable.

Metric gaming: a system that always returns the retrieved context verbatim will score 1.0 on faithfulness but 0.0 on answer relevancy. Optimise all four metrics together, not any single one.

Domain specificity: RAGAS was developed on general-domain benchmarks. Its faithfulness detection may be less reliable on highly technical domains where the judge model lacks background knowledge to assess whether a claim is supported by the context.

Long context degradation: for very long contexts (many chunks, each long), the judge model may not reliably process all content to evaluate faithfulness. Monitor the relationship between context length and faithfulness scores in your system.

Despite these limitations, RAGAS is the most practical tool available for systematic RAG evaluation without large human-labelled datasets. Used consistently, with a fixed evaluation set and stable judge model, it catches real regressions reliably enough to be worth integrating into every production RAG pipeline.