LLM evaluation: how to measure hallucination rate in production
Defining hallucination precisely
Hallucination is an overloaded term. Before you can measure it, you need a definition specific enough to be operationalised. There are at least three distinct failure modes that people call hallucination:
Factual hallucination — the model asserts something that is objectively false. "The Eiffel Tower was built in 1925" when it was built in 1889. This category is the hardest to catch automatically because it requires a ground truth knowledge source to compare against.
Grounding hallucination — in a RAG (retrieval-augmented generation) system, the model makes a claim that is not supported by any of the retrieved documents. The claim might even be factually true in the real world but it is not grounded in the evidence the model was given. This is the most common form in enterprise systems and the easiest to measure automatically.
Faithfulness violation — the model is asked to summarise or restate a document, and the output contradicts the source material. Similar to grounding hallucination but applies to tasks where the source is the full input rather than retrieved context.
For most production applications, grounding hallucination and faithfulness violation are the actionable targets. You have the source material, you have the output, and you can build automated checks. Factual hallucination requires external knowledge bases and is much harder to systematically catch.
Building a hallucination evaluation pipeline
The core pattern for automated hallucination detection is LLM-as-judge: you use a capable model (separate from the one you are evaluating) to assess whether the evaluated model's output is supported by the provided context.
Here is the structure of an evaluation record:
interface EvaluationRecord {
id: string;
query: string;
context: string[]; // Retrieved documents or input source material
response: string; // Model's output to evaluate
groundTruth?: string; // Optional reference answer for accuracy scoring
}
interface EvaluationResult {
recordId: string;
groundednessScore: number; // 0 to 1
faithfulnessScore: number; // 0 to 1
hallucinated: boolean; // Binary verdict
unsupportedClaims: string[]; // Specific claims flagged
reasoning: string; // Judge's reasoning
}
The grounding evaluation prompt:
const GROUNDING_JUDGE_PROMPT = `You are evaluating whether an AI assistant's response is fully supported by the provided context documents.
Your task:
1. Read the context documents carefully
2. Read the assistant's response
3. Identify any claims in the response that are NOT supported by the context documents
4. Return a JSON object with your evaluation
A claim is "hallucinated" if it:
- States a fact not present in the context
- Contradicts information in the context
- Makes a specific assertion (number, date, name, causal relationship) that cannot be verified from the context
A claim is NOT hallucinated if it:
- Is directly stated in the context
- Is a logical inference clearly implied by the context
- Is a general statement that doesn't require specific sourcing (e.g., "there are several approaches")
CONTEXT DOCUMENTS:
{{context}}
ASSISTANT RESPONSE:
{{response}}
Return a JSON object with this exact structure:
{
"groundednessScore": <number between 0 and 1>,
"unsupportedClaims": [<list of specific phrases from the response that are not grounded>],
"reasoning": "<brief explanation of your assessment>",
"hallucinated": <true if any claim is unsupported, false otherwise>
}`;
The evaluation function:
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
async function evaluateGrounding(
record: EvaluationRecord
): Promise<EvaluationResult> {
const contextText = record.context
.map((doc, i) => `[Document ${i + 1}]\n${doc}`)
.join("\n\n");
const prompt = GROUNDING_JUDGE_PROMPT.replace(
"{{context}}",
contextText
).replace("{{response}}", record.response);
const response = await client.messages.create({
model: "claude-opus-4-5",
max_tokens: 1024,
temperature: 0, // Deterministic evaluation
messages: [{ role: "user", content: prompt }],
});
const textContent = response.content.find((b) => b.type === "text");
if (!textContent || textContent.type !== "text") {
throw new Error("No text content in judge response");
}
// Extract JSON from response (handle potential markdown code blocks)
const jsonMatch = textContent.text.match(/\{[\s\S]*\}/);
if (!jsonMatch) {
throw new Error("No JSON found in judge response");
}
const judgeOutput = JSON.parse(jsonMatch[0]) as {
groundednessScore: number;
unsupportedClaims: string[];
reasoning: string;
hallucinated: boolean;
};
return {
recordId: record.id,
groundednessScore: judgeOutput.groundednessScore,
faithfulnessScore: judgeOutput.groundednessScore, // Same metric, different framing
hallucinated: judgeOutput.hallucinated,
unsupportedClaims: judgeOutput.unsupportedClaims,
reasoning: judgeOutput.reasoning,
};
}
Running batch evaluations at scale
Evaluating hundreds or thousands of records requires batching and rate limit handling. This runner processes a dataset concurrently with a configurable concurrency limit:
import PQueue from "p-queue";
interface BatchEvaluationReport {
totalRecords: number;
hallucinations: number;
hallucinationRate: number;
averageGroundednessScore: number;
results: EvaluationResult[];
failedRecords: string[];
}
async function runBatchEvaluation(
records: EvaluationRecord[],
concurrency: number = 5
): Promise<BatchEvaluationReport> {
const queue = new PQueue({ concurrency });
const results: EvaluationResult[] = [];
const failedRecords: string[] = [];
const tasks = records.map((record) =>
queue.add(async () => {
try {
const result = await evaluateGrounding(record);
results.push(result);
console.log(
`Evaluated ${record.id}: hallucinated=${result.hallucinated}, score=${result.groundednessScore.toFixed(2)}`
);
} catch (err) {
console.error(`Failed to evaluate ${record.id}:`, err);
failedRecords.push(record.id);
}
})
);
await Promise.all(tasks);
const hallucinations = results.filter((r) => r.hallucinated).length;
const averageGroundednessScore =
results.reduce((sum, r) => sum + r.groundednessScore, 0) / results.length;
return {
totalRecords: records.length,
hallucinations,
hallucinationRate: hallucinations / records.length,
averageGroundednessScore,
results,
failedRecords,
};
}
Sampling strategy for production monitoring
You cannot run every production query through the evaluation pipeline — the cost and latency would be prohibitive. Instead, sample intelligently.
Random baseline sampling gives you a representative cross-section. Sample 2-5% of all queries randomly and run evaluation on those. This is your baseline hallucination rate.
Stratified sampling ensures that rare query types are represented. If you have query categories (product questions, policy questions, account questions), ensure each category has adequate representation in your evaluation sample regardless of its volume proportion.
Triggered sampling on high-risk signals. Track signals that correlate with hallucination — long responses, responses containing specific numbers or dates, responses that cite or quote, queries about recent events. Sample at higher rates (20-50%) when these signals are present.
function shouldEvaluate(query: string, response: string): boolean {
// Always evaluate randomly at 3%
if (Math.random() < 0.03) return true;
// Elevate rate for high-risk patterns
const highRiskSignals = [
/\d{4}/.test(response), // Contains a year
/\$[\d,]+/.test(response), // Contains a dollar amount
/according to/i.test(response), // Citing a source
/specifically/i.test(response), // Making specific claims
response.length > 1000, // Long response
/"[^"]{20,}"/.test(response), // Extended quote
];
const riskScore = highRiskSignals.filter(Boolean).length;
if (riskScore >= 2) return Math.random() < 0.2; // 20% for high risk
if (riskScore >= 1) return Math.random() < 0.1; // 10% for medium risk
return false;
}
Tracking hallucination rate over time
A single hallucination rate measurement is a data point. What you need is a time series that lets you detect regressions and measure the impact of changes.
Store evaluation results in a database and query them over time windows. This schema works for Firestore or any document database:
interface EvaluationLogEntry {
id: string;
timestamp: Date;
modelVersion: string;
promptVersion: string;
queryCategory: string;
hallucinationRate: number; // For rolling window calculations
isHallucinated: boolean;
groundednessScore: number;
recordId: string;
}
// Query hallucination rate by time window and model version
async function getHallucinationRate(
modelVersion: string,
startDate: Date,
endDate: Date
): Promise<{ rate: number; count: number; sampleSize: number }> {
// Implementation depends on your database
// This is the query logic:
const records = await db
.collection("evaluation_logs")
.where("modelVersion", "==", modelVersion)
.where("timestamp", ">=", startDate)
.where("timestamp", "<=", endDate)
.get();
const docs = records.docs.map((d) => d.data() as EvaluationLogEntry);
const hallucinations = docs.filter((d) => d.isHallucinated).length;
return {
rate: hallucinations / docs.length,
count: hallucinations,
sampleSize: docs.length,
};
}
Create alerts when the hallucination rate exceeds thresholds:
interface HallucinationAlert {
severity: "warning" | "critical";
currentRate: number;
threshold: number;
period: string;
sampleSize: number;
}
function checkHallucinationThresholds(
rate: number,
sampleSize: number
): HallucinationAlert | null {
// Need statistical significance — don't alert on tiny samples
if (sampleSize < 50) return null;
if (rate > 0.2) {
return {
severity: "critical",
currentRate: rate,
threshold: 0.2,
period: "last 24 hours",
sampleSize,
};
}
if (rate > 0.1) {
return {
severity: "warning",
currentRate: rate,
threshold: 0.1,
period: "last 24 hours",
sampleSize,
};
}
return null;
}
Using RAGAS for structured evaluation
RAGAS (Retrieval-Augmented Generation Assessment) is a Python library that implements several LLM evaluation metrics as a framework. It provides standard implementations of faithfulness, answer relevancy, context recall, and context precision.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset
# Prepare evaluation dataset
eval_data = {
"question": [
"What is the refund policy for digital products?",
"How long does shipping take to Nigeria?",
],
"answer": [
# Model responses to evaluate
"Digital products are non-refundable once downloaded.",
"Shipping to Nigeria takes 7-14 business days via DHL.",
],
"contexts": [
# Retrieved documents for each question
[
"Section 4.2: All digital products are final sale. Refunds are not available after download has been initiated.",
"Section 4.1: Physical products may be returned within 30 days.",
],
[
"Shipping times: Europe 3-5 days, Americas 5-10 days, Africa 10-21 days depending on carrier.",
],
],
"ground_truth": [
# Reference answers (optional, needed for some metrics)
"Digital products cannot be refunded once downloaded.",
"Shipping to Nigeria typically takes 10-21 days.",
],
}
dataset = Dataset.from_dict(eval_data)
result = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy, context_recall],
llm=your_judge_model, # Uses your configured LLM as the judge
)
print(result.to_pandas())
RAGAS handles the prompt engineering and scoring logic for each metric. The faithfulness metric specifically targets hallucination: it breaks the model's response into individual claims, checks each claim against the context, and computes the fraction of claims that are supported.
Calibrating your judge model
The judge model is itself a source of error. If you use the same model you are evaluating as the judge, you risk systematic self-serving bias. If the judge model is weaker than the evaluated model, it may fail to catch sophisticated hallucinations.
Use a different, more capable model as the judge when possible. If you are evaluating a 7B model, judge with a 70B model or a frontier model. If you are evaluating a frontier model, use a different frontier model as judge (evaluate Claude with GPT, or vice versa).
Calibrate your judge against human labels before trusting its assessments. Take 200 evaluation records, label them by hand for hallucination presence, and compute agreement between your automated judge and human labels. Aim for Cohen's Kappa above 0.7.
from sklearn.metrics import cohen_kappa_score
human_labels = [1, 0, 0, 1, 0, 1, 1, 0] # 1 = hallucinated
model_labels = [1, 0, 1, 1, 0, 1, 0, 0] # Your judge's predictions
kappa = cohen_kappa_score(human_labels, model_labels)
print(f"Cohen's Kappa: {kappa:.3f}")
# > 0.7 is acceptable, > 0.8 is good
Connecting evaluation to deployment decisions
Hallucination measurement only matters if it connects to action. Establish clear gates in your deployment process:
A model or prompt change should not go to production if it causes a statistically significant increase in hallucination rate on your golden test set. Run automated evaluation as part of your CI pipeline before any release.
# .github/workflows/llm-eval.yml
name: LLM Evaluation Gate
on:
pull_request:
paths:
- "prompts/**"
- "rag/**"
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run hallucination evaluation
run: |
python scripts/evaluate.py \
--test-set data/golden_eval_set.jsonl \
--threshold 0.10 \
--fail-on-regression
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
The evaluation script should exit with a non-zero code if the hallucination rate exceeds the threshold or if it regresses significantly from the baseline stored in your repository.
Hallucination measurement is not a one-time exercise. It is an ongoing monitoring discipline that tells you whether your LLM application is actually trustworthy — and gives you the data to make it more so over time.