Building an LLM evaluation pipeline you can trust
A team of ML engineers spends two weeks tuning a prompt for a document summarization feature. They evaluate on 50 documents, compare outputs, and choose the prompt that produces the best summaries by consensus review. They ship. Three months later, an engineer updates the system prompt to fix an unrelated bug. Nobody evaluates whether the update affected summarization quality. User satisfaction scores drop 12% over the following month. The connection is not made until a support escalation surfaces three specific users who explicitly complained about "worse summaries."
The team had a good initial evaluation process. They had no ongoing evaluation process. Without a repeatable pipeline, every change to the system prompt, model version, retrieval logic, or document preprocessing is a blind deployment.
What an evaluation pipeline must do
An LLM evaluation pipeline is not a test suite in the traditional sense. Traditional software tests verify deterministic behavior: the same input produces the same output, and any deviation is a failure.
LLM evaluation measures probabilistic behavior: the expected quality of outputs across a distribution of inputs, where "quality" is typically multidimensional and not purely computable.
The minimum requirements for an evaluation pipeline that is actionable:
-
Repeatability. The same evaluation run on the same model version and system prompt must produce consistent results (within sampling variance). Without this, it is impossible to distinguish a real regression from noise.
-
Sensitivity. The pipeline must detect meaningful quality changes. A 10% drop in summarization quality should show up as a measurable change in evaluation scores.
-
Actionability. When the evaluation detects a regression, it must point to the failure type — not just "quality dropped" but "factual accuracy dropped on technical documents while creative length increased."
-
Automation. The pipeline must run without human review on every deployment and model update. Human review is a bottleneck that prevents evaluation from being integrated into the deployment process.
Building the evaluation dataset
An evaluation dataset for an LLM application has three components: inputs, expected outputs (or ground truth), and evaluation criteria.
Inputs should be sampled from production queries, not handcrafted by developers. Developer-handcrafted inputs cluster around the expected cases; production inputs represent the actual distribution. Collect and deduplicate production inputs, stratify by query type or domain, and select a representative sample:
from typing import List
import random
def build_evaluation_dataset(
production_queries: List[str],
target_size: int = 500,
stratify_by: str = "query_category"
) -> List[dict]:
"""
Builds an evaluation dataset from production queries.
Stratifies by category to ensure coverage of all query types.
"""
# Categorize queries
categorized = {}
for query in production_queries:
category = classify_query_category(query)
categorized.setdefault(category, []).append(query)
# Sample proportionally from each category
dataset = []
for category, queries in categorized.items():
# How many samples to take from this category
category_proportion = len(queries) / len(production_queries)
n_samples = max(5, int(target_size * category_proportion))
sampled = random.sample(queries, min(n_samples, len(queries)))
for query in sampled:
dataset.append({
"query": query,
"category": category,
"input_id": generate_id()
})
return dataset[:target_size]
Ground truth for LLM outputs is expensive to create and must be created before it is needed. Two approaches:
-
Human-labeled golden outputs: Have domain experts write ideal responses for a subset of evaluation inputs. These become the reference outputs for automated comparison. Time-consuming to create, high quality, does not scale to large datasets.
-
Historical best outputs: For each evaluation input, the best output from the previous system version (before any regression) serves as the reference. This enables regression detection but cannot evaluate absolute quality.
Evaluation criteria depend on the application. For a summarization feature, the criteria might be:
- Factual accuracy (does the summary contain only information from the source document?)
- Completeness (does the summary cover the key points from the source?)
- Conciseness (is the summary appropriately short relative to the source?)
- Coherence (is the summary readable and well-structured?)
Automated evaluation metrics
Deterministic metrics for objective properties
Some quality properties can be measured without a language model. Use these when possible — they are faster, cheaper, and more reliable than model-based evaluation.
def evaluate_response_length(response: str, expected_length_range: tuple) -> dict:
word_count = len(response.split())
in_range = expected_length_range[0] <= word_count <= expected_length_range[1]
return {
"metric": "response_length",
"value": word_count,
"passed": in_range,
"expected_range": expected_length_range
}
def evaluate_format_compliance(response: str, required_sections: List[str]) -> dict:
"""Checks that the response contains required structural elements."""
missing = [s for s in required_sections if s.lower() not in response.lower()]
return {
"metric": "format_compliance",
"value": 1.0 - (len(missing) / len(required_sections)),
"passed": len(missing) == 0,
"missing_sections": missing
}
def evaluate_citation_presence(response: str, source_ids: List[str]) -> dict:
"""For RAG applications — checks that the response cites retrieved sources."""
cited = [sid for sid in source_ids if sid in response]
citation_rate = len(cited) / len(source_ids) if source_ids else 1.0
return {
"metric": "citation_presence",
"value": citation_rate,
"passed": citation_rate >= 0.5,
"cited_sources": cited,
"uncited_sources": [s for s in source_ids if s not in cited]
}
LLM-as-judge for semantic properties
For properties that require understanding meaning — accuracy, relevance, coherence — use a separate LLM call to evaluate the output. This is the "LLM as judge" pattern.
The key requirement for LLM-as-judge: the judge must be a stronger or equal model than the system under evaluation, and the judge prompt must be precise enough to produce consistent judgments.
def llm_judge_factual_accuracy(
source_document: str,
summary: str,
judge_client
) -> dict:
"""
Uses an LLM to evaluate whether a summary contains only
information present in the source document.
"""
judge_prompt = f"""You are evaluating a summary for factual accuracy.
A summary is factually accurate if it contains only information that is present in the source document.
A summary is NOT factually accurate if it adds information, changes quantities/dates, or makes claims not in the source.
Source document:
---
{source_document[:3000]}
---
Summary to evaluate:
---
{summary}
---
Evaluate each sentence of the summary:
1. Is the sentence supported by the source document? (YES/NO)
2. If NO, what is the unsupported claim?
Then give an overall factual accuracy score from 0.0 to 1.0:
- 1.0: Every sentence is supported by the source
- 0.5: About half the sentences are supported
- 0.0: Most sentences contain unsupported claims
Respond in JSON:
{{
"sentence_evaluations": [{{"sentence": "...", "supported": true/false, "issue": "..."}}],
"overall_score": 0.0-1.0,
"unsupported_claim_count": 0
}}"""
response = judge_client.complete(judge_prompt, temperature=0, max_tokens=1000)
try:
result = json.loads(response)
return {
"metric": "factual_accuracy",
"value": result["overall_score"],
"passed": result["overall_score"] >= 0.9,
"details": result
}
except json.JSONDecodeError:
return {"metric": "factual_accuracy", "value": None, "error": "parse_error"}
Run LLM-as-judge evaluations at temperature=0 for consistency. Run the same evaluation 3 times and average the scores to reduce variance — LLM judges have their own sampling variance.
Running the pipeline at deployment time
The evaluation pipeline should run automatically on every deployment and block the deployment if key metrics regress beyond thresholds:
def run_evaluation_pipeline(
model_version: str,
system_prompt: str,
evaluation_dataset: List[dict],
baseline_metrics: dict
) -> dict:
"""
Runs the full evaluation pipeline and compares to baseline.
Returns evaluation results and whether the deployment should proceed.
"""
results = []
for item in evaluation_dataset:
response = llm_client.complete(
system_prompt=system_prompt,
user_message=item["query"],
temperature=0.7
)
item_results = {
"input_id": item["input_id"],
"category": item["category"],
"metrics": {}
}
# Run all evaluation metrics
item_results["metrics"]["length"] = evaluate_response_length(
response, (50, 500)
)
item_results["metrics"]["factual_accuracy"] = llm_judge_factual_accuracy(
item.get("source_document", ""), response, judge_client
)
results.append(item_results)
# Aggregate metrics by category
aggregate = compute_aggregate_metrics(results)
# Compare against baseline — flag regressions
regressions = []
for metric_name, current_value in aggregate.items():
baseline_value = baseline_metrics.get(metric_name)
if baseline_value and current_value < baseline_value - REGRESSION_THRESHOLD[metric_name]:
regressions.append({
"metric": metric_name,
"baseline": baseline_value,
"current": current_value,
"delta": current_value - baseline_value
})
deployment_approved = len(regressions) == 0
return {
"model_version": model_version,
"aggregate_metrics": aggregate,
"regressions": regressions,
"deployment_approved": deployment_approved,
"evaluation_date": datetime.utcnow().isoformat()
}
The human-in-the-loop layer
Automated evaluation catches measurable regressions. Human review catches quality degradation that metrics do not capture.
A weekly human review process complements the automated pipeline:
- Sample 20 randomly selected inputs from the week's production queries
- Run the current system and the baseline on each input
- Have a domain expert rank which output is better (blinded — the reviewer does not know which is current and which is baseline)
- Track the preference rate over time
A preference rate below 50% (the reviewer prefers the old system more often) is a signal that the automated metrics are not capturing a quality dimension that users care about. Investigate what the metrics are missing and add an appropriate automated metric.
Common mistakes in LLM evaluation pipelines
Using the initial evaluation dataset indefinitely without refresh. A dataset built from queries collected in the first month of production does not represent the queries submitted in month twelve. User behavior shifts. New use cases emerge. New failure modes appear that the initial dataset did not anticipate. Refresh the evaluation dataset quarterly by sampling recent production queries, reviewing them for new query types, and adding representative examples to each category.
Setting regression thresholds too conservatively. A pipeline configured to block deployments on any metric drop of more than 1% will block deployments on natural variance — the same system evaluated twice against the same dataset produces slightly different scores due to model sampling. Set thresholds based on observed variance: run the evaluation against the baseline multiple times, measure the standard deviation of each metric, and set the regression threshold at 2–3 standard deviations below the mean. Changes smaller than this are within noise; changes larger than this are likely real regressions.
Evaluating only the happy path. Evaluation datasets that consist entirely of well-formed, unambiguous queries do not cover the cases where LLM behavior is most variable. Include adversarial inputs (negation questions, ambiguous queries, queries that contain contradictions), edge cases (very short queries, very long queries, multilingual queries), and queries from support tickets that revealed past failures. The evaluation dataset should be weighted toward the inputs where the system is most likely to fail.
Not measuring inter-rater reliability for human evaluations. When human reviewers judge which of two responses is better, they often disagree. An evaluation process where a single reviewer makes all judgments has high variance. Measure inter-rater reliability by having two reviewers independently evaluate the same sample and comparing their judgments. Agreement below 70% (kappa score below 0.4) indicates the evaluation criteria are not specific enough to produce consistent results.
Conflating evaluation for quality with evaluation for safety. Quality evaluation (does the response answer the question accurately?) and safety evaluation (does the response contain harmful content, reveal confidential information, or assist with restricted topics?) require different datasets, different metrics, and different response strategies. Run them as separate evaluation pipelines with separate alerting thresholds.
What the evaluation pipeline enables over time
An evaluation pipeline that runs consistently over many months produces a history of quality metrics over time. This history enables decisions that blind deployment cannot support.
Attribution of quality changes to specific system changes. When quality drops, the deployment history shows what changed at that time — a new system prompt, a model version update, a change to the retrieval logic. The evaluation history narrows the root cause from "something went wrong" to "this specific change caused this specific metric to drop."
Measurement of improvement investments. When the team spends two weeks improving the retrieval pipeline, the evaluation pipeline measures whether quality actually improved. Without measurement, "improved retrieval" is a claim; with measurement, it is a fact.
Informed model version migration. When the LLM provider releases a new model version, the evaluation pipeline runs against the new version before migrating any production traffic. Quality differences — positive or negative — are measured before users are affected.
The team that builds this evaluation pipeline — automated metrics on every deployment, weekly human review, regression thresholds that block bad deployments — will catch the regression that dropped user satisfaction by 12% at deployment time, not six weeks later from support escalations. The evaluation infrastructure is the difference between an LLM application that the team can improve with confidence and one that changes unpredictably.
Handling non-determinism in evaluation scores
LLM outputs are probabilistic. The same prompt, model version, and input can produce different outputs on successive runs due to temperature sampling. This non-determinism propagates into evaluation scores: the same system evaluated twice against the same dataset will produce slightly different aggregate scores.
Accounting for this in the regression detection logic prevents false positives (blocking a good deployment because scores fluctuated down) and false negatives (approving a bad deployment because scores fluctuated up):
def compute_regression_threshold(
metric_name: str,
historical_scores: List[float],
sensitivity: float = 2.5 # Standard deviations
) -> float:
"""
Computes the regression threshold for a metric based on observed variance
across multiple evaluation runs of the same baseline system.
A score below (mean - sensitivity * std_dev) is a likely regression.
A score above this is within noise.
"""
import statistics
mean = statistics.mean(historical_scores)
std_dev = statistics.stdev(historical_scores) if len(historical_scores) > 1 else 0.05
threshold = mean - (sensitivity * std_dev)
return max(threshold, 0.0) # Never negative
def should_block_deployment(
current_metrics: dict,
baseline_metrics: dict,
variance_model: dict # metric_name -> regression_threshold
) -> tuple[bool, list]:
"""
Determines whether a deployment should be blocked based on metric regressions.
Returns (should_block, list_of_failed_metrics).
"""
failed_metrics = []
for metric_name, current_value in current_metrics.items():
if current_value is None:
continue
threshold = variance_model.get(metric_name)
if threshold is None:
# No variance model yet — use a conservative fixed threshold
threshold = baseline_metrics.get(metric_name, 0) - 0.05
if current_value < threshold:
failed_metrics.append({
"metric": metric_name,
"current": current_value,
"threshold": threshold,
"baseline": baseline_metrics.get(metric_name)
})
return len(failed_metrics) > 0, failed_metrics
Building the variance model requires running the evaluation pipeline multiple times against the known-good baseline — typically 5 to 10 runs — to measure the natural spread of scores. This investment is made once per application and updated quarterly. After this, the regression detection is calibrated to the actual non-determinism of the specific model and prompts being evaluated, rather than using an arbitrary threshold that may be either too sensitive or too permissive.
Evaluation as product infrastructure
An LLM evaluation pipeline is not a one-time investment — it is product infrastructure in the same sense that application monitoring is product infrastructure. It requires ongoing maintenance:
- The evaluation dataset must be refreshed as user behavior evolves
- Evaluation metrics must be updated when the application adds new capabilities
- Regression thresholds must be recalibrated after major model or prompt changes
- The human review process must be scheduled and tracked like any other recurring team responsibility
Teams that treat the evaluation pipeline as a project deliverable — build it, ship it, forget it — find that it becomes stale. The dataset no longer represents the user population. The metrics no longer capture the quality dimensions users care about. The thresholds are either too aggressive (blocking every deployment) or too permissive (not catching regressions). At that point, the pipeline provides false confidence rather than real safety.
The teams that get value from evaluation pipelines long-term are the ones that assign ownership explicitly — a specific engineer or small team is responsible for evaluation infrastructure health — and that track evaluation pipeline maintenance as part of the regular engineering workload, not as optional cleanup. Evaluation infrastructure is as important as test infrastructure for deterministic software, and deserves the same level of ongoing investment.