Implementing confidence scoring for LLM outputs
One of the most counterintuitive properties of large language models is that fluency and accuracy are nearly independent. A model can produce a beautifully written, grammatically perfect, confidently stated paragraph that is completely wrong. This is not a bug — it is a structural consequence of how these models are trained. They are optimised to produce the next most likely token, not to be correct.
This decoupling is manageable when a human is in the loop to sanity-check outputs. It becomes dangerous when LLM responses feed directly into automated systems: code pipelines, customer-facing chatbots, data extraction workflows, or medical information retrieval. For those use cases, you need a way to attach a confidence score to the model's output — a number that tells you how much to trust this particular response.
This article covers the main approaches to confidence scoring in practice, their tradeoffs, and how to combine them into a system that is actually calibrated.
Why model confidence is hard
Before diving into techniques, it helps to understand why this problem is non-trivial.
The model does produce internal probability signals: the logprob of each token it generates. A sequence with high token probabilities is something the model "wanted to say" strongly. But high logprobs do not equal correctness. If the model has memorised a plausible-sounding but wrong answer, it will generate that answer with high confidence. Conversely, a correct but unusual answer might have lower logprobs because the model has seen fewer examples of it.
The second issue is that models are not inherently calibrated. A perfectly calibrated classifier would be correct 70% of the time when it says it is 70% confident. Most LLMs, out of the box, are not calibrated — they tend to be overconfident on questions where they have abundant training data and underconfident on questions that are more novel.
Finally, asking a model "how confident are you?" as a follow-up prompt tends to produce optimistic answers. The model treats confidence self-reporting as a conversational task and generates a plausible-sounding confidence level rather than a calibrated one.
With those caveats in mind, here are the techniques that actually work.
Logprob-based confidence
For models that expose token logprobs (many do via API), the average log probability of the generated tokens is the most direct signal you have. Sequences with high mean logprob tend to be more reliable than those with low mean logprob, holding everything else equal.
import math
from anthropic import Anthropic
# Note: Claude's API does not currently expose logprobs.
# This example uses the OpenAI API where logprobs are available.
import openai
def get_response_with_logprob_confidence(
prompt: str,
model: str = "gpt-4o"
) -> dict:
"""
Returns the model response along with a confidence score
derived from mean token log probability.
"""
client = openai.OpenAI()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
logprobs=True,
top_logprobs=1,
max_tokens=512
)
choice = response.choices[0]
message_text = choice.message.content
# Extract log probabilities for each token
token_logprobs = [
token_data.logprob
for token_data in choice.logprobs.content
if token_data.logprob is not None
]
if not token_logprobs:
return {"text": message_text, "confidence": None}
# Mean logprob, converted to probability
mean_logprob = sum(token_logprobs) / len(token_logprobs)
# Clamp to a [0, 1] range for display
# mean_logprob is in (-inf, 0]; 0 = certain, -inf = impossible
# Map to [0, 1] with a reasonable scale (logprob of -2.3 ~ 10% prob per token)
confidence = math.exp(mean_logprob)
return {
"text": message_text,
"raw_mean_logprob": mean_logprob,
"confidence": round(confidence, 4),
"token_count": len(token_logprobs)
}
Logprob confidence has a known weakness: it captures how probable the sequence is given the model's training, not how factually correct it is. Use it as a baseline signal, not a standalone measure.
Self-critique prompting
A more broadly applicable approach — and one that works with any model API — is asking the model to critique its own answer. After generating a response, you send a second prompt asking the model to identify potential errors, missing caveats, or cases where the answer might be wrong.
from anthropic import Anthropic
import json
client = Anthropic()
def get_response_with_self_critique(
question: str,
domain_context: str = ""
) -> dict:
"""
Two-step process: generate an answer, then critique it.
Returns both the answer and a structured confidence assessment.
"""
# Step 1: Generate the answer
answer_response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
system=f"You are a knowledgeable assistant. {domain_context}",
messages=[{"role": "user", "content": question}]
)
answer = answer_response.content[0].text
# Step 2: Critique the answer
critique_response = client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
system="""You are a rigorous fact-checker. Your job is to evaluate an answer
for potential errors, hallucinations, or missing nuance. Be honest about uncertainty.
Return a JSON object with these fields:
- confidence_score: number from 0.0 to 1.0
- likely_correct: boolean
- concerns: list of strings describing potential issues
- knowledge_cutoff_risk: boolean (true if the answer might be outdated)
- requires_verification: boolean""",
messages=[
{
"role": "user",
"content": f"""Question: {question}
Answer to evaluate:
{answer}
Evaluate this answer and return the JSON assessment."""
}
]
)
# Parse the critique (handle potential JSON extraction)
critique_text = critique_response.content[0].text
try:
# Extract JSON if wrapped in markdown code blocks
if "```json" in critique_text:
json_str = critique_text.split("```json")[1].split("```")[0].strip()
elif "```" in critique_text:
json_str = critique_text.split("```")[1].split("```")[0].strip()
else:
json_str = critique_text.strip()
critique = json.loads(json_str)
except (json.JSONDecodeError, IndexError):
critique = {"confidence_score": 0.5, "concerns": ["Parse error in critique"]}
return {
"question": question,
"answer": answer,
"confidence": critique.get("confidence_score", 0.5),
"concerns": critique.get("concerns", []),
"requires_verification": critique.get("requires_verification", False),
"knowledge_cutoff_risk": critique.get("knowledge_cutoff_risk", False)
}
Self-critique works better than you might expect, because the task of evaluating an answer is different from the task of generating one. The model can often identify that it is uncertain even if it initially produced an overconfident answer. That said, it has a systematic blind spot: if the model is confidently wrong because it has memorised incorrect information, the critique step may not catch it — the model lacks the external ground truth needed to spot the error.
Ensemble voting
Ensemble methods are the most robust confidence estimator for factual questions. The idea is simple: ask the model the same question multiple times (with some temperature and prompt variation), collect the answers, and treat agreement as a proxy for confidence.
from anthropic import Anthropic
from collections import Counter
import re
client = Anthropic()
def ensemble_confidence(
question: str,
n_samples: int = 5,
temperature_range: tuple = (0.3, 0.9)
) -> dict:
"""
Sample the model multiple times and use answer agreement
as a confidence signal.
"""
import random
answers = []
for i in range(n_samples):
# Vary the prompt slightly to reduce repetition bias
prompt_variants = [
question,
f"Please answer the following: {question}",
f"Question: {question}\nAnswer:",
]
prompt = prompt_variants[i % len(prompt_variants)]
response = client.messages.create(
model="claude-haiku-4-5", # Use fast model for ensemble
max_tokens=256,
# Note: temperature parameter shown for concept;
# check current API docs for exact parameter names
messages=[{"role": "user", "content": prompt}]
)
answers.append(response.content[0].text.strip())
# For short factual answers, normalise and count agreement
normalised = [_normalise_answer(a) for a in answers]
counter = Counter(normalised)
most_common, top_count = counter.most_common(1)[0]
agreement_ratio = top_count / n_samples
return {
"question": question,
"consensus_answer": answers[normalised.index(most_common)], # Return original form
"confidence": agreement_ratio,
"n_samples": n_samples,
"answer_distribution": dict(counter),
"high_confidence": agreement_ratio >= 0.8
}
def _normalise_answer(text: str) -> str:
"""
Normalise answer text for comparison.
Strips punctuation, lowercases, and collapses whitespace.
"""
text = text.lower()
text = re.sub(r"[^\w\s]", "", text)
text = re.sub(r"\s+", " ", text).strip()
# Take just the first sentence for comparison
first_sentence = text.split(".")[0] if "." in text else text
return first_sentence[:200] # Cap at 200 chars for comparison
Ensemble voting is expensive — five API calls instead of one — but for high-stakes queries it is worth it. The 80% agreement threshold (four of five samples giving the same answer) is a reasonable heuristic for "high confidence." Tune it for your use case.
For open-ended tasks where the answer is not a single fact, you can use semantic similarity instead of exact match: embed all five answers and compute the mean pairwise cosine similarity. High average similarity indicates consistent responses even when the wording varies.
Retrieval-grounded confidence
For factual domains where you have a trusted knowledge base, the strongest confidence signal is whether the answer is grounded in retrieved documents. An answer that cites retrieved evidence is more trustworthy than one that relies on parametric knowledge alone.
from anthropic import Anthropic
client = Anthropic()
def retrieval_grounded_confidence(
question: str,
retrieved_docs: list[dict], # [{"content": str, "source": str, "relevance": float}]
) -> dict:
"""
Generate an answer grounded in retrieved documents, with a
citation-based confidence score.
"""
# Filter to documents above relevance threshold
relevant_docs = [d for d in retrieved_docs if d["relevance"] >= 0.6]
if not relevant_docs:
return {
"answer": "I don't have reliable information to answer this question.",
"confidence": 0.1,
"grounded": False,
"citations": []
}
# Format context
context = "\n\n".join(
f"[Source {i+1}: {doc['source']}]\n{doc['content']}"
for i, doc in enumerate(relevant_docs)
)
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
system="""Answer questions using ONLY the provided context.
For each claim you make, indicate which source supports it using [Source N] notation.
If the context does not contain enough information to answer, say so explicitly.
End your response with a JSON block:
```json
{"citation_count": <number>, "fully_grounded": <boolean>, "confidence": <0.0-1.0>}
```""",
messages=[
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}"
}
]
)
answer_text = response.content[0].text
# Extract the trailing JSON
confidence_data = {"confidence": 0.5, "fully_grounded": False, "citation_count": 0}
try:
if "```json" in answer_text:
json_str = answer_text.split("```json")[-1].split("```")[0].strip()
confidence_data = json.loads(json_str)
answer_text = answer_text.split("```json")[0].strip()
except Exception:
pass
# Adjust confidence based on retrieval quality
max_retrieval_score = max(d["relevance"] for d in relevant_docs)
adjusted_confidence = confidence_data["confidence"] * max_retrieval_score
return {
"answer": answer_text,
"confidence": round(adjusted_confidence, 3),
"grounded": confidence_data.get("fully_grounded", False),
"citation_count": confidence_data.get("citation_count", 0),
"retrieval_docs_used": len(relevant_docs)
}
Calibration: making scores meaningful
Raw confidence scores are only useful if they are calibrated — if a score of 0.7 actually corresponds to being right 70% of the time. Without calibration, scores are just rough ordinal rankings.
To calibrate your system, you need a held-out evaluation set: a collection of questions where you know the ground truth. Run your confidence scoring pipeline over the evaluation set and compute calibration error.
A simple calibration check: bin your scores into deciles (0–0.1, 0.1–0.2, ..., 0.9–1.0). For each bin, compute the actual accuracy of responses in that bin. If the 0.8–0.9 bin has actual accuracy of 60%, your system is overconfident in that range.
Post-hoc calibration with temperature scaling or Platt scaling can fix systematic over- or underconfidence. These are simple transformations applied to raw scores before they are returned to the application layer.
The Brier score is a useful single-number summary of calibration quality: the mean squared error between predicted confidence and actual binary correctness. Lower is better, and 0.25 is the baseline (always predicting 50% confidence on every question).
Combining signals into a final score
In production, you will typically combine multiple signals. A reasonable approach:
def combined_confidence(
logprob_confidence: float | None,
self_critique_confidence: float,
retrieval_confidence: float | None,
ensemble_confidence: float | None
) -> float:
"""
Weighted combination of available confidence signals.
Weights can be tuned on your calibration set.
"""
weights = []
scores = []
if logprob_confidence is not None:
weights.append(0.2)
scores.append(logprob_confidence)
weights.append(0.35)
scores.append(self_critique_confidence)
if retrieval_confidence is not None:
weights.append(0.30)
scores.append(retrieval_confidence)
if ensemble_confidence is not None:
weights.append(0.25)
scores.append(ensemble_confidence)
# Normalise weights
total_weight = sum(weights)
normalised = [w / total_weight for w in weights]
combined = sum(s * w for s, w in zip(scores, normalised))
return round(combined, 3)
The weights here are starting points. Tune them on your calibration set to minimise Brier score. In practice, retrieval-grounded confidence tends to be the most reliable signal when available, and self-critique is the most widely applicable.
Deciding what to do with confidence scores
A confidence score is only valuable if your application does something useful with it. Common patterns:
Threshold gating — responses below a confidence threshold are held for human review, not shown to users automatically. This is the right pattern for high-stakes domains like medical or legal information.
Uncertainty disclosure — surface the confidence to the user. "I'm fairly confident in this answer, but you may want to verify with [primary source]." This works well when users are sophisticated enough to act on it.
Retrieval fallback — if confidence is below a threshold, automatically trigger a retrieval pass even if the question seemed to be answerable from parametric knowledge.
Ensemble escalation — run a cheap single-pass first, and only invoke the expensive ensemble step when initial confidence is borderline. This keeps costs manageable while applying heavier scrutiny where it matters.
Confidence scoring is not a silver bullet, but it is a practical engineering tool that makes LLM-powered systems meaningfully safer in production. The goal is not perfect calibration — it is calibration good enough to make better decisions than random about when to trust the model.