Why your LLM application hallucinates in production even though it passed your tests

An AI product team spends three weeks building and testing a customer support assistant. They create 200 test cases covering their most common support scenarios. The assistant answers all 200 correctly. They ship to production. Within two days, the support queue fills with complaints about wrong answers, invented policies, and confidently stated incorrect information.

The team reviews the failed cases. None are similar to the 200 tests. Users asked about edge cases, used informal language, asked multi-part questions, combined topics that the test cases treated separately, and asked about topics the test set did not anticipate. The assistant had been tested on the expected cases. Users submitted unexpected cases. Hallucination appeared in the gap.

This is the structural problem with LLM evaluation. Unlike software that fails deterministically — the same input always produces the same output — an LLM's output is probabilistic. Even a test case that passes 99% of the time fails 1% of the time. At production scale, 1% failure rate on 10,000 daily queries is 100 wrong answers per day.

The three sources of production hallucination

Distribution shift between test and production

Test sets are built by developers or domain experts who know the system. They anticipate the questions users should ask. They do not anticipate the questions users actually ask.

Users ask questions that combine multiple domains. They misspell product names. They reference outdated information as if it were current. They ask negation questions ("what can't I do with the free plan?"). They follow up on previous conversation context that the assistant has lost. They submit single-word queries ("refund?") that require the assistant to infer an entire question.

Each of these is a distribution shift from the test cases. The assistant was trained implicitly on a specific distribution of well-formed queries. Production queries come from a broader, messier distribution. The gaps between these distributions are where hallucination lives.

# Test case distribution (what developers write):
TEST_QUERIES = [
    "What is the return policy for digital products?",
    "How do I cancel my subscription?",
    "Can I transfer my license to another account?",
]

# Production query distribution (what users actually submit):
PRODUCTION_EXAMPLES = [
    "refund",                                           # Under-specified
    "i bought something 2 weeks ago can i still return",  # Typos + implied question
    "what happens to my data if i cancel",             # Policy + data privacy combined
    "my friend wants to use my license is that ok",    # Framing obscures the question
    "you said earlier i could return it but now youre saying i cant",  # Context issue
    "return policy EXCEPT for digital stuff",          # Negation pattern
]

The test distribution and production distribution overlap but are not the same. The model's behavior on the non-overlapping production cases was never evaluated.

Prompt sensitivity

LLMs are sensitive to phrasing in ways that are not obvious in testing. A system prompt that produces correct answers for the test distribution may produce hallucinations for slight variations in phrasing, question structure, or domain.

# This prompt works well for direct questions:
SYSTEM_PROMPT_V1 = """
You are a customer support assistant. Answer questions about our product policies.
If you don't know the answer, say so. Do not make up information.
"""

# But fails for multi-part questions where the model "bridges" between topics:
user_query = """
I want to return a digital course I bought last month.
Also, can I use my account credits for the refund?
"""
# The model may correctly state the digital return policy
# but then hallucinate details about account credits that were never stated in the docs

The compound question requires the model to handle two separate policies and their interaction. If the system prompt or context does not explicitly address the interaction, the model fills the gap with plausible-sounding but invented information.

Temperature and sampling

Most LLM production deployments use temperature > 0 for response diversity. At temperature 0.7, the same input produces different outputs on different runs. A query that produces the correct answer 90% of the time produces a wrong answer 10% of the time — and the wrong answers can be confident hallucinations, not explicit uncertainty.

Testing with a fixed temperature and a single sample per test case does not capture this variance:

# Test case evaluation — single sample, misses variance
def evaluate_test_case(query: str, expected_answer: str) -> bool:
    response = llm.complete(query, temperature=0.7, max_tokens=500)
    return is_factually_correct(response, expected_answer)

# Production behavior evaluation — multiple samples, captures variance
def evaluate_production_case(query: str, expected_answer: str, n_samples: int = 10) -> dict:
    correct_count = 0
    hallucination_count = 0

    for _ in range(n_samples):
        response = llm.complete(query, temperature=0.7, max_tokens=500)
        if is_factually_correct(response, expected_answer):
            correct_count += 1
        elif contains_hallucination(response, expected_answer):
            hallucination_count += 1

    return {
        "query": query,
        "accuracy_rate": correct_count / n_samples,
        "hallucination_rate": hallucination_count / n_samples,
        "pass_threshold": correct_count / n_samples >= 0.9  # Requires 90%+ pass rate
    }

A query with 85% accuracy on 10 samples is a production risk that single-sample testing would classify as passing.

Structural approaches to reducing hallucination

Constrained generation

For domains where the set of valid answers is bounded, constrain the model's output to that set. A customer support assistant answering policy questions should not be able to invent policies — it should retrieve and quote the relevant policy, not generate a policy.

def answer_policy_question(user_query: str) -> dict:
    # Retrieve relevant policy chunks
    relevant_policies = policy_retriever.retrieve(user_query, top_k=3)

    if not relevant_policies:
        return {
            "answer": "I don't have information about that. Please contact support directly.",
            "sources": [],
            "confidence": "low"
        }

    # Build context with explicit instruction to not go beyond it
    context = "\n\n".join([
        f"Policy ({p.id}): {p.content}" for p in relevant_policies
    ])

    prompt = f"""Answer the customer's question using ONLY the policies provided below.
If the policies do not directly address the question, say so.
Do not add information that is not in the provided policies.
Do not paraphrase policies in ways that change their meaning.
Quote relevant policy text directly.

Policies:
{context}

Customer question: {user_query}

Your answer must:
1. Be based only on the provided policies
2. Include the policy ID when quoting
3. State if the policies do not address the question"""

    response = llm.complete(prompt, temperature=0.3, max_tokens=400)
    return {"answer": response, "sources": [p.id for p in relevant_policies]}

The retrieval-augmented approach constrains the answer space to the retrieved content. Hallucination can still occur (the model may misquote or misinterpret), but invented information has no source to appear from if the model correctly follows the instruction to quote.

Uncertainty calibration

A model that says "I'm not sure, but..." followed by a hallucination is less useful than a model that refuses to answer when it is uncertain. Training for calibrated uncertainty is possible but expensive; prompting for explicit uncertainty markers is more accessible.

UNCERTAINTY_PROMPT = """After your answer, add a confidence marker:
[CONFIDENT] if you are certain the answer is in the provided context
[UNCERTAIN] if you are inferring beyond what is explicitly stated
[NO_INFO] if the context does not address this question

Only provide answers you would mark [CONFIDENT] for factual questions.
For [UNCERTAIN] or [NO_INFO], recommend the user contact support."""

Log and monitor the distribution of uncertainty markers in production. A spike in [UNCERTAIN] answers on a specific topic is a signal that the knowledge base has a gap or the retrieval is failing for that topic.

Output validation

For structured outputs, validate the model's response against the expected schema and factual constraints before returning it to the user:

def validate_policy_response(response: str, source_policies: List[Policy]) -> bool:
    """
    Validates that the response does not contain claims not present in source policies.
    Uses a second LLM call to check for hallucinations — the "LLM as judge" pattern.
    """
    policy_content = "\n".join([p.content for p in source_policies])

    validation_prompt = f"""Given the following source policies and a generated answer,
determine whether the answer contains any claims that are NOT supported by the source policies.

Source policies:
{policy_content}

Generated answer:
{response}

Does the answer contain any claims not supported by the source policies?
Answer with only "YES" or "NO"."""

    verdict = validation_llm.complete(
        validation_prompt,
        temperature=0,
        max_tokens=10
    ).strip().upper()

    return verdict == "NO"  # True means no hallucination detected

This "LLM as judge" pattern adds latency and cost but catches hallucinations before they reach users. Applied selectively — for responses on high-stakes topics, or when the primary response shows uncertainty markers — it provides a meaningful additional check without doubling cost on every response.

What production monitoring looks like

The test suite that passed tells the team what the model does on 200 expected queries. Production monitoring tells them what it does on 10,000 unexpected queries daily.

The minimum viable production monitoring:

• Sample and review: randomly sample 50 production queries per day for human review. This takes 30-60 minutes but catches failure patterns that automated metrics miss.
• Track contradiction rate: flag responses that explicitly contradict content in the source knowledge base. These are likely hallucinations.
• Monitor user feedback signals: track thumbs-down ratings, abandoned conversations, and follow-up "you're wrong" messages. These correlate with hallucination.
• Log queries that return [UNCERTAIN] or [NO_INFO]: these are knowledge gaps that need to be filled.

The team that builds this monitoring and reviews it weekly will discover their production failure patterns within the first month. The team that ships and assumes the 200-test evaluation represents production will discover the failure patterns six months later, from user complaints.

Common mistakes that increase production hallucination rates

Using temperature 0 for testing and temperature 0.7 in production. Evaluation at temperature 0 produces deterministic outputs. Production at temperature 0.7 produces varied outputs. The test results do not predict production behavior. If the application requires lower hallucination rates, test at the production temperature and account for variance by evaluating multiple samples per test case.

Trusting negative capability claims. A system prompt instruction like "Do not make up information" reduces hallucination frequency in some contexts but does not eliminate it. The model's training optimizes for plausible, helpful responses. When the model does not have a confident answer, the easiest path is to produce a plausible-sounding one — regardless of the system prompt instruction. Instructions are a signal, not a constraint. Trust output validation, not system prompt directives, for high-stakes correctness requirements.

Not distinguishing between types of wrong answers. A response can be wrong in several ways: factually incorrect but plausible (hallucination), correct on a different interpretation of the question (misunderstanding), correct information delivered with wrong confidence (miscalibration), or correct in training knowledge but wrong for this application's domain (knowledge mismatch). Each type requires a different mitigation. A monitoring system that only tracks "wrong answer rate" does not distinguish between them and does not provide actionable information.

Evaluating only on the test set after failures. When production hallucinations are discovered, the reflex is to add the failing queries to the test set and re-evaluate. This prevents regression on those specific failures but does not address the underlying distribution gap. The test set grows to include documented failures but continues to miss the next wave of undocumented ones. Systematic sampling of production queries — not just adding documented failures — is the correct approach to bridging the distribution gap.

Deploying a model update without evaluating against existing test cases. Model providers update their models periodically. A model that passes evaluation today may behave differently after a provider update. Applications that do not pin to specific model versions and do not re-evaluate after provider updates will have unexplained quality regressions that appear to be random. Pin to specific model versions in production, and evaluate new versions against the existing test set before migrating.

The monitoring setup that catches production hallucinations early

Production hallucination monitoring requires three complementary signals:

Automated factual consistency checks. For domains with a structured knowledge base, automated checks compare the model's assertions against the knowledge base:

def check_response_consistency(response: str, query: str) -> dict:
    """
    Checks whether the response's factual claims are consistent
    with the knowledge base. Returns list of inconsistencies found.
    """
    # Extract factual claims from the response using a second LLM call
    claims_prompt = f"""Extract the specific factual claims from this response as a JSON list.
    Each claim should be a single verifiable assertion.

    Response: {response}

    Return only a JSON array of strings."""

    claims_text = validation_llm.complete(claims_prompt, temperature=0, max_tokens=300)
    claims = json.loads(claims_text)

    inconsistencies = []
    for claim in claims:
        # Check each claim against the knowledge base
        supporting_docs = knowledge_base.retrieve(claim, top_k=3)
        if not any(supports_claim(doc, claim) for doc in supporting_docs):
            inconsistencies.append({
                "claim": claim,
                "status": "unsupported",
                "retrieved_docs": [d.id for d in supporting_docs]
            })

    return {
        "total_claims": len(claims),
        "unsupported_claims": len(inconsistencies),
        "details": inconsistencies
    }

User feedback correlation. Thumbs-down ratings, escalations to human support, and "that's wrong" follow-up messages are high-precision signals for hallucination. Track the topic distribution of negative feedback signals — a cluster around specific topics identifies knowledge gaps or prompt failures that automated metrics may miss.

Weekly human review cadence. Automated metrics catch patterns but miss nuance. A 30-minute weekly review of 30-50 sampled conversations — including the model's responses and the retrieved context — surfaces failure patterns that no automated metric was built to detect. This cadence is the most effective investment in long-term quality for applications where hallucination has real consequences.

The evaluation work does not end at launch. In LLM applications, evaluation is an ongoing operational discipline, not a pre-launch checkbox.

Looking ahead: where hallucination mitigation is going

The structural approaches described here — constrained generation, output validation, production monitoring — address hallucination with the tools available today. The field is moving toward better solutions. Model providers are improving factual calibration through reinforcement learning from human feedback on factual accuracy, not just response quality. Structured output support is becoming a first-class API feature rather than a prompting technique. Retrieval-augmented pipelines are getting tighter integration with generation, making it harder for the model to drift from its sources mid-response.

None of these advances eliminate the need for the monitoring and validation practices described here. A more accurate model still benefits from output validation on high-stakes responses. Production sampling still catches failure patterns that no test set anticipates. The discipline of treating evaluation as ongoing operations rather than a pre-launch gate is the practice that will remain relevant regardless of how the underlying models improve. Building that discipline now — systematic sampling, contradiction tracking, human review cadences — creates the institutional knowledge needed to manage increasingly capable models responsibly as they are deployed in higher-stakes applications. Teams that invest in this infrastructure early accumulate a body of production failure data that informs every future model change, prompt update, and retrieval adjustment. That accumulated knowledge is the most durable competitive advantage in LLM application development.