Evaluating LLM outputs when you do not have a gold-standard test set

The Evaluation Paradox in Resource-Constrained Environments

In our health-tech lab here in Lagos, we rarely have the luxury of sitting on a pristine, hand-labeled gold-standard dataset. When you are building a tool to help community health workers triage respiratory conditions in low-connectivity areas, you don't start with a clean CSV of perfectly mapped inputs and outputs. You start with messy clinical notes, fragmented diagnostic logs, and a hardware stack that would make a Silicon Valley researcher weep.

Most mainstream advice assumes you have a GPU farm to run massive inference pipelines or a dedicated team to curate evaluation benchmarks. But for the 40+ teams I work with across the continent, the reality is different. We are deploying 7B models on repurposed laptops and edge devices using INT4 quantisation. In this environment, "evaluation" is not just a stage in the development lifecycle; it is a constant, lightweight struggle to ensure the model isn't hallucinating medical advice. When you don't have a gold-standard test set, you have to build your own evaluation harness from the ground up.

Section 1: The LLM-as-a-Judge Pattern with Local Quantisation

When you lack a ground truth dataset, the most accessible strategy is to use a more capable model to evaluate the outputs of your edge-deployed model. This is the 'LLM-as-a-Judge' paradigm. However, you cannot simply throw everything at a closed API. Between the costs and the latency of hitting OpenAI’s servers from a mobile connection in rural Nigeria, it is simply not viable. Instead, we use a larger, higher-precision model—perhaps a 30B model quantised to GGUF format—running locally to serve as an automated auditor.

By feeding the output of our primary INT4 model into a more robust, locally-hosted judge, we can establish a baseline for consistency. The judge model isn't the ground truth, but it acts as a consistent heuristic. We use YAML configuration files to define the rubrics that the judge must apply, focusing on specific clinical accuracy markers rather than conversational fluency.

evaluation_config:
  judge_model: "Llama-3-8B-Q8_0.gguf"
  criteria:
    - clinical_adherence: "Does the advice align with WHO protocols?"
    - safety_check: "Are there any dangerous omissions or hallucinations?"
    - conciseness: "Is the response actionable for a field worker?"
  scoring_scale: 1-5
  metadata: "clinical_triage_v1"

This setup allows us to cycle through thousands of generated prompts during a training run. While the judge is not perfect, it flags outliers. We focus our limited manual review time only on the outputs that receive low scores from the judge. This triage-within-a-triage approach is how we scale evaluation without the budget for human-in-the-loop oversight at scale.

Section 2: Implementing Semantic Drift Detection

If you don't have a gold standard, you can at least measure consistency. Semantic drift occurs when your model starts responding differently to variations of the same query. In health-tech, if asking "How do I treat a mild fever?" yields a vastly different instruction set than "What is the protocol for a fever?", you have a reliability problem.

We use a technique borrowed from search engineering: embedding-based distance scoring. By generating a vector representation of your model's outputs and measuring the cosine similarity between responses to paraphrased queries, you can detect when the model is losing its grounding. This is a "gold-standard-free" metric because it relies on the internal geometry of the model's language space rather than external labels. If the distance between two semantically identical queries exceeds a set threshold, the model is flagged for manual review.

Section 3: Leveraging Deterministic Constraints in Deployment

One of the biggest advantages of working with constrained hardware is the necessity of simplifying your deployment stack. We often use Grammar-Constrained Decoding (GCD) to force the model into specific formats, such as JSON or structured medical records. By enforcing this strict syntax, we eliminate a huge class of evaluation problems. You don't need a gold standard to tell you that your model failed to output a JSON object if your decoder forces it to.

In our current stack, we use Kotlin-based wrappers to handle edge-side inference where the model is quantised via GGUF, ensuring that even if the underlying weights are INT4, the structural integrity of the output remains rigid. Here is a simplified representation of how we handle output enforcement in our Kotlin interface:

class InferenceEngine(val modelPath: String) {
    fun generateValidatedResponse(prompt: String): String {
        val output = runQuantisedInference(modelPath, prompt)
        return if (isValidJSON(output)) {
            output
        } else {
            // Fallback to a predefined safety schema if validation fails
            generateDefaultRefusal()
        }
    }
}

This approach effectively turns your model into a deterministic machine. If the output isn't in the correct format, the error is immediately caught at the inference layer. This removes the need for complex natural language parsing in your evaluation pipeline.

Section 4: The 'Synthetic Gold' Approach via RAG

When you don't have a curated test set, you can build a 'synthetic' one by utilizing Retrieval-Augmented Generation (RAG). By grounding your model in a trusted document store—like a collection of verified clinical guidelines—you create a baseline for what a 'good' answer looks like. We use these documents as the 'context' for our evaluation. Any output generated by the model can then be verified against these retrieved chunks.

We measure the faithfulness of the model's output to the retrieved chunks rather than some abstract concept of correctness. This simplifies evaluation because you are comparing the model's output to a specific snippet of text that the model was supposed to summarize or follow. If the model provides a medical fact not present in the chunk, or contradicts it, the answer is flagged as hallucinated. This allows teams with zero labels to start with a directory of PDF manuals and immediately begin 'evaluating' their AI.

Section 5: Building a Culture of 'Evaluation-as-Contribution'

Ultimately, evaluation in resource-constrained settings is a community effort. When we build our quantisation tools, we include a standard CLI evaluation tool that logs these 'synthetic' scores. By sharing these logs among the 40+ teams I work with, we build a cross-team knowledge base. If Team A finds that their INT4 model struggles with dosage calculations in Swahili, they can flag it, and Team B can run a quick check on their own deployments.

We are not waiting for global benchmarks or massive datasets. We are building our own benchmarks out of our collective failures and refinements. This is the essence of the African AI ecosystem today. We iterate, we quantise, we evaluate against synthetic baselines, and we share the results.

If you find yourself stuck without a gold-standard set, stop worrying about the perfect dataset. Focus on the structural constraints you can control. Use a larger model as an auditor, enforce your output formats with rigid decoding, and ground your model in domain-specific documentation. These steps don't require high-end GPUs, just a bit of clever engineering and a willingness to accept that, in the real world, the best evaluation is the one you can actually run on the hardware you have in front of you. Every bit of accuracy you claw back through these methods is a win for the clinical outcomes you are supporting. Stay focused, keep the latency budget in mind, and keep the weights small.