← Articles

How to A/B test prompts systematically

By Tunde Olatunde · 15 July 20260 views
#llm#prompt-engineering#ab-testing#evaluation#experimentation

The problem with informal prompt iteration

Most teams iterate on prompts by making a change, testing it against a few examples, and deciding it "feels better." That workflow has two fatal flaws: the example set is cherry-picked, and there is no way to know whether the improvement holds at scale or is just noise from a small sample.

The right model is the same one product teams use for UI changes: a controlled experiment with a clear metric, sufficient sample size, and a statistical test to distinguish signal from luck. The mechanics are slightly different because the "responses" are natural language, but the discipline is identical.

Choosing the right primary metric

Before you write a single line of experiment code, decide what you are optimising for. A vague goal like "better responses" cannot be measured. Concrete metrics you can calculate automatically include:

  • Task completion rate — for agentic tasks, did the model produce output that passes downstream validation?
  • Format compliance — did the response match the expected JSON schema, length constraint, or structural template?
  • Semantic similarity to a gold-standard answer — useful when you have a reference corpus.
  • LLM-as-judge score — pass the response to a separate evaluation model and ask it to rate quality on a defined rubric.
  • User downstream action — did the user accept or edit the suggestion? (Requires product instrumentation.)

Pick one primary metric. Secondary metrics are useful for understanding trade-offs (latency, token cost) but should not drive the rollout decision.

Designing the experiment

A prompt experiment has three components: variants, traffic allocation, and evaluation criteria.

Variants. You usually compare a control (current production prompt) against one or two challengers. Testing more than three variants simultaneously increases the sample size required to reach significance without proportionally increasing what you learn.

Traffic allocation. For online experiments, allocate traffic deterministically by user ID or request ID so the same user always sees the same variant. A hash-based assignment avoids the state management overhead of a database lookup:

import { createHash } from "crypto";

type Variant = "control" | "challenger_a" | "challenger_b";

export function assignVariant(
  userId: string,
  experimentId: string,
  weights: Record<Variant, number>  // must sum to 1.0
): Variant {
  const hash = createHash("sha256")
    .update(`${experimentId}:${userId}`)
    .digest("hex");
  const bucket = parseInt(hash.slice(0, 8), 16) / 0xffffffff; // 0.0 – 1.0

  let cumulative = 0;
  for (const [variant, weight] of Object.entries(weights)) {
    cumulative += weight;
    if (bucket < cumulative) return variant as Variant;
  }

  return "control"; // unreachable in practice
}

Call this once per request and log the assigned variant alongside the result so you can join them for analysis later.

Evaluation criteria. Define before the experiment starts what a "win" looks like: a minimum relative improvement in the primary metric and a p-value threshold (0.05 is conventional; 0.01 is appropriate when rollback cost is high).

Offline evaluation: testing against a golden dataset

For many prompt changes you do not need to expose users to the experiment at all. If you have a labelled dataset of inputs and expected outputs, you can run both variants offline and compare results before touching production.

Build a simple eval harness:

import Anthropic from "@anthropic-ai/sdk";
import * as fs from "fs";

interface EvalCase {
  input: string;
  expectedOutput: string;
  tags?: string[];
}

interface PromptVariant {
  id: string;
  systemPrompt: string;
}

interface EvalResult {
  variantId: string;
  caseIndex: number;
  input: string;
  response: string;
  score: number;
  inputTokens: number;
  outputTokens: number;
}

async function runEval(
  client: Anthropic,
  variant: PromptVariant,
  cases: EvalCase[],
  scorer: (expected: string, actual: string) => number
): Promise<EvalResult[]> {
  const results: EvalResult[] = [];

  for (let i = 0; i < cases.length; i++) {
    const c = cases[i];
    const response = await client.messages.create({
      model: "claude-sonnet-4-5",
      max_tokens: 1024,
      system: variant.systemPrompt,
      messages: [{ role: "user", content: c.input }],
    });

    const text = response.content
      .filter(b => b.type === "text")
      .map(b => (b as { type: "text"; text: string }).text)
      .join("");

    results.push({
      variantId:   variant.id,
      caseIndex:   i,
      input:       c.input,
      response:    text,
      score:       scorer(c.expectedOutput, text),
      inputTokens: response.usage.input_tokens,
      outputTokens: response.usage.output_tokens,
    });
  }

  return results;
}

// Example scorer: format compliance check for JSON output.
function jsonFormatScore(expected: string, actual: string): number {
  try {
    const parsed = JSON.parse(actual);
    const expectedParsed = JSON.parse(expected);
    // Check that all expected keys are present.
    const expectedKeys = Object.keys(expectedParsed);
    const presentKeys = expectedKeys.filter(k => k in parsed);
    return presentKeys.length / expectedKeys.length;
  } catch {
    return 0;
  }
}

// Entrypoint
const client = new Anthropic();
const cases: EvalCase[] = JSON.parse(fs.readFileSync("eval-dataset.json", "utf-8"));

const control: PromptVariant = {
  id: "control",
  systemPrompt: fs.readFileSync("prompts/control.txt", "utf-8"),
};

const challenger: PromptVariant = {
  id: "challenger_a",
  systemPrompt: fs.readFileSync("prompts/challenger_a.txt", "utf-8"),
};

const [controlResults, challengerResults] = await Promise.all([
  runEval(client, control, cases, jsonFormatScore),
  runEval(client, challenger, cases, jsonFormatScore),
]);

const controlMean = controlResults.reduce((s, r) => s + r.score, 0) / controlResults.length;
const challengerMean = challengerResults.reduce((s, r) => s + r.score, 0) / challengerResults.length;

console.log(`Control:    ${(controlMean * 100).toFixed(1)}%`);
console.log(`Challenger: ${(challengerMean * 100).toFixed(1)}%`);
console.log(`Delta:      ${((challengerMean - controlMean) * 100).toFixed(1)}pp`);

Run this in CI against every prompt change, just as you would run unit tests against code changes.

LLM-as-judge: using a model to score responses

For tasks where the "correct" output is subjective — summarisation, tone matching, helpfulness — a binary scorer does not capture quality. A model judge works well here if you design the rubric carefully.

async function judgeResponse(
  client: Anthropic,
  rubric: string,
  input: string,
  response: string
): Promise<{ score: number; reasoning: string }> {
  const judgement = await client.messages.create({
    model: "claude-opus-4-5",   // use a stronger model for judging
    max_tokens: 512,
    system: `You are an impartial evaluator. Score the response on the following rubric and output ONLY valid JSON:
{"score": <integer 1-5>, "reasoning": "<one sentence>"}

Rubric:
${rubric}`,
    messages: [
      {
        role: "user",
        content: `Input: ${input}\n\nResponse: ${response}`,
      },
    ],
  });

  const text = (judgement.content[0] as { type: "text"; text: string }).text;
  return JSON.parse(text);
}

const SUMMARISATION_RUBRIC = `
5 — Captures all key points, no hallucinated facts, reads naturally.
4 — Captures most key points, minor omissions, no hallucinated facts.
3 — Captures main point only, several omissions, or minor hallucination.
2 — Misses key points or contains a significant hallucination.
1 — Off-topic, empty, or factually wrong.
`.trim();

Use a stronger model as judge than the one you are testing when feasible — otherwise the judge has a systematic bias toward its own style. Using the same model to judge itself inflates scores by 10–20% in our experience.

Statistical significance testing

Collecting scores is the easy part. Knowing whether a difference is real requires a hypothesis test. For continuous scores (like 1–5 Likert scales), Welch's t-test works well. For binary outcomes (pass/fail), use a two-proportion z-test.

import numpy as np
from scipy import stats

def welch_t_test(control_scores: list[float], challenger_scores: list[float]) -> dict:
    control = np.array(control_scores)
    challenger = np.array(challenger_scores)

    t_stat, p_value = stats.ttest_ind(challenger, control, equal_var=False)
    
    effect_size = (challenger.mean() - control.mean()) / np.sqrt(
        (control.std() ** 2 + challenger.std() ** 2) / 2
    )
    
    return {
        "control_mean":      round(float(control.mean()), 4),
        "challenger_mean":   round(float(challenger.mean()), 4),
        "relative_delta":    round((challenger.mean() - control.mean()) / control.mean(), 4),
        "p_value":           round(float(p_value), 4),
        "cohen_d":           round(float(effect_size), 4),
        "significant":       p_value < 0.05 and challenger.mean() > control.mean(),
    }

def two_proportion_z_test(
    control_successes: int, control_n: int,
    challenger_successes: int, challenger_n: int
) -> dict:
    p1 = control_successes / control_n
    p2 = challenger_successes / challenger_n
    p_pool = (control_successes + challenger_successes) / (control_n + challenger_n)
    
    z = (p2 - p1) / np.sqrt(p_pool * (1 - p_pool) * (1/control_n + 1/challenger_n))
    p_value = 2 * (1 - stats.norm.cdf(abs(z)))

    return {
        "control_rate":    round(p1, 4),
        "challenger_rate": round(p2, 4),
        "relative_lift":   round((p2 - p1) / p1, 4),
        "p_value":         round(float(p_value), 4),
        "significant":     p_value < 0.05 and p2 > p1,
    }

The minimum detectable effect (MDE) determines how many samples you need. For a 5% relative improvement in a metric with 80% baseline mean and 20% standard deviation, with 80% power and 5% significance, you need roughly 450 samples per variant. Use a power calculator before starting the experiment to avoid under-powered tests that produce ambiguous results.

Running the experiment online with traffic splitting

For online experiments where the task is too subjective or context-dependent for offline evaluation, you need real user traffic. The request flow becomes:

  1. Resolve variant assignment from user ID.
  2. Build the prompt from the variant's template.
  3. Call the LLM and emit a telemetry event that includes the variant and a requestId.
  4. Log the downstream outcome (user accepted/edited/rejected the response) with the same requestId.
  5. Join variant + outcome at analysis time.
export async function handleSummaryRequest(
  userId: string,
  document: string,
  experimentId: string
): Promise<{ summary: string; requestId: string; variant: Variant }> {
  const variant = assignVariant(userId, experimentId, {
    control:      0.50,
    challenger_a: 0.50,
    challenger_b: 0,
  });

  const prompt = PROMPT_TEMPLATES[variant];
  const requestId = crypto.randomUUID();

  const response = await llm.messages(
    {
      model: "claude-sonnet-4-5",
      max_tokens: 512,
      system: prompt,
      messages: [{ role: "user", content: document }],
    },
    { feature: `summary-${experimentId}`, userId }
  );

  const summary = (response.content[0] as { type: "text"; text: string }).text;

  await logExperimentExposure({
    experimentId,
    requestId,
    userId,
    variant,
    timestamp: new Date().toISOString(),
  });

  return { summary, requestId, variant };
}

// Called from the client when the user accepts or edits the summary.
export async function logOutcome(
  requestId: string,
  outcome: "accepted" | "edited" | "rejected"
): Promise<void> {
  await logExperimentOutcome({ requestId, outcome, timestamp: new Date().toISOString() });
}

Store exposure events and outcome events in separate BigQuery tables, then join them for analysis:

SELECT
  e.variant,
  COUNT(DISTINCT e.user_id)                       AS exposed_users,
  COUNTIF(o.outcome = "accepted") / COUNT(o.*)    AS acceptance_rate,
  COUNTIF(o.outcome = "rejected") / COUNT(o.*)    AS rejection_rate
FROM `project.experiments.exposures` e
LEFT JOIN `project.experiments.outcomes` o USING (request_id)
WHERE e.experiment_id = "summary-v2"
  AND e.timestamp >= "2026-07-01"
GROUP BY e.variant;

Automating the decision with a rollout gate

Once your experiment reaches the required sample size and statistical significance, automate the rollout decision. A simple gate script that runs nightly:

async function evaluateExperiment(experimentId: string): Promise<void> {
  const results = await queryExperimentResults(experimentId);
  const test = two_proportion_z_test(
    results.control.successes, results.control.n,
    results.challenger.successes, results.challenger.n
  );

  if (test.significant && test.relative_lift > 0.05) {
    await promoteVariant(experimentId, "challenger_a");
    await notifySlack(`Experiment ${experimentId} promoted challenger_a (+${(test.relative_lift * 100).toFixed(1)}% lift, p=${test.p_value})`);
  } else if (results.challenger.n >= REQUIRED_SAMPLE_SIZE && !test.significant) {
    await concludeExperiment(experimentId, "no-winner");
    await notifySlack(`Experiment ${experimentId} concluded with no significant winner after ${results.challenger.n} samples`);
  }
}

Promotion means updating the production prompt in your config store (a Firestore document, a feature flag service, or a simple environment variable update in CI) so the winning variant becomes the new control for the next experiment.

Common mistakes to avoid

Testing too many things at once. When a prompt change involves five different modifications simultaneously, you cannot attribute the improvement to any one of them. Make atomic changes: one structural change, one phrasing change, or one example substitution per experiment.

Running experiments for a fixed calendar period instead of until significance. Peeking at results daily and stopping when they "look good" inflates the false positive rate substantially. Either pre-commit to a sample size or use sequential testing methods (like the SPRT or always-valid inference) if early stopping is genuinely necessary.

Ignoring secondary metric regressions. A challenger that improves acceptance rate by 8% but doubles response length is not a clear win. Always check cost and latency alongside quality.

Conflating eval score improvements with user outcome improvements. An LLM judge score of 4.2 vs 3.9 is a meaningful signal in offline evaluation, but the ultimate arbiter is what users do. Build the instrumentation to capture real outcomes as soon as the feature has enough traffic.

Building a prompt registry

As experiments accumulate, you need a structured way to store versions. A simple schema in Firestore:

interface PromptRecord {
  id: string;               // "summary-v3"
  feature: string;          // "document-summary"
  systemPrompt: string;
  model: string;
  maxTokens: number;
  status: "experiment" | "production" | "retired";
  createdAt: string;
  promotedAt?: string;
  experimentId?: string;
  metrics?: {
    acceptanceRate?: number;
    avgScore?: number;
    avgCostUsd?: number;
  };
}

Fetching the production prompt at runtime is then a single Firestore read against status == "production" filtered by feature. This decouples prompt changes from deploys entirely — you can promote a winner without touching application code.

A disciplined prompt experiment process — clear metric, deterministic assignment, sufficient sample size, automated significance check — turns prompt engineering from alchemy into engineering. The harness is a one-time investment; subsequent experiments run with minimal overhead.

Comments

No comments yet. Be the first!

Sign in to leave a comment.