Building a classification pipeline with zero-shot and few-shot prompting

Classification without a training pipeline

Traditional text classification requires labelled data, a training run, model hosting, and a redeployment cycle every time your taxonomy changes. An LLM-based classifier skips all of that. You write a prompt, describe your categories, optionally add a handful of examples, and you have a classifier that can be updated in minutes.

This is not always the right trade-off — a fine-tuned BERT model will outperform few-shot LLM prompting on narrow, high-volume tasks at a fraction of the latency and cost. But for broad taxonomies, constantly evolving categories, and tasks where the label space is defined in natural language, prompt-based classification is hard to beat.

Zero-shot classification: when to use it and when it breaks

Zero-shot classification describes every category in plain text and asks the model to assign the most fitting label without seeing any examples. It works surprisingly well when:

• The category names are semantically unambiguous ("billing", "technical support", "refund request").
• The model has seen similar classification tasks in its training data.
• The label space is small (under 20 categories).

It fails when categories overlap semantically, the task involves domain-specific jargon the model has not seen, or the distinction between categories is subtle and requires expert knowledge.

A minimal zero-shot classifier:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface ClassifyOptions {
  categories: string[];
  descriptions?: Record<string, string>;
  allowMultiple?: boolean;
}

async function classifyZeroShot(
  text: string,
  options: ClassifyOptions
): Promise<{ labels: string[]; reasoning: string }> {
  const categoryList = options.categories
    .map(c => options.descriptions?.[c] ? `- ${c}: ${options.descriptions[c]}` : `- ${c}`)
    .join("\n");

  const system = `You are a text classification system. Classify the input text into one of the following categories.
Return ONLY valid JSON with shape {"labels": ["label1"], "reasoning": "one sentence"}.
${options.allowMultiple ? "Multiple labels are allowed." : "Choose exactly one label."}

Categories:
${categoryList}`;

  const response = await client.messages.create({
    model: "claude-haiku-3-5",
    max_tokens: 256,
    system,
    messages: [{ role: "user", content: text }],
  });

  const text_content = (response.content[0] as { type: "text"; text: string }).text;
  return JSON.parse(text_content);
}

Usage:

const result = await classifyZeroShot(
  "My payment was charged twice and I need a refund",
  {
    categories: ["billing", "technical-support", "refund", "account-access", "general-inquiry"],
    descriptions: {
      billing:           "Questions about invoices, charges, or payment methods",
      "technical-support": "App not working, errors, or performance problems",
      refund:            "Requests to return money for a product or service",
      "account-access":  "Login problems, password reset, account locked",
      "general-inquiry": "Any question that doesn't fit the other categories",
    },
    allowMultiple: true,
  }
);
// result.labels => ["billing", "refund"]

Adding descriptions is almost always worth the extra tokens. Without them, ambiguous category names lead to inconsistent assignments.

Few-shot classification: selecting and ordering examples

Few-shot classification augments the prompt with labeled examples before the test input. The key insight is that example selection matters far more than example count. Three well-chosen examples often outperform ten mediocre ones.

Principles for example selection:

• Coverage — include at least one example per category.
• Difficulty — include examples that represent edge cases or categories that commonly confuse the model.
• Balance — avoid a lopsided distribution that biases the model toward majority classes.
• Recency — for time-sensitive domains, use recent examples so the language matches current usage.

interface LabeledExample {
  text: string;
  label: string;
}

async function classifyFewShot(
  text: string,
  categories: string[],
  examples: LabeledExample[]
): Promise<{ label: string; confidence: number; reasoning: string }> {
  const exampleBlock = examples
    .map(e => `Input: ${e.text}\nLabel: ${e.label}`)
    .join("\n\n");

  const system = `You are a text classifier. Use the examples below to classify new inputs.
Return ONLY valid JSON: {"label": "...", "confidence": 0.0-1.0, "reasoning": "..."}.
Valid labels: ${categories.join(", ")}.

Examples:
${exampleBlock}`;

  const response = await client.messages.create({
    model: "claude-haiku-3-5",
    max_tokens: 256,
    system,
    messages: [{ role: "user", content: `Input: ${text}` }],
  });

  return JSON.parse((response.content[0] as { type: "text"; text: string }).text);
}

Dynamic example selection from a retrieval store significantly improves accuracy for large taxonomies. Instead of static examples in the prompt, fetch the k most similar examples from your labelled dataset at query time using embedding similarity:

import { OpenAI } from "openai";

const openai = new OpenAI(); // use any embedding provider

async function getEmbedding(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, v, i) => sum + v * b[i], 0);
  const normA = Math.sqrt(a.reduce((sum, v) => sum + v * v, 0));
  const normB = Math.sqrt(b.reduce((sum, v) => sum + v * v, 0));
  return dot / (normA * normB);
}

async function retrieveExamples(
  query: string,
  exampleStore: Array<LabeledExample & { embedding: number[] }>,
  k: number = 3,
  ensureAllLabels: boolean = true
): Promise<LabeledExample[]> {
  const queryEmbedding = await getEmbedding(query);

  const scored = exampleStore.map(ex => ({
    ...ex,
    similarity: cosineSimilarity(queryEmbedding, ex.embedding),
  }));

  scored.sort((a, b) => b.similarity - a.similarity);

  if (!ensureAllLabels) return scored.slice(0, k);

  // Ensure each label appears at least once in the top examples.
  const byLabel = new Map<string, typeof scored[0]>();
  for (const ex of scored) {
    if (!byLabel.has(ex.label)) byLabel.set(ex.label, ex);
  }

  const mandatory = Array.from(byLabel.values());
  const remaining = scored
    .filter(ex => !mandatory.includes(ex))
    .slice(0, Math.max(0, k - mandatory.length));

  return [...mandatory, ...remaining].slice(0, k);
}

Pre-compute embeddings for your example store offline. At inference time you only pay for the query embedding and the LLM call — not re-embedding all examples on every request.

Hierarchical classification

Many real taxonomies are trees. A support ticket might be "Technical > Mobile > iOS > Login". Classifying all levels in a single prompt degrades accuracy because the model must reason about relationships between dozens of categories at once.

A cascade approach works better: classify at the top level, then use a narrower sub-taxonomy prompt conditioned on the parent label.

interface TaxonomyNode {
  label: string;
  description?: string;
  children?: TaxonomyNode[];
}

async function classifyHierarchical(
  text: string,
  taxonomy: TaxonomyNode[],
  examples: Map<string, LabeledExample[]>,
  depth: number = 0,
  maxDepth: number = 3
): Promise<string[]> {
  if (depth >= maxDepth) return [];

  const topCategories = taxonomy.map(n => n.label);
  const topExamples = examples.get("root") ?? [];

  const result = await classifyFewShot(text, topCategories, topExamples);

  const matchedNode = taxonomy.find(n => n.label === result.label);
  if (!matchedNode?.children?.length) return [result.label];

  // Recurse into the matched child taxonomy.
  const childLabels = await classifyHierarchical(
    text,
    matchedNode.children,
    examples,
    depth + 1,
    maxDepth
  );

  return [result.label, ...childLabels];
}

// Example taxonomy
const taxonomy: TaxonomyNode[] = [
  {
    label: "technical",
    children: [
      { label: "mobile", children: [{ label: "ios" }, { label: "android" }] },
      { label: "web", children: [{ label: "login" }, { label: "performance" }] },
    ],
  },
  { label: "billing" },
  { label: "refund" },
];

The cascade adds latency (one LLM call per level), but significantly improves accuracy for deep hierarchies. Cache the top-level result aggressively if many requests share the same root label.

Confidence scoring and abstention

A binary "correct / incorrect" classifier is not always appropriate in production. Sometimes it is better to abstain — send the input to a human reviewer — than to emit a low-confidence prediction.

Ask the model to score its own confidence and define an abstention threshold:

interface ClassificationResult {
  label: string | null;
  confidence: number;
  abstained: boolean;
  reasoning: string;
}

async function classifyWithAbstention(
  text: string,
  categories: string[],
  examples: LabeledExample[],
  abstentionThreshold: number = 0.75
): Promise<ClassificationResult> {
  const raw = await classifyFewShot(text, categories, examples);

  if (raw.confidence < abstentionThreshold) {
    return {
      label:      null,
      confidence: raw.confidence,
      abstained:  true,
      reasoning:  raw.reasoning,
    };
  }

  return {
    label:      raw.label,
    confidence: raw.confidence,
    abstained:  false,
    reasoning:  raw.reasoning,
  };
}

The threshold of 0.75 is a starting point. Tune it by plotting your classifier's precision-recall curve against a validation set and choosing the operating point that matches your tolerance for false positives vs false negatives.

One important caveat: LLM-reported confidence scores are not calibrated probabilities. A model that says "confidence: 0.9" is not necessarily right 90% of the time on that statement. Calibration varies by model, prompt, and task. Measure empirical accuracy at each confidence decile on your validation set and adjust thresholds accordingly.

Multi-label classification

Some inputs belong to multiple categories simultaneously. The most reliable approach is to transform multi-label classification into a series of binary questions, one per label:

async function classifyMultiLabel(
  text: string,
  categories: Array<{ label: string; description: string }>,
  examples: LabeledExample[]
): Promise<Array<{ label: string; present: boolean; confidence: number }>> {
  // Ask for all labels in a single call using structured output.
  const categoryQuestions = categories
    .map(c => `"${c.label}": is this category present? (${c.description})`)
    .join("\n");

  const system = `For each category below, determine if it applies to the input text.
Return ONLY valid JSON: {"results": [{"label": "...", "present": true/false, "confidence": 0.0-1.0}]}

Categories:
${categoryQuestions}`;

  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    system,
    messages: [{ role: "user", content: text }],
  });

  const parsed = JSON.parse((response.content[0] as { type: "text"; text: string }).text);
  return parsed.results;
}

This compresses all binary questions into a single API call. For very large label spaces (50+ categories), split into batches of 10–15 and aggregate.

Evaluating and monitoring the pipeline

Accuracy on a one-time validation set is not sufficient. Models update, category definitions drift, and input distribution shifts over time. Build evaluation into the deployment pipeline:

interface EvalMetrics {
  accuracy: number;
  macroF1: number;
  perClassF1: Record<string, number>;
  abstentionRate: number;
}

function computeMetrics(
  predictions: ClassificationResult[],
  groundTruth: Array<{ label: string }>
): EvalMetrics {
  const total = predictions.length;
  const abstained = predictions.filter(p => p.abstained).length;
  const nonAbstained = predictions.filter(p => !p.abstained);
  
  const correct = nonAbstained.filter(
    (p, i) => p.label === groundTruth[predictions.indexOf(p)].label
  ).length;

  const labels = [...new Set(groundTruth.map(g => g.label))];
  const perClassF1: Record<string, number> = {};

  for (const label of labels) {
    const tp = nonAbstained.filter((p, i) =>
      p.label === label && groundTruth[predictions.indexOf(p)].label === label
    ).length;
    const fp = nonAbstained.filter((p, i) =>
      p.label === label && groundTruth[predictions.indexOf(p)].label !== label
    ).length;
    const fn = groundTruth.filter((g, i) =>
      g.label === label && predictions[i].label !== label
    ).length;

    const precision = tp / (tp + fp || 1);
    const recall = tp / (tp + fn || 1);
    perClassF1[label] = precision + recall > 0
      ? 2 * (precision * recall) / (precision + recall)
      : 0;
  }

  const macroF1 = Object.values(perClassF1).reduce((s, f) => s + f, 0) / labels.length;

  return {
    accuracy: correct / (nonAbstained.length || 1),
    macroF1,
    perClassF1,
    abstentionRate: abstained / total,
  };
}

Run this evaluation daily against a held-out test set and alert when accuracy drops more than 3 percentage points below the baseline. Also track per-class F1 — a drop in overall accuracy often masks a catastrophic regression on a minority class that would be invisible in the aggregate.

Productionising with caching and rate limiting

Classification requests for identical or near-identical inputs waste tokens. A simple hash-based cache covers exact duplicates:

import { createHash } from "crypto";
import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });

async function cachedClassify(
  text: string,
  categories: string[],
  examples: LabeledExample[],
  ttlSeconds: number = 3600
): Promise<ClassificationResult> {
  const key = `classify:${createHash("sha256")
    .update(JSON.stringify({ text, categories }))
    .digest("hex")}`;

  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const result = await classifyWithAbstention(text, categories, examples);
  await redis.set(key, JSON.stringify(result), { EX: ttlSeconds });
  return result;
}

For a batch processing pipeline, parallelize calls within the concurrency limits of your LLM provider. Most providers allow 60–600 requests per minute per API key. A simple semaphore-based rate limiter:

class Semaphore {
  private permits: number;
  private queue: Array<() => void> = [];

  constructor(permits: number) {
    this.permits = permits;
  }

  async acquire(): Promise<void> {
    if (this.permits > 0) {
      this.permits--;
      return;
    }
    return new Promise(resolve => this.queue.push(resolve));
  }

  release(): void {
    if (this.queue.length > 0) {
      this.queue.shift()!();
    } else {
      this.permits++;
    }
  }
}

async function classifyBatch(
  inputs: string[],
  categories: string[],
  examples: LabeledExample[],
  concurrency: number = 20
): Promise<ClassificationResult[]> {
  const sem = new Semaphore(concurrency);

  return Promise.all(inputs.map(async text => {
    await sem.acquire();
    try {
      return await cachedClassify(text, categories, examples);
    } finally {
      sem.release();
    }
  }));
}

A classification pipeline built on few-shot LLM prompting gives you taxonomy flexibility that fine-tuned models cannot match. The trade-off is higher per-call cost and latency. Mitigate both with a fast, cheap model (Haiku or a comparable small model), aggressive caching, and batched async processing. The examples, confidence thresholds, and evaluation harness described here generalise across domains — adapt the taxonomy and the example store, and the rest of the pipeline is reusable.