Intent classification with embeddings: a production approach

Why embeddings for intent classification

Intent classification is one of the most common NLP tasks in production systems: given a user's message, determine what they want to do. Route them to the right handler, trigger the right workflow, or select the right response template.

The traditional approach — training a fine-tuned classifier — requires labelled examples, a training pipeline, model hosting, and periodic retraining as intent distributions shift. For teams without ML infrastructure, this is a significant investment.

Embedding-based classification offers a compelling alternative. You embed a set of canonical examples for each intent, store them in a vector database, and classify new inputs by finding their nearest neighbours. Adding a new intent is as simple as adding example sentences. No retraining, no deployment ceremony.

This approach isn't always better than fine-tuned models — dedicated classifiers outperform it when you have thousands of labelled examples. But for systems with 10-100 intents and limited training data, embedding-based classification is fast to build, easy to maintain, and performs surprisingly well.

This article covers the full production path: embedding model selection, example library design, vector storage, threshold calibration, confidence-scored output, fallback handling, and monitoring.

Embedding model selection

Your embedding model determines the quality ceiling for the entire system. The model needs to produce vectors where semantically similar text clusters together, which sounds obvious but varies enormously across models.

For English-language intent classification, several models work well:

• text-embedding-3-small (OpenAI): Fast, cheap, good general quality. 1536 dimensions.
• text-embedding-3-large (OpenAI): Higher quality, 3072 dimensions, slower and more expensive.
• voyage-large-2 (Voyage AI): Strong performance on retrieval benchmarks, 1024 dimensions.
• all-MiniLM-L6-v2 (Sentence Transformers): Open-source, runs locally, lower quality than API models.

The critical thing is to use the same model for indexing your canonical examples and for embedding incoming queries. Never mix models — the vector spaces are incompatible.

For a self-hosted approach, all-MiniLM-L6-v2 via the @xenova/transformers package runs in Node.js without an API dependency:

import { pipeline } from "@xenova/transformers";

let embedder: any = null;

async function getEmbedder() {
  if (!embedder) {
    embedder = await pipeline(
      "feature-extraction",
      "Xenova/all-MiniLM-L6-v2"
    );
  }
  return embedder;
}

async function embed(text: string): Promise<number[]> {
  const model = await getEmbedder();
  const output = await model(text, { pooling: "mean", normalize: true });
  return Array.from(output.data as Float32Array);
}

For API-based embedding, use the provider's batch endpoint to embed multiple texts efficiently:

import OpenAI from "openai";

const openai = new OpenAI();

async function embedBatch(texts: string[]): Promise<number[][]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: texts,
  });

  return response.data
    .sort((a, b) => a.index - b.index)
    .map((item) => item.embedding);
}

Designing the intent library

The quality of your example library matters more than almost anything else in this system. Five excellent, diverse examples per intent outperform fifty similar ones.

The key principle: examples should cover the range of ways users phrase the intent, not just the ideal phrasing. Users say "how do I cancel" and "cancel my account" and "I want to stop my subscription" and "delete my account please" — all the same intent, phrased differently.

interface IntentExample {
  text: string;
  weight?: number; // optional: down-weight ambiguous examples
}

interface Intent {
  id: string;
  label: string;
  description: string;
  examples: IntentExample[];
}

const INTENTS: Intent[] = [
  {
    id: "cancel_subscription",
    label: "Cancel Subscription",
    description: "User wants to cancel their subscription or account",
    examples: [
      { text: "I want to cancel my subscription" },
      { text: "How do I cancel?" },
      { text: "Cancel my account" },
      { text: "I'd like to stop my membership" },
      { text: "Delete my account" },
      { text: "I don't want to be charged anymore" },
      { text: "How do I unsubscribe?" },
      { text: "End my plan" },
    ],
  },
  {
    id: "billing_question",
    label: "Billing Question",
    description: "User has a question about their bill or charges",
    examples: [
      { text: "Why was I charged twice?" },
      { text: "I have a question about my invoice" },
      { text: "What does this charge mean?" },
      { text: "Can you explain my bill?" },
      { text: "I was overcharged" },
      { text: "When does my billing cycle renew?" },
      { text: "How much does the plan cost?" },
    ],
  },
  {
    id: "technical_support",
    label: "Technical Support",
    description: "User is experiencing a technical problem",
    examples: [
      { text: "The app isn't working" },
      { text: "I'm getting an error" },
      { text: "Something is broken" },
      { text: "I can't log in" },
      { text: "The feature isn't loading" },
      { text: "I found a bug" },
      { text: "Nothing is working" },
    ],
  },
];

Common mistakes to avoid:

• Too many similar examples. If six of your eight examples say "cancel subscription" with minor wording variations, you've wasted slot space and created a tight cluster that won't generalize.
• Ambiguous examples. "I need help" could be any intent. Either exclude it or give it a low weight.
• Missing edge cases. "delete everything" might be cancel_subscription or something else. Add edge cases explicitly.

Indexing: computing and storing embeddings

Compute and store intent embeddings at startup (or when the intent library changes). In production, persist these to your vector store rather than recomputing on every deployment:

interface IndexedIntent {
  intentId: string;
  exampleText: string;
  embedding: number[];
  weight: number;
}

async function buildIndex(intents: Intent[]): Promise<IndexedIntent[]> {
  const allExamples = intents.flatMap((intent) =>
    intent.examples.map((ex) => ({
      intentId: intent.id,
      text: typeof ex === "string" ? ex : ex.text,
      weight: typeof ex === "string" ? 1.0 : (ex.weight ?? 1.0),
    }))
  );

  const texts = allExamples.map((e) => e.text);
  const embeddings = await embedBatch(texts);

  return allExamples.map((example, i) => ({
    intentId: example.intentId,
    exampleText: example.text,
    embedding: embeddings[i],
    weight: example.weight,
  }));
}

For production use with hundreds of intents and thousands of examples, a vector database like pgvector, Qdrant, or Cloudflare Vectorize will outperform in-memory search. For smaller systems (under 10,000 vectors), in-memory search with a well-optimized cosine similarity function is fast enough:

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

interface ScoredExample {
  intentId: string;
  exampleText: string;
  similarity: number;
  weight: number;
}

function findNearestNeighbours(
  queryEmbedding: number[],
  index: IndexedIntent[],
  topK: number = 10
): ScoredExample[] {
  return index
    .map((item) => ({
      intentId: item.intentId,
      exampleText: item.exampleText,
      similarity: cosineSimilarity(queryEmbedding, item.embedding),
      weight: item.weight,
    }))
    .sort((a, b) => b.similarity - a.similarity)
    .slice(0, topK);
}

Aggregating scores: from neighbours to intent

Raw nearest-neighbour results give you the most similar individual examples, but you need to aggregate across examples to get an intent-level score. A weighted average works well:

interface IntentScore {
  intentId: string;
  score: number;
  topExample: string;
  topSimilarity: number;
  matchedExamples: number;
}

function aggregateIntentScores(
  neighbours: ScoredExample[],
  intents: Intent[]
): IntentScore[] {
  // Group by intent
  const byIntent = new Map<
    string,
    { totalScore: number; count: number; topExample: string; topSim: number }
  >();

  for (const neighbour of neighbours) {
    const existing = byIntent.get(neighbour.intentId) ?? {
      totalScore: 0,
      count: 0,
      topExample: "",
      topSim: 0,
    };

    existing.totalScore += neighbour.similarity * neighbour.weight;
    existing.count++;

    if (neighbour.similarity > existing.topSim) {
      existing.topSim = neighbour.similarity;
      existing.topExample = neighbour.exampleText;
    }

    byIntent.set(neighbour.intentId, existing);
  }

  return Array.from(byIntent.entries())
    .map(([intentId, data]) => ({
      intentId,
      score: data.totalScore / data.count,
      topExample: data.topExample,
      topSimilarity: data.topSim,
      matchedExamples: data.count,
    }))
    .sort((a, b) => b.score - a.score);
}

Threshold calibration: the most important tuning step

The threshold is where most embedding classifiers fail in production. Set it too high and you reject legitimate matches; too low and you misclassify ambiguous inputs. You need empirical calibration, not a gut-feel value.

Build a labeled evaluation set — at minimum 50 examples per intent, ideally 100+, plus out-of-scope examples that should trigger no-match:

interface EvalExample {
  text: string;
  expectedIntentId: string | null; // null = out of scope
}

interface ThresholdEvalResult {
  threshold: number;
  precision: number;
  recall: number;
  f1: number;
  outOfScopeAccuracy: number;
}

async function evaluateThresholds(
  evalSet: EvalExample[],
  index: IndexedIntent[],
  intents: Intent[],
  thresholds: number[] = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8]
): Promise<ThresholdEvalResult[]> {
  // Embed all eval examples
  const embeddings = await embedBatch(evalSet.map((e) => e.text));

  const predictions = evalSet.map((example, i) => {
    const neighbours = findNearestNeighbours(embeddings[i], index, 10);
    const scores = aggregateIntentScores(neighbours, intents);
    return {
      expected: example.expectedIntentId,
      topScore: scores[0]?.score ?? 0,
      topIntent: scores[0]?.intentId ?? null,
    };
  });

  return thresholds.map((threshold) => {
    let truePositive = 0;
    let falsePositive = 0;
    let falseNegative = 0;
    let trueNegativeOutOfScope = 0;
    let falsePositiveOutOfScope = 0;

    for (const pred of predictions) {
      const predicted =
        pred.topScore >= threshold ? pred.topIntent : null;

      if (pred.expected === null) {
        // Out of scope example
        if (predicted === null) trueNegativeOutOfScope++;
        else falsePositiveOutOfScope++;
      } else {
        if (predicted === pred.expected) truePositive++;
        else if (predicted !== null) falsePositive++;
        else falseNegative++;
      }
    }

    const precision =
      truePositive / Math.max(truePositive + falsePositive, 1);
    const recall =
      truePositive / Math.max(truePositive + falseNegative, 1);
    const f1 =
      precision + recall > 0
        ? (2 * precision * recall) / (precision + recall)
        : 0;

    const outOfScopeTotal = trueNegativeOutOfScope + falsePositiveOutOfScope;
    const outOfScopeAccuracy =
      trueNegativeOutOfScope / Math.max(outOfScopeTotal, 1);

    return { threshold, precision, recall, f1, outOfScopeAccuracy };
  });
}

Run this evaluation and pick the threshold that maximises F1 while keeping out-of-scope accuracy above an acceptable floor (typically 90%+). Most embedding classifiers on well-designed intent libraries perform best in the 0.65-0.75 range, but this varies significantly by domain.

Classification interface with confidence tiers

The final classifier exposes a structured result with confidence tiers rather than a binary yes/no:

type ConfidenceTier = "high" | "medium" | "low" | "none";

interface ClassificationResult {
  intentId: string | null;
  intentLabel: string | null;
  confidence: number;
  tier: ConfidenceTier;
  alternativeIntents: Array<{ intentId: string; confidence: number }>;
  topMatchExample: string | null;
}

class IntentClassifier {
  private index: IndexedIntent[] = [];
  private intents: Intent[] = [];
  private intentMap: Map<string, Intent> = new Map();
  private thresholds = {
    high: 0.78,
    medium: 0.68,
    low: 0.58,
  };

  async initialize(intents: Intent[]): Promise<void> {
    this.intents = intents;
    this.intentMap = new Map(intents.map((i) => [i.id, i]));
    this.index = await buildIndex(intents);
    console.log(
      `Classifier initialized with ${this.index.length} examples across ${intents.length} intents`
    );
  }

  async classify(text: string): Promise<ClassificationResult> {
    const embedding = await embedBatch([text]);
    const neighbours = findNearestNeighbours(embedding[0], this.index, 15);
    const scores = aggregateIntentScores(neighbours, this.intents);

    const top = scores[0];

    if (!top || top.score < this.thresholds.low) {
      return {
        intentId: null,
        intentLabel: null,
        confidence: top?.score ?? 0,
        tier: "none",
        alternativeIntents: [],
        topMatchExample: null,
      };
    }

    const tier: ConfidenceTier =
      top.score >= this.thresholds.high
        ? "high"
        : top.score >= this.thresholds.medium
          ? "medium"
          : "low";

    const intent = this.intentMap.get(top.intentId);

    return {
      intentId: top.intentId,
      intentLabel: intent?.label ?? top.intentId,
      confidence: top.score,
      tier,
      alternativeIntents: scores.slice(1, 3).map((s) => ({
        intentId: s.intentId,
        confidence: s.score,
      })),
      topMatchExample: top.topExample,
    };
  }
}

Applications use the confidence tier to decide how to handle the result: high-confidence classifications route automatically, medium-confidence may add a confirmation step, and low-confidence or no-match routes to a human or fallback handler.

Handling out-of-scope and fallback to LLM

No intent library covers every possible user message. When the classifier returns tier: "none", you have two options: hard-reject with an error message, or escalate to an LLM for free-form handling.

For chatbot applications, the LLM fallback is usually the right choice:

async function handleClassificationResult(
  userMessage: string,
  result: ClassificationResult
): Promise<string> {
  if (result.tier === "high" || result.tier === "medium") {
    // Route to intent handler
    return await routeToIntentHandler(result.intentId!, userMessage);
  }

  if (result.tier === "low") {
    // Confirm before routing
    return `I think you want to ${result.intentLabel?.toLowerCase()}. Is that right?`;
  }

  // Fallback to LLM for out-of-scope
  return await handleWithLLM(userMessage);
}

async function handleWithLLM(message: string): Promise<string> {
  const response = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 512,
    system: "You are a helpful customer support agent. Answer the user's question directly and concisely.",
    messages: [{ role: "user", content: message }],
  });
  return response.content[0].type === "text" ? response.content[0].text : "";
}

Monitoring and continuous improvement

Track these metrics in production to keep the classifier accurate over time:

interface ClassificationMetrics {
  intentId: string | null;
  confidence: number;
  tier: ConfidenceTier;
  userMessage: string;
  timestamp: Date;
  wasOverridden?: boolean; // human corrected the classification
  correctIntentId?: string; // what the correct intent was
}

The wasOverridden and correctIntentId fields are populated when humans correct the classifier — this is your signal for which intents need more examples. If billing_question is consistently being mis-classified as technical_support, your examples are too similar or you need to add more distinguishing ones.

Set up alerts for:

• Increasing none tier rate: More inputs falling outside your intent library — either users are asking new things or the library needs expansion
• High override rate on specific intents: Those intents have poor examples or overlapping coverage
• Sudden drop in high-confidence rate: Could indicate model degradation, prompt injection, or a shift in user language

Embedding-based intent classification earns its place in production because the improvement loop is tight. Adding five examples to an under-performing intent takes minutes, deploys instantly (just rebuild the index), and immediately improves classification quality. That velocity advantage over fine-tuned models is substantial for teams moving quickly.