Building a customer support bot that escalates gracefully
Customer support bots have a terrible reputation, and most of them deserve it. The pattern is familiar: you explain your problem, the bot fires back a list of FAQ links, you say "talk to a human," it asks you to rephrase your query, you type "AGENT" in all caps, and eventually you give up and call the phone number buried at the bottom of the contact page. The bot failed not because AI is incapable of handling support queries — it is — but because the designers treated escalation as a failure state to be avoided rather than a first-class feature to be engineered.
This article walks through building a support bot that actually works: one that resolves straightforward queries quickly, recognises when it is out of its depth, and hands off to a human agent with full context so the customer never has to repeat themselves.
Architecture overview
A production-ready support bot has five distinct layers working together:
- Intent classification — understand what the user actually wants
- Knowledge retrieval — pull relevant information from your support corpus
- Response generation — produce accurate, on-brand replies
- Escalation detection — decide when to hand off
- Context handoff — transfer everything the human agent needs
Most tutorials cover layers 2 and 3 and ignore the rest. The escalation layer is where real products succeed or fail.
The data flow looks like this: the user message arrives, gets classified, triggers a retrieval step if needed, passes through the LLM for a response, then goes through an escalation scorer. If the score exceeds a threshold, the bot initiates a handoff instead of sending the response directly.
Intent classification and routing
Before the LLM writes a single word, you need to know what category of request you are dealing with. Some requests — order status, password reset, return policy — are safe for full automation. Others — billing disputes, account compromises, emotionally charged complaints — should go straight to a human queue.
from anthropic import Anthropic
import json
client = Anthropic()
INTENT_SCHEMA = {
"type": "object",
"properties": {
"intent": {
"type": "string",
"enum": [
"order_status",
"return_request",
"billing_dispute",
"technical_support",
"account_security",
"general_inquiry",
"complaint",
"cancellation"
]
},
"sentiment": {
"type": "string",
"enum": ["positive", "neutral", "frustrated", "angry"]
},
"complexity": {
"type": "string",
"enum": ["simple", "moderate", "complex"]
},
"urgency": {"type": "boolean"},
"contains_pii_risk": {"type": "boolean"}
},
"required": ["intent", "sentiment", "complexity", "urgency", "contains_pii_risk"]
}
def classify_intent(message: str, conversation_history: list[dict]) -> dict:
history_text = "\n".join(
f"{turn['role'].upper()}: {turn['content']}"
for turn in conversation_history[-6:] # last 3 exchanges
)
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
system="""You are an intent classifier for a customer support system.
Analyse the user's message and conversation history, then return a JSON object
matching the provided schema. Be conservative: when in doubt, mark complexity
as higher, not lower.""",
messages=[
{
"role": "user",
"content": f"""Conversation history:
{history_text}
Latest user message: {message}
Classify this message and return valid JSON."""
}
],
tools=[{
"name": "classify",
"description": "Classify the user intent",
"input_schema": INTENT_SCHEMA
}],
tool_choice={"type": "tool", "name": "classify"}
)
tool_use = next(b for b in response.content if b.type == "tool_use")
return tool_use.input
The key insight here is using Claude's tool-use mode with a strict schema. This eliminates the ambiguity you get when asking the model to "return JSON" in a text response — tool use enforces the schema at the API level, so you never get malformed output in production.
Building the knowledge retrieval layer
The bot needs a way to answer factual questions about your product without hallucinating. A simple vector search over your support documentation works well for most use cases. Here is a minimal implementation using embeddings and a local FAISS index:
import numpy as np
import faiss
from anthropic import Anthropic
client = Anthropic()
class SupportKnowledgeBase:
def __init__(self, documents: list[dict]):
"""
documents: list of {"id": str, "content": str, "metadata": dict}
"""
self.documents = documents
self.index = None
self.embeddings = None
self._build_index()
def _embed(self, texts: list[str]) -> np.ndarray:
# Use your preferred embedding model here.
# For production, consider Cohere embed or OpenAI text-embedding-3-small.
# This stub returns random vectors for illustration.
return np.random.rand(len(texts), 1536).astype("float32")
def _build_index(self):
texts = [d["content"] for d in self.documents]
self.embeddings = self._embed(texts)
dimension = self.embeddings.shape[1]
self.index = faiss.IndexFlatIP(dimension)
faiss.normalize_L2(self.embeddings)
self.index.add(self.embeddings)
def search(self, query: str, top_k: int = 4) -> list[dict]:
query_vec = self._embed([query])
faiss.normalize_L2(query_vec)
distances, indices = self.index.search(query_vec, top_k)
results = []
for score, idx in zip(distances[0], indices[0]):
if idx == -1:
continue
results.append({
**self.documents[idx],
"relevance_score": float(score)
})
return results
def format_context(self, results: list[dict]) -> str:
sections = []
for r in results:
sections.append(
f"[Source: {r['metadata'].get('title', r['id'])}]\n{r['content']}"
)
return "\n\n---\n\n".join(sections)
Once you have retrieval wired up, the response generation step is a standard RAG call. The system prompt defines the bot's personality and safety guardrails, and the retrieved context is injected into the user turn.
Escalation detection: the heart of the system
This is where most bots fail. Escalation detection needs to catch several distinct signals:
Sentiment degradation — the user started neutral but is getting frustrated. Track sentiment turn-by-turn and trigger escalation when you see two consecutive angry or frustrated classifications.
Repeated rephrasing — if the user is asking the same question in three different ways, the bot is not resolving their need. Count semantic near-duplicates across the conversation.
Direct escalation requests — "talk to a human," "I want to speak to someone," "get me a manager." These should be honoured immediately, without the bot trying one more time.
High-stakes intents — billing disputes, legal threats, account compromises, and cancellations should skip the bot entirely or get very short resolution windows before escalation.
Confidence deficit — if the retrieval layer returns nothing above a relevance threshold, the bot is operating without grounding and the risk of hallucination spikes.
from dataclasses import dataclass
from enum import Enum
class EscalationReason(Enum):
USER_REQUESTED = "user_requested"
SENTIMENT_DEGRADED = "sentiment_degraded"
REPEATED_REPHRASING = "repeated_rephrasing"
HIGH_STAKES_INTENT = "high_stakes_intent"
LOW_CONFIDENCE = "low_confidence"
TURN_LIMIT_EXCEEDED = "turn_limit_exceeded"
@dataclass
class EscalationDecision:
should_escalate: bool
reason: EscalationReason | None
confidence: float # 0-1, how sure we are this needs a human
suggested_queue: str # "billing", "technical", "general", "urgent"
HIGH_STAKES_INTENTS = {"billing_dispute", "account_security", "cancellation"}
ESCALATION_PHRASES = [
"speak to", "talk to", "human", "agent", "manager",
"supervisor", "person", "representative", "escalate"
]
def check_escalation(
message: str,
intent: dict,
conversation_history: list[dict],
retrieval_scores: list[float],
max_bot_turns: int = 8
) -> EscalationDecision:
# 1. Explicit user request
msg_lower = message.lower()
if any(phrase in msg_lower for phrase in ESCALATION_PHRASES):
return EscalationDecision(
should_escalate=True,
reason=EscalationReason.USER_REQUESTED,
confidence=1.0,
suggested_queue=_queue_for_intent(intent["intent"])
)
# 2. High-stakes intent
if intent["intent"] in HIGH_STAKES_INTENTS:
return EscalationDecision(
should_escalate=True,
reason=EscalationReason.HIGH_STAKES_INTENT,
confidence=0.95,
suggested_queue=_queue_for_intent(intent["intent"])
)
# 3. Sentiment degradation (two consecutive angry/frustrated turns)
recent_sentiments = [
t.get("sentiment") for t in conversation_history[-4:]
if t["role"] == "user"
]
angry_count = sum(1 for s in recent_sentiments if s in ("angry", "frustrated"))
if angry_count >= 2:
return EscalationDecision(
should_escalate=True,
reason=EscalationReason.SENTIMENT_DEGRADED,
confidence=0.85,
suggested_queue="urgent"
)
# 4. Low retrieval confidence
if retrieval_scores and max(retrieval_scores) < 0.45:
return EscalationDecision(
should_escalate=True,
reason=EscalationReason.LOW_CONFIDENCE,
confidence=0.75,
suggested_queue=_queue_for_intent(intent["intent"])
)
# 5. Turn limit
bot_turns = sum(1 for t in conversation_history if t["role"] == "assistant")
if bot_turns >= max_bot_turns:
return EscalationDecision(
should_escalate=True,
reason=EscalationReason.TURN_LIMIT_EXCEEDED,
confidence=0.9,
suggested_queue=_queue_for_intent(intent["intent"])
)
return EscalationDecision(
should_escalate=False,
reason=None,
confidence=0.0,
suggested_queue=""
)
def _queue_for_intent(intent: str) -> str:
mapping = {
"billing_dispute": "billing",
"account_security": "urgent",
"cancellation": "retention",
"technical_support": "technical",
}
return mapping.get(intent, "general")
Notice that the turn limit is a hard backstop. Even if none of the other signals fire, a conversation that has gone eight rounds without resolution is almost certainly stuck. Ending it gracefully and handing off is better than grinding the customer down with more bot responses.
The handoff experience
The escalation decision is only half the battle. How you communicate the handoff determines whether the customer stays calm or hangs up in disgust.
Do not apologise for the bot. "I'm sorry I couldn't help you" frames the entire bot interaction as a failure. Instead, frame it as a natural step: "I'm connecting you with our billing team who can look into this directly."
Set accurate wait time expectations. If your ticketing system has queue depth data, use it. "You're number 4 in queue, estimated wait 7 minutes" is infinitely better than "please hold."
Confirm that nothing will be repeated. The customer's biggest fear is having to re-explain everything. Lead with "I've shared our full conversation with the agent, so they're already up to speed."
Collect the context package. Before handing off, build a structured summary for the agent:
def build_handoff_context(
conversation_history: list[dict],
intent: dict,
escalation: EscalationDecision,
customer_id: str | None = None
) -> dict:
"""
Returns a structured context object to attach to the support ticket.
"""
# Summarise the conversation for the agent
summary_response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=400,
system="Summarise this customer support conversation in 3-5 bullet points. "
"Focus on: what the customer needs, what was already tried, "
"and any relevant account details mentioned.",
messages=[
{
"role": "user",
"content": "\n".join(
f"{t['role'].upper()}: {t['content']}"
for t in conversation_history
)
}
]
)
return {
"customer_id": customer_id,
"escalation_reason": escalation.reason.value if escalation.reason else None,
"suggested_queue": escalation.suggested_queue,
"detected_intent": intent["intent"],
"detected_sentiment": intent["sentiment"],
"conversation_turns": len(conversation_history),
"agent_summary": summary_response.content[0].text,
"full_transcript": conversation_history,
"metadata": {
"bot_version": "2.1.0",
"escalation_confidence": escalation.confidence
}
}
Use a fast model (Haiku or equivalent) for the summary generation — you want this to be nearly instantaneous from the user's perspective.
Handling the edge cases
A few scenarios catch teams off guard in production:
The user says "never mind" after the bot offers escalation. Always give the user an out. "Would you still like me to connect you, or is there something else I can try?" If they want to continue with the bot, reset the escalation state and try again — but log it, because two aborted escalations in one session usually means you should escalate anyway.
After-hours escalation. If no agents are available, the bot should say so clearly and offer async options: email callback, scheduled chat, or a support ticket with guaranteed response time. Never leave the user in a queue that will never be answered.
Escalation loops. Sometimes an agent resolves part of an issue and the customer comes back to the bot with the remainder. Detect returning customers, pull their recent ticket history, and surface it in the greeting rather than treating them as a new conversation.
Language and accessibility. Escalation phrases vary by language and culture. Maintain a localised list of escalation signals and test it with native speakers. A frustrated French speaker is unlikely to type "speak to an agent."
Testing your escalation logic
Escalation logic is easy to write and hard to tune. A bot that escalates too aggressively wastes agent time; one that escalates too late creates angry customers. Build an evaluation harness:
Create a labelled dataset of 50-100 historical conversations, annotated with the "correct" escalation point (human-labelled). Then run your escalation logic over the replayed conversations and measure precision (how often escalations were actually needed) and recall (how many cases that needed escalation were caught). Aim for recall above 90% — missing an escalation is more costly than a false positive.
Run this evaluation on every change to your escalation thresholds, and re-run it quarterly as your customer base and product evolve. The signals that predict escalation for a B2C e-commerce store are quite different from those for a B2B SaaS product.
Deployment considerations
A few practical notes from running support bots in production:
Keep the LLM call latency below 2 seconds for the first response. Users tolerate longer waits for follow-ups but judge the bot immediately on the first message. Use streaming responses to show progress indicators rather than blank screens.
Log every conversation with enough metadata to reconstruct the escalation decision path. When a customer complains about the bot, you need to be able to replay exactly what happened and why the bot responded the way it did.
Rate-limit per-user to prevent abuse. Some users discover that typing "I want to speak to a manager" immediately skips the queue and use it routinely. Track this pattern and adjust.
Instrument the handoff success rate separately from the bot resolution rate. A handoff where the customer resolves their issue with the agent is a success, not a failure. Track the combined resolution rate (bot + agent) as your primary metric.
The best support bots are invisible in retrospect — customers remember getting their problem solved, not whether a bot or human did it. Escalation done well is the feature that makes that possible.