Tool use in production: why LLM agent reliability is a systems problem
A developer builds an AI assistant that can search a knowledge base, query a database, and send emails. In development, it works smoothly. In production, it fails in ways that were not anticipated: the knowledge base search returns a 503 when the embedding service is under load; the database query times out on a complex join; the email service rate limits the assistant during a bulk action. The assistant returns unhelpful error messages to users, or worse, silently fails and tells the user the action was completed when it was not.
The developer had thought about what the assistant should do. He had not thought about what it should do when each external service fails. This is the reliability gap in LLM agent development: the happy path is built carefully, and the failure paths are discovered in production.
Why agent reliability is harder than application reliability
A standard web application that calls external services has predictable failure modes. The service fails, the application returns an error, the user retries. The application is stateless across requests; a failure in one request does not affect the next.
An LLM agent has additional failure modes that compound:
Sequential tool call failures. An agent that executes a multi-step plan — search, then retrieve, then format, then send — must handle failures at each step. A failure at step 3 of 5 has already consumed the cost of steps 1 and 2. The agent must decide whether to retry from the failure point, retry from the beginning, or abort with a partial result.
Ambiguous tool call results. A tool that returns a 200 with a partial result — the database returned the first 100 rows of a potentially 10,000-row query — is not clearly a success or a failure. The agent must interpret the result and decide whether to request more data, proceed with partial results, or escalate to the user.
Stateful side effects. If the agent sends an email at step 3 and then fails at step 4, retrying from step 3 sends the email twice. Tools that have side effects (writes, sends, transactions) require idempotency handling that is usually not built into the tool definition.
Context pollution. A failed tool call that returns an error in an unexpected format can pollute the agent's context. The model may interpret the error as data and continue reasoning from incorrect information.
Designing tools for reliability
The tool interface is the contract between the agent and the external world. A well-designed tool interface makes agent reliability achievable; a poorly designed one makes it difficult.
Structured, predictable return types. Tools should return typed, predictable structures. An error should be clearly distinguishable from a partial result, which should be clearly distinguishable from a complete result:
from dataclasses import dataclass
from typing import Optional, List, Any
from enum import Enum
class ToolResultStatus(Enum):
SUCCESS = "success"
PARTIAL = "partial" # Tool succeeded but result is incomplete
ERROR = "error" # Tool failed — caller should handle
RATE_LIMITED = "rate_limited" # Should retry after delay
NOT_FOUND = "not_found" # Query returned no results (not an error)
@dataclass
class ToolResult:
status: ToolResultStatus
data: Optional[Any]
error_message: Optional[str]
retry_after_seconds: Optional[int] # Set for RATE_LIMITED status
is_complete: bool # False for PARTIAL status — more data available
continuation_token: Optional[str] # For paginated results
def search_knowledge_base(query: str, limit: int = 10) -> ToolResult:
try:
results = search_service.search(query, limit=limit)
has_more = search_service.total_count(query) > limit
return ToolResult(
status=ToolResultStatus.SUCCESS if not has_more else ToolResultStatus.PARTIAL,
data=results,
error_message=None,
retry_after_seconds=None,
is_complete=not has_more,
continuation_token=results[-1].id if has_more else None
)
except RateLimitError as e:
return ToolResult(
status=ToolResultStatus.RATE_LIMITED,
data=None,
error_message=f"Search service rate limited",
retry_after_seconds=e.retry_after,
is_complete=False,
continuation_token=None
)
except ServiceUnavailableError:
return ToolResult(
status=ToolResultStatus.ERROR,
data=None,
error_message="Search service temporarily unavailable. Try again in a moment.",
retry_after_seconds=10,
is_complete=False,
continuation_token=None
)
Idempotency keys for write operations. Every tool that has side effects should accept an idempotency key. The agent generates the key before calling the tool; if the call is retried, the same key ensures the operation is only executed once:
def send_email(
to: str,
subject: str,
body: str,
idempotency_key: str
) -> ToolResult:
"""
Sends an email. If called multiple times with the same idempotency_key,
the email is only sent once.
"""
# Check whether this idempotency key has been used
existing = email_log.get(idempotency_key)
if existing:
# Already sent — return the original result
return ToolResult(
status=ToolResultStatus.SUCCESS,
data={"message_id": existing.message_id, "already_sent": True},
error_message=None,
retry_after_seconds=None,
is_complete=True,
continuation_token=None
)
# Send the email
try:
result = email_service.send(to=to, subject=subject, body=body)
email_log.record(idempotency_key, message_id=result.message_id)
return ToolResult(
status=ToolResultStatus.SUCCESS,
data={"message_id": result.message_id, "already_sent": False},
...
)
except Exception as e:
return ToolResult(status=ToolResultStatus.ERROR, error_message=str(e), ...)
Timeout enforcement. Every tool should have a hard timeout. Without one, a slow external service can cause the entire agent to hang:
import asyncio
from functools import wraps
def with_timeout(timeout_seconds: float):
def decorator(func):
@wraps(func)
async def wrapper(*args, **kwargs):
try:
return await asyncio.wait_for(func(*args, **kwargs), timeout=timeout_seconds)
except asyncio.TimeoutError:
return ToolResult(
status=ToolResultStatus.ERROR,
data=None,
error_message=f"Tool timed out after {timeout_seconds}s",
retry_after_seconds=5,
is_complete=False,
continuation_token=None
)
return wrapper
return decorator
@with_timeout(10.0)
async def database_query(sql: str, params: list) -> ToolResult:
result = await db.execute(sql, params)
return ToolResult(status=ToolResultStatus.SUCCESS, data=result.rows, ...)
The agent execution framework
The framework that runs tool calls should handle retries, timeouts, and error escalation — not the agent prompt:
class AgentExecutionFramework:
MAX_RETRIES = 3
MAX_TOTAL_STEPS = 15 # Prevent infinite loops
async def execute_tool_call(
self,
tool_name: str,
tool_args: dict,
step_number: int
) -> ToolResult:
for attempt in range(self.MAX_RETRIES):
result = await self.call_tool(tool_name, tool_args)
if result.status == ToolResultStatus.SUCCESS:
return result
if result.status == ToolResultStatus.RATE_LIMITED:
if attempt < self.MAX_RETRIES - 1:
await asyncio.sleep(result.retry_after_seconds or 5)
continue
if result.status == ToolResultStatus.NOT_FOUND:
return result # Not an error — don't retry
if result.status == ToolResultStatus.ERROR:
if attempt < self.MAX_RETRIES - 1:
await asyncio.sleep(2 ** attempt) # Exponential backoff
continue
else:
# All retries exhausted
self.log_tool_failure(tool_name, tool_args, result, step_number)
return result
return result
async def run_agent(self, user_message: str, session_id: str) -> str:
conversation = [{"role": "user", "content": user_message}]
step_count = 0
while step_count < self.MAX_TOTAL_STEPS:
response = await self.llm_client.complete_with_tools(
messages=conversation,
tools=AVAILABLE_TOOLS
)
if response.finish_reason == "stop":
return response.content # Agent completed
if response.finish_reason == "tool_calls":
step_count += 1
tool_results = []
for tool_call in response.tool_calls:
result = await self.execute_tool_call(
tool_call.name, tool_call.args, step_count
)
tool_results.append({
"tool_call_id": tool_call.id,
"result": result
})
conversation.append({"role": "assistant", "tool_calls": response.tool_calls})
conversation.append({"role": "tool", "tool_results": tool_results})
continue
return "I was unable to complete this task — it required too many steps."
Observability for agent systems
Standard application observability (request logs, error rates) is insufficient for agents. An agent that succeeds at the HTTP layer but fails semantically — it called the wrong tool, misinterpreted a result, or completed the wrong task — is invisible to standard metrics.
Agent-specific observability:
@dataclass
class AgentTrace:
session_id: str
user_message: str
steps: List[AgentStep]
final_response: str
total_tool_calls: int
failed_tool_calls: int
total_duration_ms: int
success: bool # Determined by user feedback or output validation
@dataclass
class AgentStep:
step_number: int
tool_name: Optional[str]
tool_args: Optional[dict]
tool_result_status: Optional[str]
duration_ms: int
tokens_used: int
Tracing every tool call, its arguments, its result status, and its duration provides the observability needed to debug agent failures. A trace where the agent called search_knowledge_base three times with the same query, received ERROR each time, and then gave up provides a clear picture: the search service was failing and the agent's retry logic hit the maximum retries.
Without this trace, the failure looks like "agent gave a poor response" with no indication of whether it was a retrieval failure, a reasoning failure, or a tool implementation bug.
Common mistakes in LLM agent implementations
Not bounding the total number of tool call steps. An agent that can make tool calls indefinitely will occasionally enter a loop — calling the same tool repeatedly with slightly different arguments, making no progress toward completing the task. The MAX_TOTAL_STEPS guard in the execution framework is the most important safety mechanism in any agent system. Set it aggressively (15 steps is usually more than enough for any task a user would submit) and handle the "too many steps" case gracefully.
Returning raw tool errors to the model without guidance. When a tool returns an error, the error message becomes part of the model's context. If the error is technical ("Connection refused to database host: db-primary-east:5432"), the model may attempt to reason about the database connection rather than handling the failure gracefully. Tool errors in the model's context should be written for the model, not for an engineer: "The database is temporarily unavailable. Please try again in a moment or contact support." The technical error is logged; the model receives the user-friendly version.
Allowing the model to choose between tools it cannot distinguish. Tool descriptions that are ambiguous — two tools that both "retrieve information about the user" — lead to the model guessing which tool to call. Tool descriptions should be specific enough that the model can determine the correct tool without ambiguity. Each tool's description should include: what it does, when to use it, what it returns, and when NOT to use it (if there is a common confusion with another tool).
Not testing the agent against realistic failure scenarios. Agent testing typically covers the happy path. Tests that inject tool failures — the search returns a 503, the database times out, the email rate limit is hit — reveal whether the retry logic, error handling, and graceful degradation work as designed. Build a test harness that allows injecting failures at specific steps:
class AgentTestHarness:
def __init__(self, failure_scenario: dict):
"""
failure_scenario: {tool_name: {step_number: error_to_raise}}
e.g., {"search_knowledge_base": {2: ServiceUnavailableError()}}
"""
self.failures = failure_scenario
self.call_counts: dict = {}
async def call_tool(self, tool_name: str, tool_args: dict) -> ToolResult:
step = self.call_counts.get(tool_name, 0) + 1
self.call_counts[tool_name] = step
if tool_name in self.failures and step in self.failures[tool_name]:
raise self.failures[tool_name][step]
return await real_tool_implementations[tool_name](**tool_args)
Treating agent observability as optional. An agent that fails without a trace of its steps, tool calls, and tool results cannot be debugged effectively. The trace is the primary debugging artifact. Teams that add tracing after encountering production failures spend far more time debugging those failures than teams that had tracing from the beginning.
The distinction between agent reliability and agent capability
Reliability and capability are independent dimensions. An agent with sophisticated reasoning but poor reliability engineering will fail unpredictably in production. An agent with simple reasoning but robust reliability engineering will succeed at its limited scope consistently.
For most production use cases, reliability is more valuable than capability at the margin. A support agent that can reliably answer 70% of support tickets and gracefully escalate the other 30% is more valuable than one that can potentially handle 90% but fails unpredictably on some percentage of cases. Users calibrate their trust to the worst case, not the best case. An agent that fails unpredictably will not be trusted for any case, even the ones it handles well.
Reliability engineering for agents — tool interface design, retry logic, timeout handling, idempotency, step limits, and observability — is the foundation on which capability improvements are built. A capable agent on a poor reliability foundation is a demo. A reliable agent on a solid foundation is a product.
The reliability work for LLM agents is engineering work, not prompt work. Getting the prompt right is necessary. Getting the tool interfaces, retry logic, timeout handling, idempotency, and observability right is what makes the system reliable. These are the same problems that appear in any distributed system that calls external services. They have the same solutions.
Versioning tool interfaces for agent compatibility
Tool interfaces in production agent systems change as requirements evolve. A tool that returns a simple string today may need to return structured data tomorrow. Managing these changes without breaking running agent sessions requires versioning:
from typing import Callable, Dict, Any
class ToolRegistry:
"""
Registry for versioned tool implementations.
Allows running multiple versions of a tool simultaneously during migrations.
"""
def __init__(self):
self._tools: Dict[str, Dict[str, Callable]] = {}
def register(self, name: str, version: str, handler: Callable):
if name not in self._tools:
self._tools[name] = {}
self._tools[name][version] = handler
def get(self, name: str, version: str = "latest") -> Callable:
if name not in self._tools:
raise KeyError(f"Tool '{name}' not registered")
versions = self._tools[name]
if version == "latest":
version = sorted(versions.keys())[-1]
if version not in versions:
raise KeyError(f"Tool '{name}' version '{version}' not registered")
return versions[version]
# Register tool versions
registry = ToolRegistry()
def search_v1(query: str) -> ToolResult:
"""Original version — returns plain text results."""
results = search_service.search(query)
return ToolResult(
status=ToolResultStatus.SUCCESS,
data="\n".join(r.text for r in results),
...
)
def search_v2(query: str, limit: int = 10, include_metadata: bool = False) -> ToolResult:
"""New version — returns structured results with metadata."""
results = search_service.search(query, limit=limit)
return ToolResult(
status=ToolResultStatus.SUCCESS,
data=[{
"text": r.text,
"source": r.source if include_metadata else None,
"score": r.relevance_score if include_metadata else None
} for r in results],
...
)
registry.register("search_knowledge_base", "v1", search_v1)
registry.register("search_knowledge_base", "v2", search_v2)
Sessions that were created with a v1 tool definition continue using v1. New sessions use v2. The migration happens at session creation time, not at deployment time — agents mid-conversation are not disrupted by a tool version upgrade.
Planning for graceful degradation
A production agent that is 100% dependent on every tool being available will have availability equal to the product of each tool's availability. An agent that calls four tools, each with 99.9% availability, has a combined availability of approximately 99.6% — worse than any individual tool.
Graceful degradation improves this by defining fallback behaviors for each tool failure:
TOOL_FALLBACK_STRATEGIES = {
"search_knowledge_base": {
"fallback": "acknowledge_and_continue",
"user_message": "The knowledge base is temporarily unavailable. "
"Please describe your question in more detail and "
"the assistant will help from its training knowledge."
},
"database_query": {
"fallback": "escalate_to_human",
"user_message": "The database is unavailable. A support agent will "
"follow up within 2 hours."
},
"send_email": {
"fallback": "queue_for_retry",
"retry_after_seconds": 300
}
}
Defining fallback strategies per tool acknowledges that different tool failures have different business impacts and appropriate responses. A knowledge base failure can often be handled gracefully; a payment failure requires a different response entirely. The agent's prompt should include instructions for each fallback — what to tell the user, what to do next — so the model produces coherent, helpful responses even when tools fail.
The trajectory of agent reliability
Agent systems are maturing rapidly. Tool calling, once an experimental feature, is now a standard capability of production LLM APIs. The tooling around agent reliability — distributed tracing for multi-step workflows, purpose-built orchestration frameworks, managed execution environments with built-in retry and idempotency — is catching up with the capability.
The reliability gap that exists today — agents that work in demos and fail in production — is an engineering problem, not a fundamental limitation of the technology. The solutions are known: typed tool interfaces, step limits, idempotency keys, observability, and testing against realistic failure scenarios. Teams that apply these solutions consistently will find that agent reliability approaches the reliability of well-engineered synchronous services. The path from demo to product runs through reliability engineering, and the engineering is not exotic — it is the same distributed systems discipline that has governed service reliability for decades.