Structured Output from LLMs: JSON Mode vs Function Calling

As large language models (LLMs) gain ground in automating various tasks, producing structured outputs becomes vital. This article explores two primary approaches to generating structured outputs: JSON mode and function calling. Given our company's recent success in reducing customer support ticket volume by 55% with effective tool-use design, understanding these methods is essential for efficient LLM implementation in customer support scenarios.

The Customer Support Volume Problem

Customer service teams are often inundated with repetitive queries, leading to increased ticket volumes. The challenge lies in crafting responses that not only resolve these queries but also allow for efficient handling by support agents. Ultimately, reducing customer support volume is a business priority. When we deployed an LLM-powered customer support bot, the goal was clear: reduce the number of tickets while enhancing customer satisfaction. Thus, understanding how to design structured outputs helps in building context-aware bots that can interact meaningfully with customers.

JSON Mode: Standardization and Familiarity

JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and easy for machines to parse and generate. Leveraging JSON mode in LLMs offers several advantages:

1. Standardized Response Format: By producing structured outputs in JSON, we ensure consistency across responses. This is particularly beneficial in customer support, where predictability can enhance efficiency.
2. Seamless Integration: JSON is universally understood and integrates well with existing systems. Many software applications and databases can directly accept JSON, making it easier to process outputs from LLMs.
3. Readability: The key-value pair structure of JSON enhances readability, which aids both developers and support staff who interact with outputs.

For example, generating a structured response for a ticket status inquiry could look like this:

{
  "status": "success",
  "data": {
    "ticket_id": "12345",
    "status": "resolved",
    "last_updated": "2023-10-15T12:05:00Z"
  }
}

Potential Pitfalls of JSON Mode

While JSON mode has significant advantages, there are some pitfalls to consider:

• Inflexibility: Dependence on a predefined schema can limit the adaptability of responses, making it challenging to address edge cases not covered by the schema.
• Overhead: Generating JSON can introduce performance overhead, especially if not optimized for specific tasks. High ticket volumes necessitate efficient processing to maintain response times.

Function Calling: Dynamic and Contextual

Contrasting JSON mode, function calling allows LLMs to execute defined functions within a conversational context. This approach introduces dynamic behavior that can significantly enhance user interaction:

1. Real-Time Execution: Function calling enables the LLM to act based on user input dynamically, leading to context-aware responses that are highly relevant.
2. Tool-Use Capabilities: With function calling, LLMs can access and utilize various tools and APIs—essential in performing real-world actions like checking an order status or initiating a refund.
3. Improved Interaction: By allowing function execution, the model can engage in a more interactive dialogue that simplifies the user’s experience.

An example of a function call for processing a refund might look like this:

void processRefund(String ticketId) {
  // Logic to process the refund
  return {"status": "success", "message": "Refund processed successfully."};
}

Potential Pitfalls of Function Calling

Although function calling has distinct advantages, it also comes with challenges that need consideration:

• Complexity: Function management can add complexity to the system. You need a robust framework to handle functions efficiently and ensure smooth integration with the LLM.
• Error Handling: Function execution may lead to failures if incorrect parameters are passed or if dependencies are not met. Robust error handling mechanisms should be in place.

Retrieval Integration for Enhanced Responses

An effective way to augment either JSON mode or function calling is through retrieval integration. By enabling LLMs to access external databases or APIs, the scope of responses can expand significantly:

• Context Retrieval: LLMs can fetch information based on historical interactions or current context, providing more relevant and personalized responses.
• Dynamic Knowledge Base Utilization: By interfacing with knowledge bases, LLMs can answer queries not explicitly programmed into their function set.

An integration example for context retrieval might be:

retrieveTicketDetails:
  parameters:
    ticket_id: string
  response:
    status: string
    data:
      response_code: int
      ticket_summary: string

Context Window Management: Maintaining Relevance

The context window—essentially the amount of information an LLM can process at once—plays a crucial role in generating any structured output. Effective context window management ensures that LLMs can generate outputs based on recent interactions or specific data:

• State Awareness: Keeping track of the user's state allows the bot to build context-aware responses.
• Attention on Relevant Context: Adjusting the context window dynamically ensures that high-priority details are emphasized in generation, leading to relevant interactions.

Balancing the context window for the highest ticket deflection is essential for optimizing how the LLM surfaces structured outputs. If too much irrelevant information is processed, the response could veer off course; too little may lead to repetitive or simple responses.

Ticket Deflection Measurement: Evaluating Success

The ultimate goal of employing these structured output methods in LLMs is ticket deflection. Monitoring ticket deflection metrics provides insight into the effectiveness of either JSON mode or function calling in your solution:

• Analytics Integration: Coupling analytics tools with your LLM can track user interactions and define success based on reduced ticket volumes.
• Feedback Mechanisms: Implementing a feedback mechanism allows users to rate interactions, giving you qualitative and quantitative data to assess ticket deflection.

By focusing on these metrics, continuous improvements can be made in LLM design and tool-use strategies. By adapting both JSON mode and function calling according to user feedback and performance data, a business can fine-tune their customer support bot to address challenges efficiently.

Conclusion

While both structured output methods—JSON mode and function calling—have their merits, the strategic choice between them ultimately hinges on the specific needs of your customer support framework. Understanding their strengths and pitfalls can lead to a well-designed tool-use schema that remains adaptable and performs efficiently under operational demands. In our case, the integration of both methods contributed significantly to the 55% reduction in customer support ticket volumes. Designing tools to work seamlessly with LLMs is not just beneficial; it is essential for transforming the landscape of customer support.