Building a Semantic Changelog Diff Tool with Embeddings

The 55% Deflection Catalyst: Moving Beyond Keyword Search

When we launched our first iteration of the AI-powered support agent at our Tema-based startup, we were obsessed with one metric: ticket volume. Specifically, how much of the repetitive "where is my feature?" or "why did the dashboard change?" noise could we strip out of our human agents' queues. Achieving a 55% ticket deflection rate in the first month wasn’t about throwing a large language model at a pile of PDFs. It was about engineering precise tool-use interfaces that bridge the gap between our internal engineering changelogs and the customer’s natural language intent.

Most companies treat their changelogs as static documents. When they update a feature, they push a markdown file to GitHub, and customers are left to search through dense, technical jargon. This is a primary driver of support ticket inflation. If a customer can't parse your release notes, they open a ticket. If they open a ticket, your support load increases, and your product team loses their feedback loop. By building a semantic changelog diff tool, we empowered our LLM to act as a translator between raw engineering commits and customer-facing context. This approach is not just about automation; it’s about strategic augmentation—giving the LLM the tools to solve problems, not just echo documentation.

Designing the Tool-Use Schema for Changelog Retrieval

The core of our system relies on function calling that allows the LLM to query our internal database of changes. We don’t just dump the entire changelog into the context window. That’s a recipe for hallucinations and context-window bloat. Instead, we define a granular schema that allows the model to perform a targeted search based on the user's inquiry.

We utilize a specific tool-use schema that looks for version ranges, feature categories, and affected modules. By forcing the model to select the correct tool, we reduce the ambiguity in how it retrieves information. Below is an example of the YAML configuration we use to define the available tools for the model to access our change data:

tools:
  - name: get_changelog_diff
    description: Retrieves specific feature changes between two versions or dates.
    parameters:
      type: object
      properties:
        start_version:
          type: string
          description: The semantic version to start from (e.g., v1.2.0).
        end_version:
          type: string
          description: The semantic version to end at (e.g., v1.5.0).
        category:
          type: string
          enum: ["ui", "api", "billing", "performance"]
          description: The domain of the change requested.
    required: ["start_version", "end_version"]

When a customer asks, "Why does my billing dashboard look different?", the LLM doesn't hallucinate. It triggers the get_changelog_diff function with the category set to 'billing', pulls the relevant entries from the last two weeks, and synthesizes a response. This targeted retrieval is the backbone of our 55% deflection rate.

Vector Embeddings: The Semantic Glue

Standard keyword search fails when a customer describes a feature by its function rather than its technical name. A user might say, "The button to export my data moved," while the engineering log says, "Refactored CSV download utility location." To bridge this semantic gap, we transform our changelog entries into vector embeddings using a pre-trained model (like OpenAI’s text-embedding-3-small or an open-source alternative like bge-large).

By storing these embeddings in a vector database like Pinecone or Weaviate, we can perform semantic searches. When an incoming ticket comes in, we convert the intent into a vector. The system then finds the most relevant changelog entries, even if the phrasing doesn't match perfectly. This ensures that the context provided to the LLM is always relevant to the user’s specific issue.

If we have 500 entries in the changelog, the model only gets the top 3-5 most similar items. This keeps the context window lean and minimizes latency. The engineering team manages the raw data, but the embedding pipeline manages the understanding. This decoupling is crucial for scale. As your product grows, your changelog will get longer, but your response quality remains high because the semantic retrieval layer does the heavy lifting before the LLM even sees the data.

Managing Context Window and Response Accuracy

Context window management is the most underestimated aspect of building reliable AI support. If you dump 20,000 tokens of release notes into a prompt, your model will eventually lose focus. Our approach is to treat context as a finite, precious resource. We use a RAG (Retrieval-Augmented Generation) pipeline that dynamically generates the context based on the tool-use output.

Below is a snippet of the logic in Kotlin, illustrating how we handle the context injection for our internal microservice that coordinates the support requests:

fun buildSystemPrompt(userQuery: String, retrievedChanges: List<ChangelogEntry>): String {
    val changeSummary = retrievedChanges.joinToString("\n") { entry ->
        "[${entry.version}] ${entry.title}: ${entry.description}"
    }
    
    return """
    You are a technical support assistant. Use the following changelog data to answer 
    the user's question. If the information is not in the context, do not guess. 
    
    Context:
    $changeSummary
    
    Question: $userQuery
    """.trimIndent()
}

This simple, rigorous template ensures that the LLM has exactly what it needs to solve the customer's problem. We’ve found that by limiting the model’s scope to only the retrieved ChangelogEntry objects, we effectively eliminate the "hallucination problem" for feature inquiries. If the data isn't there, the model is instructed to hand the ticket over to a human, which is an acceptable fall-back that maintains high customer satisfaction while still supporting our 55% deflection goal.

Measuring Success: Why Ticket Deflection is the North Star

In our startup, everything circles back to the 55% figure. Why? Because ticket volume is the ultimate proxy for product friction. If your product is intuitive and well-documented, the AI agent can easily resolve inquiries using the changelog diff tools we've built. If the volume stays high, the AI agent highlights specific areas of the product where users are constantly confused.

We measure the success of the semantic changelog tool through two key mechanisms:

1. Resolution Accuracy: We track whether the LLM's suggested response contained a link to the specific release note entry that matched the user's intent. If the model uses a tool to retrieve a changelog, we log that link as a "deflection event."

2. Feedback Loops: Every response from the bot includes a simple binary feedback button (helpful/not helpful). We correlate negative feedback with the specific tool-use retrieval. If a user says "not helpful," but the retrieval step was correct, we know the model needs better instructions on how to synthesize the answer. If the retrieval step was wrong, we know we need to tune our vector embeddings.

This is the power of a tool-use-first product engineering mindset. We aren't just building a chatbot; we are building an intelligent system that learns the anatomy of a support ticket. By mapping the changelog to the user's language, we have effectively removed the need for a human to mediate the vast majority of "update-related" support inquiries. The 55% isn't just a number; it represents the hours our engineers and support staff save every week to focus on building new features rather than explaining old ones.

Building this requires patience in the pipeline design. It requires obsessing over the schemas you pass to the model and the way you vectorize your internal documents. But the ROI is immediate. By treating your changelog as a dynamic, queryable database rather than a static document, you create a bridge between the engineering department and the customer. You turn raw commits into clear, actionable, and customer-friendly answers. That is the true impact of LLM integration in a modern support stack: moving from being a ticket-processing machine to being a bridge for customer understanding.