LLM cost tracking and budget controls in production

Why LLM costs compound faster than you expect

A single GPT-4o call that processes a 10-page document might cost $0.05. That sounds negligible. Run that call 50,000 times a month — which is modest for a SaaS product with a few thousand active users — and you are looking at $2,500 before you have counted retries, streaming overhead, or the supporting infrastructure. Add a second model for re-ranking or classification and the number doubles again.

The problem is structural: LLM costs scale with tokens, not requests. A 1,000-token context and a 50,000-token context issued to the same endpoint look identical to your application code but carry a 50x cost difference. Standard cloud cost dashboards show you API call counts and aggregate spend, but they cannot tell you which user, which feature, or which prompt variant is burning the budget.

This article gives you the instrumentation layer you need to answer those questions precisely.

Instrumenting every LLM call at the SDK level

The most reliable way to capture cost data is to wrap your LLM client once and emit structured telemetry at every call site automatically. No individual developer needs to remember to log anything.

The following example wraps the Anthropic SDK in TypeScript. The same pattern applies to OpenAI, Gemini, or any provider with a typed response that exposes token usage.

import Anthropic from "@anthropic-ai/sdk";
import type { MessageParam } from "@anthropic-ai/sdk/resources/messages";

// Pricing table — update whenever Anthropic publishes new rates.
const ANTHROPIC_PRICES: Record<string, { inputPer1M: number; outputPer1M: number }> = {
  "claude-opus-4-5":          { inputPer1M: 15.00, outputPer1M: 75.00 },
  "claude-sonnet-4-5":        { inputPer1M: 3.00,  outputPer1M: 15.00 },
  "claude-haiku-3-5":         { inputPer1M: 0.80,  outputPer1M: 4.00 },
};

export interface LLMCallRecord {
  model: string;
  feature: string;
  userId?: string;
  inputTokens: number;
  outputTokens: number;
  cacheReadTokens: number;
  cacheWriteTokens: number;
  costUsd: number;
  durationMs: number;
  timestamp: string;
}

export type CostEmitter = (record: LLMCallRecord) => void;

function calcCost(
  model: string,
  inputTokens: number,
  outputTokens: number,
  cacheReadTokens: number,
  cacheWriteTokens: number
): number {
  const price = ANTHROPIC_PRICES[model];
  if (!price) return 0;

  const inputCost  = (inputTokens  / 1_000_000) * price.inputPer1M;
  const outputCost = (outputTokens / 1_000_000) * price.outputPer1M;
  // Cache reads are typically 10% of input price; writes are 25% extra.
  const cacheReadCost  = (cacheReadTokens  / 1_000_000) * price.inputPer1M * 0.10;
  const cacheWriteCost = (cacheWriteTokens / 1_000_000) * price.inputPer1M * 0.25;

  return inputCost + outputCost + cacheReadCost + cacheWriteCost;
}

export function buildTrackedClient(
  client: Anthropic,
  emit: CostEmitter,
  defaults: { feature: string; userId?: string }
) {
  return {
    async messages(
      params: Anthropic.MessageCreateParamsNonStreaming,
      context?: { feature?: string; userId?: string }
    ) {
      const start = Date.now();
      const response = await client.messages.create(params);
      const durationMs = Date.now() - start;

      const usage = response.usage as {
        input_tokens: number;
        output_tokens: number;
        cache_read_input_tokens?: number;
        cache_creation_input_tokens?: number;
      };

      const inputTokens       = usage.input_tokens;
      const outputTokens      = usage.output_tokens;
      const cacheReadTokens   = usage.cache_read_input_tokens   ?? 0;
      const cacheWriteTokens  = usage.cache_creation_input_tokens ?? 0;

      const record: LLMCallRecord = {
        model:            params.model,
        feature:          context?.feature ?? defaults.feature,
        userId:           context?.userId  ?? defaults.userId,
        inputTokens,
        outputTokens,
        cacheReadTokens,
        cacheWriteTokens,
        costUsd:          calcCost(params.model, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens),
        durationMs,
        timestamp:        new Date().toISOString(),
      };

      emit(record);
      return response;
    },
  };
}

You instantiate this once at application startup:

import Anthropic from "@anthropic-ai/sdk";
import { buildTrackedClient } from "./llm-client";
import { publishToDatadog } from "./telemetry";

const raw = new Anthropic();

export const llm = buildTrackedClient(raw, publishToDatadog, {
  feature: "unknown",
});

Then at every call site you pass a feature tag:

const result = await llm.messages(
  { model: "claude-sonnet-4-5", max_tokens: 1024, messages },
  { feature: "document-summary", userId: session.userId }
);

The emitter receives a fully-typed record with exact costs. You can send it to Datadog custom metrics, BigQuery, Cloudflare Analytics Engine, or any sink you already use.

Persisting cost records in BigQuery for per-feature analysis

Flat-file logs work for debugging but are hard to aggregate over time. BigQuery is cheap for append-only analytical workloads and lets you answer questions like "what did the document-summary feature cost last week, broken down by model and user cohort?"

Create the table once:

CREATE TABLE IF NOT EXISTS `my-project.llm_costs.calls` (
  model             STRING NOT NULL,
  feature           STRING NOT NULL,
  user_id           STRING,
  input_tokens      INT64  NOT NULL,
  output_tokens     INT64  NOT NULL,
  cache_read_tokens INT64  NOT NULL,
  cache_write_tokens INT64 NOT NULL,
  cost_usd          FLOAT64 NOT NULL,
  duration_ms       INT64  NOT NULL,
  ts                TIMESTAMP NOT NULL
)
PARTITION BY DATE(ts)
CLUSTER BY feature, model;

Then emit records from your Node.js backend using the BigQuery client:

import { BigQuery } from "@google-cloud/bigquery";

const bq = new BigQuery();
const table = bq.dataset("llm_costs").table("calls");

export async function publishToBigQuery(record: LLMCallRecord): Promise<void> {
  await table.insert([{
    model:              record.model,
    feature:            record.feature,
    user_id:            record.userId ?? null,
    input_tokens:       record.inputTokens,
    output_tokens:      record.outputTokens,
    cache_read_tokens:  record.cacheReadTokens,
    cache_write_tokens: record.cacheWriteTokens,
    cost_usd:           record.costUsd,
    duration_ms:        record.durationMs,
    ts:                 record.timestamp,
  }]);
}

The PARTITION BY DATE(ts) clause keeps query costs low: a dashboard querying the last 30 days scans only 30 partitions. The CLUSTER BY feature, model makes filtering on those two dimensions fast.

A useful daily aggregation query you can materialise into a view:

SELECT
  DATE(ts)   AS day,
  feature,
  model,
  COUNT(*)             AS calls,
  SUM(input_tokens)    AS total_input_tokens,
  SUM(output_tokens)   AS total_output_tokens,
  SUM(cost_usd)        AS total_cost_usd,
  AVG(duration_ms)     AS avg_latency_ms
FROM `my-project.llm_costs.calls`
WHERE ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY 1, 2, 3
ORDER BY total_cost_usd DESC;

Enforcing per-user and per-team budgets

Observability alone is reactive. Budget enforcement is proactive. The architecture that works at scale is a token-budget counter stored in Redis (or Firestore if you are on GCP without Redis) that gets checked before every LLM call and decremented after.

import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const BUDGET_PREFIX = "llm:budget:";

// Budget in USD cents to avoid float arithmetic.
export async function checkAndReserveBudget(
  userId: string,
  estimatedCostUsd: number,
  limitUsd: number
): Promise<{ allowed: boolean; remainingUsd: number }> {
  const key = `${BUDGET_PREFIX}${userId}:${currentMonthKey()}`;
  const limitCents = Math.round(limitUsd * 100);
  const estimatedCents = Math.round(estimatedCostUsd * 100);

  // WATCH + MULTI/EXEC gives optimistic locking.
  await redis.watch(key);
  const spentCents = parseInt((await redis.get(key)) ?? "0", 10);

  if (spentCents + estimatedCents > limitCents) {
    await redis.unwatch();
    return { allowed: false, remainingUsd: (limitCents - spentCents) / 100 };
  }

  const multi = redis.multi();
  multi.incrBy(key, estimatedCents);
  // Expire the key a week after the month ends so cleanup is automatic.
  multi.expireAt(key, endOfNextMonthTimestamp());
  await multi.exec();

  return { allowed: true, remainingUsd: (limitCents - spentCents - estimatedCents) / 100 };
}

function currentMonthKey(): string {
  const d = new Date();
  return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, "0")}`;
}

function endOfNextMonthTimestamp(): number {
  const d = new Date();
  d.setUTCMonth(d.getUTCMonth() + 2, 1);
  d.setUTCHours(0, 0, 0, 0);
  return Math.floor(d.getTime() / 1000);
}

Integrate this into the tracked client:

async messages(params, context) {
  const estimatedCost = estimateCallCost(params); // rough estimate before the call

  const budget = await checkAndReserveBudget(
    context?.userId ?? "anonymous",
    estimatedCost,
    USER_MONTHLY_LIMIT_USD
  );

  if (!budget.allowed) {
    throw new BudgetExceededError(
      `Monthly LLM budget exhausted. Remaining: $${budget.remainingUsd.toFixed(4)}`
    );
  }

  const response = await client.messages.create(params);
  // Reconcile: adjust the reservation with the actual cost.
  const actual = calcCost(params.model, ...extractUsage(response));
  await reconcileBudget(context?.userId, estimatedCost, actual);

  return response;
}

The two-phase approach (reserve an estimate, reconcile after) prevents race conditions where multiple concurrent requests each see the budget as available but together would exceed it. The reconciliation step corrects the over- or under-estimate without blocking the response.

Alerting when costs spike unexpectedly

Budget limits protect individual users, but they will not warn you when a new feature that passed review turns out to be 10x more expensive than projected. You need anomaly detection at the team level.

A simple but effective approach is a Cloud Scheduler job that queries BigQuery every hour and compares rolling 24-hour spend against a configurable threshold:

// Cloud Function: check-llm-cost-anomaly
import { BigQuery } from "@google-cloud/bigquery";
import { WebClient } from "@slack/web-api";

const bq = new BigQuery();
const slack = new WebClient(process.env.SLACK_BOT_TOKEN);

export async function checkCostAnomaly(): Promise<void> {
  const [rows] = await bq.query(`
    SELECT
      feature,
      SUM(cost_usd) AS cost_last_24h,
      AVG(SUM(cost_usd)) OVER (
        PARTITION BY feature
        ORDER BY UNIX_SECONDS(TIMESTAMP_TRUNC(ts, HOUR))
        RANGE BETWEEN 168 PRECEDING AND 25 PRECEDING  -- prior 7 days same hour
      ) AS baseline_cost
    FROM \`my-project.llm_costs.calls\`
    WHERE ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 25 HOUR)
    GROUP BY feature, TIMESTAMP_TRUNC(ts, HOUR)
  `);

  for (const row of rows) {
    const ratio = row.cost_last_24h / (row.baseline_cost ?? 0.01);
    if (ratio > 3.0) {
      await slack.chat.postMessage({
        channel: "#platform-alerts",
        text: [
          `*LLM cost spike detected* on \`${row.feature}\``,
          `Last 24h: $${row.cost_last_24h.toFixed(2)}`,
          `Baseline: $${(row.baseline_cost ?? 0).toFixed(2)}`,
          `Ratio: ${ratio.toFixed(1)}x`,
        ].join("\n"),
      });
    }
  }
}

A ratio of 3x is a reasonable starting point. Tune it based on your traffic patterns; features with naturally high variance need a looser threshold, while stable batch jobs can be set to 1.5x.

Modelling cost projections for capacity planning

Once you have 30 days of data you can project future spend with reasonable accuracy. A simple linear model that accounts for week-over-week growth is usually sufficient.

import pandas as pd
from google.cloud import bigquery
from sklearn.linear_model import LinearRegression
import numpy as np

client = bigquery.Client()

df = client.query("""
  SELECT
    DATE(ts) AS day,
    feature,
    SUM(cost_usd) AS daily_cost
  FROM `my-project.llm_costs.calls`
  WHERE ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  GROUP BY 1, 2
""").to_dataframe()

projections = []

for feature, group in df.groupby("feature"):
    group = group.sort_values("day")
    group["day_index"] = range(len(group))

    X = group[["day_index"]].values
    y = group["daily_cost"].values

    model = LinearRegression().fit(X, y)
    future_indices = np.array([[len(group) + i] for i in range(30)])
    projected = model.predict(future_indices)

    projections.append({
        "feature": feature,
        "projected_monthly_usd": projected.sum(),
        "daily_growth_rate": model.coef_[0],
    })

print(pd.DataFrame(projections).sort_values("projected_monthly_usd", ascending=False))

Run this monthly, or trigger it automatically as part of your budget review. Features with a steep positive daily_growth_rate are candidates for either optimisation (shorter prompts, faster models, caching) or explicit budget provisioning.

Reducing costs without sacrificing quality

Tracking spend tells you where money goes. The next step is systematic reduction. The most impactful levers, roughly in order of effort:

Model routing by task complexity. Not every task needs your most capable model. A simple classification prompt that routes support tickets to teams does not need Opus; Haiku is 18x cheaper per token and equally accurate on narrow classification tasks. Add a modelTier field to your feature registry and route accordingly.

const MODEL_TIER: Record<string, string> = {
  "ticket-classification":  "claude-haiku-3-5",
  "contract-analysis":      "claude-opus-4-5",
  "document-summary":       "claude-sonnet-4-5",
};

function modelForFeature(feature: string): string {
  return MODEL_TIER[feature] ?? "claude-sonnet-4-5";
}

Prompt caching. Anthropic's cache API lets you mark stable portions of the system prompt as cacheable. Subsequent calls that reuse the cached prefix pay only 10% of the normal input price for those tokens. For long system prompts that stay constant across requests — a 5,000-token constitution document, for instance — this cut is immediate and requires zero quality trade-off.

const messages: Anthropic.MessageParam[] = [
  {
    role: "user",
    content: [
      {
        type: "text",
        text: LONG_STATIC_CONTEXT,
        // Mark the static portion as cacheable.
        cache_control: { type: "ephemeral" },
      } as any,
      {
        type: "text",
        text: userQuery,
      },
    ],
  },
];

Response length controls. Setting max_tokens conservatively on tasks that produce short outputs is free. Monitor the 99th percentile of output_tokens per feature over a week, then set max_tokens to p99 + 20%. You will reject almost no valid responses while capping runaway generations.

Semantic caching. For use cases where many users ask semantically similar questions, a vector cache over recent responses can serve cached answers without hitting the API. Libraries like GPTCache handle this, or you can build it with pgvector and cosine similarity thresholds. A threshold of 0.95 cosine similarity typically yields answers indistinguishable from a fresh generation.

Summary

Cost visibility for LLMs requires three things working together: per-call instrumentation that records exact token counts and feature attribution, a persistent store that supports aggregation queries over time, and active enforcement via pre-call budget checks. Anomaly alerting bridges the gap between the instrumentation layer and the finance team. With this stack in place you can answer "which feature is most expensive, and why did it spike on Tuesday?" in seconds rather than days.

The patterns above are provider-agnostic: swap the pricing table and SDK wrapper for any provider and the rest of the pipeline remains identical. Start with the instrumentation wrapper and BigQuery sink — those alone will make your next cost review meeting significantly more productive.