← Articles

LLM observability: tracing, logging, and alerting for AI applications

By Nkechi Obiora · 16 July 20260 views
#llm#observability#tracing#logging#monitoring

What makes LLM observability different

Traditional backend observability instruments request latency, error rate, and throughput. These metrics still matter for LLM applications, but they capture only a fraction of what can go wrong. A request that returns HTTP 200 in 800ms may contain a hallucinated answer, a truncated response, or output that violates your system's safety policy. None of that appears in a latency histogram.

LLM observability requires three additional layers:

  • Semantic logging — capturing the prompt, system prompt, and raw model response alongside standard request metadata.
  • Quality metrics — scores derived from the model output that indicate whether the response was useful, safe, and correct.
  • Pipeline tracing — for multi-step agentic systems, understanding which step in a chain caused a bad outcome.

These layers interact: a latency spike in an agentic pipeline is meaningless unless you can trace it to the specific tool call or model invocation that caused it.

Structured logging for LLM requests

Unstructured log lines that contain prompt text are nearly impossible to query at scale. Every LLM request should emit a structured JSON event with a consistent schema:

interface LLMSpanEvent {
  // Trace correlation
  traceId:       string;
  spanId:        string;
  parentSpanId?: string;

  // Request metadata
  service:       string;
  feature:       string;
  model:         string;
  userId?:       string;
  sessionId?:    string;

  // Timing
  startedAt:     string;  // ISO-8601
  durationMs:    number;

  // Prompt data
  systemPrompt?: string;
  messages:      Array<{ role: string; content: string }>;

  // Response data
  response:      string;
  stopReason:    string;

  // Token accounting
  inputTokens:       number;
  outputTokens:      number;
  cacheReadTokens:   number;
  cacheWriteTokens:  number;
  costUsd:           number;

  // Quality signals
  outputValidated:   boolean;
  qualityScore?:     number;
  flags:             string[];  // e.g. ["format-failure", "content-filter"]

  // Error
  error?:            string;
}

Emit this event synchronously (before returning the response to the caller) so you never lose telemetry on a crash. Use a fire-and-forget emitter for the actual write to avoid adding latency:

import { LLMSpanEvent } from "./types";

const emitQueue: LLMSpanEvent[] = [];
let flushing = false;

export function emit(event: LLMSpanEvent): void {
  emitQueue.push(event);
  if (!flushing) scheduleFlush();
}

function scheduleFlush(): void {
  flushing = true;
  setImmediate(async () => {
    const batch = emitQueue.splice(0, 100);
    try {
      await sendToBigQuery(batch);
    } catch (err) {
      console.error("Failed to emit LLM telemetry:", err);
    }
    flushing = emitQueue.length > 0;
    if (flushing) scheduleFlush();
  });
}

Distributed tracing for multi-step pipelines

A RAG pipeline might involve: query classification → retrieval → re-ranking → answer generation → citation grounding. If the final answer is wrong, which step failed? Without trace context propagated through each step, you have to guess.

Use W3C trace context headers (traceparent, tracestate) to correlate spans across steps. If you are on OpenTelemetry, the SDK handles this automatically. For a custom tracer:

import { randomBytes } from "crypto";

function generateId(): string {
  return randomBytes(8).toString("hex");
}

export interface TraceContext {
  traceId: string;
  spanId:  string;
}

export function rootContext(): TraceContext {
  return { traceId: randomBytes(16).toString("hex"), spanId: generateId() };
}

export function childContext(parent: TraceContext): TraceContext {
  return { traceId: parent.traceId, spanId: generateId() };
}

export class PipelineSpan {
  private startMs: number;
  private ctx: TraceContext;
  private name: string;
  private attributes: Record<string, unknown> = {};
  private error?: string;

  constructor(name: string, parent: TraceContext) {
    this.name = name;
    this.ctx = childContext(parent);
    this.startMs = Date.now();
  }

  set(key: string, value: unknown): this {
    this.attributes[key] = value;
    return this;
  }

  fail(err: unknown): this {
    this.error = err instanceof Error ? err.message : String(err);
    return this;
  }

  end(): { traceId: string; spanId: string; durationMs: number } & Record<string, unknown> {
    const durationMs = Date.now() - this.startMs;
    const record = {
      name:       this.name,
      traceId:    this.ctx.traceId,
      spanId:     this.ctx.spanId,
      durationMs,
      error:      this.error,
      ...this.attributes,
    };
    // Emit asynchronously.
    emitSpan(record);
    return record as any;
  }

  get context(): TraceContext {
    return this.ctx;
  }
}

Using this in a pipeline:

async function runRagPipeline(
  query: string,
  userId: string
): Promise<{ answer: string; citations: string[] }> {
  const root = rootContext();

  // Step 1: classify intent
  const classifySpan = new PipelineSpan("classify-intent", root);
  const intent = await classifyIntent(query, classifySpan.context);
  classifySpan.set("intent", intent).end();

  // Step 2: retrieve documents
  const retrieveSpan = new PipelineSpan("retrieve-documents", root);
  let documents: string[];
  try {
    documents = await retrieveDocuments(query, intent, retrieveSpan.context);
    retrieveSpan.set("docCount", documents.length).end();
  } catch (err) {
    retrieveSpan.fail(err).end();
    throw err;
  }

  // Step 3: generate answer
  const generateSpan = new PipelineSpan("generate-answer", root);
  const { answer, citations } = await generateAnswer(
    query,
    documents,
    generateSpan.context
  );
  generateSpan
    .set("answerLength", answer.length)
    .set("citationCount", citations.length)
    .end();

  return { answer, citations };
}

Each span emits independently and carries the same traceId. Querying by traceId in BigQuery or your log aggregator gives you the full timeline of a single request across all steps, with per-step durations and any errors.

Capturing prompts safely

Raw prompt logging raises privacy concerns. A user's query about their medical history or financial details should not sit unredacted in a log store accessible to your entire engineering team. At minimum:

  • Apply PII detection before logging and redact sensitive entities.
  • Store prompt logs in a separate BigQuery dataset with restricted IAM access.
  • Set retention to 30 days (or whatever your data retention policy requires) using BigQuery table expiration.
  • Hash user IDs in logs using a one-way HMAC so you can correlate logs for debugging without storing the raw ID.
import { createHmac } from "crypto";

const LOG_HMAC_KEY = process.env.LOG_HMAC_KEY!;

export function pseudonymise(userId: string): string {
  return createHmac("sha256", LOG_HMAC_KEY).update(userId).digest("hex").slice(0, 16);
}

// Simple entity masking — replace with a proper NLP-based PII detector in production.
export function maskPii(text: string): string {
  return text
    .replace(/\b[\w.+-]+@[\w-]+\.[a-z]{2,}\b/gi, "[EMAIL]")
    .replace(/\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g, "[PHONE]")
    .replace(/\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g, "[CARD]");
}

Alerting: distinguishing model problems from infrastructure problems

The hardest part of LLM observability is setting alerts that fire on real problems without flooding on-call with noise. The key is separating three failure categories:

Infrastructure failures — timeouts, 5xx errors from the LLM provider, rate limit errors. These look like any other backend failure and standard APM alerts handle them well.

Format failures — the model returned output that could not be parsed as the expected format (JSON, a specific schema, etc.). These are LLM-specific and require custom instrumentation. Track outputValidated = false and alert when the rate exceeds 2% over a 15-minute window.

Quality regressions — the model returns syntactically valid output that is factually wrong, off-topic, or low quality. These are the hardest to detect automatically. Track quality scores from your LLM-as-judge pipeline and alert when the rolling average drops below threshold.

interface AlertRule {
  name: string;
  metric: string;
  windowMinutes: number;
  threshold: number;
  comparison: "gt" | "lt";
  severity: "warning" | "critical";
}

const ALERT_RULES: AlertRule[] = [
  {
    name:           "high-format-failure-rate",
    metric:         "format_failure_rate",
    windowMinutes:  15,
    threshold:      0.02,
    comparison:     "gt",
    severity:       "warning",
  },
  {
    name:           "critical-error-rate",
    metric:         "error_rate",
    windowMinutes:  5,
    threshold:      0.05,
    comparison:     "gt",
    severity:       "critical",
  },
  {
    name:           "quality-score-drop",
    metric:         "avg_quality_score",
    windowMinutes:  60,
    threshold:      0.75,
    comparison:     "lt",
    severity:       "warning",
  },
  {
    name:           "p99-latency-spike",
    metric:         "p99_duration_ms",
    windowMinutes:  10,
    threshold:      8000,
    comparison:     "gt",
    severity:       "warning",
  },
];

Run alert evaluation on a schedule (every 5 minutes) using a Cloud Scheduler job that queries BigQuery:

-- Format failure rate over last 15 minutes
SELECT
  feature,
  COUNTIF(NOT output_validated) / COUNT(*) AS format_failure_rate,
  COUNT(*) AS sample_count
FROM `project.llm_telemetry.spans`
WHERE started_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 15 MINUTE)
GROUP BY feature
HAVING sample_count >= 10  -- ignore features with sparse traffic
ORDER BY format_failure_rate DESC;

Alert suppression is important: if the provider is having an incident, every feature will fire simultaneously. Add a check that fires only if the problem is isolated to specific features — a provider-wide outage produces uniform error rates across all features, while a prompt regression produces elevated error rates on one feature.

OpenTelemetry integration

If your organisation uses OpenTelemetry, emit LLM spans as OTEL spans with LLM semantic conventions. The GenAI semantic conventions (in preview as of 2026) define standard attribute names:

import { trace, context, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("llm-pipeline", "1.0.0");

export async function tracedLlmCall(
  feature: string,
  model: string,
  messages: Array<{ role: string; content: string }>,
  system?: string
) {
  return tracer.startActiveSpan(`${feature} llm call`, async (span) => {
    span.setAttributes({
      "gen_ai.system":             "anthropic",
      "gen_ai.request.model":      model,
      "gen_ai.operation.name":     "chat",
      "llm.feature":               feature,
      "gen_ai.request.max_tokens": 1024,
    });

    try {
      const response = await client.messages.create({
        model,
        max_tokens: 1024,
        system,
        messages,
      });

      span.setAttributes({
        "gen_ai.usage.input_tokens":  response.usage.input_tokens,
        "gen_ai.usage.output_tokens": response.usage.output_tokens,
        "gen_ai.response.finish_reasons": [response.stop_reason ?? ""],
      });

      span.setStatus({ code: SpanStatusCode.OK });
      return response;
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      span.recordException(err as Error);
      throw err;
    } finally {
      span.end();
    }
  });
}

OTEL spans integrate with Datadog, Grafana Tempo, Google Cloud Trace, and any other backend that ingests OTLP. You get waterfall views of multi-step pipelines without building a custom trace viewer.

Building a debug console for prompt investigations

When a user reports a bad LLM response, your on-call engineer needs to be able to pull up the full trace — system prompt, messages, raw response, quality scores, and timing — within seconds. Build a simple admin query endpoint backed by BigQuery:

// GET /admin/llm-trace/:traceId
export async function getLlmTrace(traceId: string): Promise<LLMSpanEvent[]> {
  const [rows] = await bq.query({
    query: `
      SELECT *
      FROM \`project.llm_telemetry.spans\`
      WHERE trace_id = @traceId
      ORDER BY started_at
    `,
    params: { traceId },
  });

  return rows as LLMSpanEvent[];
}

Augment the UI with a diff view that compares the system prompt at the time of the incident against the current production prompt. A prompt that silently changed between the incident and the investigation is a common source of confusion.

Dashboards that matter

Three BigQuery views that become the foundation for your LLM observability dashboard:

Quality trend by feature:

CREATE VIEW `project.llm_telemetry.quality_by_feature` AS
SELECT
  DATE(started_at) AS day,
  feature,
  model,
  COUNT(*)                        AS requests,
  AVG(CAST(output_validated AS INT64)) AS format_success_rate,
  AVG(quality_score)              AS avg_quality_score,
  AVG(duration_ms)                AS avg_latency_ms,
  APPROX_QUANTILES(duration_ms, 100)[OFFSET(99)] AS p99_latency_ms,
  SUM(cost_usd)                   AS total_cost_usd
FROM `project.llm_telemetry.spans`
WHERE started_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY 1, 2, 3;

Error breakdown:

CREATE VIEW `project.llm_telemetry.errors_by_type` AS
SELECT
  DATE(started_at) AS day,
  feature,
  CASE
    WHEN error IS NOT NULL                     THEN "api_error"
    WHEN NOT output_validated                  THEN "format_failure"
    WHEN quality_score IS NOT NULL
     AND quality_score < 0.6                   THEN "low_quality"
    ELSE "success"
  END AS outcome,
  COUNT(*) AS count
FROM `project.llm_telemetry.spans`
GROUP BY 1, 2, 3;

User-level cost attribution:

CREATE VIEW `project.llm_telemetry.cost_by_user` AS
SELECT
  DATE(started_at) AS day,
  user_id,
  feature,
  COUNT(*) AS requests,
  SUM(cost_usd) AS total_cost_usd
FROM `project.llm_telemetry.spans`
WHERE user_id IS NOT NULL
GROUP BY 1, 2, 3;

These three views answer the questions your product and platform teams ask most often: Is quality getting worse? What types of failure are increasing? Which users and features are driving cost?

Operationalising the full stack

The observability stack described here has five components:

  1. Structured span emitter — wraps every LLM call and emits a consistent event.
  2. BigQuery sink — stores events partitioned by day for low-cost historical queries.
  3. Alert evaluator — runs on a 5-minute schedule and fires Slack/PagerDuty notifications.
  4. Trace UI — lets engineers pull up the full context for any request by trace ID.
  5. Dashboard — three BigQuery views exposed through Looker, Grafana, or your BI tool of choice.

Each component can be deployed incrementally. Start with the structured emitter and BigQuery sink — that alone unlocks retrospective analysis. Add the alert evaluator once you have a week of data and know which thresholds make sense for your traffic patterns. The trace UI and dashboard come last and turn the data into a tool your whole team can use without writing SQL.

LLM applications are not web APIs that either work or do not. They exist on a quality spectrum, and that spectrum requires instrumentation beyond what traditional observability tools provide. Build the logging layer from day one and you will have the data you need when something goes wrong — not after spending two days reconstructing what happened.

Comments

No comments yet. Be the first!

Sign in to leave a comment.