Observability before alerting: the monitoring approach that surfaces real problems
An on-call engineer is paged at 2am: error rate exceeds 1%. She opens the monitoring dashboard. The alert is there. The error rate is 2.3%. There is one metric: error rate. She opens the application logs and searches for errors. She finds them — 500 errors on the checkout endpoint. She looks at the checkout endpoint's logs. They show database connection errors. She looks at the database dashboard. It is green. She is now in the weeds of an incident without the information to understand it quickly.
The monitoring system told her something was wrong. It did not help her understand what was wrong or why. The distinction between alerting and observability determines how long incidents last.
An hour and forty minutes later, the incident is resolved. The problem was not the database. A third-party payment processor was rejecting requests due to an expired API key, which caused the application to retry repeatedly, filling the database connection pool. None of that causality was visible in the monitoring. She pieced it together by reading raw log lines, one at a time, in a browser interface that times out on large result sets.
This is a monitoring system that alerts without observing. It is unfortunately common.
The three pillars
Observability is the property of a system that allows you to understand its internal state from its external outputs. The practical implementation rests on three data types:
Metrics — numerical measurements aggregated over time (error rate, p99 latency, request count, memory usage). Metrics answer "how much" and "how often." They are efficient to store and query but lose the detail needed to explain root causes. A metric spike tells you there is a problem. It does not tell you which users are affected, which code path is failing, or what the error message is.
Logs — timestamped, structured records of discrete events. Logs answer "what happened and when." They are expensive to store at high volume but contain the detail needed to understand specific incidents. Unstructured free-text logs are searchable but not queryable — you can grep for a string but you cannot aggregate by field, join across services, or filter by a combination of conditions efficiently.
Traces — records of a request's path through multiple services, with timing for each step. Traces answer "where did the time go and which service failed." They are essential for distributed systems and difficult to add after the fact because they require instrumentation at every service boundary. A trace ties together a user action (add to cart), a frontend request, a backend function call, a database query, and an external API call into a single timeline with timing for every step.
Teams that add alerts before building these foundations are alerting on symptoms without having the tools to investigate causes. The result is the 2am scenario: you know something is wrong, but you cannot understand it quickly.
Building structured logs before adding alerts
The on-call engineer's problem was unstructured logs. Searching for errors in free-text logs is slow. Structured logs — JSON records with consistent fields — can be queried programmatically:
// UNSTRUCTURED: Hard to query, hard to aggregate
console.log(`Error processing checkout for user ${userId}: ${error.message}`);
// STRUCTURED: Queryable, aggregatable, correlatable
import { logger } from 'firebase-functions';
function logError(context: {
operation: string;
userId?: string;
orderId?: string;
error: Error;
durationMs?: number;
}) {
logger.error({
operation: context.operation,
userId: context.userId,
orderId: context.orderId,
errorMessage: context.error.message,
errorCode: (context.error as any).code,
errorStack: context.error.stack,
durationMs: context.durationMs,
timestamp: new Date().toISOString(),
severity: 'ERROR',
});
}
// Usage in a Cloud Function
try {
await processCheckout(order);
} catch (error) {
logError({
operation: 'processCheckout',
userId: order.userId,
orderId: order.id,
error: error as Error,
durationMs: Date.now() - startTime,
});
throw error;
}
With structured logs in Cloud Logging (Firebase/GCP), the on-call engineer can query:
# Cloud Logging query — find all checkout errors in the last hour
gcloud logging read \
'resource.type="cloud_function"
AND jsonPayload.operation="processCheckout"
AND severity="ERROR"
AND timestamp >= "2026-07-15T01:00:00Z"' \
--format=json | jq '.[] | {time: .timestamp, error: .jsonPayload.errorMessage, user: .jsonPayload.userId}'
This returns specific error messages, affected users, and timing — not a count of errors but the errors themselves.
Beyond error logging, every meaningful operation should emit a structured log entry at completion, success or failure:
function logOperation(context: {
operation: string;
userId?: string;
durationMs: number;
success: boolean;
metadata?: Record<string, unknown>;
}) {
const entry = {
operation: context.operation,
userId: context.userId,
durationMs: context.durationMs,
success: context.success,
timestamp: new Date().toISOString(),
severity: context.success ? 'INFO' : 'WARNING',
...context.metadata,
};
if (context.success) {
logger.info(entry);
} else {
logger.warn(entry);
}
}
With this pattern, a query for jsonPayload.operation="processCheckout" AND jsonPayload.durationMs > 5000 surfaces all slow checkout operations — potential leading indicators before errors begin — without waiting for an alert to fire.
Adding distributed tracing to Cloud Functions
Firebase/GCP functions can emit trace data to Cloud Trace with the OpenTelemetry SDK:
import { NodeTracerProvider } from '@opentelemetry/sdk-node';
import { TraceExporter } from '@google-cloud/opentelemetry-cloud-trace-exporter';
import { trace, context, SpanStatusCode } from '@opentelemetry/api';
const provider = new NodeTracerProvider({
exporter: new TraceExporter(),
});
provider.register();
const tracer = trace.getTracer('checkout-service');
async function processCheckout(order: Order): Promise<void> {
const span = tracer.startSpan('processCheckout', {
attributes: {
'order.id': order.id,
'order.userId': order.userId,
'order.itemCount': order.items.length,
}
});
const ctx = trace.setSpan(context.active(), span);
try {
await context.with(ctx, async () => {
// Each operation within gets its own child span
await validateInventory(order.items); // Creates child span if instrumented
await chargePayment(order); // Creates child span if instrumented
await updateOrderStatus(order.id, 'confirmed');
});
span.setStatus({ code: SpanStatusCode.OK });
} catch (error) {
span.recordException(error as Error);
span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
throw error;
} finally {
span.end();
}
}
With traces, the 2am incident investigation becomes: open the trace for a failing checkout request → see that validateInventory succeeded (200ms), chargePayment failed after 4500ms (timeout). The database is not the problem — the payment processor is. The investigation that took 45 minutes takes 3.
Child spans should be created for any operation that crosses a service boundary or that could independently be slow:
async function chargePayment(order: Order): Promise<void> {
const parentSpan = trace.getActiveSpan();
const span = tracer.startSpan('chargePayment', {
attributes: {
'payment.provider': 'stripe',
'payment.amount': order.totalAmount,
'payment.currency': order.currency,
}
});
const ctx = trace.setSpan(context.active(), span);
try {
const result = await context.with(ctx, () =>
stripe.paymentIntents.create({
amount: order.totalAmount,
currency: order.currency,
payment_method: order.paymentMethodId,
confirm: true,
})
);
span.setAttribute('payment.intentId', result.id);
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (error) {
span.recordException(error as Error);
span.setAttribute('payment.errorCode', (error as any).code ?? 'unknown');
span.setStatus({ code: SpanStatusCode.ERROR });
throw error;
} finally {
span.end();
}
}
The trace now shows exactly which external call failed, how long it took, and what error code the payment processor returned — without reading a single log line.
Connecting metrics to logs and traces
Metrics alone lead to alert fatigue and slow investigations. The value comes from being able to jump from a metric anomaly to the underlying log entries and traces that explain it. This requires a correlation strategy.
The standard approach is a trace ID that appears in both log entries and traces:
import { trace } from '@opentelemetry/api';
function logWithTrace(level: 'info' | 'warn' | 'error', data: Record<string, unknown>) {
const span = trace.getActiveSpan();
const traceId = span?.spanContext().traceId;
const spanId = span?.spanContext().spanId;
const entry = {
...data,
traceId,
spanId,
timestamp: new Date().toISOString(),
};
logger[level](entry);
}
With traceId in the log entry, a Cloud Logging query can filter to all log entries from a single request: jsonPayload.traceId="abc123". The investigation path becomes: metric spike → find an affected trace → read the trace to see which span failed → query logs for that traceId to see the detailed error message. The entire investigation is correlated.
The alerting layer: alert on symptoms, not causes
With observability in place, alerting should fire on symptoms (what users experience) not causes (what the system is doing). A high error rate is a symptom; high database CPU is a cause. Alert on the symptom first — that is what matters to users.
# Example: Google Cloud Monitoring alert on symptom
resource.type="cloud_function"
AND metric.type="cloudfunctions.googleapis.com/function/execution_count"
# Alert when:
# - Error rate (error executions / total executions) > 1% for 5 minutes
# - p99 latency > 3000ms for 5 minutes
# - Function execution count drops to 0 for 5 minutes (dead function)
Cause-based alerts (database CPU > 80%, memory > 85%) are useful as diagnostic aids — they can help an engineer investigate after a symptom alert fires. They should not page anyone at 3am. A database at 90% CPU is not an incident if the error rate is zero. It becomes an incident if the error rate climbs. Page on the error rate; display the CPU metric as context.
Alert fatigue: the cost of alerting before observability
A team that adds alerts before building observability ends up adding too many alerts. Because the monitoring lacks context, engineers try to catch problems earlier by alerting on more signals. Each new alert adds noise. Alerts that fire regularly without a clear remediation are muted or ignored. The on-call rotation burns out.
The diagnostic for alert fatigue is simple: for each alert, ask "what does the on-call engineer do when this fires at 3am?" If the answer is "investigate whether it's a real problem," the alert should not page. If the answer is "execute this specific runbook step," the alert is actionable and appropriate.
Not everything needs an alert. Before adding an alert, answer: if this fires at 3am, what will the on-call engineer do? If the answer is "investigate and confirm it is a real user-facing problem before escalating," the alert should fire later in the day when the engineer is available, or not at all. Pages should only fire for conditions that require immediate action.
Building observability incrementally
Adding full observability to an existing system without structured logs or traces is a multi-month project. The order matters:
-
Structured logging first. Add structured JSON output to all Cloud Functions and backend services. This requires the least instrumentation effort and provides the most immediate investigative value.
-
Log-based metrics second. Cloud Logging can create metrics from structured log fields. An error rate metric derived from
severity="ERROR"log entries is more accurate than a metric that counts function execution failures, because it captures errors that are caught and logged but do not cause function invocations to fail. -
Distributed tracing third. Add OpenTelemetry instrumentation to the critical paths first (checkout, user registration, data import). The instrumentation overhead is low; the investigation value is high for the flows that matter most.
-
Alerts last. Once the first three layers are in place, add alerts for symptoms. Define runbooks that point to the observability tooling as the investigation starting point.
This sequence means the team has investigation tools before alerts fire. Every alert is actionable from the moment it is added.
What good looks like after the investment
The on-call engineer's 2am investigation improved after the team added structured logging and tracing. The next time the checkout error rate spiked, she opened a trace for a failing request, saw the payment processor timing out, confirmed with a structured log query that the pattern was consistent across multiple requests, and called the payment processor's support line. Total time: 8 minutes. The observability layer did not prevent the incident — it made the incident fast to understand.
More precisely: within two minutes she had a trace showing the payment processor call failing at 5 seconds. Within four minutes she had a log query confirming that the error was payment_key_expired on every affected request. The incident was diagnosed before she had finished her coffee. The prior investigation — without structured logs or traces — had taken 100 minutes for a simpler root cause.
The difference is not cleverness or experience. It is tooling. A system that is observable gives any engineer the information to diagnose an incident quickly. A system that merely alerts requires the engineer to piece together causality from insufficient signals under pressure.
Observability is infrastructure, not a feature. It should be present before the first production alert fires, not added reactively after the third midnight page.
Service level objectives and error budgets
Observability without a clear definition of acceptable service quality leads to alert tuning arguments. What error rate is too high? What latency is unacceptable? These questions should be answered before incidents, not during them.
Service Level Objectives (SLOs) define the target: "99.5% of checkout requests should succeed" or "p99 latency for the checkout endpoint should be below 2000ms over a 30-day window." The error budget is what remains: if the SLO is 99.5%, there is a 0.5% budget for failures.
// Tracking SLO compliance with structured logs
// Cloud Logging metric: count of successful vs. failed checkout requests
function logCheckoutResult(result: {
success: boolean;
durationMs: number;
userId: string;
orderId: string;
errorCode?: string;
}) {
logger.info({
operation: 'checkout',
success: result.success,
durationMs: result.durationMs,
userId: result.userId,
orderId: result.orderId,
errorCode: result.errorCode,
// This field is used by the Cloud Logging metric filter
sloRelevant: true,
});
}
In Cloud Monitoring, create a log-based metric on sloRelevant=true AND operation="checkout" with a label for success. The ratio of successful to total gives real-time SLO compliance. When the error budget is more than 50% consumed for the month, deployments slow down. When it is exhausted, no new features ship until reliability is restored.
This is the discipline that connects observability to business outcomes: the metrics are not just numbers on a dashboard, they are the measure of whether the service is delivering on its promise to users.
Common mistakes when building observability
Logging at the wrong level. Logging every database query at INFO level floods the log sink with noise. Logging nothing at INFO means every investigation starts at ERROR and works backward. The right approach is to log operation completions at INFO, unexpected conditions at WARN, and errors at ERROR. Reserve DEBUG for temporary investigation logs that should not ship to production.
Missing context in log entries. A log entry that says "payment failed" without a user ID, order ID, or error code requires correlation with other log entries to be useful. Every log entry should carry enough context to be useful in isolation.
Alerting on absolute values instead of rates. An alert that fires when error count exceeds 100 in a minute behaves differently at 10 requests per minute (10% error rate, probably bad) and at 1000 requests per minute (1% error rate, probably fine). Alert on rates and percentiles, not absolute counts.
No runbook linked to the alert. An alert without a runbook asks the on-call engineer to remember how to investigate this specific type of failure at 2am. Every alert should link to a runbook that explains what the alert means, what to check first, and what actions to take for common root causes.
These mistakes are correctable. The observability investment produces returns even when imperfect — a noisy alert that links to a runbook is better than no alert. The goal is a system that incrementally improves investigation speed with each incident, not one that is perfect from the start.