The outbox pattern: reliable event publishing without distributed transactions

A payments engineer writes what looks like a straightforward service. After a successful payment is saved to the database, she publishes a payment.completed event to the message queue. The feature ships. In the first week, the on-call rotation catches two incidents where payment records exist in the database but no payment.completed event was ever published. Downstream consumers — the notification service, the analytics pipeline, the fraud detection system — never saw the payment. The events were lost.

She reviews the code. The database write succeeded. The message queue publish failed silently in a network timeout. The retry logic caught the queue error but the retry itself failed after three attempts. The payment is in the database. The event is gone.

She adds a try-except around the publish call, logs the error, and adds an alert. Two weeks later, a second variant appears. The database write succeeds, the publish call returns success, but the queue acknowledges receipt and then the broker restarts before the message is persisted to disk. The message is gone from the broker's memory and never flushed to disk. The consumer never sees it.

Both scenarios have the same root cause: the database write and the message queue publish are two separate operations. There is no guarantee they will both succeed or both fail together.

The dual-write problem

Writing to two different systems atomically — a database and a message broker — requires a distributed transaction. Distributed transactions are expensive, complex, and introduce availability concerns (both systems must be available at the commit point). Most systems do not implement them.

Without a distributed transaction, any sequence of write to database, then publish to queue has a failure window. The database can succeed and the queue can fail. The queue can succeed and the database can fail. The queue can acknowledge receipt but fail to persist the message.

Retrying the publish after a failure closes one of these windows but not all. If the database write succeeded and the publish fails on every retry, the event is permanently lost unless the application has a separate reconciliation mechanism. If the publish retries succeed but the first publish also succeeded (the failure was in the acknowledgment, not the delivery), the consumer receives the event twice — which is only safe if the consumer is idempotent.

The outbox pattern eliminates the failure window by making the event write part of the database write.

How the outbox pattern works

Instead of writing to the database and then publishing to the queue, the application writes to the database and writes the event to an outbox table in the same transaction. The outbox table is in the same database. The write is atomic — if the main write succeeds, the event is written. If the main write fails, the event is not written. There is no window where one happens and the other does not.

A separate process — the outbox relay — reads from the outbox table and publishes to the message queue. When the queue acknowledges receipt, the relay marks the row as published. If the relay fails or the queue is unavailable, the row stays in the outbox table and the relay retries on its next run.

-- Outbox table in the same database as the application tables
CREATE TABLE outbox (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type  TEXT NOT NULL,
    payload     JSONB NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    published_at TIMESTAMPTZ,         -- NULL until the relay publishes it
    CONSTRAINT outbox_event_type_check CHECK (event_type IN (
        'payment.completed',
        'payment.refunded',
        'payment.failed'
    ))
);

-- Index for the relay to efficiently find unpublished events
CREATE INDEX idx_outbox_unpublished ON outbox (created_at)
    WHERE published_at IS NULL;

The application write:

def process_payment(order_id: str, payment_data: dict) -> Payment:
    with db.transaction():
        # Primary write
        payment = Payment.create(
            order_id=order_id,
            amount=payment_data["amount"],
            status="completed",
            processor_id=payment_data["processor_id"]
        )

        # Outbox write — same transaction, atomic with the primary write
        db.execute(
            """
            INSERT INTO outbox (event_type, payload)
            VALUES (%s, %s)
            """,
            (
                "payment.completed",
                json.dumps({
                    "payment_id": str(payment.id),
                    "order_id": str(payment.order_id),
                    "amount": str(payment.amount),
                    "processor_id": payment.processor_id,
                    "completed_at": payment.created_at.isoformat()
                })
            )
        )

        return payment
        # Both writes committed atomically.
        # If this transaction fails, neither the payment nor the outbox row exists.
        # There is no window where the payment exists without the outbox row.

The relay process:

import time
import logging

def run_outbox_relay(db: Database, queue: MessageQueue, batch_size: int = 100) -> None:
    logger = logging.getLogger("outbox_relay")

    while True:
        try:
            rows = db.execute(
                """
                SELECT id, event_type, payload, created_at
                FROM outbox
                WHERE published_at IS NULL
                ORDER BY created_at ASC
                LIMIT %s
                """,
                (batch_size,)
            ).all()

            if not rows:
                time.sleep(1)  # No unpublished events — wait briefly
                continue

            for row in rows:
                try:
                    queue.publish(
                        topic=row["event_type"],
                        message=row["payload"],
                        message_id=str(row["id"])  # Deterministic ID for deduplication
                    )
                    db.execute(
                        "UPDATE outbox SET published_at = NOW() WHERE id = %s",
                        (row["id"],)
                    )
                    logger.info("Published outbox event %s: %s", row["id"], row["event_type"])

                except Exception as e:
                    logger.error(
                        "Failed to publish outbox event %s: %s",
                        row["id"], str(e)
                    )
                    # Do not mark as published — will retry on next poll

        except Exception as e:
            logger.error("Outbox relay error: %s", str(e))
            time.sleep(5)

The event will be published. It may be published more than once — if the relay publishes but fails before marking the row as published, the next relay run will publish again. This is why consumers must be idempotent: the relay provides at-least-once delivery. The message ID (set to the outbox row's UUID) enables consumers to detect and discard duplicates.

Idempotent consumers

The outbox pattern guarantees that events are published at least once. Consumers that cannot safely process the same event twice will have problems.

An idempotent consumer checks whether it has already processed an event with the given ID before processing it:

def handle_payment_completed(event_id: str, event_data: dict) -> None:
    # Check whether this event has already been processed
    if ProcessedEvent.exists(event_id=event_id):
        logger.info("Skipping duplicate event %s", event_id)
        return

    with db.transaction():
        # Record the event as processed — part of the same transaction
        # as the business logic below, so they commit atomically
        ProcessedEvent.create(
            event_id=event_id,
            event_type="payment.completed",
            processed_at=datetime.utcnow()
        )

        # Business logic — safe to execute once
        order_id = event_data["order_id"]
        Order.update(order_id, fulfillment_status="confirmed")
        FulfillmentQueue.enqueue(order_id=order_id)
        NotificationQueue.enqueue(
            type="payment_confirmed",
            order_id=order_id
        )

The ProcessedEvent write and the business logic writes are in the same transaction. If the business logic fails, the ProcessedEvent is not written, and the event will be retried. If the business logic succeeds, the ProcessedEvent is written, and the event will not be retried. The combination of outbox relay and idempotent consumers produces exactly-once semantics from an at-least-once delivery mechanism.

Variants of the outbox pattern

Change data capture

Instead of a relay process that polls the outbox table, CDC reads the database's replication log (WAL in Postgres, binlog in MySQL) and publishes changes as events. Debezium is the most common implementation.

CDC has lower latency than polling (events are published as soon as they are written to the log) and is more efficient (no polling query, no separate outbox table). It is more complex to operate — the CDC connector must be configured, monitored, and kept in sync with the database schema.

For teams with the operational capacity, CDC is preferable. For teams that want a simpler starting point, polling the outbox table is reliable and requires no additional infrastructure.

Transactional outbox with optimistic locking

In high-throughput systems, multiple relay instances may compete to publish the same outbox rows. An optimistic locking approach:

-- Claim rows for processing with a lease, preventing other relays from claiming them
UPDATE outbox
SET relay_claimed_at = NOW(), relay_claimed_by = %s
WHERE id IN (
    SELECT id FROM outbox
    WHERE published_at IS NULL
      AND (relay_claimed_at IS NULL OR relay_claimed_at < NOW() - INTERVAL '30 seconds')
    ORDER BY created_at ASC
    LIMIT 10
    FOR UPDATE SKIP LOCKED
)
RETURNING id, event_type, payload;

SKIP LOCKED ensures relay instances do not block each other. The lease timeout ensures that if a relay instance dies mid-processing, another instance picks up the rows after 30 seconds.

What the outbox pattern does not solve

The outbox pattern solves the dual-write problem. It does not solve:

Business logic failures. If the payment succeeds but an error in the business logic causes the transaction to roll back, neither the payment record nor the outbox row is written. The event is correctly not published. This is correct behavior — the payment did not succeed from the database's perspective.

Message queue unavailability. If the message queue is unavailable for an extended period, outbox rows accumulate. The queue processes them when it comes back online, potentially out of the order they were written, and in a burst that may overwhelm consumers. The outbox provides durability; it does not provide flow control.

Consumer failures. The outbox pattern ensures the event is published. It does not ensure the consumer successfully processes it. Consumer failure handling — dead-letter queues, retry policies, alerting on consumer lag — is separate.

Monitoring the outbox relay

The outbox relay is infrastructure that must be monitored. Specific signals to track:

Relay lag. The time between an outbox row's created_at and its published_at indicates how far behind the relay is. A relay that is consistently 30+ seconds behind the write rate is a relay that is either overwhelmed or encountering persistent failures.

Unpublished row count. A monotonically growing count of rows where published_at IS NULL indicates the relay has stopped making progress. This should trigger an alert — the relay is not running, the queue is unavailable, or every publish is failing.

Error rate. Track how often the relay encounters exceptions during publish attempts. A nonzero error rate that clears is transient queue unavailability. A sustained error rate is a persistent problem.

def relay_metrics(db: Database) -> dict:
    """Report outbox health metrics for monitoring."""
    result = db.execute("""
        SELECT
            COUNT(*) FILTER (WHERE published_at IS NULL) AS pending_count,
            COUNT(*) FILTER (WHERE published_at IS NULL
                             AND created_at < NOW() - INTERVAL '60 seconds') AS stale_count,
            AVG(EXTRACT(EPOCH FROM (published_at - created_at)))
                FILTER (WHERE published_at IS NOT NULL
                        AND created_at > NOW() - INTERVAL '1 hour') AS avg_relay_lag_seconds
        FROM outbox
    """).first()

    return {
        "pending_count": result["pending_count"],
        "stale_count": result["stale_count"],
        "avg_relay_lag_seconds": result["avg_relay_lag_seconds"]
    }

The outbox pattern is the correct solution to the specific problem of reliably publishing events after a successful database write, without distributed transactions. It is simple to implement, simple to operate, and straightforwardly testable. In most systems that need reliable event publishing, it is the right starting point — far simpler than distributed transactions, far more reliable than fire-and-forget publish calls, and far more observable than either.

Common mistakes when implementing the outbox pattern

Publishing the event before the transaction commits. A common mistake is to write the outbox row inside the transaction but then call the relay's publish logic synchronously before COMMIT. If the publish call blocks or fails, it can cause the transaction to hold a lock longer than necessary — or in some implementations, the event is published before the database confirms the commit, meaning the event can be published for a write that ultimately rolls back. The relay process must read from committed rows only, never from within an open transaction.

Not setting a relay concurrency limit. A relay that picks up all unpublished rows at once on every poll cycle may attempt to publish thousands of events simultaneously after a queue outage ends. The LIMIT clause in the relay query is essential — it controls the burst rate and prevents the relay from overwhelming the message broker with a flood of backed-up events.

Using auto-increment IDs as message IDs for deduplication. Message IDs used for deduplication must be stable across relay retries. An auto-increment ID assigned at relay time will be different on each retry. The outbox row's UUID — assigned at write time, before any relay activity — is the correct stable identifier. Consumers that use this UUID for idempotency checks will correctly detect and discard duplicate deliveries.

Not cleaning up published rows. The outbox table grows indefinitely if published rows are never removed. For a service that processes thousands of payments per day, the outbox table will accumulate millions of rows over months, causing relay queries to slow down even with the partial index on published_at IS NULL. Schedule a periodic cleanup:

-- Delete published rows older than 7 days
-- Run as a daily job or via a background process
DELETE FROM outbox
WHERE published_at IS NOT NULL
  AND published_at < NOW() - INTERVAL '7 days';

A partial index on WHERE published_at IS NULL keeps relay queries fast regardless of the total table size, but the cleanup job prevents unbounded table growth that could eventually affect vacuum performance.

Treating the relay as infallible. The relay is infrastructure. It can crash, lose its database connection, or encounter bugs in the publish logic. A relay that is not running should trigger an alert within minutes — not hours. The alert threshold should be based on the stale_count metric (rows that were created more than N seconds ago and are still unpublished). For a payments service where downstream consumers expect events within 30 seconds, a stale count above zero for more than 60 seconds is an alertable condition.

Integrating the outbox with integration testing

The outbox pattern is straightforward to test because it separates two concerns that can be verified independently: the write (do the outbox row and the business record appear atomically?) and the relay (does the relay publish outbox rows and mark them as published?).

def test_payment_creates_outbox_row_atomically():
    """
    Both the payment record and the outbox row must exist after process_payment,
    or neither should exist if the transaction fails.
    """
    payment = process_payment(order_id="order-abc", payment_data=valid_payment_data())

    # Verify the payment was created
    assert payment is not None

    # Verify the outbox row was created in the same transaction
    outbox_rows = db.execute(
        "SELECT * FROM outbox WHERE event_type = 'payment.completed' AND published_at IS NULL"
    ).all()
    assert len(outbox_rows) == 1
    assert json.loads(outbox_rows[0]["payload"])["payment_id"] == str(payment.id)


def test_relay_publishes_and_marks_as_published():
    """
    The relay should publish the outbox row and set published_at.
    """
    # Insert an outbox row directly for testing the relay in isolation
    db.execute(
        "INSERT INTO outbox (event_type, payload) VALUES (%s, %s)",
        ("payment.completed", json.dumps({"payment_id": "test-payment-1"}))
    )

    mock_queue = MockMessageQueue()
    run_outbox_relay(db=db, queue=mock_queue, batch_size=10)

    # Verify the message was published
    assert len(mock_queue.published_messages) == 1
    assert mock_queue.published_messages[0]["topic"] == "payment.completed"

    # Verify the outbox row is marked as published
    row = db.execute("SELECT published_at FROM outbox WHERE event_type = 'payment.completed'").first()
    assert row["published_at"] is not None

Testing the relay in isolation — by inserting outbox rows directly and running the relay against a mock queue — keeps the tests fast and focused. Testing the write side in isolation verifies the atomic write guarantee. Integration tests that run both together verify the end-to-end flow.

The outbox pattern in context

The outbox pattern addresses one specific problem in event-driven systems: the gap between a successful database write and a guaranteed event publication. It does not replace the other reliability concerns in event-driven systems — consumer idempotency, dead-letter queue management, and consumer lag monitoring remain the team's responsibility.

What the pattern does do is eliminate the most common source of silent data loss in event-driven systems: the fire-and-forget publish call that succeeds on the happy path and fails silently on the error path. With the outbox in place, the question changes from "did the event get published?" to "is the relay running and is the queue available?" — both of which are straightforwardly monitored. That shift from silent loss to observable infrastructure is the pattern's most important contribution.