The Dual-Write Problem and the Transactional Outbox

Saving and publishing must not disagree

A service that saves an order to its database and publishes OrderPlaced to a broker performs a dual write. There is no shared transaction: if the database commits and the publish fails (or the process crashes in between), the order exists but nobody hears about it; if you publish first and the commit fails, consumers react to an order that does not exist. The transactional outbox pattern fixes this. In the same database transaction as the business change, insert the event into an outbox table. A separate relay then reads the outbox and publishes to the broker, marking rows as sent. Because the relay may publish a row and crash before marking it, delivery is at-least-once, which consumers already handle. The relay is either a polling publisher or change data capture (CDC), for example Debezium reading the database's write-ahead log.

The outbox pattern

Business row and event row commit together; a relay moves events to the broker afterwards.

Writing to the outbox in the same transaction

Either both rows commit or neither does.

BEGIN;

INSERT INTO orders (id, customer_id, total_minor, currency, status)
VALUES ('o-1001', 'c-42', 149900, 'INR', 'PLACED');

INSERT INTO outbox (id, aggregate_type, aggregate_id, type, payload, created_at)
VALUES (gen_random_uuid(), 'order', 'o-1001', 'OrderPlaced.v1',
        '{"orderId":"o-1001","customerId":"c-42","totalMinor":149900,"currency":"INR"}',
        now());

COMMIT;

-- relay (polling version), run in a loop:
-- SELECT * FROM outbox WHERE published_at IS NULL ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED;
-- publish each row, then UPDATE outbox SET published_at = now() WHERE id = ...;

Use the aggregate ID as the message key

Publishing outbox rows with the aggregate ID as the partition key keeps each entity's events in order downstream, as long as the relay publishes rows for one aggregate in creation order.

त्वरित जाँच: What problem does the transactional outbox solve?

• Slow SQL queries
• Schema evolution
• Database changes and published events getting out of sync because there is no shared transaction
• Consumer lag