Delivery Semantics and Idempotent Consumers

Duplicates are normal

Networks fail between "processed" and "acknowledged", so brokers give one of three guarantees. At-most-once: acknowledge before processing; a crash loses the message. At-least-once: acknowledge after processing; a crash causes a redelivery, so the same message may be processed twice. This is the default for most systems. Exactly-once end to end is not something a broker can promise on its own once side effects leave it (sending an email, charging a card, writing to another database). Kafka's transactions give exactly-once within Kafka (read-process-write between topics), but the practical goal is effectively-once: at-least-once delivery plus an idempotent consumer, whose effect is the same whether a message is handled once or five times. Techniques: a processed-messages table keyed by event ID, written in the same transaction as the business change; natural idempotency ("set status to SHIPPED"); and idempotency keys on downstream APIs.

An idempotent consumer with a dedupe table

The insert into processed_events and the business update commit together, so a redelivery is skipped.

def handle_payment_captured(event, db):
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO processed_events (consumer, event_id) VALUES (%s, %s) "
            "ON CONFLICT DO NOTHING",
            ("orders-service", event["id"]),
        ).rowcount
        if inserted == 0:
            return  # duplicate delivery: already handled

        tx.execute(
            "UPDATE orders SET status = 'PAID', paid_at = %s WHERE id = %s",
            (event["time"], event["data"]["orderId"]),
        )
    # acknowledge to the broker only after the transaction commits

Stamping a cheque

A bank clerk stamps "PAID" on every cheque they process. If the same cheque is handed in twice, the stamp shows it was already paid. The dedupe table is that stamp.

Quick check: With at-least-once delivery, what must a consumer that sends money be?

• Fast
• Stateless
• Written in the same language as the producer
• Idempotent, so duplicate deliveries do not repeat the payment