Redelivery, Ordering and Idempotent Consumers

At-least-once means duplicates

With manual acks, RabbitMQ provides at-least-once delivery: if a consumer processes a message but its ack is lost (connection dropped just after processing), the message is redelivered with the redelivered flag set, and may be processed twice. Make consumers idempotent: record processed message IDs (set by the publisher in the message_id property) in your database in the same transaction as the business change, and skip IDs you have seen; or design operations that are naturally idempotent, such as "set status to SHIPPED". Ordering: a single queue delivers in publish order to one consumer, but with several consumers, requeues and retries, order across messages is not guaranteed. If per-entity order matters, route each entity's messages to the same queue and consumer (for example with the consistent hash exchange plugin or single active consumer), and carry a version number so consumers can ignore stale updates.

Deduplicating by message ID

The insert fails for a duplicate, so the business change is skipped.

def handle(ch, method, props, body):
    msg_id = props.message_id          # set by the publisher, stable across retries
    with db.transaction() as tx:
        first_time = tx.execute(
            "INSERT INTO processed_messages (consumer, message_id) VALUES ('billing', %s) "
            "ON CONFLICT DO NOTHING", (msg_id,)
        ).rowcount == 1
        if first_time:
            apply_invoice(tx, json.loads(body))
    ch.basic_ack(delivery_tag=method.delivery_tag)   # ack after commit, duplicate or not

Publishers must set message_id

Deduplication only works if the same logical message keeps the same ID when it is republished. Generate the ID when the event is created (for example in an outbox row), not at each publish attempt.

त्वरित जाँच: Why can a consumer receive the same message twice with manual acknowledgements?

• RabbitMQ always sends two copies
• Prefetch duplicates messages
• If the ack is lost after processing, the broker redelivers the message
• Topic exchanges duplicate every message