Durability and Persistence

Surviving a broker restart

Several settings must line up for a message to survive a broker restart. The exchange and queue must be durable (their definitions survive restarts). The message must be persistent (delivery_mode=2), so it is written to disk. And the publisher should use confirms to know it was stored before considering it sent. A durable queue with transient messages loses those messages on restart; a persistent message in a non-durable queue disappears with the queue. Quorum queues and streams are always durable and replicate data across nodes, so they also survive the loss of a minority of nodes. Classic queues (version 2 storage) live on one node: durable classic queues survive a restart of that node, but not its permanent loss. Exclusive and auto-delete queues are useful for temporary consumers such as RPC reply queues, but never for important data.

Durable topology and persistent messages

Every piece needed for messages to survive restarts.

channel.exchange_declare("invoices", exchange_type="direct", durable=True)
channel.queue_declare(
    "invoices.generate",
    durable=True,                                  # required for quorum queues anyway
    arguments={"x-queue-type": "quorum"},          # replicated across cluster nodes
)
channel.queue_bind("invoices.generate", "invoices", routing_key="generate")

channel.confirm_delivery()
channel.basic_publish(
    "invoices", "generate", body=payload,
    properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, message_id=msg_id),
)

Writing it in ink, in a fireproof cabinet

A durable queue is the fireproof cabinet; a persistent message is writing in ink instead of pencil. You need both, and a receipt from the clerk (confirm) proving the paper went into the cabinet.

Quick check: A durable queue receives a non-persistent message, then the broker restarts. What happens to the message?

• It survives because the queue is durable
• It is lost because the message itself was not persisted
• It moves to a dead-letter exchange
• It is converted to persistent automatically