RabbitMQ Streams

An append-only log inside RabbitMQ

A stream (available since RabbitMQ 3.9) is a replicated, persistent, append-only log. Unlike a queue, consuming a message does not remove it: many consumers can read the same stream independently, each from its own offset (first, last, next, a specific offset or a timestamp), and messages are kept according to a retention policy by size or age. That gives replay, fan-out to many consumers without copying messages into many queues, and very high throughput, especially through the dedicated stream protocol (port 5552) with client libraries for Java, .NET, Go, Python and others; streams can also be used through AMQP 0-9-1 with some limits. Super streams partition a logical stream into several streams for scale, with ordering per partition key. Single active consumer on streams lets a group of consumers fail over while preserving order. Streams resemble Kafka topics, while keeping RabbitMQ's routing and operational model.

Declaring and reading a stream over AMQP 0-9-1

Stream consumers must set a prefetch and choose where to start reading.

channel.queue_declare(
    queue="clicks",
    durable=True,
    arguments={
        "x-queue-type": "stream",
        "x-max-age": "7D",                       # keep a week of data
        "x-stream-max-segment-size-bytes": 100_000_000,
    },
)

channel.basic_qos(prefetch_count=500)          # required for stream consumers
channel.basic_consume(
    queue="clicks",
    on_message_callback=handle_click,
    arguments={"x-stream-offset": "first"},    # replay from the beginning
)

Streams versus queues

Use a queue when each message is a task to be done once by one worker. Use a stream when many consumers need the same events, when you need replay, or when the data is a large, high-rate log.

Quick check: What happens to a message in a RabbitMQ stream after a consumer reads it?

• It is deleted immediately
• It moves to a dead-letter queue
• It stays in the stream until retention removes it, so other consumers can read it too
• It is converted into a classic queue message