Queues, Topics and Logs

Two families of brokers

Message queues such as RabbitMQ, Amazon SQS and Azure Service Bus deliver each message to one consumer among competing workers, remove it once acknowledged, and offer routing, priorities, delays and dead-letter queues. Publish-subscribe is built on top with exchanges or topics that copy messages into one queue per subscriber. Log-based brokers such as Apache Kafka, Amazon Kinesis, Azure Event Hubs and Redis Streams append events to a durable, ordered log split into partitions. Consumers track their own offset; events stay for a retention period whether or not they were read, so new consumers can replay history. A consumer group spreads partitions across instances, so each event is processed once per group, and different groups each see every event. Queues suit task distribution and commands; logs suit event streams, many independent consumers and replay.

Queue versus partitioned log

A queue hands each message to one worker and forgets it; a log keeps events and lets each group read at its own offset.

Consumer groups on a log

Two groups each receive every event; instances inside a group split the partitions.

topic orders.placed  (3 partitions, key = customerId)

  partition 0: e1  e4  e7  e9  ...
  partition 1: e2  e5  e8  ...
  partition 2: e3  e6  ...

group "billing"   (2 instances)  -> instance A: p0, p1   instance B: p2
group "analytics" (1 instance)   -> instance C: p0, p1, p2

adding a 4th billing instance with 3 partitions leaves one instance idle:
parallelism within a group is capped by the partition count

Partitions cap parallelism

In Kafka-style logs, a group cannot use more active consumers than there are partitions. Choose partition counts with future throughput in mind, because changing them later reshuffles key-to-partition mapping.

त्वरित जाँच: A new analytics service must process the last 7 days of order events. Which broker style makes that straightforward?

• A log-based broker with retention, read from an earlier offset
• A classic queue that deletes acknowledged messages
• A synchronous REST call
• An in-memory event bus