# Queues, Topics and Logs — Event-Driven Architecture & CQRS

Source: https://www.skillbyai.com/en/event-driven-architecture/m-brokers

> Compare message queues with log-based brokers and consumer groups.

## Two families of brokers

**Message queues** such as RabbitMQ, Amazon SQS and Azure Service Bus deliver each message to **one consumer** among competing workers, remove it once **acknowledged**, and offer routing, priorities, delays and dead-letter queues. Publish-subscribe is built on top with exchanges or topics that copy messages into one queue per subscriber. **Log-based brokers** such as Apache Kafka, Amazon Kinesis, Azure Event Hubs and Redis Streams append events to a durable, ordered **log** split into **partitions**. Consumers track their own **offset**; events stay for a retention period whether or not they were read, so new consumers can **replay** history. A **consumer group** spreads partitions across instances, so each event is processed once per group, and different groups each see every event. Queues suit task distribution and commands; logs suit event streams, many independent consumers and replay.

## Queue versus partitioned log

A queue hands each message to one worker and forgets it; a log keeps events and lets each group read at its own offset.

![Top: a pipe with items flowing to whichever of three workers is free. Bottom: three parallel numbered strips with two readers at different positions along them.](assets/figures/event-driven-architecture/section-3-map.svg) — Figure 3.1 — Competing consumers on a queue and consumer groups on a log.

## Consumer groups on a log

Two groups each receive every event; instances inside a group split the partitions.

```text
topic orders.placed  (3 partitions, key = customerId)

  partition 0: e1  e4  e7  e9  ...
  partition 1: e2  e5  e8  ...
  partition 2: e3  e6  ...

group "billing"   (2 instances)  -> instance A: p0, p1   instance B: p2
group "analytics" (1 instance)   -> instance C: p0, p1, p2

adding a 4th billing instance with 3 partitions leaves one instance idle:
parallelism within a group is capped by the partition count
```

## Partitions cap parallelism

In Kafka-style logs, a group cannot use more active consumers than there are partitions. Choose partition counts with future throughput in mind, because changing them later reshuffles key-to-partition mapping.

**Quiz:** A new analytics service must process the last 7 days of order events. Which broker style makes that straightforward?

- [x] A log-based broker with retention, read from an earlier offset
- [ ] A classic queue that deletes acknowledged messages
- [ ] A synchronous REST call
- [ ] An in-memory event bus

*Answer:* A log-based broker with retention, read from an earlier offset. Logs retain events, so a new consumer group can start from an earlier position and replay.