Lesson 24 / 25
Case Study: Order Processing with RabbitMQ
Design a reliable order-processing topology end to end.
By Rahul Agarwal · 15 years in web development
Putting the pieces together
An online shop needs to process orders reliably. The orders service writes the order and an outbox row in one database transaction; a relay publishes order.placed.<region> events to the orders topic exchange with persistent delivery, a stable message_id and publisher confirms. Billing, inventory and email each own a quorum queue bound with their own patterns. Each consumer uses manual acks, a prefetch suited to its work (billing 10, email 50), and idempotent handling keyed by message_id. On transient failures the consumer republishes the message to a 30-second TTL retry queue with an attempt counter and acks the original; after 3 attempts, or if the quorum delivery limit of 5 is hit by repeated crashes, the message goes to a parking-lot queue with an alert and a replay script. Inventory uses single active consumer for per-warehouse ordering. The cluster has three nodes across zones, metrics flow to Prometheus and Grafana, alerts fire on growing queues and missing consumers, and topology lives in a definitions file deployed through CI.
The topology at a glance
Exchanges, queues and the failure paths for each consumer.
orders (topic exchange)
order.placed.* -> billing.orders (quorum, prefetch 10)
order.# -> inventory.orders (quorum, single active consumer)
order.placed.* -> email.orders (quorum, prefetch 50)
failure path per consumer (example: billing)
consumer fails -> publish to billing.orders.retry (x-retries + 1), ack original
retry TTL 30 s -> default exchange, key billing.orders -> billing.orders
x-retries >= 3 or delivery-limit 5 hit -> billing.orders.parked (alert, replay tool)
publishing: outbox relay, persistent, message_id, confirms, mandatory
cluster: 3 nodes / 3 zones, Prometheus + Grafana, definitions in GitTest the failure paths
Deliberately throw errors in a consumer in staging and watch messages flow through retry and into the parking lot. A failure path that has never run is a guess.
Quick check: In the case study, what prevents a poison message from being retried forever?
- An attempt counter plus the quorum delivery limit, which send it to a parking-lot queue
- A larger prefetch
- Auto-ack mode
- Publisher confirms
Answer
An attempt counter plus the quorum delivery limit, which send it to a parking-lot queue — After a bounded number of attempts the message is parked for inspection instead of looping.