Case Study: Order Processing with RabbitMQ

Putting the pieces together

An online shop needs to process orders reliably. The orders service writes the order and an outbox row in one database transaction; a relay publishes order.placed.<region> events to the orders topic exchange with persistent delivery, a stable message_id and publisher confirms. Billing, inventory and email each own a quorum queue bound with their own patterns. Each consumer uses manual acks, a prefetch suited to its work (billing 10, email 50), and idempotent handling keyed by message_id. On transient failures the consumer republishes the message to a 30-second TTL retry queue with an attempt counter and acks the original; after 3 attempts, or if the quorum delivery limit of 5 is hit by repeated crashes, the message goes to a parking-lot queue with an alert and a replay script. Inventory uses single active consumer for per-warehouse ordering. The cluster has three nodes across zones, metrics flow to Prometheus and Grafana, alerts fire on growing queues and missing consumers, and topology lives in a definitions file deployed through CI.

The topology at a glance

Exchanges, queues and the failure paths for each consumer.

orders (topic exchange)
  order.placed.*      -> billing.orders     (quorum, prefetch 10)
  order.#             -> inventory.orders   (quorum, single active consumer)
  order.placed.*      -> email.orders       (quorum, prefetch 50)

failure path per consumer (example: billing)
  consumer fails  -> publish to billing.orders.retry (x-retries + 1), ack original
  retry TTL 30 s  -> default exchange, key billing.orders -> billing.orders
  x-retries >= 3 or delivery-limit 5 hit -> billing.orders.parked (alert, replay tool)

publishing: outbox relay, persistent, message_id, confirms, mandatory
cluster: 3 nodes / 3 zones, Prometheus + Grafana, definitions in Git

Test the failure paths

Deliberately throw errors in a consumer in staging and watch messages flow through retry and into the parking lot. A failure path that has never run is a guess.

त्वरित जाँच: In the case study, what prevents a poison message from being retried forever?

• An attempt counter plus the quorum delivery limit, which send it to a parking-lot queue
• A larger prefetch
• Auto-ack mode
• Publisher confirms