पाठ 18 / 30

Queues, Partitions and Backlog

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Size message-queue partitions and consumers, and compute how long a backlog takes to drain.

लेखक: · 15 साल से वेब डेवलपर

Queues absorb bursts, consumers set the pace

A message queue or log (for example Kafka, RabbitMQ or a cloud queue service) decouples producers from consumers. It does not create capacity; it buys time. Two estimates matter. Consumer throughput: total consumer rate must exceed the average produce rate, otherwise the backlog grows forever. In partitioned logs, parallelism is bounded by partition count, since within a consumer group each partition is read by at most one consumer at a time; so partitions = target throughput / per-consumer throughput, with headroom. Backlog drain time: after an outage or spike leaves B messages waiting, drain time = B / (consume rate - produce rate). If consumers are only slightly faster than producers, the backlog clears painfully slowly, which is another face of the utilisation curve from the previous section. Also estimate retention storage: messages per second x size x retention seconds x replication.

Draining a backlog

After a 30-minute consumer outage, how long until the system catches up?

produce_rate = 8_000        # msgs/s, continues during recovery
consume_rate = 10_000       # msgs/s with all consumers healthy
outage_s = 30 * 60

backlog = produce_rate * outage_s
drain_s = backlog / (consume_rate - produce_rate)
print(f"backlog: {backlog:,} messages")
print(f"drain time: {drain_s / 60:.0f} minutes")

Output:

backlog: 14,400,000 messages
drain time: 120 minutes

Clearing a traffic jam

When a signal fails for half an hour, the jam does not vanish when it starts working again. Cars keep arriving, so the queue shrinks only by the difference between how fast cars leave and how fast new ones join.

त्वरित जाँच: A backlog of 600,000 messages exists. Producers send 4,000/s and consumers process 5,000/s. How long to drain it?

  • 2 minutes
  • About 67 seconds
  • 150 seconds
  • 10 minutes
Answer

10 minutes — Net drain rate is 5,000 - 4,000 = 1,000/s; 600,000 / 1,000 = 600 seconds, which is 10 minutes.