Queues, Load Levelling and Back-Pressure

Turning spikes into steady work

Not every task must finish before the response. Putting work on a queue (sending email, resizing images, generating invoices) lets the web tier answer quickly while workers process at a steady rate: the queue absorbs the spike, a pattern called load levelling. Workers scale on queue depth, and a slow downstream system no longer slows users. Queues are not infinite, though. When producers outpace consumers for long periods, the backlog and latency grow without limit, so systems need back-pressure: signals that make producers slow down, such as bounded queues that reject or block, HTTP 429/503 with Retry-After, or flow control in streaming protocols. Batching improves throughput by amortising per-request overhead: one database insert of 500 rows instead of 500 inserts, at the cost of a little latency.

A bounded buffer that applies back-pressure

When the buffer is full, the producer is told to back off instead of growing memory forever.

import asyncio

queue: asyncio.Queue = asyncio.Queue(maxsize=1000)   # bounded

async def accept_upload(job):
    try:
        queue.put_nowait(job)
        return 202, "accepted"
    except asyncio.QueueFull:
        return 503, "busy, retry later"            # back-pressure to the client

async def worker():
    while True:
        batch = [await queue.get()]
        while len(batch) < 100 and not queue.empty():
            batch.append(queue.get_nowait())        # batch for efficiency
        await save_thumbnails(batch)
        for _ in batch:
            queue.task_done()

A dam on a river

A dam stores the monsoon flood and releases it steadily through the turbines. But if it keeps raining for weeks, you must open the spillways (back-pressure) or the dam fails.

Quick check: What does back-pressure achieve in an overloaded pipeline?

• It makes the queue infinitely large
• It signals producers to slow down so the backlog and memory stay bounded
• It deletes the oldest data silently
• It disables batching