Job Queues: Streams vs Lists

Two ways to queue work

Before Streams, the classic Redis job queue was a list: producers LPUSH, workers BRPOP. The problem: once a worker pops a job and crashes, the job is gone. The reliable queue pattern fixes this with BLMOVE (formerly BRPOPLPUSH): atomically move the job from the main list to a per-worker processing list, remove it with LREM after success, and have a reaper move jobs back from processing lists of dead workers. It works, but you build the bookkeeping yourself. Streams with a consumer group provide that bookkeeping natively: the PEL tracks in-progress jobs, delivery counts and owners, XAUTOCLAIM recovers abandoned jobs, several groups can consume the same jobs for different purposes, and history is available for debugging. Libraries such as BullMQ (Node.js) build full job systems with retries, delays and priorities on Redis. For new designs, prefer Streams or an established library over hand-rolled list queues.

List queue and stream queue

A list hands each job out and forgets it; a stream remembers who is working on what.

Reliable list queue with BLMOVE

The job is never only in the worker's memory; it sits in a processing list until removed.

def worker(worker_id):
    processing = f"jobs:processing:{worker_id}"
    while True:
        job = r.blmove("jobs", processing, timeout=5, src="RIGHT", dest="LEFT")
        if job is None:
            continue
        try:
            do_work(json.loads(job))
            r.lrem(processing, 1, job)          # done: remove from processing list
        except Exception:
            r.lmove(processing, "jobs:failed", "LEFT", "LEFT")

# a separate reaper moves jobs from processing lists of dead workers back to "jobs"
# Streams + consumer groups provide this bookkeeping (PEL, XAUTOCLAIM) built in

Do not reinvent a job system

Retries with backoff, delayed jobs, priorities, rate limits and dashboards take months to build well. If you need them, use a mature library on Redis (BullMQ, Sidekiq, RQ, Celery with Redis) or a dedicated broker.

Quick check: What problem does BLMOVE solve compared with a plain BRPOP job queue?

• It makes jobs smaller
• It removes the need for workers
• It keeps the job in a processing list so it is not lost if the worker crashes
• It sorts jobs by priority