Build a Redis-backed task queue by choosing the right Redis structure, running a fixed number of asynchronous workers, and treating every job as potentially repeatable. Use Redis lists for a straightforward one-worker-per-job queue; choose Streams when you need retained history, replay, or independent consumer groups. Neither design makes arbitrary external side effects exactly once: workers must acknowledge only after successful processing, and failed or abandoned work needs a recovery plan.
Choose a Redis list or a Stream
The main design choice is whether you need a queue for background work or an ordered event log that workers can replay. Redis documents both approaches; their recovery and fan-out semantics differ.
| Decision | Redis list-based job queue | Redis Streams consumer group |
|---|---|---|
| Main shape | A job moves from a pending list to a processing list when a worker claims it. | Ordered entries are tracked by a consumer-group cursor and pending-entry list. |
| Recovery | A reclaimer returns jobs abandoned beyond a visibility timeout. | Workers can transfer sufficiently idle pending entries with XCLAIM or XAUTOCLAIM. |
| History and replay | Job state and retention are managed by the application. | Entries remain in the Stream for replay, subject to trimming. |
| Fan-out | In the documented queue pattern, one worker claims each job. | Workers in one group share entries; separate groups read the Stream independently. |
| Additional capabilities | Sorted sets can support delayed execution and priorities. | Ordered IDs, group acknowledgements, pending inspection, and retention controls. |
| Best fit | Background jobs are the main concern. | Replay, retained history, or independent downstream consumers matter. |
For the list pattern, Redis describes atomically moving jobs from pending to processing with LPUSH and BRPOPLPUSH or BLMOVE. For Streams, producers append with XADD; consumers read through a group with XREADGROUP, acknowledge completed entries with XACK, and inspect outstanding entries with XPENDING. See Redis’s job queue pattern and streaming concepts.
A Stream is not simply a list with a different name: one consumer group distributes entries among its consumers, while multiple groups each receive the stream independently. Redis Pub/Sub is different again; it is fire-and-forget and does not retain messages for subscribers that are disconnected. Choose based on the work’s required semantics, not just the word “queue.”
Recommended Free Tools
#1 Best Overall
Design the job lifecycle before writing workers
Use at-least-once handling, not an exactly-once assumption
In a Stream consumer group, an entry remains pending until acknowledged. A worker might complete an external action—such as sending an email or updating another service—and then crash before issuing XACK. Redis will still regard the entry as unfinished, so recovery can cause it to be processed again. Make handlers safe to retry: use a stable job ID and an application-level idempotency record or another durable deduplication mechanism for side effects.
Set retry limits and decide what happens to permanently invalid jobs, such as moving them to a dead-letter or quarantine path. Keep transient failures distinct from malformed or otherwise unrecoverable payloads. Acknowledging before the side effect is complete avoids a pending entry but risks losing the job if the worker then fails; acknowledge only after successful processing.
Separate producer retries from consumer idempotency
Redis 8.6 documents idempotent message production for retries of XADD when a response may have been lost. This addresses duplicate insertion at the producer boundary; it does not make a consumer’s payment, email, or other external side effect exactly once. Confirm that the Redis server version supports this feature before relying on it. See Redis’s idempotent message processing documentation.
Rank #2
Run a bounded set of asyncio workers
Use a fixed worker count rather than spawning a new asyncio task for every incoming job. Under backlog, unbounded task creation shifts overload into process memory and scheduling. Choose worker count and batch size against job duration, Redis capacity, CPU availability, and downstream service limits; there is no universal throughput number that applies to every queue.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →For Python 3.11 and later, asyncio.TaskGroup provides structured lifetime management: leaving the context waits for child tasks, and a child’s non-cancellation failure cancels its siblings and raises an exception group. A worker loop should handle expected per-job failures without accidentally hiding failures that ought to stop the worker service.
Use try/finally for cleanup during cancellation and generally propagate asyncio.CancelledError after cleanup. Swallowing cancellation can interfere with structured-concurrency features such as TaskGroup and asyncio.timeout(). Python documents these cancellation practices in its asyncio coroutines and tasks guide.
Make shutdown and recovery work together
- Stop accepting new work from Redis.
- Allow in-flight handlers a bounded period to finish and acknowledge completed jobs.
- Cancel remaining workers, run their cleanup, and close Redis connections.
- Leave unfinished Stream entries pending, or list jobs in the processing list, so a recovery process can reclaim them rather than marking them complete.
For Streams, set the reclaim idle threshold to accommodate realistic job duration and heartbeat behavior. If it is too short, a healthy long-running job may be reclaimed while its original worker is still active; if too long, crashed work takes longer to recover. The appropriate value depends on the application, not a universal Redis default.
Start consumer groups deliberately
When creating a Stream consumer group, choose whether it should process existing entries or only new arrivals. In the Redis Python guide, start ID 0-0 means beginning with the existing Stream; $ is used for entries that arrive after group creation. Treat this as a deployment decision: accidentally choosing the wrong start point can either skip work or trigger a large first-time backlog. See Redis’s Streams guide for redis-py.
Give consumers stable, distinguishable names and plan how restarted consumers handle pending work. A restarted consumer using the same name can explicitly revisit its own pending entries; a separate recovery sweep can claim sufficiently idle entries from failed consumers. Do not assume that creating a new consumer name automatically recovers another consumer’s pending work.
A blocking read with a timeout avoids busy-looping when a Stream is idle. A blocked read occupies its client connection while waiting, so account for connection use when sizing the consumer pool. Use the async API supported by the redis-py release installed in your application, and verify its connection lifecycle and cancellation behavior against that release; API signatures can vary by version.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Recover pending work and control retention
For Streams, inspect pending entries and reclaim those that have been idle long enough with XAUTOCLAIM or XCLAIM. For a list-based queue, use the processing list and visibility-timeout reclaimer to return abandoned jobs to pending work. In both designs, recovery is safe only when handlers tolerate retries.
Stream trimming keeps retained history bounded, but the policy must account for consumer lag and pending work. Redis’s Python guide demonstrates approximate trimming with MAXLEN ~; approximate trimming does not promise an exact maximum. Avoid deleting entries that consumers still need for replay or that undermine the recovery behavior you expect.
Free tools Windows power users keep installed
One-click scans. No signup required.
Best Value
Redis documents additional stream deletion and retention coordination options beginning with Redis 8.2: KEEPREF, DELREF, and ACKED for trimming or deletion interactions with consumer groups, plus XDELEX and XACKDEL. Their effects on pending references differ, so use them only after checking the behavior supported by the server version you deploy. The version-specific options are described in the Redis Streams documentation.
Monitor queue health, not just whether workers are running
A worker process can be alive while its queue is falling behind. Track the signals that reveal backlog, stuck work, and repeated failures:
- Stream length and whether it is growing.
- Consumer-group lag and pending-entry count.
- Age of the oldest pending entry and reclaim counts.
- Job processing latency, retry counts, and dead-letter volume.
- Consumer availability and worker errors.
Redis documents XPENDING, XINFO STREAM, XINFO GROUPS, and XINFO CONSUMERS for inspecting Stream and consumer-group state. Pair those Redis-level checks with application metrics for retries and handler outcomes; queue depth alone cannot explain why work is delayed.
Quick Recap
Practical implementation checklist
- Choose a list for a direct claim-and-complete queue, or a Stream when retained history, replay, or independent consumer groups are required.
- Define how a job is considered complete, how retries work, and where permanently failed jobs go.
- Make external side effects idempotent before enabling automatic recovery.
- Set a fixed worker count, and tune it to Redis and downstream capacity rather than creating tasks without a limit.
- Choose the consumer-group start ID and consumer naming scheme intentionally.
- Set reclaim timing to suit job duration; ensure healthy work is not reclaimed prematurely.
- Plan shutdown so unfinished jobs remain recoverable and Redis connections close cleanly.
- Set a retention policy that preserves required replay and pending-work recovery.
- Check Python and Redis versions before using version-specific features;
TaskGrouprequires Python 3.11 or later, Stream enhancements vary by Redis release.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
