Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Use a bounded buffered channel for pending jobs and a fixed set of worker goroutines to process them. This gives you backpressure and a firm limit on concurrent work without adding a package dependency. It is an in-memory design: jobs are not preserved if the process exits, and it does not coordinate multiple application instances.
What this queue does—and what it does not
A Go channel can hold pending jobs; workers receive from that channel and call a handler. The channel’s capacity limits how many jobs can wait, while the worker count limits how many handlers run at once. This is suitable for work that may wait briefly inside one running process.
It is not a durable or distributed queue. A process crash loses pending jobs and any work that was in progress. Multiple instances have separate channels and do not share queue state. If you need persisted job state, crash recovery, or coordination across instances, use a broker or durable store with explicit recovery and state-transition logic. The Redis Go job-queue example illustrates pending and processing states, completion and failure paths, recovery after worker failure, and retry and idempotency considerations.
Choose the queue’s behavior before writing it
- When the queue is full: block the caller, reject immediately, or let the caller’s context cancel the enqueue. Blocking is the simplest direct backpressure policy.
- How much work can run: choose a fixed worker count. A worker pool bounds simultaneous handler calls; starting a new goroutine for every job does not.
- What happens to errors: return them to the caller, record them, retry within a limit, or route failed work elsewhere. Do not retry immediately without a bound.
- How shutdown works: decide whether to drain queued jobs or cancel processing, and stop producers before closing the queue.
- Whether jobs must survive a crash: if yes, an in-memory channel is not sufficient.
Implement a bounded queue and fixed worker pool
This minimal example uses a string job payload and a handler function. The queue blocks submissions when its buffer is full. Closing it stops workers after they finish jobs already in the buffer. The queue owner must ensure no goroutine can still submit when shutdown begins.
#1 Best Overall
package main
import (
"context"
"errors"
"fmt"
"sync"
)
type Job struct {
ID string
}
type Queue struct {
jobs chan Job
handle func(context.Context, Job) error
workers sync.WaitGroup
}
func NewQueue(workerCount, capacity int, handle func(context.Context, Job) error) (*Queue, error) {
if workerCount <= 0 {
return nil, errors.New("worker count must be positive")
}
if capacity <= 0 {
return nil, errors.New("capacity must be positive")
}
if handle == nil {
return nil, errors.New("handler must not be nil")
}
q := &Queue{
jobs: make(chan Job, capacity),
handle: handle,
}
for i := 0; i < workerCount; i++ {
q.workers.Add(1)
go func() {
defer q.workers.Done()
for job := range q.jobs {
if err := q.handle(context.Background(), job); err != nil {
// Replace with application logging or error reporting.
fmt.Printf("job %s failed: %vn", job.ID, err)
}
}
}()
}
return q, nil
}
func (q *Queue) Submit(job Job) {
q.jobs <- job
}
// CloseAndWait must be called only after all Submit calls have stopped.
func (q *Queue) CloseAndWait() {
close(q.jobs)
q.workers.Wait()
}In this example, handler errors are logged and then discarded; there is no retry, result returned to the submitter, or persistent record. Adapt that policy to the application rather than treating the sample error path as a complete failure strategy. The context passed to the handler is a background context, so this version does not cancel running work during shutdown.
Submit jobs and shut down in the right order
- Create the queue with a positive worker count and buffer capacity. Select values based on the application’s workload and acceptable waiting behavior; this example does not prescribe performance numbers.
- Submit jobs while producers are active. A send to the buffered channel waits when the buffer is full until a worker receives a job.
- During shutdown, stop or join all producers so none can call
Submitagain. - Call
CloseAndWait. Workers ranging over the closed channel finish buffered jobs, then exit; the wait group lets shutdown wait for them.
Only the queue owner should close the channel. Sending to a closed channel panics, so closing it while producers can still submit is unsafe.
Make the full-queue policy explicit
The sample’s blocking Submit is often useful because it pushes pressure back to callers instead of allowing an unbounded backlog. But a blocked caller may need a timeout or cancellation path. For a context-aware send, expose an enqueue method that selects between the channel send and ctx.Done(); for immediate rejection, use a non-blocking send with a default case and return a full-queue error. Choose one contract and make callers handle it.
Recommended Free Tools
Capacity counts waiting jobs, not jobs already assigned to workers. With a capacity of N and W workers, the queue can hold up to N pending jobs while up to W jobs are being handled. A larger buffer absorbs more bursts but also permits more work to wait in memory.
Handle failures and duplicate execution deliberately
An error from a handler does not automatically put a job back in the channel. Decide whether the caller needs the outcome, whether the application should record the failure, and whether a retry is safe. If you retry, use a limit and an intentional delay or backoff rather than an unbounded immediate loop.
A handler may perform an external side effect and then fail before the queue can record success. Retrying can therefore perform that side effect again. Make such operations idempotent where possible, or use an appropriate idempotency key. Redis’s queue guidance also highlights idempotency and retry counts for non-idempotent actions.
Rank #4
Choose drain or cancellation at shutdown
The sample drains: closing the jobs channel lets workers complete every buffered job before returning. If a handler hangs, the wait can also hang. To support cancellation, pass a context to the handler and define whether shutdown cancels active work, abandons pending jobs, or waits for a grace period. Cancellation alone does not make queued work durable; any abandoned in-memory jobs disappear with the process.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Keep channel ownership simple: the component responsible for accepting submissions should coordinate producer shutdown, close the channel once, and wait for workers. Go’s official concurrency guidance expresses the underlying design principle as “Do not communicate by sharing memory; instead, share memory by communicating.” See Effective Go: Concurrency.
Best Value
When to move beyond an in-process queue
Use the channel pattern when work only needs to be deferred within one process and losing it on process termination is acceptable. Move to a durable queue when jobs must survive restarts, be recovered after worker failure, be shared among instances, or have controlled at-least-once delivery. Those guarantees require persisted job state and recovery logic, not just a different channel capacity.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
