How to Use Celery with RabbitMQ to Queue Tasks on a VPS

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

Antihost offers Linux VPS plans with full root access, unlimited traffic on a 1 Gbps port and a 14-day free trial.

Run RabbitMQ as the message broker and Celery as the producer/worker layer: your web application publishes a task, RabbitMQ routes and stores the message, and a Celery worker consumes and executes it. On a VPS, the reliable pattern is a dedicated RabbitMQ user and virtual host, an AMQP broker_url kept out of source control, a systemd-managed worker, and firewall and monitoring controls around the broker.

What Celery and RabbitMQ each do

Celery is not the queue database itself. It provides the Python application interface for declaring tasks, sending messages, starting workers, retrying work and inspecting execution. RabbitMQ is the broker: it accepts messages from producers, applies exchange and queue routing, stores messages until they can be delivered, and acknowledges them as workers finish.

#	Preview	Product	Price
1		RabbitMQ in Action: Distributed Messaging for Everyone	$24.98	Buy on Amazon
2		RabbitMQ With C#	$19.95	Buy on Amazon
3		The World of Peter Rabbit (The Original Peter Rabbit, Books 1-23, Presentation Box)	$57.71	Buy on Amazon
4		Hurray for Hattie Rabbit: Story and pictures (An Early I can read book)	$23.87	Buy on Amazon
5		Clojure in Action	$58.99	Buy on Amazon

A typical request follows this path:

1. Your web process calls a task asynchronously, such as generate_report.delay(report_id).
2. Celery serializes the task name and arguments into an AMQP message.
3. RabbitMQ receives the message on the configured virtual host and routes it to a queue.
4. A Celery worker subscribed to that queue reserves and executes the task.
5. The worker acknowledges successful delivery. If you use a result backend, task state and return data are stored there separately from the broker.

Celery describes RabbitMQ as its default broker and documents it as stable, durable and suitable for production. RabbitMQ also supports queue monitoring and Celery remote-control features. Those capabilities do not remove the need to monitor disk, memory, queues and failed tasks.

Plan the VPS deployment

Use a supported Debian or Ubuntu Linux VPS and keep the roles clear even if they share one machine. The web application publishes tasks, RabbitMQ brokers them, and one or more worker processes execute them. A small installation can run all three roles on one VPS; larger workloads can move workers or RabbitMQ to separate hosts without changing the task model.

#1 Best Overall

Sale

RabbitMQ in Action: Distributed Messaging for Everyone

$24.98

Buy on Amazon

• Update the operating system before installing services.
• Create a non-root account for the application and worker.
• Reserve enough memory and disk for RabbitMQ, worker processes, logs and any result storage.
• Decide whether the broker is local or reached over a private network. A local broker normally uses localhost and does not expose AMQP publicly.
• Choose where task results live and how long they should be retained. Broker delivery persistence is not the same thing as result-backend retention.

Do not put RabbitMQ administration, application code and worker commands behind an ambiguous shared account. Explicit ownership makes permissions, systemd hardening and incident recovery much easier.

Install and verify RabbitMQ

Update the VPS and install the service

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install rabbitmq-server

The package creates a system service. Check it before configuring Celery:

sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
sudo systemctl status rabbitmq-server

A healthy status should show the service as active (running). If it fails, inspect the service journal before moving on:

sudo journalctl -u rabbitmq-server -b --no-pager

RabbitMQ must be running before a worker can connect. Starting a Celery worker against a stopped broker usually produces connection-refused or retry-loop messages rather than a useful task error.

Create a dedicated user and virtual host

Do not use RabbitMQ’s default guest account for remote production access. Create an account specifically for this application, give it a separate virtual host, and grant permissions only there:

sudo rabbitmqctl add_user myuser mypassword
sudo rabbitmqctl add_vhost myvhost
sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Replace the example password with a long, unique secret. The three permission expressions control configure, write and read access. The broad expressions above are a working starting point inside the dedicated vhost; tighten them to the exact exchanges and queues your application needs when your naming scheme is established.

Confirm the objects and permissions:

sudo rabbitmqctl list_users
sudo rabbitmqctl list_vhosts
sudo rabbitmqctl list_permissions -p myvhost

A virtual host is a broker namespace, not an operating-system user. It separates exchanges, queues and permissions while the RabbitMQ service itself remains one daemon.

Rank #2

RabbitMQ With C#

$19.95

Buy on Amazon

Configure Celery with an AMQP broker URL

Use the URL format correctly

Celery’s general broker URL form is:

transport://userid:password@hostname:port/virtual_host

For the local RabbitMQ installation above, an example is:

amqp://myuser:mypassword@localhost:5672/myvhost

Use amqps:// when the connection crosses an untrusted network and TLS is configured. URL-encode reserved characters in usernames, passwords and virtual-host names; for example, a literal slash in a vhost is represented as %2F. A password containing @, : or / must likewise be encoded.

Keep the complete URL in an environment variable or a file readable only by the service account, never in a public repository:

export CELERY_BROKER_URL='amqp://myuser:replace-with-a-long-secret@localhost:5672/myvhost'

In a Python settings module:

import os
from celery import Celery

broker_url = os.environ["CELERY_BROKER_URL"]
celery_app = Celery("vps_tasks", broker=broker_url)

If the application uses a framework settings object, assign the same value to its Celery configuration rather than creating a second, conflicting URL. Protect the environment file with ownership and mode settings appropriate to the worker account.

Define a task and enqueue it

Create a minimal Celery application

Save this as tasks.py in the application directory:

import os
import time
from celery import Celery

app = Celery(
    "vps_tasks",
    broker=os.environ["CELERY_BROKER_URL"],
)

@app.task
def build_report(report_id):
    # Replace this with real, bounded work.
    time.sleep(2)
    return {"report_id": report_id, "status": "complete"}

Start a worker by pointing Celery at the module and application object:

celery -A tasks:app worker --loglevel=INFO

The -A tasks:app value means “import app from tasks.py.” Run the command from the directory containing the module, or set the working directory explicitly in your service unit.

Rank #3

Sale

The World of Peter Rabbit (The Original Peter Rabbit, Books 1-23, Presentation Box)

• Note: The Hardcover edition (2006) from Warne that you originally viewed is the one you'll receive which is manufactured in china. The Look Inside option is viewing Hardcover edition (2007) from Warne which is printed in Singapore

$57.71

Buy on Amazon

Send work from the producer

from tasks import build_report

result = build_report.delay("report-123")
print(result.id)

.delay() is a shortcut for asynchronous dispatch. Use .apply_async() when you need execution options such as an ETA, countdown, routing key or custom queue:

build_report.apply_async(args=["report-123"], countdown=30)

Task arguments must be serializable by the configured serializer. Pass IDs, strings, numbers and small structured values rather than open database connections, request objects or filesystem handles. Fetch large payloads from durable storage inside the task instead of putting them into AMQP messages.

Make tasks safe to retry

Workers can lose a connection or be restarted after a message is delivered but before your side effect is complete. Design tasks to be idempotent: running the same task twice should not charge a customer twice, create duplicate records or corrupt a file. Use a unique operation key, a database uniqueness constraint or an explicit “already completed” check around non-repeatable work. Keep external calls bounded with timeouts and record enough context to diagnose a failed attempt.

Run the worker with systemd

An interactive shell is useful for a first test, but it does not restart a worker after a crash or reboot. Create a dedicated systemd unit so the process has a fixed account, directory, environment and dependency order.

Create the service account and unit

Assume the code lives in /srv/myapp and the Linux account is myapp. Store the broker URL in an environment file readable by that account, such as /etc/myapp/celery.env, and then create /etc/systemd/system/celery-worker.service:

[Unit]
Description=Celery worker for myapp
Requires=rabbitmq-server.service
After=rabbitmq-server.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
EnvironmentFile=/etc/myapp/celery.env
ExecStart=/srv/myapp/.venv/bin/celery -A tasks:app worker --loglevel=INFO
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Requires and After express the local broker dependency. They do not guarantee RabbitMQ will remain healthy forever; the worker’s restart policy and monitoring still matter. If RabbitMQ is on another host, remove the local service dependency and use network and health checks instead.

Load, enable and test the unit

1. Reload unit files: sudo systemctl daemon-reload.
2. Start the worker: sudo systemctl start celery-worker.
3. Check its state: sudo systemctl status celery-worker.
4. Enable boot startup: sudo systemctl enable celery-worker.
5. Follow logs: sudo journalctl -u celery-worker -f.

After changing ExecStart, EnvironmentFile or dependencies, run daemon-reload and restart the service. Verify a reboot test in a maintenance window: after the VPS returns, both RabbitMQ and the worker should be active and a newly submitted task should be consumed.

Rank #4

Hurray for Hattie Rabbit: Story and pictures (An Early I can read book)

$23.87

Buy on Amazon

Secure RabbitMQ on a VPS

Celery’s security guidance treats an exposed broker as a serious risk. Anyone who can publish or consume messages may execute application tasks, read sensitive arguments or disrupt work. Apply these controls together:

• Restrict network access. Bind RabbitMQ to private interfaces where possible. In the VPS firewall, allow TCP port 5672 only from the application and worker networks. Do not leave AMQP open to the public internet.
• Use separate credentials. Give each application or environment its own RabbitMQ user and vhost. Remove unused users and rotate secrets on a schedule and after staff or host changes.
• Protect files. Limit read access to the environment file and any TLS private keys. Do not print the broker URL in debug logs or share it in issue reports.
• Use TLS where appropriate. Configure amqps and certificate validation when traffic crosses an untrusted or shared network. A TLS tunnel without server and certificate verification does not provide adequate authentication.
• Patch the stack. Keep the operating system, Celery, Kombu, Erlang and RabbitMQ current within versions supported by your application. Test upgrades before applying them to the production worker.
• Back up recovery data. Back up RabbitMQ definitions and any persistent data according to the recovery-time and recovery-point objectives for the application.

The RabbitMQ management interface, if enabled, should be restricted as carefully as AMQP. Monitoring access is not a reason to expose the broker or administration ports publicly.

Understand persistence, acknowledgements and queue types

What “persistent” does and does not promise

Celery’s documented default delivery mode is persistent, which asks the broker to write messages durably. End-to-end durability still depends on several components: durable queue and exchange declarations, publisher confirms, healthy broker disk, consumer acknowledgements and the retention policy of any result backend. A persistent message in a non-durable queue is not a complete recovery design.

Decide what should happen during each failure:

• If a producer loses its connection before a publish is confirmed, can it safely retry?
• If a worker dies during a task, should the task be redelivered, discarded or reconciled by an idempotency check?
• How long should completed results remain available, and where are they backed up?
• What queue depth or disk watermark should page an operator?

Classic queues versus quorum queues

Celery supports RabbitMQ quorum queues by setting x-queue-type to quorum and enabling publisher confirms with:

broker_transport_options = {"confirm_publish": True}

Quorum queues are a deliberate architecture choice, not a universal performance switch. They can improve failure tolerance through replicated queue state, but they add operational complexity and have Celery-specific limitations. The current Celery guidance warns that quorum queues disable global QoS and that autoscaling and prefetch-count reduction behavior have limitations when quorum queues are detected.

Choose based on failure tolerance, throughput, operational skill and Celery feature compatibility. Test queue declaration and recovery in a non-production environment before changing an existing queue type.

Monitor the broker and workers

Automatic restart is not a substitute for monitoring. A worker can be running while tasks fail, a queue can grow while consumers are alive, or RabbitMQ can throttle publishers because of memory or disk alarms.

Best Value

Clojure in Action

$58.99

Buy on Amazon

Minimum checks

• Worker service state: systemctl status celery-worker.
• Worker logs and exception rates: journalctl -u celery-worker.
• RabbitMQ service state: systemctl status rabbitmq-server.
• Queue depth, consumers, message rates and unacknowledged messages using RabbitMQ’s monitoring tools.
• RabbitMQ memory and disk alarms, plus free space on the VPS.
• Task latency, retry counts and permanently failed tasks in the application or result backend.

Alert on sustained queue growth rather than a single transient message. Correlate queue depth with worker count, task duration and broker resource alarms so you can distinguish a slow workload from a disconnected consumer.

Stop services cleanly

For maintenance, stop producers first when possible, allow workers to finish or revoke work according to your task policy, and then stop the broker cleanly. RabbitMQ documents rabbitmqctl stop for a clean shutdown. Automatic restart helps recovery but does not replace an operator’s review of why the service stopped.

Diagnose common failures

“Connection refused” or repeated broker retries

• Check systemctl status rabbitmq-server and its journal.
• Confirm the hostname and port in CELERY_BROKER_URL.
• From a remote worker, verify that the firewall permits port 5672 only along the intended private path.
• Check that the RabbitMQ listener is bound to an interface reachable from the worker.

Authentication or “access refused” errors

• Verify the username and password without exposing the secret in logs.
• Confirm the vhost exists and is spelled exactly as in the URL.
• Check permissions with rabbitmqctl list_permissions -p myvhost.
• URL-encode reserved characters in credentials or vhost names.

Tasks publish but never execute

• Confirm the worker is attached to the same broker URL and virtual host as the producer.
• Read the worker startup log to verify that the expected task module was imported.
• Compare the queue and routing configuration used by the producer and worker.
• Inspect queue consumers and unacknowledged messages. A growing ready count usually means no available consumer or insufficient worker capacity.

Tasks run twice or remain incomplete after a crash

Review acknowledgement timing and task side effects. A redelivered message is possible when a worker disappears before acknowledging. Make the operation idempotent, record a unique job key, and reconcile partial work rather than assuming one delivery equals one execution.

The worker starts manually but not after reboot

• Run sudo systemctl enable celery-worker.
• Check that WorkingDirectory, ExecStart and the virtual-environment path are absolute and valid.
• Confirm the environment file is readable by User=myapp.
• Inspect journalctl -u celery-worker -b after reboot.
• Verify the unit was reloaded with systemctl daemon-reload after edits.

When RabbitMQ is the right Celery broker

RabbitMQ is a strong default when you need a mature AMQP broker, durable delivery controls, queue monitoring and Celery remote control on infrastructure you operate. Celery’s broker overview marks RabbitMQ and Redis as stable. Amazon SQS is also marked stable, but Celery’s overview notes that it lacks Celery monitoring and remote-control support.

There is no universal winner without workload data. Compare delivery and persistence behavior, monitoring, remote control, managed-service availability, operational burden and message-size profile. For a self-managed VPS where those Celery controls matter, RabbitMQ is usually the most direct fit.

Production readiness checklist

• RabbitMQ is installed, enabled and healthy.
• A dedicated user and vhost exist; the default guest account is not used for remote production access.
• The broker URL is stored in protected configuration, not source control.
• The firewall limits AMQP access to trusted application and worker networks.
• TLS is used and certificates are validated when traffic crosses an untrusted network.
• Tasks use serializable arguments, bounded external calls and idempotent side effects.
• The worker runs as a non-root account under systemd with an explicit working directory.
• The unit is enabled for boot, has a restart policy and has been tested after reboot.
• Queue depth, message rates, unacknowledged tasks, worker failures, memory alarms and disk alarms are monitored.
• Durability choices cover queues, exchanges, publisher confirms, acknowledgements, broker storage and result retention.
• Definitions, persistent data and application state are backed up according to recovery objectives.

Final architecture

The smallest dependable arrangement is a web process that publishes to RabbitMQ through an AMQP URL, a dedicated RabbitMQ user and vhost, and a Celery worker managed by systemd. Secure the broker at the network boundary, keep secrets out of code, make task effects safe to repeat, and monitor both queue behavior and broker health. Add quorum queues, TLS, multiple workers or a separate broker host only when a stated failure, security or capacity requirement justifies the additional configuration.

Quick Recap

SaleBestseller No. 1

RabbitMQ in Action: Distributed Messaging for Everyone

$24.98

Bestseller No. 2

RabbitMQ With C#

$19.95

SaleBestseller No. 3

The World of Peter Rabbit (The Original Peter Rabbit, Books 1-23, Presentation Box)

$57.71

Bestseller No. 4

Hurray for Hattie Rabbit: Story and pictures (An Early I can read book)

$23.87

Bestseller No. 5

Clojure in Action

$58.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.