python - how to configure and run celery worker on remote system

Question

i am working on celery and using rabbitmq server and created a project in django project in a server(where message queue,database exists) and it is working fine, i have created multiple workers also

from kombu import Exchange, Queue
CELERY_CONCURRENCY = 8

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

CELERY_RESULT_BACKEND = 'amqp'
CELERYD_HIJACK_ROOT_LOGGER = True
CELERY_HIJACK_ROOT_LOGGER = True
BROKER_URL = 'amqp://guest:guest@localhost:5672//'

CELERY_QUEUES = (
  Queue('default', Exchange('default'), routing_key='default'),
  Queue('q1', Exchange('A'), routing_key='routingKey1'),
  Queue('q2', Exchange('B'), routing_key='routingKey2'),
)
CELERY_ROUTES = {
 'my_taskA': {'queue': 'q1', 'routing_key': 'routingKey1'},
 'my_taskB': {'queue': 'q2', 'routing_key': 'routingKey2'},
}


AMQP_SERVER = "127.0.0.1"
AMQP_PORT = 5672
AMQP_USER = "guest"
AMQP_PASSWORD = "guest"
AMQP_VHOST = "/"`


CELERY_INCLUDE = ('functions')

`

but i want to run workers from another server.so i need some information regarding how to run a worker in another system when i referred few sites it is saying that we need to run the django project on the remote system also is it necessary?

Answer 1

Here is the gist of the idea:

On Machine A:

1. Install Celery & RabbitMQ.
2. Configure rabbitmq so that Machine B can connect to it.
3. Create my_tasks.py with some tasks and put some tasks in queue.

On Machine B:

1. Install Celery.
2. Copy my_tasks.py file from machine A to this machine.
3. Run a worker to consume the tasks

I had the same requirement and experimented with celery. It is a lot easier to do that. I wrote a detailed blog post on that few days back. Check out how to send tasks to remote machine?