Chains, Groups and Chords in Celery
If you have used Celery with Django before then this tutorial will help you understand a few important concepts. If Celery is new to you , I suggest you check out their official documentation first just to get familiar with it.
We are going to use this example task in our examples
@app.task def add(x, y): return x + y
When we chain tasks together, the second task will take the results of the first task as its first argument, for example
from celery import chain res = chain(add.s(1, 2), add.s(3)).apply_async()
In the above example, you can notice the second task has only one argument , this is because the return value of the first task which in our example is 3 will be the first argument of the second task , the second task will now look like this
Hope this image below helps you understand it better
Notice the first argument of second task is 3 and the first argument of third task is 6 which is the result of the second task.
Another way of chaining tasks is using this syntax with pipe symbol ( " | " ).
res2 = (add.s(1, 2) | add.s(3)).apply_async()
The above code is just the same as this code below, you can choose which one you prefer.
res2 = chain(add.s(1, 2), add.s(3)).apply_async()
Celery documentation says:
Groups are used to execute tasks in parallel. The group function takes in a list of signatures.
Example code :
>>> from celery import group >>> from tasks import add >>> job = group([ ... add.s(2, 2), ... add.s(4, 4), ... ]) >>> result = job.apply_async() >>> result.ready() # have all subtasks completed? True >>> result.successful() # were all subtasks successful? True >>> result.get() [4, 8 ]
Example image to illustrate how groups work
A Chord usually has two parts, header and callback
The syntax looks like this :
The header here is simply a group of tasks, the callback is run or executed after the groups of tasks have finished.
For us to demonstrate how chord works we will need a function which will act as our callback.
We will call our callback function tsum, it takes in a list of numbers and adds them up.
@app.task def tsum(numbers): return sum(numbers)
>>> callback = tsum.s() >>> header = [ add.s(2, 2), add.s(4, 4) ] >>> result = chord(header)(callback) >>> result.get() 12
The above code is similar to this mathematical expression:
∑ ((2 + 2) + (4 + 4))
Image below illustrates how chords work
Celery documentation says we should avoid using chords as much as possible, but you can still use it if you really have to, read more on why you should not use chords on the Celery documentation.
Canvas: Designing Work-flows
How to Use Celery and RabbitMQ with Django
Header photo by @vivekdoshi on unsplash
- Django Class Based Views vs Function Based Views
- What is super() in python