In this tutorial, we are going to look at how to Use Django and Celery with RabbitMQ as our broker. we will cover how to install celery and RabbitMQ and a simple example to demonstrate its importance in a Django project.

Common Use Case of Celery

Sometimes you can have a function that takes a long time to finish, let's say it takes 10 seconds to finish, an example could be a function that filters spam from comments. Instead of making your users wait, you could direct them to the next page immediately and give the heavy task of finding bad words in users comments to celery to be processed later. Celery receives the task and processes it in the background, This way, the heavy task will not make users wait before proceeding to the next page.

Other examples of tasks that can take a long time include sending emails to users, imagine a case where you have to send emails to 500 people, that can take some time, Another example is sending push notifications, taking backups, resizing and editing images this can also take a long time especially if the images are of high resolution.

Terms

Task queue

For celery to work it needs to record tasks to a database and then process these tasks one by one, there are many tasks queues which include the following RabbitMQ, Redis, MongoDB and CouchDB. These tasks to be executed are distributed by a broker (e.g RabbitMQ ) to different workers which will then process these tasks asynchronously.

Producing

Producing has the same meaning as sending. The function or program that sends messages can be called a producer.

Consuming

Consuming means to receive a message. It is a program that receives messages.

Task

Task queue's input is a unit of work called a task. This can be your python function that takes time to finish (e.g a function that removes spam from comments or a function that resizes images ) and the one we outsource to celery.

Worker

Workers can be one or multiple, they constantly check the database (e.g rabbitMQ) and find tasks that should be executed, and when it finds a task it will process it.

Installation of Celery on Linux

You can install Celery by using the following command.

pip install Celery

Installation of RabbitMQ on Linux

Since RabbitMQ is written in erlang programming language, we have to install it first before installing RabbitMQ

apt-get install -y erlang

apt-get install rabbitmq-server

To enable RabbitMQ use this command below

systemctl enable rabbitmq-server

Start RabbitMQ server with the following command

systemctl start rabbitmq-server

You can also check the status RabbitMQ server

systemctl status rabbitmq-server 

Celery and Django demo

We will use a simple Django project to show how celery works. you can find that demo project on GitHub here django celery example

We need to setup CELERY_BROKER_URL , inside our settings.py we have

CELERY_BROKER_URL = 'amqp://guest:guest@localhost' 

we created a celery.py file next to settings.py file

#celery.py

import os
from celery import Celery



# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django apps.
app.autodiscover_tasks()




@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

We edited __init__ file which is found next to settings.py file and added the following in it

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

Celery Task Example

Create tasks.py file next to models.py file, inside our task.py file we added the following

# Create your tasks here
from app.models import Comment
from celery import shared_task
import time

@shared_task   
def spam_filter(comment_id):
    # sleep 5 seconds
    time.sleep(5)
    comment = Comment.objects.get(pk=comment_id)
    bad_words = ['stupid', 'idiot', 'loser']
    is_spam = True if any(word in comment.comment for word in bad_words) else False 
    if is_spam:
        comment.delete()   
    return is_spam    

The above task is used to remove comments that contain bad words, this is done asynchronously which means users don't have to wait for this filtering to finish, users continue to the next page immediately and celery will handle the process of removing bad comments.

In our case, the words that we are targeting are 'stupid', 'idiot', 'loser' . If those words are in a particular comment, we are going to delete that comment. We assume that this process will take 5 seconds to finish. So if a user posts a comment with bad words, that comment will be deleted after 5 seconds. we have also used Python's Ternary Operator

# our ternary operator 
is_spam = True if any(word in comment.comment for word in bad_words) else False 

This returns True if bad words are found in a particular comment and False if the comment does not have bad words. To learn more about Python's Ternary Operator click here . This Ternary Operator helps us condense if-else statements into one line.

We have also used Python any() function , learn more about this function here python any function tutorial .

Our models.py file looks like this:

from django.db import models


class Comment(models.Model):
    comment = models.TextField()
    pub_date = models.DateTimeField(auto_now_add=True)

our forms.py

from django import forms
from app.models import Comment



class CommentForm(forms.ModelForm):

    class Meta:
        model = Comment
        fields = ['comment',] 

views.py file

from django.shortcuts import render, redirect
from app.forms import CommentForm
from app.models import Comment
from app import tasks



def home(request):
    comments = Comment.objects.all()
    if request.method == 'POST':
        form = CommentForm(request.POST)
        if form.is_valid():
            comment = form.save()
            # Check spam asynchronously.
            tasks.spam_filter.delay(comment_id=comment.id)
            return redirect('home')
    else:
        form = CommentForm() 

    return render(request, "app/home.html", {'form': form, 'comments': comments})

Take a look at this line tasks.spam_filter.delay(comment_id=comment.id) in the above code, we have used .delay() method which comes from celery, this means that we are delaying the function and calling it asynchronously. This means the code doesn't wait until it has the result.

Starting the worker

We can now start Celery worker using the command below

$ celery -A project worker -l INFO

After starting the worker you will see something like

Also, make sure you have started the Django development server, so far we have

• rabbitmq-server running on localhost using the default ports.
• We have started the worker using celery -A project worker -l INFO
• Django development server is running on localhost on port 8000, http://127.0.0.1:8000/

For Celery to work, we must have the above three things running.

The homepage of our Django app looks like this

We have a funny looking image and a comment box, so if a user submits a comment that has bad words, in this case, the bad words are 'stupid', 'idiot', ' loser, the comment will be deleted after 5 seconds but the user does not have to wait for celery to finish filtering spam comments before moving to the next page. Celery does this process Asynchronously

Conclusion

In this tutorial, we covered how to set up and use Celery with Django on Linux. I did not include how to set up on Windows because at the moment I don't have access to Windows PC but you can find instructions online on how to do so.

We also used a simple Django example to demonstrate how celery works, we created a Celery task to check for people leaving abusive comments. Our spam filter is not perfect but the main goal of this tutorial was to show how celery works with Django.

Resources

• Header photo by Lagos Techie on unsplash
• rabbitmq tutorial on official website
• First steps with django using celery on official website