Celery for background task processing and deferred executionPiyush Kumar & Konark Modi

Show of Hands : How many of us• Have used or know about celery / djcelery .• Have worked with queues .

Setup details :System config. we'd be using:

Centos 6.2 : Hosted on Virtual boxPython version : Python 2.6.6MySQLDjango-1.4

Packages installed :1. Virutal env : pip install virtualenv2. RabbitMQ 3. Celery : pip install -U celery4. Django-Celery : pip install django-celery5. MySQLdb module for python : easy_install MySQL , yum

install mysql-devel python-devel6. pytz : pip install pytz

Tutorial sections: • Understanding the basics – Level 0• Getting started with celery – Level 1• Next Steps with Celery – Level 2• Recap of what we’ve discussed till now• Q & A

Understanding the basics – Level 0• Need of job queues• What is celery ?• Understanding AMQP (Advanced Message Queueing

Protocol )• Choosing a broker : Why RabbitMQ ?• Moving to console : Understanding RabbitMQ better.

Getting started with Celery – Level 1• Celery architecture• Setting up celery : Installation of all the components• Hands-On : A simple application• How are we using Celery• Integrating Celery with a Web-Framework : Django using

djcelery• Django-celery architecture• Hands-On : Web application using celery

Next steps with Celery – Level 2• Adding multiple worker• Retrying tasks• Routing to multiple queues• Monitoring• Security

Lets get started now ☺Understanding the basics – Level 0

Need of Job Queues:• For executing tasks asynchronusly.• Distributed tasks.• Scheduling.• Retries.• Handling errors.• Dynamic work load.• Enhancing user experience.• Some of the use cases include:

• Clickstream ( No. of hotels being watched, likes etc.)• Generate graphs based on a large data set for display on a site’s home page / Dashboard

every 15 minute• Admin applications like

• Thumbnails generation • Re-generate static CSS files by examining when certain items in the admin have been modified

• Connecting to third-party APIs• Mass e-mailers

What is Celery ?• Asynschronus task queue/job queue.• Uses distributed message passing.• Supports both real-time processing + schedule jobs.• Tasks can be run concurrently on a single or more worker

servers taking advantage of multiprocessing.

AMQP (Advanced Message Queueing Protocol )• Open standard application layer protocol for message-

oriented middleware• Features of AMQP

• Message orientation,• Queuing,• Routing (including point-to-point and publish-and-subscribe),• Reliability • Security• Interoperable: AMQP mandates the behaviour of the messaging

provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems.

Choosing a broker : Why RabbitMQ ?

Moving to console : Understanding RabbitMQ better

Getting started with celery – Level 1

Installing Celery :• RabbitMQ (http://www.rabbitmq.com/download.html )• pip install -U celery

A simple application• Create tasks.py• Run celeryd and understand its various options• Configure the result backend• Fetch the result• Call the task using file, python interpreter

• Delay()• Apply_async() : Expiration, countdown

• Periodic tasks

Our use cases:• Poll approx 300+ DB queries every 15 minutes.• Custom metric collection• Inventory management system :

• Asynchronus web-interface• Periodic tasks

Integrating celery with a web-framework : Django with djcelery• Installation• Understanding the DB schema djcelery uses to store

tasks/result sets• Writing a simple web-app

Next steps• Adding multiple workers• Retrying tasks• Monitoring various components of celery:

• Exploring various components of our celery setup.• Setting up celery-cam Monitoring RabbitMQ

• Running celeryd as a daemon in production• Security :

• Signals , Tracking of tasks as they transition through different states, and inspecting return values.

• Message signing.

Recap of what we’ve discussed till now

Thanks : Q & A