By

Os Shiowattana (@djshiow)

Who am I

● Os Shiowattana (@djshiow)● Software engineer for ~10 years● Currently at Origami Logic

○ We’re looking for great developers ;-)● Python experience: 3 months

○ Teaching is a good way to learn

Outline

● What is Celery?● What is it for?● Simple Example

http://www.myhdiet.com/sharons-raw-celery-soup/

What is celery?

● Task queue○ Distributed○ Asynchronous

● Create tasks and add it to the queue

● Workers pick them up & work on them

How does it work?

Broker

Worker

Worker

Worker

….

Producer

ResultStore

What is it for?

● Long running asynchronous task● Execute a task “reliably”

○ Asynchronous + Retries○ e.g. Interactions with external APIs

● Scheduling Periodic tasks

How does Origami Logic use it?

● Fetch marketing metrics from all your social network daily○ Periodic○ Reliable (with retries) / Traceable○ Distributed (multiple machines) & Scalable○ Rate limits

Instagram

● Populate follower feed

>7 M followers

Source: http://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-celery-and-rabbitmq.html

Simple Example

● Adding two numbers asynchronously !!● Setup:

○ Install RabbitMQ○ Install Celery

From: http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html

First, define Celery app & tasks from celery import Celery

#setup app

app = Celery('tasks', backend='amqp' , broker='amqp://guest@localhost//')

#Define task

@app.task

def add(x, y):

return x + y

Then, run celery worker

celery -A tasks worker --loglevel=info

Decorator to create a task class

Lastly, trigger the taskfrom tasks import add

#run the add task asynchronously 3 seconds later

#passing (4,4) as paramadd.apply_async((4, 4) , countdown=3)

[2015-03-15 14:34:10,397: INFO/MainProcess] Received task: tasks.add[b68b0109-7e99-473b-8991-1da8adaa50e8] eta:[2015-03-15 14:34:13.385448+00:00]

[2015-03-15 14:34:14,836: INFO/MainProcess] Task tasks.add[b68b0109-7e99-473b-8991-1da8adaa50e8] succeeded in 0.0209855769999s: 8

Reference● http://www.slideshare.net/idangazit/an-introduction-to-celery● http://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-

celery-and-rabbitmq.html● http://celery.readthedocs.org/en/latest/