Why Task Queues inPython, Ruby and More!

~ a talk by Bryan Helmig

task queue noun \ˈtask ˈkyü\a system for parallel execution of discrete tasks in a non-blocking fashion.

celery, resque, or home-grown...

broker noun \broh-ker\the middle man holding the tasks (messages) themselves.

rabbitmq, gearman, redis, etc...

producer noun \pruh-doo-ser\the code that places the tasks to be executed later in the broker.

your application code!

consumer noun \kuhn-soo-mer\also called the worker, takes tasks fromthe broker and perform them.

usually a daemon under supervision.

imagine this...your app: posting a new message triggers “new message” emails to all your friends.

def new_message(request): user = get_user_or_404(request) message = request.POST.get('message', None)

if not message: raise Http404

user.save_new_message(message)

for friend in user.friends.all(): friend.send_email(message)

return redirect(reverse('dashboard'))

def new_message(request): user = get_user_or_404(request) message = request.POST.get('message', None)

if not message: raise Http404

user.save_new_message(message)

for friend in user.friends.all(): friend.send_email(message)

return redirect(reverse('dashboard'))

the problem:that works good for like 0-6 friends... but what if you have 100,000? or more? will your user ever get a response?

bad idea #1: ignoremake your users wait through your long request/response cycle.

your users are important, right?

bad idea #2: ajaxreturn the page fast with JS code that calls another script in the browser’s background.

duplicate calls & http cycles are not cool.

bad idea #3: cronjobmake a email_friends table with user_id & message column. cron every 5/10 minutes.

backlogs will destroy you.

good idea: queuesthe task to potentially email thousands of users is put into a queue to be dealt with later, leaving you to return the response.

@taskdef alert_friends(user_id, message): user = User.objects.get(id=user_id)

for friend in user.friends.all(): friend.send_email(message)

def new_message(request): user = get_user_or_404(request) message = request.POST.get('message', None) if not message: raise Http404 user.save_new_message(message)

alert_friends.delay(user.id, message)

return redirect(reverse('dashboard'))

adding a task to a queue should be faster than performing the task itself.

RULE #1:

you should consume tasks faster than you produce them. if not, add more workers.

RULE #2:

queue libraries:1. celery (python) earlier...

2. resque (ruby) up next...

3. build your own grand finalé!

class MessageSend def self.perform(user_id, message) user = User.find(user_id) user.friends.each do |friend| friend.send_email(message) end endend

class MessageController < ActionController::Base def new_message render(file: 'public/404.html', status: 404) unless params[:message] current_user.save_new_message(params[:message])

Resque.enqueue(MessageSend, current_user.id, params[:message])

redirect_to dashboard_url endend

let’s build our own!and let’s do it with redis, in php and

in less than 50 lines of code!(not including the task code)

quick redis aside:so, redis is a key-value store. values can be strings, blobs, lists or hashes. we’ll use lists and RPUSH and LPOP.

step #1: task runneri’m envisioning this as a Task class with various methods named after each task.each method accepts an array $params.

class Tasks { public function email_friends($params) { $user_id = $params->user_id; $message = $params->message;

# get $friends from a db query... $friends = array('paul@example.com', 'john@example.com');

foreach ($friends as $friend) { echo "Fake email ".$friend. " with ".$message."\n"; } }}

step #2: workeri think an infinite loop with a blocking redis BLPOP will work. then it needs to run the right method on the Task object.

include_once 'redis.php'; # sets up $redis, $queueinclude_once 'tasks.php';

$tasks = new Tasks();

while (true) { # BLPOP will block until it gets an item.. $data = $redis->blpop($queue, 0); $obj = json_decode($data[1]); # grab json...

$method = $obj->task_name; $params = $obj->params;

# calls the task: Task->$task_name($params)... call_user_func(array($tasks, $method), $params);}

step #3: run it!we will need to run the worker:

➜ php worker.php

step #4: producerlet’s make a handy add_task function for easy placement of tasks into the queue.

include_once 'redis.php'; # sets up $redis & $queue

function add_task($task_name, $params) { global $redis, $queue; $data = Array('task_name' => $task_name, 'params' => $params); $json = json_encode($data) $redis->rpush($queue, $json);}

# an example of our task api...add_task('email_friends', array('user_id' => 1234, 'message' => 'I just bought a car!'));

step #5: watch ita command line that runs a worker is standard. use something like supervisord or god to run it & monitor it.

things missing:our php example does not: store return results, handle errors, route tasks, degrade gracefully, log activity, etc...

the endquestion time!