Drupal 7Queues

Drupal Queues

New to Drupal 7

First in first out data structure

Used internally by Drupal

Uses OO principles

Fully customisable

Why Use Queues?

Batch processing of large amounts of data/items

Delay processing of complex calculations so system load is more stable

Sequential processing of items

Preventing API service black listing

Drupal 7 Implementations

Batch API

Cron

Aggregator Module

Update Module

Drupal Queues

Found in the file:

/modules/system/system.queue.inc

Classes are used to wrap queue functionality

Reliable vs Non-reliable

Reliable

Generally kept in database table

Every item will be executed at least once

Items exist over several requests

If the request fails then queue remains intact

Non-reliable

Generally kept in memory only

All items might exist in a single request

No guarantee that all queue items will be executed

No guarantee that all items will be executed in order

If request fails then queue might be lost

Drupal Queue Classes

DrupalQueue

Single static method get()

DrupalQueue::get('my_queue');

Returns a queue object of a given name and a given type (reliable or non-reliable)

Force reliable queue by passing TRUE as second parameter

Default returned class is SystemQueue

Essentially a queue object factory

SystemQueue

Example of a reliable queue class

Implements DrupalReliableQueueInterface

Uses the database table queue to store and retrieve the queue

Items are 'leased' to ensure no two processes get the same queue item

Default class for new queues

queue Table

Created at Drupal install

MemoryQueue

Example of an non-reliable queue class

All queue items are stored in memory

$queue parameter in class contains the queue

Also implements item leasing

System Variables

Three system variables are used by DrupalQueue to decide what sort of object to return

'queue_class_' . $nameDefault is NULL

queue_default_classDefault is SystemQueue

queue_default_reliable_classDefault is SystemQueue

Using Drupal Queues

Create A Queue

$queue = DrupalQueue::get('my_queue', TRUE);

$item = array('dataitem1' => 'something','int' => 123);

$queue->createItem($item);

echo $queue->numberOfItems(); // 1

Create A MemoryQueue

variable_set('queue_default_class', 'MemoryQueue');

$queue = DrupalQueue::get('my_queue');

$item = array('dataitem1' => 'something','int' => 123);

$queue->createItem($item);

echo $queue->numberOfItems(); // 1

Get Item From Queue

$queue = DrupalQueue::get('my_queue');

$got_item = $queue->claimItem();

echo $got_item->data['dataitem1'];

Retrieved Item Structure

stdClass Object( [data] => Array ( [dataitem1] => something [qwe] => 123 )

[item_id] => 89)

Change Lease Length

Can pass the lease time (in seconds) to claimItem()

$got_item = $queue->claimItem(100);

Release Or Delete

releaseItem() resets the lease time

$queue->releaseItem($got_item);

deleteItem() removes item from queue

$queue->deleteItem($got_item);

Customizing

Create custom queue class

Create variable called queue_class_ . $name

Value is the class name of your queue class

Call DrupalQueue::get() with the variable $name as a string

Customizing

Reliable class MUST implement DrupalReliableQueueInterface

Can also extend SystemQueue

All Queue classes should at least implement DrupalQueueInterface

Using Custom Classes

variable_set('queue_class_mycustom', 'MyCustomQueueClass');

$queue = DrupalQueue::get('mycustom');

Stack Class

Last in first out implementation

class Stack extends SystemQueue {

public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name ORDER BY created DESC', 0, 1, array(':name' => $this->name))->fetchObject();// ...

EventQueue Class

Required as part of the NWDUG website

Any items added would not be available for 45 minutes to give a period of grace after creating an event node

class EventQueue extends SystemQueue {

public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name AND created >= UNIX_TIMESTAMP(DATE_ADD(NOW(), INTERVAL 2700 SECOND)) ORDER BY created ASC', 0, 1, array(':name' => $this->name))->fetchObject();

WatchdogSystemQueue

Extends SystemQueue and creates a log every time anything is done

Maintains SystemQueue functionality

WatchdogSystemQueue

class WatchdogSystemQueue extends SystemQueue {

public function __construct($name) { watchdog('queue', '%name queue created', array('%name' => $name)); parent::__construct($name); }

public function createItem($data) { watchdog('queue', 'Item created : %data', array('%data' => print_r($data, TRUE))); parent::createItem($data); }

public function claimItem($lease_time = 3600) { $return_value = parent::claimItem($lease_time); watchdog('queue', 'Item claimed %item (lease time = %lease)', array('%item' => print_r($return_value, TRUE),'%lease' => $lease_time)); return $return_value; }//....

RandomMemoryQueue

Extends the MemoryQueue class

Items are added in order but are retrieved in random order

RandomMemoryQueue

class RandomMemoryQueue extends MemoryQueue {

public function claimItem($lease_time = 30) { $available_items = array(); // Extract the remining available items foreach ($this->queue as $key => $item) { if ($item->expire == 0) { $available_items[] = $item; } } // Randomly select one (if available) if (count($available_items) > 0) { $queue_length = count($this->queue); $rand_item = rand(0, $queue_length - 1); $item = $available_items[$rand_item]; $item->expire = time() + $lease_time; return $item; }

return FALSE; }}

Creating And Destroying

DrupalQueueInterface has two methods available for creating and destroying the queue

createQueue()Should be called within an install hook

deleteQueue()Should be called within an uninstall hook

Tips

Unless you really need to rewrite the entire class it is best to extend SystemQueue or MemoryQueue

For open source projects try to keep the same retrieved item structure as the system queues

No checks for unique items in default system queues

Resources

Drupal Queues APIhttp://api.drupal.org/api/drupal/modules--system--system.queue.inc/group/queue/7

http://bit.ly/kuphnc

Queue UI Modulehttp://drupal.org/project/queue_ui

Source code is well documented/modules/system/system.queue.inc

Full write up of this talk on #! codehttp://www.hashbangcode.com/

Questions?

$queue = DrupalQueue::get('questions');

$question = $queue->claimItem();

echo $question->data['question'];

Bloghttp:///www.norton42.org.uk/

Twitter@philipnorton42

#! codehttp://www.hashbangcode.com/

#! code on Twitter@hashbangcode

Philip Norton

Philip Nortonwww.hashbangcode.com

#!