asynchronous task queues with celery
TRANSCRIPT
Kishor Kumar MahatoPython Lead at Eagle Vision IT
Python User Group Nepal Meetup #72nd August 2015
CeleryAsynchronous Task Queues
Distributed Asynchronous Task Queue For Python
It is a task queue which focus on real-time processing
It also supports task scheduling
• Anything that needs to run asynchronously, eg. outside of the request-response cycle.
• Background computation of ‘expensive queries’
• Interactions with external API’s (eg. Twitter)
• Periodic tasks (instead of cron & scripts)• Long-running actions with results
displayed via AJAX.
Potential Uses
Work Flow
Celery Workers Run at least 1 celery worker server
( Redis / RabbitMQ)
Can be on different machines
Celery guarantees that tasks are only executed once
Taskfrom celery.task import Task
from celery.registry import tasks
class FetchUserInfoTask(Task):
def run(self, screen_name, **kwargs):
logger = self.get_logger(**kwargs)
try:
user = twitter.users.show(id=screen_name)
logger.debug("Successfully fetched {0}".format(screen_name))
except TwitterError:
logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError))
raise
return user
tasks.register(FetchUserInfoTask)
>>> from myapp.tasks import FetchUserInfoTask
>>> result = FetchUserInfoTask.delay(“kishor_14”)
Run It!
Periodic Tasks from celery.task import PeriodicTask
from celery.registry import tasks
from datetime import timedelta
class FetchMentionsTask(Task):
run_every = timedelta(seconds=60)
def run(self, **kwargs):
logger = self.get_logger(**kwargs)
mentions = twitter.statuses.mentions()
for m in mentions:
ProcessMentionTask.delay(m)
return len(mentions)
tasks.register(FetchMentionsTask)
Periodic Tasksfrom datetime import timedelta
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'order-reminder-every-1-hours': {
'task’: 'application.views.cron_job_for_application_processing',
'schedule': crontab(minute=0, hour='*/1')
# 'schedule': timedelta(seconds=30)
},
}
CELERY_TIMEZONE = 'UTC'
Support
http://groups.google.com/group/celery-users/
http://celery.readthedocs.org/en/latest/index.html
