celery vs mrq

Celery -> MRQ

LeadCooker2016

Task Queue

LeadCooker2016

Celery

LeadCooker2016

is good

Celery

LeadCooker2016

is bad

queues.io

LeadCooker2016

RestMQ Huey Zaqar RQ Nats

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

GeventRedis

MongoDB

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

• mrq-run • mrq-worker • mrq-dashboard

MRQ (task)

LeadCooker2016

https://github.com/pricingassistant/mrq

from mrq.task import Task import urllib2

class Fetch(Task):

def run(self, params):

with urllib2.urlopen(params["url"]) as f: t = f.read() return len(t)

MRQ (enqueue)

LeadCooker2016

https://github.com/pricingassistant/mrq

from mrq.context import set_current_config from mrq import config from mrq.job import queue_job

cfg = config.get_config( parser=None, config_type="run", sources=("file", "env") ) set_current_config(cfg)

queue_job('leadcooker.tasks.Fetch',[])

MRQ (schedule)

LeadCooker2016

https://github.com/pricingassistant/mrq

SCHEDULER_TASKS = [

# [every day]: Check trial user { "path": "leadcooker.tasks.CheckTrialUsersTask", "params": {}, "interval": 24 * 3600 },

# [every day]: Check expired campaigns { "path": "leadcooker.tasks.CheckExpiredCampaignsTask", "params": {}, "interval": 24 * 3600 },

# [every 6 hours]: Update mail answers/undelivers { "path": "leadcooker.tasks.UpdateAllMailStatusTask", "params": {}, "interval": 6 * 3600 } ]

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

• Integrated dashboard • gevent worker • Job management • IO monitoring • Greenlet tracing • Builtin scheduler • Supervisord integration • Integrated memory leak debugger

Benefits

MRQ

LeadCooker2016

is not good enough

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": rawparam } } }, }

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

/!\ MRQ is not yet ready for public use. Soon!

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": rawparam } } }, }

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

from mrq.job import queue_raw_jobs import time

queue_raw_jobs("leadcooker_timed_set", { 'param': time.time()+3600, })

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

from mrq.job import queue_raw_jobs import time

queue_raw_jobs("leadcooker_timed_set", { '{"lead_id": 234}': time.time()+3600, })

MRQ

LeadCooker2016

https://github.com/pricingassistant/mrq

RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": ujson.loads(rawparam) } } }, }

MRQ

LeadCooker2016

OR

• python-rq • rq-scheduler • rq-gevent-worker • rq-dashboard

Alexander Lifanov

LeadCooker2016

lifanov.a.v@gmail.com