Download - Celery vs MRQ
Celery -> MRQ
LeadCooker2016
Task Queue
LeadCooker2016
Celery
LeadCooker2016
is good
Celery
LeadCooker2016
is bad
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
GeventRedis
MongoDB
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
• mrq-run • mrq-worker • mrq-dashboard
MRQ (task)
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.task import Task import urllib2
class Fetch(Task):
def run(self, params):
with urllib2.urlopen(params["url"]) as f: t = f.read() return len(t)
MRQ (enqueue)
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.context import set_current_config from mrq import config from mrq.job import queue_job
cfg = config.get_config( parser=None, config_type="run", sources=("file", "env") ) set_current_config(cfg)
queue_job('leadcooker.tasks.Fetch',[])
MRQ (schedule)
LeadCooker2016
https://github.com/pricingassistant/mrq
SCHEDULER_TASKS = [
# [every day]: Check trial user { "path": "leadcooker.tasks.CheckTrialUsersTask", "params": {}, "interval": 24 * 3600 },
# [every day]: Check expired campaigns { "path": "leadcooker.tasks.CheckExpiredCampaignsTask", "params": {}, "interval": 24 * 3600 },
# [every 6 hours]: Update mail answers/undelivers { "path": "leadcooker.tasks.UpdateAllMailStatusTask", "params": {}, "interval": 6 * 3600 } ]
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
• Integrated dashboard • gevent worker • Job management • IO monitoring • Greenlet tracing • Builtin scheduler • Supervisord integration • Integrated memory leak debugger
Benefits
MRQ
LeadCooker2016
is not good enough
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": rawparam } } }, }
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
/!\ MRQ is not yet ready for public use. Soon!
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": rawparam } } }, }
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.job import queue_raw_jobs import time
queue_raw_jobs("leadcooker_timed_set", { 'param': time.time()+3600, })
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.job import queue_raw_jobs import time
queue_raw_jobs("leadcooker_timed_set", { '{"lead_id": 234}': time.time()+3600, })
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": ujson.loads(rawparam) } } }, }
MRQ
LeadCooker2016
OR
• python-rq • rq-scheduler • rq-gevent-worker • rq-dashboard
