background processing in ruby on rails
Post on 18-Oct-2014
12.764 views
DESCRIPTION
Background processing is an important tool in web development. Somethings just can't optimized enough for the normal request/response cycleof a web site and need to be run asynchronously. Ruby on Rails doesn't provideyou with any real out of the box solution. There are a lot of externaloptions available and this part of Rails is lacking any real conventionor standard.In this presentation, we will talk about how to choose and implement asolution that fits your needs. We will start with some basic optionsusing built in Rails tools and we will cover some of the more popularsolutions solutions out there such as BackgrounDRb, Background Job,Delayed Job, Workling and more.Rob Mack has been working with Rails professionally on and off since2005. Rob currently works for VitalSource Technologies.TRANSCRIPT
Background Processing
Rob MackAustin on Rails - April 2009
Thursday, April 30, 2009
Why Background Processing?
Thursday, April 30, 2009
Simple Background Processing
Thursday, April 30, 2009
script/runner
• script/runner lib/do_work.rb
• script/runner ‘Worker.do_work’
From RAILS_ROOT:
Thursday, April 30, 2009
Rake
• Episode #127 Railscasts
def call_rake(task,options={}) options[:rails_env] = RAILS_ENV args = options.map{|n,v| "#{n.to_s.upcase}='#{v}'"} system "rake #{task} #{args.join(' ')} &" end
Thursday, April 30, 2009
Daemons
http://daemons.rubyforge.org/
http://github.com/dougal/daemon_generator/
Thursday, April 30, 2009
Generate A Daemon
• script/generate daemon my_worker
create lib/daemons create script/daemons create lib/daemons/my_worker.rb create lib/daemons/my_worker_ctl create config/daemons.yml
Thursday, April 30, 2009
Do Some Work
$running = trueSignal.trap("TERM") do $running = falseend
while($running) do # Do some work sleep 10end
lib/daemons/my_worker.rb
Thursday, April 30, 2009
Kick It Off
lib/daemons/my_worker_ctl startlib/daemons/my_worker_ctl stop
Thursday, April 30, 2009
Spawn
http://github/tra/spawn
Thursday, April 30, 2009
Spawn - Do Some Work
def my_action flash[:notice] = "Processing ... " spawn do # do some processing end end
Thursday, April 30, 2009
Simple Queues
Thursday, April 30, 2009
ar_mailer
http://seattlerb.rubyforge.org/ar_mailer/
setup:
gem install ar_mailerar_sendmail --create-migrationar_sendmail --create-model
Thursday, April 30, 2009
ar_mailer Schema
create_table "emails" do |t| t.string "from" t.string "to" t.integer "last_send_attempt", :default => 0 t.text "mail", :limit => 16777215 t.datetime "created_on" end
Thursday, April 30, 2009
ar_mailer Setup
class MyMailer < ActionMailer::ARMailer
Mail Model:
environment.rbconfig.gem 'ar_mailer', :version => '1.3.1',
:lib => 'action_mailer/ar_mailer'
config.action_mailer.delivery_method = :activerecord
Thursday, April 30, 2009
Mail Something
MyMailer.deliver_newsletter @recipients
Thursday, April 30, 2009
The Worker Process
ar_sendmail --daemonize --max-age 0
Thursday, April 30, 2009
BackgrounDRb
http://backgroundrb.rubyforge.orghttp://github.com/gnufied/backgroundrb
Thursday, April 30, 2009
BackgrounDRb Setup
• sudo gem install chronic packet
• script/plugin install git://github.com/gnufied/backgroundrb.git
• rake backgroundrb:setup
• rake db:migrate
• edit config/backgroundrb.yml
• script/generate worker my_worker
Thursday, April 30, 2009
BackgrounDRb Schema
create_table "bdrb_job_queues" do |t| t.text "args" t.string "worker_name" t.string "worker_method" t.string "job_key" t.integer "taken" t.integer "finished" t.integer "timeout" t.integer "priority" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.datetime "archived_at" t.string "tag" t.string "submitter_info" t.string "runner_info" t.string "worker_key" t.datetime "scheduled_at" end
Thursday, April 30, 2009
Do Some Work
class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end
def do_work(user_id = nil) # do some work endend
def my_action flash[:notice] = "Processing ..." MiddleMan.worker(:my_worker).async_do_work(:arg => @user.id)end
Thursday, April 30, 2009
Persistent Work
def my_action MiddleMan(:my_worker).enq_do_work(:job_key => "unique_key")end
Thursday, April 30, 2009
Poll Status
class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end
def do_work(user_id = nil) # do some work register_status "almost done" endend
Thursday, April 30, 2009
Poll Status
def my_action @job_key = MiddleMan.new_worker( :worker => :my_worker, :job_key => "unique_key" ) MiddleMan.worker(:my_worker, @job_key). do_work(@user.id)end
def check_status @status = MiddleMan.worker(:my_worker, @job_key). ask_statusend
Thursday, April 30, 2009
The Worker Process
script/backgroundrb start
Thursday, April 30, 2009
BJ - Background Job
http://codeforpeople.rubyforge.org/svn/bj
Thursday, April 30, 2009
Bj - Setup
script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
script/bj setup
gem install bjrequire ‘bj’
bj setup
Thursday, April 30, 2009
create_table "bj_job", :primary_key => "bj_job_id" do |t| t.text "command" t.text "state" t.integer "priority" t.text "tag" t.integer "is_restartable" t.text "submitter" t.text "runner" t.integer "pid" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.text "env" t.text "stdin" t.text "stdout" t.text "stderr" t.integer "exit_status" end
Bj Schema
Thursday, April 30, 2009
create_table "bj_job_archive", :primary_key => "bj_job_archive_id" do |t| t.text "command" t.text "state" t.integer "priority" t.text "tag" t.integer "is_restartable" t.text "submitter" t.text "runner" t.integer "pid" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.datetime "archived_at" t.text "env" t.text "stdin" t.text "stdout" t.text "stderr" t.integer "exit_status" end
Bj Schema
Thursday, April 30, 2009
Do Some Work
def my_action flash[:notice] = "Processing ..." Bj.submit './script/runner /lib/do_some_work.rb'end
def my_other_action flash[:notice] = "Processing ..." Bj.submit 'echo foobar', :tag => 'sample job'end
Thursday, April 30, 2009
The Worker Process
bj run --forever --rails_env=development
Thursday, April 30, 2009
Delayed Job
http://github.com/tobi/delayed_job/
Thursday, April 30, 2009
Delayed Job Setup
• script/plugin install git://github.com/tobi/delayed_job.git
• create a delayed jobs table
Thursday, April 30, 2009
create_table "delayed_jobs" do |t| t.integer "priority", :default => 0 t.integer "attempts", :default => 0 t.text "handler" t.string "last_error" t.datetime "run_at" t.datetime "locked_at" t.datetime "failed_at" t.string "locked_by" t.datetime "created_at" t.datetime "updated_at" end
Delayed Job Schema
Thursday, April 30, 2009
Do Some Work
class MyWorker attr_accessor :user_id
def initialize(user) self.user_id = user_id end
def perform User.find user_id # do some work endend
Thursday, April 30, 2009
Do Some Work
def my_action flash[:notice] = "Processing ..." Delayed::Job.enqueue MyWorker.new(@user.id)end
def my_action flash[:notice] = "Processing ..." MyMailer.new(@user.id).send_later(:deliver_newsletter)end
Thursday, April 30, 2009
The Worker Process
rake jobs:work
Thursday, April 30, 2009
Workling
http://github.com/purzelrakete/workling/
Thursday, April 30, 2009
Workling Install
script/plugin install git://github.com/purzelrakete/workling.git
Thursday, April 30, 2009
Do Some Work
class MyWorker < Workling::Base def do_something(options) # do some work endend
def my_action flash[:notice] = "Processing ..." MyWorker.async_do_something :user_id => @user.idend
Thursday, April 30, 2009
The Worker Process
with spawn:script/workling_client start
with starling:sudo starling -dscript/workling_client start
Thursday, April 30, 2009
Starling
http://github.com/starling/starling/
Thursday, April 30, 2009
Even More Options
• Background Fu - http://github.com/ncr/background-fu
• Roofus Scheduler - http://github.com/jmettraux/rufus-scheduler
• AP4R - http://rubyforge.org/projects/ap4r/
Thursday, April 30, 2009
Advanced Queues
Thursday, April 30, 2009
Kestrel
http://github.com/robey/kestrel/
Thursday, April 30, 2009
Rabbit MQ
http://www.rabbitmq.com/
Thursday, April 30, 2009
Active MQ
http://activemq.apache.org/
Thursday, April 30, 2009
Amazon SQS
http://aws.amazon.com/sqs/
Thursday, April 30, 2009
Monitoring
• god - http://github.com/mojombo/god/
• launchd - OS X
• monit - http://mmonit.com/monit/
Thursday, April 30, 2009
Persistant?
Scaling concerns or fear of commitment?
Yes
Delayed JobBackground Job
No
Workling
Yes
Are you the next Twitter?
StarlingKestrel
Rabbit MQProbably Not Yes
Spawn
No
Daemons
I enjoy managing a
process?
NoYes
Thursday, April 30, 2009
Thanks
twitter.com/robmack
Thursday, April 30, 2009