job queues with gearman
If you can't read please download the document
TRANSCRIPT
- 1. CakeFest 2012 Manchester, UK Job QueuesLetting your servers breathe@mgiglesias
2. So you build a signupclass UsersController extends AppController {public function add() {if ($this->request->is(post)) {$this->User->save($this->request->data);$email = new CakeEmail();$email->to($this->request->data[User][email])->subject("Validate em emails")->template(activate)->viewVars(array(id => $this->User->id))->send();$this->Session->setFlash("Giddy up!");return $this->redirect(array(action =>index)); } else { $this->Session->setFlash("Fix em typos"); }}} 3. PLEASE STOP!BAD IDEA 4. Bad ideas Email sending Notifications Messaging To multiple users Imports Oh yes, imports Counter caches 5. The #1 Rule of Fight ClubYOU DO NOT MAKE THE USER WAIT 6. The solution 7. Show me how, commarade Could someone pleasesend this email?Controller{{to: [email protected], to: [email protected],subject: FYI,subject: Yo, dawg,body: Unicorns sux body: Whatup}} 8. Ingredients Channel for dispatching tasks Job queue Workers that wait for jobs Beer CakeResque: github://kamisama/Cake-Resque 9. A Harry Potter momentGEARMANMANAGER 10. Break it down Client API to dispatch tasks Job server (Gearmand) Worker API to, well, work on jobs Multi-language Multi-platform Synchronous execution Job status callbacks 11. Super scalableGearmand #1 Gearmand #2Worker #1 Worker #2 Worker #2 Worker #2 Worker #2 12. Client & Worker$client = new GearmanClient();$client->addServer(127.0.0.1);$result = $client->doBackground(hello, Mariano);var_dump($result);$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(hello, function($job) {$who = $job->workload();sleep(1);echo "Hello {$who}!" . "n";sleep(1);return true;});while($worker->work()) { usleep(50000);} 13. Getting jiggy with it Start $ gearmand --verbose DEBUG $ telnet 4730 and type in STATUS Gives you registered functions, number of workersand items in queue 14. Show me the money 15. Get the money back$client = new GearmanClient();$client->addServer(127.0.0.1);$result = $client->doNormal(hello, Mariano);var_dump($result);$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(hello, function($job) {$who = $job->workload();return "Hello {$who}!";});while($worker->work()) { usleep(50000);} 16. Show me the money 17. I got 99 problems 18. addFunction(process, function($job) {$files = json_decode($job->workload(), true);$i = 0;$count = count($files);foreach($files as $file) { echo "Processing {$file}... "; $job->sendStatus(++$i, $count); echo "DONEn"; sleep(1);}});while($worker->work()) { usleep(50000);} 22. Job statuses$client = new GearmanClient();$client->addServer(127.0.0.1);$handle = $client->doBackground(process, json_encode(array( file1.jpg, file2.jpg, file3.jpg)));$done = 0;do { list($queued, $running, $processed, $total) = $client->jobStatus($handle); if (!$queued) { break; } if ($processed != $done) { $done = $processed; echo "PROCESSED {$done} of {$total}n"; } usleep(50000);} while(true);echo DONE; 23. Parallel tasks They are not as unusual as you think Resource usage optimization Divide and conquer Map &Reduce Can even prioritize the tasks Its parallel, so its faster :) 24. Parallel tasks$client->setStatusCallback(function($task) {echo "PROCESSED {$task->taskNumerator()} of {$task->taskDenominator()}n";});$client->setCompleteCallback(function($task) {echo "COMPLETED: {$task->unique()}: {$task->data()}n";});$client->addTask(process, json_encode(array( file1.jpg, file2.jpg, file3.jpg)));$client->addTask(process, json_encode(array( a1.jpg, b.jpg)));$client->runTasks(); 25. Parallel tasks worker-1.log worker-2.logProcessing a1.jpg... DONE Processing file1.jpg... DONEProcessing b.jpg... DONEProcessing file2.jpg... DONEProcessing file3.jpg... DONE 26. What about CakePHP? 27. Bake it!class JobsController extends AppController { public function trigger() { $client = new GearmanClient(); $client->addServer(127.0.0.1); $client->doNormal(run, json_encode(array(action => get_called_class() . ::hello,params => array(Mariano) ))); echo DONE; exit; }public static function hello($who) { echo "ttHello {$who}!n";}} 28. Bake it!class WorkerShell extends AppShell { public function start() { $this->out(Connecting to Gearman... , 0, Shell::VERBOSE);$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(run, array($this, _run));$this->out(DONE, 1, Shell::VERBOSE);$this->out(Waiting for jobs, 1, Shell::VERBOSE);while($worker->work()) { usleep(50000);}}} 29. Bake it!public function _run(GearmanJob $job) {$params = json_decode($job->workload(), true);$this->out(Got new job:. $job->workload(), 1, Shell::VERBOSE);if (empty($params) || !isset($params[action])) {throw new InvalidArgumentException(Invalid job request);} else if (preg_match(/^(.+?Controller)::/, $params[action], $match) &&!class_exists($match[1])) {App::uses($match[1], Controller);}$result = call_user_func_array($params[action], $params[params]);$this->out(Job done, 1, Shell::VERBOSE);return $result;} 30. Lessons learned Processes stay alive, which mean resourcetimeouts Check your DB for reconnection! Workers can be killed, easily Be mindful of your memory usage Account for process switches 31. BONUS TRACK 32. I can haz non-blocking! (Sort of)$worker->addOptions(GEARMAN_WORKER_NON_BLOCKING);while ($worker->work() || ( $worker->returnCode() == GEARMAN_IO_WAIT || $worker->returnCode() == GEARMAN_NO_JOBS)) { if ($worker->returnCode() == GEARMAN_SUCCESS) { continue; } if (!$worker->wait()) { if ($worker->returnCode() == GEARMAN_NO_ACTIVE_FDS) {echo "Got disconnected, so waiting forserver...n";sleep(5);continue; } break; }} 33. CakeFest 2012 Manchester, UKQuestions?Beers? @mgiglesias