gearman for beginners

Gearman

Sunday, 25 October 2009

Gearman: a technology for distributed

computingGiuseppe Maxia

MySQL Community Team Lead Sun Microsystems

Mainframe

operating system

hardware

application

terminal

client

USER FRIENDLINESS

terminal

Mini computers

Mainframe

operating system

hardware

application

client

USER FRIENDLINESS

Mini terminal

terminal

Networks

server

hardware

application

client

USER FRIENDLINESS

personal computer

hardware

operating system

hardware

Web applications

web server

hardware

application

client

USER FRIENDLINESS

browser

operating system

hardware

operating system

hardware

operating system

Cloud applications

web server

hardware

application

client

USER FRIENDLINESS

browser

operating system

hardware

operating system

hardware

service provider

applicationapplication

service providerservice

providerservice provider IN

Some actors

• memcached

• gearman

Used in production by Facebook and Digg

MAG ANE R

MG ANER!=

MAG ANE R ?

M A GA N E R

server worker client

taskjob

request

http://gearman.orgSunday, 25 October 2009

Distributed

Multiple operating systems

multiple languages

freedom of choice

redundancy

USING GEARMAN

• Server: gearmand

• Client libraries:

• C/C++

• Java

• Perl

• PHP

• Python

Simple usage GEARMAN

• Command line client and worker

starting the server/usr/local/sbin/gearmand -d

# started as daemon.# No feedback given on the command line

starting the server (2)/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6)

# started as normal application# verbose output requested

starting the workergearman -w -h hostname -p 4730 \ -f conta wc

# -w = act as worker# -f = function# conta = function name# wc = command to execute when function # 'conta' is called

what the server says/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) …

INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected

starting the clientgearman -h hostname -p 4730 \ -f conta < ~/.bashrc 57 135 2149 # <- output # from worker

# -f = function# conta = function name# < ~/.bashrc = input data

what the server says/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) …

INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected

What happened

1 server start

listen to port 4730

What happened

2 worker starts

registers function 'conta' to server

What happened

3 client starts

requires function 'conta' from server

provides input data

What happened

4 server sends client request to worker

passes all input data to worker

What happened

5 worker receives request and data

processes input

What happened

6 worker returns processed data

server passes it to client

What happened

7 client receives processed data

client displays result

A simple Perl worker

A simple perl worker

1. add server

2. add function

3. loop

4. function definition

simple worker (1)use strict;use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Worker;

my $host = '127.0.0.1';my $port = 4730;

my $worker = new Gearman::XS::Worker;

my $ret = $worker->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error()); exit(1);}

simple worker (2)my $options = '';

$ret = $worker->add_function( "reverse", # public function name 0, # timeout \&myreverse, # reference to function $options); # function argumentsif ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error());}

simple worker (3)while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error()); }}

simple worker (4)sub myreverse { my ($job) = @_;

my $workload = $job->workload(); my $result = reverse($workload);

printf("Job=%s F_Name=%s Workload=%s Result=%s\n", $job->handle(), $job->function_name(), $job->workload(), $result); return $result;}

A simple Perl client

A simple perl client

• add server

• run a task

simple client (1)use strict;use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Client;

my $client = new Gearman::XS::Client;

my $host = '127.0.0.1';my $port = 4730;my $ret = $client->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $client->error()); exit(1);}

simple client (2)my $input = shift || 'teststring';

my ($return, $result) = $client->do("reverse", $input);if ($return == GEARMAN_SUCCESS) { printf("Result=%s\n", $result);}

A sample run

host 1perl worker.pl

host 2perl client.plResult=gnirtstset

host 1perl worker.plJob=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset

more client functions

• do_background

• add_task

• run_tasks

Some advanced usage

• DBIx::SQLCrosstab

• Data cubes

• Perl only

Image processing

• CPU intensive

• Large storage needed

• Application is OS specific

Remote sandboxes

Using GEARMANclient server architectureN serversN workersN clientsSupports multiple languages

host4host3

host1 host2worker1 worker2

worker4

func1func2

worker3

worker5

func1func3

worker0func1func3

client

Gearmanserver

host4host3

host1 host2worker1 worker2

worker4

func1func2

worker3

worker5

func1func3

worker0func1func3

Master

Slave2

Slave1

client

Gearmanserver

Remote sandbox worker

•add server

•add function

•loop

•function definition

remote sandbox worker (2)$ret = $worker->add_function( "make_sandbox", 0, \&make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error());}

remote sandbox worker (4)sub make_sandbox { my ($job, $options) = @_;

my $command = $job->workload(); chomp $command; print STDERR "<$command>\n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "\n"; return "Error ($!)\n"; } else { return "ok\n" }}

Client to install Remote sandboxes

•add server

•run remote installation on master

•run remote installation on each slave

•run configuration queries on slaves

sample call to remote workermy $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ;

# using the gearman command line utility

gearman for beginners

simple client

simple worker

worker gearman w

client functions

new gearman

client gearman h hostname

gearmand client libraries

function reverse

Technology

the gearman cookbook

gearman: a job server made for scale

trading for beginners - forex profit acceleratortrading for...

gearman bundle, warszawa 2013 edition

spa, inc. · 2016. 8. 1. · gearman.. within this...

knitting for beginners: 6 easy free knitting patterns for...

gearman and asynchronous processing in php applications

thai for beginners

distributed queue system using gearman

mapreduce using perl and gearman

gearman - job queue

nagios conference 2012 - jason cook - nagios and mod-gearman

wilkes a -german for beginners languages for beginners

gearman to distribute server load

map/reduce and queues for mysql using gearman

spanish courses for beginners, spanish language courses for...

terence gordon, mcluhan for beginners, for beginners llc

entre na fila. processamento distribuído usando gearman

ao infinito e além com php memcached e gearman

gearman 200907 oscon tutorial