gearman for beginners

63
Gearman Sunday, 25 October 2009

Upload: giuseppe-maxia

Post on 15-Jan-2015

6.028 views

Category:

Technology


1 download

DESCRIPTION

With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details.No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more.

TRANSCRIPT

Page 1: Gearman For Beginners

Gearman

Sunday, 25 October 2009

Page 2: Gearman For Beginners

Gearman: a technology for distributed

computingGiuseppe Maxia

MySQL Community Team Lead Sun Microsystems

Sunday, 25 October 2009

Page 3: Gearman For Beginners

Mainframe

Mainframe

operating system

hardware

application

terminal

client

USER FRIENDLINESS

0 100

terminal

terminal

terminal

Sunday, 25 October 2009

Page 4: Gearman For Beginners

Mini computers

Mainframe

operating system

hardware

application

client

USER FRIENDLINESS

0 100

Mini terminal

terminal

terminal

terminal

Mini

Sunday, 25 October 2009

Page 5: Gearman For Beginners

Networks

server

hardware

application

client

USER FRIENDLINESS

0 100

personal computer

personal computer

personal computer

personal computer

hardware

operating system

operating system

hardware

Sunday, 25 October 2009

Page 6: Gearman For Beginners

Web applications

web server

hardware

application

client

USER FRIENDLINESS

0 100

browser

browser

browser

browser

operating system

hardware

operating system

operating system

hardware

operating system

INT

ERN

ET

Sunday, 25 October 2009

Page 7: Gearman For Beginners

Cloud applications

web server

hardware

application

client

USER FRIENDLINESS

0 100

browser

browser

browser

browser

operating system

hardware

operating system

operating system

hardware

service provider

applicationapplication

service providerservice

providerservice provider IN

TER

NET

Sunday, 25 October 2009

Page 8: Gearman For Beginners

Some actors

• memcached

• gearman

Used in production by Facebook and Digg

Sunday, 25 October 2009

Page 9: Gearman For Beginners

MAG ANE R

Sunday, 25 October 2009

Page 10: Gearman For Beginners

MG ANER!=

Sunday, 25 October 2009

Page 11: Gearman For Beginners

MAG ANE R ?

Sunday, 25 October 2009

Page 12: Gearman For Beginners

M A GA N E R

Sunday, 25 October 2009

Page 13: Gearman For Beginners

server worker client

taskjob

request

http://gearman.orgSunday, 25 October 2009

Page 14: Gearman For Beginners

Distributed

Sunday, 25 October 2009

Page 15: Gearman For Beginners

Multiple operating systems

Sunday, 25 October 2009

Page 16: Gearman For Beginners

multiple languages

Sunday, 25 October 2009

Page 17: Gearman For Beginners

freedom of choice

Sunday, 25 October 2009

Page 18: Gearman For Beginners

redundancy

Sunday, 25 October 2009

Page 19: Gearman For Beginners

USING GEARMAN

• Server: gearmand

• Client libraries:

• C/C++

• Java

• Perl

• PHP

• Python

Sunday, 25 October 2009

Page 20: Gearman For Beginners

Simple usage GEARMAN

• Command line client and worker

Sunday, 25 October 2009

Page 21: Gearman For Beginners

starting the server/usr/local/sbin/gearmand -d

# started as daemon.# No feedback given on the command line

Sunday, 25 October 2009

Page 22: Gearman For Beginners

starting the server (2)/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6)

# started as normal application# verbose output requested

Sunday, 25 October 2009

Page 23: Gearman For Beginners

starting the workergearman -w -h hostname -p 4730 \ -f conta wc

# -w = act as worker# -f = function# conta = function name# wc = command to execute when function # 'conta' is called

Sunday, 25 October 2009

Page 24: Gearman For Beginners

what the server says/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) …

INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected

Sunday, 25 October 2009

Page 25: Gearman For Beginners

starting the clientgearman -h hostname -p 4730 \ -f conta < ~/.bashrc 57 135 2149 # <- output # from worker

# -f = function# conta = function name# < ~/.bashrc = input data

Sunday, 25 October 2009

Page 26: Gearman For Beginners

what the server says/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) …

INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected

Sunday, 25 October 2009

Page 27: Gearman For Beginners

What happened

1 server start

listen to port 4730

Sunday, 25 October 2009

Page 28: Gearman For Beginners

What happened

2 worker starts

registers function 'conta' to server

Sunday, 25 October 2009

Page 29: Gearman For Beginners

What happened

3 client starts

requires function 'conta' from server

provides input data

Sunday, 25 October 2009

Page 30: Gearman For Beginners

What happened

4 server sends client request to worker

passes all input data to worker

Sunday, 25 October 2009

Page 31: Gearman For Beginners

What happened

5 worker receives request and data

processes input

Sunday, 25 October 2009

Page 32: Gearman For Beginners

What happened

6 worker returns processed data

server passes it to client

Sunday, 25 October 2009

Page 33: Gearman For Beginners

What happened

7 client receives processed data

client displays result

Sunday, 25 October 2009

Page 34: Gearman For Beginners

A simple Perl worker

Sunday, 25 October 2009

Page 35: Gearman For Beginners

A simple perl worker

1. add server

2. add function

3. loop

4. function definition

Sunday, 25 October 2009

Page 36: Gearman For Beginners

simple worker (1)use strict;use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Worker;

my $host = '127.0.0.1';my $port = 4730;

my $worker = new Gearman::XS::Worker;

my $ret = $worker->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error()); exit(1);}

Sunday, 25 October 2009

Page 37: Gearman For Beginners

simple worker (2)my $options = '';

$ret = $worker->add_function( "reverse", # public function name 0, # timeout \&myreverse, # reference to function $options); # function argumentsif ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error());}

Sunday, 25 October 2009

Page 38: Gearman For Beginners

simple worker (3)while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error()); }}

Sunday, 25 October 2009

Page 39: Gearman For Beginners

simple worker (4)sub myreverse { my ($job) = @_;

my $workload = $job->workload(); my $result = reverse($workload);

printf("Job=%s F_Name=%s Workload=%s Result=%s\n", $job->handle(), $job->function_name(), $job->workload(), $result); return $result;}

Sunday, 25 October 2009

Page 40: Gearman For Beginners

A simple Perl client

Sunday, 25 October 2009

Page 41: Gearman For Beginners

A simple perl client

• add server

• run a task

Sunday, 25 October 2009

Page 42: Gearman For Beginners

simple client (1)use strict;use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Client;

my $client = new Gearman::XS::Client;

my $host = '127.0.0.1';my $port = 4730;my $ret = $client->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $client->error()); exit(1);}

Sunday, 25 October 2009

Page 43: Gearman For Beginners

simple client (2)my $input = shift || 'teststring';

my ($return, $result) = $client->do("reverse", $input);if ($return == GEARMAN_SUCCESS) { printf("Result=%s\n", $result);}

Sunday, 25 October 2009

Page 44: Gearman For Beginners

A sample run

Sunday, 25 October 2009

Page 45: Gearman For Beginners

host 1perl worker.pl

Sunday, 25 October 2009

Page 46: Gearman For Beginners

host 2perl client.plResult=gnirtstset

Sunday, 25 October 2009

Page 47: Gearman For Beginners

host 1perl worker.plJob=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset

Sunday, 25 October 2009

Page 48: Gearman For Beginners

more client functions

• do_background

• add_task

• run_tasks

Sunday, 25 October 2009

Page 49: Gearman For Beginners

Some advanced usage

• DBIx::SQLCrosstab

• Data cubes

• Perl only

Sunday, 25 October 2009

Page 50: Gearman For Beginners

Sunday, 25 October 2009

Page 51: Gearman For Beginners

Image processing

• CPU intensive

• Large storage needed

• Application is OS specific

Sunday, 25 October 2009

Page 52: Gearman For Beginners

Sunday, 25 October 2009

Page 53: Gearman For Beginners

Sunday, 25 October 2009

Page 54: Gearman For Beginners

Remote sandboxes

Using GEARMANclient server architectureN serversN workersN clientsSupports multiple languages

Sunday, 25 October 2009

Page 55: Gearman For Beginners

host4host3

host1 host2worker1 worker2

worker4

func1func2

worker3

worker5

func1

func1func3

func3

func1func3

func1func3

worker0func1func3

client

Gearmanserver

Gearmanserver

Sunday, 25 October 2009

Page 56: Gearman For Beginners

host4host3

host1 host2worker1 worker2

worker4

func1func2

worker3

worker5

func1

func1func3

func3

func1func3

func1func3

worker0func1func3

Master

Slave2

Slave1

client

Gearmanserver

Gearmanserver

Gearmanserver

Sunday, 25 October 2009

Page 57: Gearman For Beginners

Remote sandbox worker

•add server

•add function

•loop

•function definition

Sunday, 25 October 2009

Page 58: Gearman For Beginners

remote sandbox worker (2)$ret = $worker->add_function( "make_sandbox", 0, \&make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error());}

Sunday, 25 October 2009

Page 59: Gearman For Beginners

remote sandbox worker (4)sub make_sandbox { my ($job, $options) = @_;

my $command = $job->workload(); chomp $command; print STDERR "<$command>\n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "\n"; return "Error ($!)\n"; } else { return "ok\n" }}

Sunday, 25 October 2009

Page 60: Gearman For Beginners

Client to install Remote sandboxes

•add server

•run remote installation on master

•run remote installation on each slave

•run configuration queries on slaves

Sunday, 25 October 2009

Page 61: Gearman For Beginners

sample call to remote workermy $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ;

# using the gearman command line utility

Sunday, 25 October 2009

Page 62: Gearman For Beginners

More on similar matters

• FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009

• CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009

http://datacharmer.blogspot.com

http://gearman.orgSunday, 25 October 2009

Page 63: Gearman For Beginners

THANKS

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.

Let's talk!

Sunday, 25 October 2009