gearman work queue in php

Gearman work queue in php用 Gearman打造 PHP排程

2013.10.05

@appleboy

2

Who am I

Bo-Yi Wu @appleboyhttp://blog.wu-boy.comhttps://github.com/appleboy

任職於瑞昱半導體 RealTek(IC Design House)

3

今日來聊聊排程系統

4

排程使用時機 ?

5

不需要即時回覆都可以使用排程

6

哪些應用需要排程 ?

7

使用者註冊系統

8

使用者註冊

填寫表單完成

寄發認證信

使用者收到回覆

未使用排程

處理資料

透過第三方寄信Amazone SES

9

缺點 Response time過長使用者不需要等伺服器寄信

(來自知名插畫家彎彎 )

10

使用者註冊

填寫表單完成

寄發認證信

使用者收到回覆

使用排程

處理資料

寫入排程

11

Email Queue?Github Social Coding

12

Application Server

Database Database Database

Work Queue

13

執行排程Linux crontab

Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/

14

<?php $link = mysqli_connect("host", "user", "passw", "db"); $query = 'SELECT * FROM jobs where status = "on"'; $result = $link->query($query); while($row = mysqli_fecth_array($result)) { // ….... // do work // ….... $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); }

15

每分鐘執行一次*/1 * * * * root cron.sh

16

優點

● 容易瞭解

● 任何程式語言都可以實現

● 快速 Deploy到任何機器

17

缺點

● 每分鐘只能執行一次

● Race condition● 浪費 Database資源

18

換個寫法來試試看

19

每 10秒執行一次

<?php$link = mysqli_connect("myhost","myuser","mypassw","mybd");$query = 'SELECT * FROM jobs where status = "on"';$done = false;while(!$done) { $result = $link->query($query); if ($result->num_rows > 0) { // do work $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); } else { sleep(10); }}

20

缺點

● 真的 10秒執行一次 ?● Race condition● 更浪費 Database資源

21

如果沒有 Race Condition

22

#!/bin/sh(sleep 5 && /path/script.sh) &(sleep 15 && /path/script.sh) &(sleep 25 && /path/script.sh) &(sleep 35 && /path/script.sh) &(sleep 45 && /path/script.sh) &(sleep 55 && /path/script.sh) &

每 10秒執行一次

23

如何分配處理這些問題 ?

24

Using Gearman

25

What is Gearman?

It allows you to do work in parallel, to load balance processing, and to call

functions between languages.

26

Client inPerl,Python,PHPJava,C#(.Net)

27

evenMySQL,PostgreSQL

28

How Does Gearman Work?

29

Gearman優點

30

Work Queues● 淺顯易懂

Got job do job repeat→ →

Application

Gearmand

Worker

Got job

Do job

31

Work Queues● 淺顯易懂

Get job do job repeat→ →

● 跨語言

任何平台只要能存入字串即可

32

Multiple Language

Application

Gearmand

Application

Worker Worker

33

Work Queues● 淺顯易懂

Get job do job repeat→ →

● 跨語言

任何平台只要能存入字串即可

● 跨平台

當 Client寫入工作時，Worker不需要啟動

34

跨平台

Application

Gearmand

Application

Gearmand

35

Work Queues● 淺顯易懂

Get job do job repeat→ →

● 跨語言

任何平台只要能存入字串即可

● 跨平台

當 Client寫入工作時，Worker不需要啟動● 即時性

可以為同步或非同步

36

Gearman Architecture佈署架構

37

Application

Gearmand

Job

38

Application

Gearmand

Job

Worker

Job

39

Application

Gearmand

Job

Worker

Job Response

40

Application

Gearmand

Job

Worker

Job Response

Response

41

Application

Gearmand

Worker

Worker

Worker

42

Application

Gearmand

Worker

Worker

Gearmand

43

Application

Gearmand

Worker

Worker

Gearmand

Application

44

結論 :Gearman擴充性佳

45

不論要幾台 gearmand跟數個 worker都可以

46

Application只會跟Gearmand溝通

47

多個 Application不管丟出多少個 Jobs

48

最終由多個 Job Server分配工作

49

Synchronous vs

Asynchronous

50

Synchronous Application

Gearmand

Job

Worker

Job Response

Response

51

AsynchronousApplication

Gearmand

Job

Worker

Job Done!

52

Installation安裝方式

53

Installation

● Job Server (C)● Client library (PHP,Python ...)● Worker library (PHP,Python ...)

54

Job Server安裝方式

55

Ubuntu/Debianaptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0

56

CentOS/Fedorayum -y install libgearman gearmand libgearman-devel libdrizzle

57

Tarball installhttps://gist.github.com/appleboy/6774735

58

Gearmand enabled usingMySQL(Mariadb)

PostgresqlSQL Lite

Memcached

59

MySQL Example/etc/default/gearman-job-server

-q mysql --mysql-host=localhost

--mysql-user=xx --mysql-password=xx

--mysql-db=ovoq --mysql-table=gearman_queue

60

Client Librarypecl install channel://pecl.php.net/gearman-1.1.2echo 'extension=gearman.so' > /path/gearman.ini

61

Quick Testphp -i | grep 'gearman'

62

Gearman in PHP Client

63

PHP Client# Create a client object

$gm = new GearmanClient();

# Add default server, default is localhost:4730

$gm->addServer();

# Do Job

$result = $gm->doNormal("reverse", "Hello!");

echo "Success: $result\n";

64

Do background Job$job = $gm->doBackground("reverse", "apple");

65

Gearman in PHP Worker

66

PHP Worker

# Define worker job function

function reverse_fn($job) { //.... }

# fetch message from client

function reverse_fn($job) {

$data = $job->workload();

}

67

PHP Worker# Create our worker object.

$gm= new GearmanWorker();

# Add default server (localhost).

$gm->addServer();

# Register function "reverse"

$gm->addFunction("reverse", "reverse_fn");

# Waiting for job...

while($gm->work());

68

Multiple Jobs$gm->addFunction("reverse", "reverse_fn");

$gm->addFunction("resize", "resize_fn");$gm->addFunction("queue", "queue_fn");

…...

69

Anonymous function$gm->addFunction("reverse", function($job){

$data = $job->workload(); });

70

Gearman in CodeIgniter

71

CodeIgniter Gearman Libraryhttps://github.com/appleboy/CodeIgniter-Gearman-Library

72

Installation via Sparkphp tools/spark install -v1.0.1 codeigniter-gearman

73

Setup Gearmand Server and Portconfig/gearman.php

$config['gearman_server'] = array('localhost'); $config['gearman_port'] = array('4730');

74

CodeIgniter Client

75

Implement Image Resize Jobcreate app controller

76

class App extends CI_Controller{

public function resize_sync()public function resize_async()

}

77

public function resize_sync(){ $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); do {

$this->lib_gearman->do_job_foreground('resize', $data); switch($client->returnCode()) { //.... }

} while ($client->returnCode() != GREARMAN_SUCCESS);}

78

class App extends CI_Controller{

public function resize_sync()public function resize_async()

}

79

public function resize_async(){ $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); $this->lib_gearman->do_job_background('resize', $data));}

80

CodeIgniter Workercreate cli controller

81

class Cli extends CI_Controller{

public static function image_resize()public static function email_queue()public function worker()

}

82

public static function image_resize($job){ $data = json_decode($job->workload(), true); $ci =& get_instance(); $ci->image_lib->initialize($config); $ci->image_lib->resize(); $ci->image_lib->clear();}

83

Define Worker

84

public function worker(){ $worker = $this->lib_gearman->gearman_worker(); $this->lib_gearman ->add_worker_function('resize', 'Cli::image_resize'); $this->lib_gearman ->add_worker_function('queue', 'Cli::email_queue'); while ($this->lib_gearman->work());}

85

Start Worker$ php index.php cli worker

86

Running workers via script#!/bin/shscript="index.php cli worker"folder="/script_path/"max_worker="5"php_bin="/usr/bin/php"count=0current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l)

if test $current_count -lt $max_workerthen

cd $folder$php_bin $folder$script &

elseecho "The worker count has been limited."

fihttps://gist.github.com/appleboy/6806323

87

About Job DataDon't pass 2GB object.

Gearman is not data center.

88

Keep your message data smallAmazone SQS message siez: 256K

http://goo.gl/0MBq8W

89

Scalability

90

Client Client

Job Server

Client

ClientWorker Worker Worker Worker

Job Server

Client

91

you can scale out your clients and workers

as needed

92

Client SettingClient1, Client2: Gearman1Client3, Client4: Gearman2

…...

93

Worker SettingWorker[1-4]:Server[1-2]

…...

94

Add new Job Server...

95

Reconfigure all setting(client and worker program)

96

Reconfigure all setting(client and worker program)

evenDeploy Tool

97

如何控制易怒的情緒?

98

How to improve your architecture?

99

Client Client

Job Server

Client

ClientWorker Worker Worker Worker

Job Server

Client

Proxy (Haproxy ...)

100

Live DemoEmail Queue.

101

Any Questions?

102

Thanks全體工作人員和聽眾