q4m microblogcon
DESCRIPTION
Introduction and Tutorial of Q4M - a pluggable message queue for MySQLTRANSCRIPT
![Page 1: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/1.jpg)
Q4Ma high-performance message queue for MySQL
Q4Ma high-performance message queue for MySQL
Cybozu Labs, Inc.Kazuho Oku
![Page 2: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/2.jpg)
2008 8 27年 月 日 Q4M 2
What is Q4M?
![Page 3: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/3.jpg)
2008 8 27年 月 日 Q4M 3
Introduction of Q4MIntroduction of Q4M
Q4M stands for Queue for MySQLQ4M の意は「 Queue for MySQL 」
A pluggable storage engine of MySQL 5.1MySQL 5.1 用のプラガブルストレージエンジン
Access & administration by SQLSQL を用いたアクセスと管理
![Page 4: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/4.jpg)
2008 8 27年 月 日 Q4M 4
Design Goals of Q4MDesign Goals of Q4M
Robust ( 堅牢性 )No data loss on OS crash or power failureOS クラッシュや停電時にデータロスが発生しな
い
Fast ( 高速性 )Optimized for speed
Flexible ( 柔軟性 )Use of SQL for access and administration, includin
g JOINs with other storage engines
![Page 5: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/5.jpg)
2008 8 27年 月 日 Q4M 5
What is a Message Queue?from Distributed Systems (Tanenbaum / Van Steen)
![Page 6: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/6.jpg)
2008 8 27年 月 日 Q4M 6
What is a Message Queue?What is a Message Queue?
Middleware for persistent asynchronous communication
持続的な非同期コミュニケーションのためのミドルウェア
A.k.a. Message-Oriented Middleware別名 : MOM
![Page 7: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/7.jpg)
2008 8 27年 月 日 Q4M 7
Message Queue (cont.)Message Queue (cont.)
MQ is an intermediate storageMQ は中間的なストレージ
vs. RDBMS (long-term persistent storage)vs. RDBMS ( 永続的なストレージ )
Senders and/or receivers may go down送受信プロセスが任意に停止可能
![Page 8: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/8.jpg)
2008 8 27年 月 日 Q4M 8
Minimal Configuration of a MQ ( 最小構成 )Minimal Configuration of a MQ ( 最小構成 )
Senders and receivers access a single queue
送信プロセスと受信プロセスが同一のキューにアクセス
Sender Receiver
Queue
![Page 9: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/9.jpg)
2008 8 27年 月 日 Q4M 9
MQ and RelaysMQ and Relays
Separate queue for sender and receiver送信プロセスと受信プロセスが別個にキューをもつ
Messages relayed between queuesメッセージリレーでキュー間を接続
Sender
Queue
Receiver
Queue
Relay
![Page 10: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/10.jpg)
2008 8 27年 月 日 Q4M 10
Merits of Message Relays ( リレーの意義 )Merits of Message Relays ( リレーの意義 )
Destination can be easily changed宛先の変更が容易
Relays may transfer messages to different locations depending on the headerヘッダを見て転送先を変えるとか
Robustness against network failureネットワーク障害の影響を受けない
Logging and Multicasting, etc.ロギングやマルチキャストが可能
![Page 11: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/11.jpg)
2008 8 27年 月 日 Q4M 11
Message BrokersMessage Brokers
Transform (filter) messages within the relay agent
リレーエージェント内でメッセージの変換を行う
Publish / subscribe modelSeparation between logic and data transfer
ロジックとデータ転送の分離
![Page 12: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/12.jpg)
2008 8 27年 月 日 Q4M 12
Q4M as a Message Queue (MQ としての Q4M)Q4M as a Message Queue (MQ としての Q4M)
Q4M (can) support all modelsQ4M は全モデルに対応 ( 可能 )
Message transfer by q4m-forwardメッセージ転送は q4m-forward で
APIs for routers and message brokersルータやメッセージブローカ用の APIQ4M-forward is built upon those APIs
Q4M-forward は、それら API を使用して開発
![Page 13: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/13.jpg)
2008 8 27年 月 日 Q4M 13
Performance
![Page 14: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/14.jpg)
2008 8 27年 月 日 Q4M 14
Internal Architecture of Q4MInternal Architecture of Q4M
Q4M maps queries from multiple threads to single I/O requestsQ4M は複数スレッドからの SQL クエリを単一
の I/O 処理にグループ化
Spawn multiple clients for maximum performance多数のクライアントを同時実行することで、パ
フォーマンスを最大限に発揮
![Page 15: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/15.jpg)
2008 8 27年 月 日 Q4M 15
BenchmarkBenchmark
Troughput Test7,655 mess./sec. (avg. 256 bytes/mess.)Linux 2.6; x86_64; Opteron 2218x2--with-sync=fdatasync --with-delete=msync
$ VAR_LENGTH=504 CONCURRENCY=50 MESSAGES=100000 DBI='dbi:mysql:test;mysql_socket=/tmp/mysql51.sock' DBI_USER=root MYSQL_SOCKET='/tmp/mysql51.sock' USE_C_CLIENT=1 t/05-multirw.t 1..4ok 1 - check number of messagesok 2 - min value of received messageok 3 - max value of received messageok 4 - should have no rows in table
Multi-reader-writer benchmark result: Number of messages: 100000 Number of readers: 50 Elapsed: 13.063 seconds Throughput: 7655.070 mess./sec.
![Page 16: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/16.jpg)
2008 8 27年 月 日 Q4M 16
Benchmark (cont.)Benchmark (cont.)
No sync. to disk23,637 mess./sec. (256 bytes/mess.)--with-sync=no --with-delete=msync
![Page 17: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/17.jpg)
2008 8 27年 月 日 Q4M 17
Tutorial
![Page 18: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/18.jpg)
2008 8 27年 月 日 Q4M 18
Sending a MessageSending a Message
INSERT into queue values (1,’Hello World!’);
![Page 19: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/19.jpg)
2008 8 27年 月 日 Q4M 19
Receiving a MessageReceiving a Message
while (1) {SELECT queue_wait(‘queue’);;my @row = SELECT ROW * FROM queue;
or next;…
}SELECT queue_end(‘queue’);;
![Page 20: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/20.jpg)
2008 8 27年 月 日 Q4M 20
OWNER Mode and NON-OWNER ModeOWNER Mode and NON-OWNER Mode
QuickTime˛ Ç∆TIFFÅiîÒà≥èkÅj êLí£ÉvÉçÉOÉâÉÄ
ǙDZÇÃÉsÉNÉ`ÉÉÇ å©ÇÈÇΩÇflÇ…ÇÕïKóvÇ≈Ç∑ÅB
![Page 21: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/21.jpg)
2008 8 27年 月 日 Q4M 21
OWNER Mode and NON-OWNER Mode (cont.)OWNER Mode and NON-OWNER Mode (cont.)
Within OWNER mode, only the owned row are visible
OWNER モードでは、所有中の行のみが見える
Within NON-OWNER mode, rows owned by other connections are invisible
NON-OWNER モードでは、他の接続が所有している行は見えない
![Page 22: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/22.jpg)
2008 8 27年 月 日 Q4M 22
Function for Entering OWNER ModeFunction for Entering OWNER Mode
queue_waitUsed to enter OWNER mode
OWNER モードに ( 再 ) 切替するために使用When called within OWNER mode, the owned row i
s deleted
![Page 23: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/23.jpg)
2008 8 27年 月 日 Q4M 23
queue_waitqueue_wait
Two formsqueue_wait(tbl_cond)queue_wait(tbl_cond,[tbl_cond…,timeout])
![Page 24: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/24.jpg)
2008 8 27年 月 日 Q4M 24
queue_wait(tbl_cond)queue_wait(tbl_cond)
Wait max. 60 seconds until any data becomes available on given table under given condition
最大 60 秒間、指定された条件に合致する行が指定されたテーブルに登録されるまでブロック
Returns 1 if successful, 0 if no data成功すれば 1 、データがなければ 0
Enters OWNER mode even if no dataデータがない場合でも OWNER モードへ切り替わる
![Page 25: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/25.jpg)
2008 8 27年 月 日 Q4M 25
queue_wait(tbl_cond)queue_wait(tbl_cond)
Table name (and optionally condition) should be specified
テーブル名と (オプションで条件 ) を指定
Only numeric columns may be used within condition
条件節では数値型のカラムのみ使用可能See queue_share_t::init_fixed_fields
tbl_cond ::= table [ “:” cond]
![Page 26: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/26.jpg)
2008 8 27年 月 日 Q4M 26
queue_wait(tbl_cond) (cont.)queue_wait(tbl_cond) (cont.)
SELECT queue_wait(‘table:v<3’);
![Page 27: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/27.jpg)
2008 8 27年 月 日 Q4M 27
queue_wait(tbl_cond,[tbl_cond…,timeout])queue_wait(tbl_cond,[tbl_cond…,timeout])
Accepts multiple tables and timeout複数のテーブルとタイムアウトを指定可能
Data searched from leftmost table to right
データは左端のテーブルから右方向に探索
Returns table index (the leftmost table is 1) of the newly owned row
獲得した行の属するテーブルのインデックス (左端のテーブルが 1) を返す
![Page 28: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/28.jpg)
2008 8 27年 月 日 Q4M 28
Functions for Exiting OWNER ModeFunctions for Exiting OWNER Mode
queue_endDeletes the owned row and exits OWNER mode
所有中の行を削除して OWNER モードを脱出
queue_abortReleases (instead of deleting) the owned row and
exits OWNER mode所有中の行を非所有状態に変更して OWNER モードを脱出
Close of a MySQL connection does the same thingMySQL の接続切断時も同様
![Page 29: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/29.jpg)
2008 8 27年 月 日 Q4M 29
Receiving a Message (revisited)Receiving a Message (revisited)
while (1) {SELECT queue_wait(‘queue’);;my @row = SELECT ROW * FROM queue;
or next;if (consume_row(@row) != SUCCESS) {
exit(1);}
}
![Page 30: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/30.jpg)
2008 8 27年 月 日 Q4M 30
Restrictions in SQL of Q4MRestrictions in SQL of Q4M
No indexesNo UPDATE
Why need UPDATE a queue row?キューのデータを UPDATE する必要なんてない
SELECT and DELETE are supported for queue administration
SELECT DELETE を使ってキューを管理可能
ex. select count(*) from queue;
![Page 31: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/31.jpg)
2008 8 27年 月 日 Q4M 31
Why no Indexes Support?Why no Indexes Support?
There is generally no reason to support indexes with mysql-based queues, since mysql cannot handle a large number of connections requesting data under various conditions
インデックスが必要になるほど多様な条件を指定して同時に購読できるほど、 mysql の同時接続数は大きくない
![Page 32: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/32.jpg)
2008 8 27年 月 日 Q4M 32
Why no Indexes Support?Why no Indexes Support?
Instead use conditional subscription Conditional subscription 機能を使うべき
Or route data after reading from Q4Mあるいは Q4M からデータを読んだ後にルーティング
すべき
![Page 33: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/33.jpg)
2008 8 27年 月 日 Q4M 33
Configuration Options of Q4MConfiguration Options of Q4M
--with-sync=no|fsync|fdatasync|fcntlControls synchronization to disk, see --help for det
ail
--enable-mmapMmap’ed reads lead to higher throughput
--enable-mmap-writesDELETEs commited using mmap(PROT_WRITE) and
msync, recommended on linux>=2.6.20 if you need really high performance
![Page 34: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/34.jpg)
2008 8 27年 月 日 Q4M 34
My Build OptionsMy Build Options
On CentOS 5 (x86_64) ./configure --prefix=/var/mq/mysql --with-mysql=/home/k
azuho/install/mq/mysql-5.1.24-rc CFLAGS=’-O2 -g -Wall -fpic’ CXXFLAGS=’-O2 -g -Wall -fpic’
![Page 35: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/35.jpg)
2008 8 27年 月 日 Q4M 35
Applications of Q4M
![Page 36: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/36.jpg)
2008 8 27年 月 日 Q4M 36
Connecting Distant ServersConnecting Distant Servers
Pathtraq uses Q4M queues and a relay to communicate with its content analysis service running in a different iDC
Pathtraq では、 Q4M キューとメッセージリレーを使用して、別の iDC にあるコンテンツ分析サービスと接続している
ContentAnalysisService
Queue
PathtraqDB
Queue
MySQL conn.over SSL,gzip
![Page 37: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/37.jpg)
2008 8 27年 月 日 Q4M 37
User NotificationsUser Notifications
For sending notifications from web services
ウェブサービスでユーザ通知送信のために使用可能
DB
Queue(s)
App. Logic SMTP Agent
IM Agent
![Page 38: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/38.jpg)
2008 8 27年 月 日 Q4M 38
Asynchronous UpdatesAsynchronous Updates
Asynchronous updates leads to faster response of web services
非同期更新に利用することでウェブサービスの応答性を向上可能
DB
Queue
App. Logic
DB
![Page 39: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/39.jpg)
2008 8 27年 月 日 Q4M 39
Scheduling Web CrawlersScheduling Web Crawlers
Web crawlers with retry-on-errorリトライ機能つき Web クローラーExample included in the Q4M distribution
URLDB
Request Queue
Spiders
Retry Queue
Re-scheduler
Store Result
Read URL
If failed to fetch, store URL in retry queue
![Page 40: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/40.jpg)
2008 8 27年 月 日 Q4M 40
Prioritized Task SchedulingPrioritized Task Scheduling
Used for scheduling keyword analysis in Pathtraq
Pathtraq でキーワード解析のスケジューリングに使用High prio. for new content, low prio. for re-analysis
Content &Keyword
DBRequest Queue
KeywordAnalyzer
Store Keywords Read URL ID
URL ID Prio0
1
0Read Content
![Page 41: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/41.jpg)
2008 8 27年 月 日 Q4M 41
Prioritized Task Scheduling (cont.)Prioritized Task Scheduling (cont.)
select queue_wait(’queue:prio<0’, ’queue:prio=0’,’queue:prio>0’);
# ugly part of not having indexes support…
Content &Keyword
DBRequest Queue
KeywordAnalyzer
Store Keywords Read URL ID
URL ID Prio0
1
0Read Content
![Page 42: Q4M Microblogcon](https://reader035.vdocuments.net/reader035/viewer/2022062418/556ae906d8b42a86218b4b6e/html5/thumbnails/42.jpg)
2008 8 27年 月 日 Q4M 42