handlersocket 20110517

1. HandlerSocket plugin for MySQL 2011/05/17NoSQL @ DeNAIT

2. 3. HandlerSocket plugin

MySQL SQL

4. HandlerSocket plugin

InnoDB SQL

TCP/IP

C++ Perl

PHP, Java, Python, Ruby, JavaScript(node.js), Scala

Linux/FreeBSD/MacOS

https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL

5. mysqld client app Handler Interface Innodb MyISAM Other storage engines SQL Layer HandlerSocket Plugin Listener for libmysql libmysql libhsclient Applications 6.

7. : SQL for MySQL (HandlerSocket )

mycached

http://developer.cybozu.co.jp/kazuho/2009/08/mycached-memcac.html

HandlerSocket handler

memcached

NDB API

http://dev.mysql.com/doc/ndbapi/en/index.html

ndbcluster

HandlerSocket ( )

2010/03

2010/04

2010/06 DeNA Technology Seminar #2

2010/08 github

2010/08 mobage

9. (1)

Percona Server HandlerSocket

Percona Server now both SQL and NOSQL

http://www.mysqlperformanceblog.com/2010/12/14/percona-server-now-both-sql-and-nosql/

10. (2)

2011/04 MySQL Conference & Expo 2011

DeNA HandlerSocket

DeNA Corporate Contributor of the Year 2011

DeNA is Japan's biggest social game provider and run hundreds of MySQL servers to provide their services. So it is not surprising they also employ some of Japan's best and world famous MySQL experts to work on their LAMP stack. This team produced one of the most interesting innovations MySQL has seen for a while:HandlerSocket is MySQL's answer to the NoSQL trend, and with a vengeance. With read-transactions now running over 700% faster, it makes MySQL the fastest NoSQL key-value solution out there. And you get best of both worlds: MySQL and NoSQL in the same package.

->MySQL + HandlerSocket KVS 11. (3)

2011/04 MySQL memcached

NoSQL to MySQL with Memcached

http ://dev.mysql.com/tech-resources/articles/nosql-to-mysql-with-memcached.html

... The HandlerSocket development at DeNA is a great example of community innovation, with a solution implemented as a custom plug-in and protocol for the MySQL server daemon.

->MySQL NoSQL 12. mobage 13. mobage

2010/08 mobage

mysqld

HandlerSocket

mysqld

MySQL + memcached

memcached

MySQL memcached

20% 35%

16. HandlerSocket 17. (libmysql )

18. HandlerSocket

19. HandlerSocket

HandlerSocket

CPU CPU

Disk IO

20. 21. HandlerSocket ( )

Primary Key Unique Key

=, >=, >, CPU

->CPU IO

29. CPU 30. oprofile libmysql/mysqld

SELECT v from table where k = ? oprofile CPU

samples|%| ------------------ 9669940 53.1574 mysqld 4438098 24.3970 vmlinux 1835976 10.0927 libpthread-2.5.so 16806569.2389 libc-2.5.so 3979702.1877 e1000e 891360.4900 oprofiled 428810.2357 oprofile 31. oprofile libmysql/mysqld

mysqld CPU

samples%symbol name 7480227.7355MYSQLparse(void*) 2197022.2720my_pthread_fastmutex_lock 2056062.1262make_join_statistics(JOIN*, TABLE_LIST*, 1982342.0500btr_search_guess_on_hash 1807311.8690JOIN::optimize() 1771201.8317row_search_for_mysql 1711851.7703lex_one_token(void*, void*) 1626831.6824alloc_root 1318231.3632read_view_open_now 1227951.2699mysql_select(THD*, Item***, TABLE_LIST*, 1002761.0370open_table(THD*, TABLE_LIST*, st_mem_root*, 995751.0297mem_pool_fill_free_list 964340.9973build_template(row_prebuilt_struct*, THD*, 863490.8930get_hash_symbol(char const*, unsigned int, 32. oprofile libmysql/mysqld

samples%symbol name 2043934.6054schedule 1186482.6734tcp_sendmsg 1158322.6099tcp_recvmsg 1065372.4005tcp_v4_rcv 1039152.3414tcp_ack 1035342.3328system_call 938642.1150dev_queue_xmit 868311.9565__mod_timer 858911.9353tcp_rcv_established 840831.8946.text.task_rq_lock 33. oprofile libmysql/mysqld

libmysql/mysqld CPU :

mysqld CPU

SQL CPU

34. oprofile HandlerSocket

SQL HandlerSocket oprofile CPU

samples|%| ------------------ 1919039 51.0453 vmlinux 811998 21.5987 mysqld 421215 11.2041 libpthread-2.5.so 2071665.5105 e1000e 1915665.0955 HandlerSocket.so 1886185.0171 libc-2.5.so 136220.3623 oprofiled 57070.1518 oprofile 35. oprofile HandlerSocket

mysqld CPU

samples%symbol name 11968414.7394btr_search_guess_on_hash 582027.1678row_search_for_mysql 469465.7815mutex_delay 386174.7558my_pthread_fastmutex_lock 377074.6437buf_page_get_known_nowait 365284.4985rec_get_offsets_func 346254.2642build_template(row_prebuilt_struct*, THD*, TABLE*,200242.4660row_sel_store_mysql_rec 193472.3826btr_cur_search_to_nth_level 167012.0568row_sel_convert_mysql_key_to_innobase 133431.6432cmp_dtuple_rec_with_match 113811.4016ha_innobase::index_read(unsigned char*,111761.3764dict_index_copy_types 107621.3254mtr_memo_slot_release 107341.3219ha_innobase::init_table_handle_for_HANDLER() 36. oprofile HandlerSocket

samples%symbol name 1290386.7241tcp_sendmsg 800804.1729tcp_v4_rcv 696583.6298dev_queue_xmit 661713.4481.text.skb_release_data 633163.2994__qdisc_run 602793.1411tcp_recvmsg 597033.1111ip_output 584623.0464.text.skb_release_head_state 488762.5469tcp_ack 487332.5394__alloc_skb 456602.3793ip_queue_xmit 446712.3278tcp_transmit_skb 37. oprofile HandlerSocket

HandlerSocket CPU :

network CPU

mysqld innodb

adaptive hash index

38. 39.

mysqld :

MySQL 5 1

MySQL 6

HandlerSocket :

41. HandlerSocket DB readview DB (1/ ) HandlerSocket 42. HandlerSocket DB DB HandlerSocket 43.

(durable)

sync_binlog = 1

innodb_flush_log_at_trx_commit= 1

innodb_support_xa = 1

write-back cache SSD

mysql: 1000 qps

innodb pluginsync_binlog = 0 group commit

HandlerSocket: 30000 qps

44. HandlerSocket

MyISAM

shared-exclusive lock

InnoDB

HandlerSocket

45. 46. C/S

write(3, "L0003select column0,column1,column2,column3,column4 from db_1.table_1 where k=15", 80) = 80

read(3, "100100560023def4 db_1 7 table_1 7 table_1 7 column0 7 column0 fr0