PERFORMANCE_SCHEMA in MySQL 5.6:

The Good, The Bad and The Ugly

Valerii Kravchuk, Principal Support Engineervalerii.kravchuk@percona.com

www.percona.com

Who am I?

Valerii (aka Valeriy) Kravchuk (you may call me “Larry”):● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012

○ Bugs Verification Team all this time○ Support issues related to bugs, InnoDB, performance and more○ Trainings (mostly informal) for new team members○ All kinds of decision making committees…

● Principal Support Engineer in Percona, 2012 - ...○ Doing more or less the same as before, but better (I hope)...○ Plus I speak and write about MySQL in public now!

● http://mysqlentomologist.blogspot.com - my blog about MySQL bugs● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about

MySQL (mostly bugs…)● http://bugs.mysql.com - my personal playground● Yes, I am Orthodox, in many senses...

www.percona.com

What is this session about?

“The Good, the Bad and the Ugly (Italian title: Il buono, il brutto, il cattivo) is a 1966 Italian epic Spaghetti western film directed by Sergio Leone, starring Clint Eastwood, Lee Van Cleef, and Eli Wallach in the title roles respectively.

...

It is the third film in the Dollars Trilogy following A Fistful of Dollars (1964) and For a Few Dollars More (1965). The plot revolves around three gunslingers competing to find a fortune in buried Confederate gold amid the violent chaos of gunfights, hangings, American Civil War battles and prison camps.”

http://en.wikipedia.org/wiki/The_Good,_the_Bad_and_the_Ugly

www.percona.com

So, what is this session really about?

● How to find the root causes of MySQL server performance problems?○ http://method-r.com/faq/35-what-is-method-r○ Where time is spent?

● Proper instrumentation of the code, is it really important?○ http://tkyte.blogspot.com/2005/06/instrumentation.html○ http://en.wikipedia.org/wiki/Observer_effect_%28physics%29

● OS-level debug, trace and profiling tools, aren’t they allow to pinpoint any performance problem in MySQL?

● Instrumentation in MySQL - PERFORMANCE_SCHEMA○ History and engineers involved○ Features in MySQL 5.6○ How to use it?○ Any alternatives in MySQL world?○ Sources of additional information and tools

● The Good, The Bad and The Ugly in PERFORMANCE_SCHEMA

www.percona.com

Method R - common sense approach

1. Identify the most important task (the bottleneck)2. Measure its response time (R) in detail3. Optimize that response time in the most (economically)

efficient way4. Repeat until your system is (economically) optimal

Percona also tries to apply common sense:http://www.mysqlperformanceblog.com/2008/11/24/how-percona-does-a-mysql-performance-audit/

http://www.mysqlperformanceblog.com/2011/07/27/whats-required-to-tune-mysql/

http://www.mysqlperformanceblog.com/2011/05/05/the-two-even-more-fundamental-performance-metrics/

www.percona.com

Instrumentation

“Code instrumentation, what is it? To me, it is the fine art of making approximately every other line of your developed code “debug” of some sort. Trace information. With timestamps. And meaningful information.” - Tom Kyte

“In science, the term observer effect refers to changes that the act of observation will make on a phenomenon being observed. This is often the result of instruments that, by necessity, alter the state of what they measure in some manner.” - Wikipedia

www.percona.com

Trace files and debug printouts?

● Readily available in MySQL for years...○ http://dev.mysql.com/doc/refman/5.6/en/making-trace-files.html○ http://dev.mysql.com/doc/refman/5.6/en/dbug-package.html○ … --debug=d:t:i:o,/tmp/mysqld.trace○ -debug binaries only

● Additional tracing features are available in MySQL 5.6 for non-debug binaries also (for optimizer decision making, for example)○ http://dev.mysql.com/doc/internals/en/tracing-example.html

● Still far behind Oracle’s SQL Trace and trace events in general○ http://www.juliandyke.com/Diagnostics/Events/EventReference.html

● Let’s try to produce trace while test case for Bug #68079 is running:

mysql> set session debug='d:t:i:o,/tmp/mysqld.trace';Query OK, 0 rows affected (0.00 sec)

www.percona.com

Content of the trace file......T@9 : | | | | THD::enter_stage: /home/openxs/bzr2/percona-5.6/Percona-Server-5.6.10-alpha60.2/sql/sql_parse.cc:5383T@9 : | | | | >PROFILING::status_changeT@9 : | | | | <PROFILING::status_change 374T@9 : | | | | >close_thread_tablesT@9 : | | | | <close_thread_tables 1469T@9 : | | | | >stmt_causes_implicit_commitT@9 : | | | | <stmt_causes_implicit_commit 279T@9 : | | | | >MDL_context::release_transactional_locksT@9 : | | | | | >MDL_context::release_locks_stored_beforeT@9 : | | | | | <MDL_context::release_locks_stored_before 2753T@9 : | | | | | >MDL_context::release_locks_stored_beforeT@9 : | | | | | <MDL_context::release_locks_stored_before 2753T@9 : | | | | <MDL_context::release_transactional_locks 2908T@9 : | | | <mysql_execute_command 5431T@9 : | | | >~opt_trace_startT@9 : | | | <~opt_trace_start 230T@9 : | | | THD::enter_stage: /home/openxs/bzr2/percona-5.6/Percona-Server-5.6.10-alpha60.2/sql/sql_parse.cc:6676T@9 : | | | >PROFILING::status_changeT@9 : | | | <PROFILING::status_change 374...

● Debug printouts/traces/assertions/error log, is this enough?● Think about concurrency!● Too intrusive, changes behavior, serializes and wastes resources...

www.percona.com

Example of the problem to solve: Bug #68079

http://bugs.mysql.com/bug.php?id=68079CREATE TABLE `incident` ( `sys_id` char(32) NOT NULL DEFAULT '', `category` varchar(40) DEFAULT NULL,PRIMARY KEY (`sys_id`),KEY `incident_category` (`category`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;

insert into incident(sys_id) values (rand()*1000000);insert into incident(sys_id) select rand()*1000000 from incident; -- 13 times

CREATE TABLE `task` ( `sys_id` char(32) NOT NULL DEFAULT '', `u_root_cause` char(32) DEFAULT NULL, `u_business_impact_description` mediumtext, `u_business_impact_category` mediumtext,PRIMARY KEY (`sys_id`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;

insert into task(sys_id) values(rand()*1000000);insert into task(sys_id) select rand()*1000000 from task; -- 13 timesinsert into task(sys_id) select sys_id from incident limit 100;update incident set category=rand()*100000;

www.percona.com

Example of the problem to solve: Bug #68079

These queries scale well:select count(*), category from incident group by category;select * from task where sys_id=rand()*1000000;

This one does not scale even to 10 threads on 12 cores:select count(*), category from task inner join incident on task.sys_id=incident.sys_id group by incident.category;

mysql> explain select count(*), category from task inner join incident on task.sys_id=incident.sys_id group by incident.category;

| 1 | SIMPLE | incident | index | PRIMARY | incident_category | 123 | NULL | 8210 | Using index || 1 | SIMPLE | task | eq_ref | PRIMARY | PRIMARY | 96 | test.incident.sys_id | 1 | Using index |

www.percona.com

Example of the problem to solve: Bug #68079

mysqlslap -uroot --iterations=10 --concurrency=N --create-schema=test --no-drop --number-of-queries=1000 --query='select count(*), category from task inner join incident on task.sys_id=incident.sys_id group by incident.category'

where N in (1, 2, 4, 8, 10, 12, 24 … 2*number_of_cores … 1024 … 8192)

Results:

Benchmark Average number of seconds to run all queries: 8.512 seconds Minimum number of seconds to run all queries: 8.248 seconds Maximum number of seconds to run all queries: 8.844 seconds Number of clients running queries: 8 Average number of queries per client: 125…

Benchmark Average number of seconds to run all queries: 9.399 seconds Minimum number of seconds to run all queries: 9.137 seconds Maximum number of seconds to run all queries: 9.749 seconds Number of clients running queries: 10 Average number of queries per client: 100

www.percona.com

Example of the problem to solve: Bug #68079

Simple (?) join does NOT scale even to 10 threads...● Why does this happen?● Where time is spent?● Is there any way to tune MySQL server to improve

scalability for this query?● Maybe we need different hardware?● Any changes at schema level that may help?● How to get information to answer the questions above?● This is a performance tuning task to check tools,

approaches and code improvement ideas!

www.percona.com

Sources of information on performance

What sources of information and tools we can try to use?● show engine innodb status\G● show engine innodb mutex;● InnoDB-related tables in the INFORMATION_SCHEMA (?)● show global status like ‘innodb%’ (?)● userstat (Percona Server and others) (?)● show profiles● Poor man’s profiler (pt-pmp)● perf● oprofile● DTrace (?)● Maybe… PERFORMANCE_SCHEMA?● Discussion of the need for PERFORMANCE_SCHEMA and alternative

approaches - hence all the above

www.percona.com

SHOW ENGINE INNODB STATUS● Do you know how to read that?

○ http://dev.mysql.com/doc/refman/5.6/en/innodb-monitors.html - fine manual○ http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/ - the

best thing ever since Worcestershire sauce○ http://www.percona.com/doc/percona-server/5.6/diagnostics/innodb_show_status.html - just

better in Percona Server

● Not very useful for Bug #68079 on my QuadCore:

Per second averages calculated from the last 36 seconds-----------------BACKGROUND THREAD-----------------srv_master_thread loops: 121 srv_active, 0 srv_shutdown, 2473 srv_idlesrv_master_thread log flush and writes: 2594----------SEMAPHORES----------OS WAIT ARRAY INFO: reservation count 9477OS WAIT ARRAY INFO: signal count 3703Mutex spin waits 28946906, rounds 25108917, OS waits 8475RW-shared spins 209, rounds 5999, OS waits 182RW-excl spins 0, rounds 9, OS waits 0Spin rounds per wait: 0.87 mutex, 28.70 RW-shared, 9.00 RW-excl

www.percona.com

SHOW ENGINE INNODB STATUS...Purge done for trx's n:o < 2384 undo n:o < 0 state: running but idleHistory list length 20LIST OF TRANSACTIONS FOR EACH SESSION:---TRANSACTION 143558, not startedMySQL thread id 1, OS thread handle 0x7f01cc057700, query id 10678 localhost msandbox initshow engine innodb status...Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0Pending flushes (fsync) log: 0; buffer pool: 0285 OS file reads, 49 OS file writes, 49 OS fsyncs0.00 reads/s, 0 avg bytes/read, 0.06 writes/s, 0.06 fsyncs/sHash table size 276671, node heap has 0 buffer(s)0.00 hash searches/s, 142759.09 non-hash searches/s...Log sequence number 4732500...Last checkpoint at 47324800 pending log writes, 0 pending chkp writes50 log i/o's done, 0.06 log i/o's/second...Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000...Main thread process no. 5202, id 139645286401792, state: sleepingNumber of rows inserted 0, updated 0, deleted 0, read 891675610.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 144695.09 reads/s

www.percona.com

SHOW ENGINE INNODB MUTEX

● http://dev.mysql.com/doc/refman/5.6/en/show-engine.html● -debug builds give more information

○ count - how many times the mutex was requested○ spin_waits - how many times the spinlock had to run○ spin_rounds - the number of spinlock rounds○ os_waits - indicates the number of operating system waits. The mutex

was not unlocked during the spinlock and it was necessary to yield to the operating system and wait. You have only this value in non-debug binaries.

○ os_wait_times - indicates the amount of time (in ms) spent in operating system waits, if the timed_mutexes system variable is 1 (ON). If timed_mutexes is 0 (OFF, default), timing is disabled, so os_wait_times is 0.

| InnoDB | rw_lock_mutexes | count=0, spin_waits=0, spin_rounds=0, os_waits=0, os_yields=0, os_wait_times=0 |

www.percona.com

SHOW ENGINE INNODB MUTEX

● Looks like we have something immediately useful here for Bug #68079:

mysql [localhost] {msandbox} (test) > show engine innodb mutex;+--------+-------------------------+----------------+| Type | Name | Status |+--------+-------------------------+----------------+| InnoDB | log0log.cc:737 | os_waits=2 || InnoDB | buf0buf.cc:1242 | os_waits=2 || InnoDB | combined buf0buf.cc:992 | os_waits=66475 || InnoDB | dict0dict.cc:896 | os_waits=1 || InnoDB | log0log.cc:799 | os_waits=16 || InnoDB | combined buf0buf.cc:993 | os_waits=1 |+--------+-------------------------+----------------+6 rows in set (0.02 sec)

vi +992 /home/openxs/bzr2/mysql-5.6/storage/innobase/buf/buf0buf.cc mutex_create(PFS_NOT_INSTRUMENTED, &block->mutex, SYNC_BUF_BLOCK);

● Hardly is maintained properly in MySQL 5.6. Check my Bug #70819● Percona Server shows mutex name, not file and line number…| InnoDB | &buf_pool->LRU_list_mutex | os_waits=13 | InnoDB | &fil_system->mutex | os_waits=5073 | InnoDB | combined &block->mutex | os_waits=162396

www.percona.com

INFORMATION_SCHEMA?

InnoDB-related tables in the INFORMATION_SCHEMA:

● http://dev.mysql.com/doc/refman/5.6/en/innodb-i_s-tables.html - there are many...

● But as we know from INNODB STATUS it’s not about I/O or locks, it seems they do not add much useful for the case of Bug #68079…

● If only INNODB_METRICS - http://dev.mysql.com/doc/refman/5.6/en/innodb-metrics-table.html:

| NAME | varchar(193) | NO | | | || SUBSYSTEM | varchar(193) | NO | | | || COUNT | bigint(21) | NO | | 0 | || MAX_COUNT | bigint(21) | YES | | NULL | || MIN_COUNT | bigint(21) | YES | | NULL | || AVG_COUNT | double | YES | | NULL | |...| STATUS | varchar(193) | NO | | | || TYPE | varchar(193) | NO | | | || COMMENT | varchar(193) | NO | | | |

www.percona.com

INNODB_METRICS...

mysql> select name, count from information_schema.innodb_metrics where status='enabled' and type='status_counter';+---------------------------------+-----------+| name | count |+---------------------------------+-----------+...| buffer_pool_reads | 271 || buffer_pool_read_requests | 336246061 |...| buffer_pages_read | 270 || buffer_data_reads | 6623232 || buffer_data_written | 141312 || os_data_reads | 285 || os_data_writes | 89 || os_data_fsyncs | 89 || os_log_bytes_written | 40960 || os_log_fsyncs | 83 |...| innodb_rwlock_s_spin_waits | 210 || innodb_rwlock_x_spin_waits | 0 || innodb_rwlock_s_spin_rounds | 6029 || innodb_rwlock_x_spin_rounds | 9 || innodb_rwlock_s_os_waits | 183 || innodb_rwlock_x_os_waits | 0 |...

www.percona.com

SHOW GLOBAL STATUS LIKE ‘innodb%’● pt-mext helps - http://www.percona.com/doc/percona-toolkit/2.1/pt-mext.html● Let’s monitor the counters (51 starts with Innodb_) over 10 seconds:[openxs@chief msb_5_6_14]$ pt-mext -r -- /home/openxs/5.6.14/bin/mysqladmin --socket=/tmp/mysql_sandbox5614.sock -uroot ext -i10 -c3 >/tmp/mext.txt

Innodb_buffer_pool_read_requests 686556718 14726570...Innodb_buffer_pool_write_requests 160 3Innodb_data_fsyncs 176 3...Innodb_data_read 6623232 0Innodb_data_reads 285 0Innodb_data_writes 176 3...Innodb_data_written 250368 1536...Innodb_log_writes 160 3Innodb_num_open_files 7 0Innodb_os_log_fsyncs 166 3...Innodb_rows_read 342861007 7353994

www.percona.com

Percona Server and userstat?

http://www.percona.com/doc/percona-server/5.6/diagnostics/user_stats.html:

● Introduced in Google patch● Additional tables in INFORMATION_SCHEMA:

○ CLIENT_STATISTICS○ THREAD_STATISTICS○ USER_STATISTICS○ TABLE_STATISTICS○ INDEX_STATISTICS

● Easy to configure: ○ http://dom.as/2013/04/17/on-performance-schemas/

● Low performance impact we say:○ http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-user_statistics/

● Hardly this can help with mutexes or waits…● But some says it adds overhead:

○ “and then, once userstat=1 the regression is horrible.. over 30%... “ - guess who says that and why?

www.percona.com

SHOW PROFILE

● http://bugs.mysql.com/bug.php?id=24795 - SHOW PROFILE contribution by Jeremy Cole (MySQL 5.0)

● http://dev.mysql.com/doc/refman/5.6/en/show-profile.html - details…● Let’s try to profile the query while test for Bug #68079 is running:mysql [localhost] {msandbox} (test) > set profiling=1;Query OK, 0 rows affected, 1 warning (0.02 sec)

mysql [localhost] {msandbox} (test) > show warnings\G*************************** 1. row *************************** Level: Warning Code: 1287Message: '@@profiling' is deprecated and will be removed in a future release.1 row in set (0.00 sec)

mysql [localhost] {msandbox} (test) > select count(*), category from task inner join incident on task.sys_id=incident.sys_id group by incident.category;...103 rows in set (0.08 sec)

mysql [localhost] {msandbox} (test) > show profiles;

mysql [localhost] {msandbox} (test) > show profile for query 3;

www.percona.com

SHOW PROFILE● Profile of the query while test for Bug #68079 is running:| starting | 0.000063 || checking permissions | 0.000004 || checking permissions | 0.000004 || Opening tables | 0.000092 || init | 0.000023 || System lock | 0.000007 || optimizing | 0.000009 || statistics | 0.000037 || preparing | 0.000015 || Creating tmp table | 0.000023 || Sorting result | 0.000003 || executing | 0.000002 || Sending data | 0.081722 || Creating sort index | 0.000141 || end | 0.000005 || removing tmp table | 0.000008 || end | 0.000004 || query end | 0.000008 || closing tables | 0.000013 || freeing items | 0.000553 || cleaning up | 0.000017 |+----------------------+----------+21 rows in set, 1 warning (0.00 sec)

www.percona.com

Poor man’s profiler (gdb tricks)

● http://poormansprofiler.org/● http://www.percona.com/doc/percona-toolkit/2.1/pt-pmp.html● http://www.mysqlperformanceblog.com/2011/12/02/three-ways-that-the-

poor-mans-profiler-can-hurt-mysql/○ mysqld freezes for the duration of the process○ They say mysqld can crash (?)○ They say mysqld can be left in an unstable state (?)

● I’ve tried recently on QuadCore with test from bug #68079, not impressed:

10 libaio::??,os_aio_linux_collect,os_aio_linux_handle,fil_aio_wait,io_handler_thread,start_thread,clone 2 __memcpy_ssse3,store_key_field::copy_inner,copy,cp_buffer_from_ref,cmp_buffer_with_ref,join_read_key,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_execute_select,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,do_handle_one_connection,handle_one_connection,pfs_spawn_thread,start_thread,clone 1 update_ref_and_keys,make_join_statistics,JOIN::optimize,mysql_execute_select,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,do_handle_one_connection,handle_one_connection,pfs_spawn_thread,start_thread,clone

www.percona.com

Poor man’s profiler - Domas rulez!

● So, I was not impressed (found nothing enlightening), but it depends:○ “Usually pmp (poor man's profiler, making stacktrace snapshots)

producing more relevant information than performance schema reports.”● Google for something like site:bugs.mysql.com Mark Callaghan pmp:

○ http://bugs.mysql.com/bug.php?id=58037 - fixed in 5.5+ with MDL○ http://bugs.mysql.com/bug.php?id=57021 - it was used to find a problem

in Facebook’s patch…○ http://bugs.mysql.com/bug.php?id=61545 - still “Verified”, commit stalls

when mysql_binlog_send does disk read● If you are on Linux and don’t run MySQL with real profiler, try use pt-pmp

when server stalls, hangs or seems overloaded and unusable● Key to success for all kinds of profilers - open source of MySQL server!

www.percona.com

Let’s try real profiler - perf

● https://perf.wiki.kernel.org/index.php/Main_Page:○ “Among others, it provides per task, per CPU and per-workload

counters, sampling on top of these and source code event annotation”● http://bugs.mysql.com/bug.php?id=69236 - Single-threaded workloads can

be much slower on 5.6 than on 4.0. perf was used to prove that!● Sample session while test for Bug #68079 was running:

[openxs@chief ~]$ perf record -p 3310 sleep 20[ perf record: Woken up 1 times to write data ][ perf record: Captured and wrote 2.006 MB perf.data (~87654 samples) ]

[openxs@chief ~]$ perf report...

12.76% mysqld mysqld [.] btr_cur_search_to_nth_level(d 12.32% mysqld mysqld [.] buf_page_get_gen(unsigned lon 10.39% mysqld mysqld [.] my_strnncollsp_utf8

www.percona.com

perf output continued...

● Sample session while test for Bug #68079 was running:

... 7.28% mysqld mysqld [.] rec_get_offsets_func(unsigned 6.68% mysqld mysqld [.] mtr_memo_slot_release_func(mt 5.76% mysqld mysqld [.] page_cur_search_with_match(bu 4.56% mysqld mysqld [.] cmp_dtuple_rec_with_match_low 4.22% mysqld mysqld [.] row_search_for_mysql(unsigned 2.87% mysqld mysqld [.] ut_delay(unsigned long) 2.79% mysqld mysqld [.] pfs_mutex_enter_func(ib_mutex 2.68% mysqld mysqld [.] innobase_mysql_cmp(int, unsig 2.33% mysqld mysqld [.] cmp_whole_field(unsigned long 1.97% mysqld mysqld [.] row_mysql_store_col_in_innoba 1.86% mysqld mysqld [.] get_internal_charset 1.83% mysqld mysqld [.] mutex_spin_wait(ib_mutex_t*, 1.81% mysqld mysqld [.] pfs_mutex_exit_func(ib_mutex_ ...

● This was on QuadCore with 4 concurrent threads, MySQL 5.6.14● Now tell me that PERFORMANCE_SCHEMA impact with default settings is

low on MySQL 5.6...

www.percona.com

Let’s try another profiler - oprofile

● http://www.fromdual.ch/mysql-oprofile:○ Nice summary from Oli on how to use it!

● Sample session while test for Bug #68079 was running:[root@chief openxs]# opreport --demangle=smart --symbols --long-filenames --merge tgid /home/openxs/5.6.14/bin/mysqld | head -n 20...Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000samples % image name symbol name1208437 12.5180 /home/openxs/5.6.14/bin/mysqld btr_cur_search_to_nth_level(dict_index_t*, unsigned long, dtuple_t const*, unsigned long, unsigned long, btr_cur_t*, unsigned long, char const*, unsigned long, mtr_t*)1151258 11.9257 /home/openxs/5.6.14/bin/mysqld buf_page_get_gen(unsigned long, unsigned long, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned long, mtr_t*)1115969 11.5601 /home/openxs/5.6.14/bin/mysqld my_strnncollsp_utf8...242608 2.5131 /home/openxs/5.6.14/bin/mysqld pfs_mutex_enter_func(ib_mutex_t*, char const*, unsigned long)...142685 1.4780 /lib64/libc-2.13.so __memcpy_ssse3132334 1.3708 /home/openxs/5.6.14/bin/mysqld pfs_mutex_exit_func(ib_mutex_t*)...

www.percona.com

DTrace? One day, maybe...

● I wish I remember how to use it and have MacOS X or Solaris, or FreeBSD 9.x (on some good hardware) at hand…

● but I don’t. So, I had not tried to use DTrace to find out what’s wrong with that Bug #68079.

● Some useful references, still, if you ever plan to try:

○ http://dev.mysql.com/doc/refman/5.6/en/dba-dtrace-server.html - it’s documented in the manual, including static probes added

○ http://dom.as/tag/dtrace/ - check what Domas used to do with DTrace

○ http://cdn.oreillystatic.com/en/assets/1/event/21/Deep-inspecting%20MySQL%20with%20DTrace%20Presentation.pdf - from one of User Conferences...

○ http://dev.mysql.com/tech-resources/articles/getting_started_dtrace_saha.html - “native” Sun engineers also contributed

● I really miss Sun...

www.percona.com

Time to try PERFORMANCE_SCHEMA● We know functions where most of the time was spent (profiling)● We know that some mutex waits were long enough to force OS wait (switch

to other thread)● We need more details, specifically time spent waiting… So, time to use

PERFORMANCE_SCHEMA it seems● Sample output while test for Bug #68079 was running, naive attempt (the

time is in picoseconds):

mysql> select event_name, count_star, sum_timer_wait from performance_schema.events_waits_summary_global_by_event_name where event_name like 'wait/synch/mutex/innodb%' and count_star > 0 order by sum_timer_wait desc limit 5;+--------------------------------------------+------------+----------------+| event_name | count_star | sum_timer_wait |+--------------------------------------------+------------+----------------+| wait/synch/mutex/innodb/os_mutex | 71754726 | 11380485535846 || wait/synch/mutex/innodb/trx_sys_mutex | 60057 | 4191727883 || wait/synch/mutex/innodb/buf_pool_mutex | 8655 | 468303388 || wait/synch/mutex/innodb/mutex_list_mutex | 10310 | 427385546 || wait/synch/mutex/innodb/rw_lock_list_mutex | 8304 | 316946749 |+--------------------------------------------+------------+----------------+5 rows in set (0.00 sec)

www.percona.com

Basic ideas and terms

PERFORMANCE_SCHEMA (aka PS, P_S, PFS):● Inspired by Oracle Wait Interface● Separate database and storage engine● Records run time statistics via built-in instrumentation points● Designed to let us get the answers to the following questions:

○ Where was time spent?○ Who has used most of resources/time?○ What was executed to to use these?○ When it was executed?

● Appeared in MySQL 5.5, improved and ON by default in MySQL 5.6, and is documented and explained in details already:○ http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html

● Implemented by Marc Alff (http://marcalff.blogspot.com/)● Deal with instruments (including wait events), consumers, objects (tables for

now) that are monitored, threads (some of them background) that run and actors (users) that execute SQL statements (in stages).

www.percona.com

So, how to use it?

● The amount of information in the manual is huge, it’s not easy to start using P_S○ I wish it would be as easy as setting TIMED_STATISTICS to TRUE with ALTER

SYSTEM + few queries to V$SESSTATS, V$SYSSTATS or V$SESSION_WAIT● Marc Alff explains in http://marcalff.blogspot.com/2013/04/on-configuring-

performance-schema.html:○ “Define your goals” - this is clear, we have waits and we need to find out what

waits are most important based on time wasted (or waited)○ “Define what to instrument” - we need to instrument InnoDB waits mutex waits

and internal lock waits, but how to do this? Details next, but:■ "wait/synch/mutex/%", for mutexes■ "wait/synch/rwlock/%", for read write locks

○ “Define how much detail to collect” - well, we need counts and time spent for all InnoDB-related waits

○ “Provide sizing data” - autosizing based on max_connections?| performance_schema_max_mutex_instances | 15600 || performance_schema_max_rwlock_instances | 9000 |

○ “Monitor sizing problems” - memory used vs dropped statistics and impact

:

www.percona.com

How instruments are named?● For those who want solutions, not complains or comments...● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-instrument-

naming.html:○ "wait/io/file/%" - file io○ "wait/io/socket/%" - network io○ "wait/io/table/%" - table io○ "wait/lock/table/%" - table locks○ "wait/synch/cond/%" - conditions○ "wait/synch/mutex/%" - mutexes○ "wait/synch/rwlock/%" - read write locks○ "stage/%" - stages of statement execution (see show profile)○ "idle" - clients idle time.○ "statement/com/%" - commands in the client/server protocol○ "statement/sql/%" - SQL queries (and some more statement/…)

Hint: bold means “new in MySQL 5.6”, blue color means enabled by default

www.percona.com

Setup for Bug #68079 (waits/…)

● http://mysqlentomologist.blogspot.com/2013/01/how-to-use-performanceschema-to-check.html - my experience back at MySQL 5.6 RC stage...

● To set up InnoDB mutex waits monitoring via PERFORMANCE_SCHEMA in general with MySQL 5.6.x you should do at least the following:1. Start MySQL server with all mutex related instruments enabled at startup

(performance_schema=ON by default on recent 5.6.x), like this:

mysqld_safe --performance_schema_instrument='wait/synch/mutex/innodb/%=on' &

2. Connect to server and set up proper consumers explicitly, like this:

UPDATE performance_schema.setup_consumers SET enabled = 'YES' WHERE name like 'events_waits%';

3. Run your problematic load.4. Check waits using whatever tables you need

www.percona.com

Summary of my setup experience...● It's great to have Oracle engineers reading your Facebook posts and

replying (especially in a real time). Thank you, Mark Leith!● PERFORMANCE_SCHEMA in MySQL 5.6 can be used to study InnoDB

mutex waits● MySQL 5.6 manual is not clear enough about steps and details on

instruments involved to do the study explained above● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-filtering.html:

“Modifications to any of these tables affect monitoring immediately, with the exception of setup_actors. Modifications to setup_actors affect only foreground threads created

thereafter.”● Now compare to reality, complain in public, report a bug:

○ http://bugs.mysql.com/bug.php?id=68097 - “Manual does not explain that some P_S instruments must be enabled at startup”

● Get a reply from Dimitri Kravtchuk:“All instrumentation is dynamic EXCEPT mutexes which are created mainly on the MySQL server startup, so to get their instrumentation initialized you have to enable it on startup”

www.percona.com

Not that clear and easy to use?

www.percona.com

ps_helper To the Rescue!

ps_helper:● Developed by Mark Leith, MySQL expert @ Oracle

○ worked in MySQL Support team since 2005○ senior software development manager @Oracle now (MEM, maybe MEB)○ good Oracle RDBMS background

● Easier to use comparing to “raw” PERFORMANCE_SCHEMA● Ready to use collection of views, stored routines and...● Recently even an easy way to create ps, P_S, PFS or whatever synonym you like!● git clone https://github.com/MarkLeith/dbahelper.git dbahelper - if you prefer...● Installed in a separate schema (ps_helper)● IO/latency/waits/statement digests

Links:● http://www.markleith.co.uk/ps_helper/● https://github.com/MarkLeith/dbahelper/archive/master.zip● https://www.slideshare.net/Leithal/performance-schema-andpshelper

www.percona.com

ps_helper: basicsInstall:openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot test < ps_helper_56.sql

Check:mysql> show tables;+-------------------------------------------------+| Tables_in_ps_helper |+-------------------------------------------------+| _digest_95th_percentile_by_avg_us || _digest_avg_latency_by_avg_us || check_lost_instrumentation |...53 rows in set (0,00 sec)

mysql> select routine_name from information_schema.routines where routine_schema='ps_helper';+-------------------------------+| routine_name |+-------------------------------+| analyze_statement_digest |...

20 rows in set (0,00 sec)

www.percona.com

ps_helper: why is it useful?Have some real fun:openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot ps_helper < procedures/create_synonym_db.sqlWarning: Using a password on the command line interface can be insecure.

mysql> call create_synonym_db('performance_schema', 'ps');Query OK, 0 rows affected (2,35 sec)

mysql> use psReading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -A

Database changedmysql> show tables;+----------------------------------------------------+| Tables_in_ps |+----------------------------------------------------+| accounts |...

Use ps_helper.format_time(SUM(sum_timer_wait)):● Check Bug #70252 to find out what may happen if you don’t use it● Check https://www.slideshare.net/Leithal/performance-schema-andpshelper

www.percona.com

Some history behind...

● “The Good, The Bad and The Ugly”:“The film's title has entered the English language as an idiomatic expression. Typically used when describing something thoroughly, the respective phrases refer to upsides, downsides and the parts that could, or should have been done better, but were not.” - Wikipedia

● “For Three Men The Civil War Wasn’t Hell. It Was Practice!”○ Guess who are these men, in case of PERFORMANCE_SCHEMA…○ Maybe more than three…

● Read the WorkLogs:○ http://dev.mysql.com/worklog/task/?id=2360 - “Why we're doing this”○ http://dev.mysql.com/worklog/task/?id=4674 - “Setup for Actors”○ http://dev.mysql.com/worklog/task/?id=4513 - “Implement it…”

● Where the name comes from?“Proposals for the schema name were DYNAMIC (but that is a reserved word in standard SQL), PERFORMANCE_SCHEMA (Jim Winstead's suggestion), PERFORMANCE (Peter's preference), and DBTrace because it sounds like DTrace. There was a vote (see dev-private thread "WL#2360 Vote: schema name and table name"). The winner was PERFORMANCE_SCHEMA.” - Peter Gulutzan

“

www.percona.com

PERFORMANCE_SCHEMA in MySQL 5.6

Useful reading (and names to remember):● http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html - article from Mark

Alff on how to configure PERFORMANCE_SCHEMA

● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-configuration.html - RTFM

● https://www.facebook.com/notes/mysql-at-facebook/my-mysql-is-faster-than-your-mysql/10151250402570933 - influence of P_S in 5.6.10 and... 5.1.63 or 5.1.52+FB patch are still fastest in many cases

● http://dimitrik.free.fr/blog/archives/2013/07/mysql-performance-why-performance-schema-overhead.html - overhead explained and measured by Dimitri Kravtchuk:

● http://www.percona.com/webinars/using-mysql-56-performance-schema-troubleshoot-typical-workload-bottlenecks - webinar by Peter Zaitsev (May 15, 2013)

● http://mysql.wisborg.dk/category/mysql/performance_schema/ - useful posts from Jesper Krogh on the topic. He had created yet another set of tools to make out life easier, ps_tools: http://mysql.wisborg.dk/2013/10/10/mysql-connect-2013-ps_tools/

● http://mysqlblog.fivefarmers.com/tag/performance_schema/ - practical use of PERFORMANCE_SCHEMA, by Todd Farmer

www.percona.com

PERFORMANCE_SCHEMA - The Good

● Monitoring is available on all platforms supported by MySQL● Activating the Performance Schema causes no (few) changes in server behavior

○ No persistent on-disk storage○ Memory allocation at server startup○ Processing happens mostly at events retrieval○ No new keywords or threads added

● Overhead is small comparing to some other ways to monitor● Problems/failures in P_S code should not prevent server from working normally● It is easy to add new instrumentation points (they say)● Under active development:

○ Very few bugs - 4 active in MySQL 5.6 (of 47) as of November 1, 2013○ Bug #70018 (PFS Overhead on frequent connect/disconnect) fixed in 5.6.15!○ Even more features in MySQL 5.7: check Bug #69527 (MDL locks!)

● Used by Oracle internally a lot for real things (MEM, performance improvements)● As it’s based on proper ideas, eventually it’s going to become an ultimate solution

for MySQL performance monitoring (and replace everything else)

www.percona.com

Let me produce some flame...

www.percona.com

PERFORMANCE_SCHEMA - The Bad

● Performance overhead in default configuration is notable in some cases (10-30%, see bug reports and 3rd party studies)

● Additional memory use is notable (see Bug #68514):○ “Why is this not a bug? On my x86-64 with 5.6.12 when

performance_schema=1 then the extra RAM is:max_connections * 196kb” - Mark Callaghan

● Mutex and rw-lock waits instrumentation must be enabled at server startup

● How complete is the instrumentation?● There are bugs and missing parts in the manual● There are bugs in implementation● Is this kind of instrumentation really good? (SQL, no trace)● Any critical posts and even some bug reports are sometimes

considered as a kind of a personal insult: "godfather" case...

www.percona.com

Bugs in P_S and fine manual

Bug #68413 - Facebook: “performance_schema overhead is at least 10%”Bug #69782 - Old files not being removed from performance_schema.file_instancesBug #70628 - Wrong instrumentation interface for mysql_cond_timedwait Bug #70697 - Probably not a bug, but indication that manual for SHOW ENGINE PERFORMANCE_SCHEMA STATUS is still not clear

Bug #68574 - “No instrumentation for InnoDB files in P_S on windows” - same with 5.6.14, by the way...Bug #68785 - new features (connection attributes) work well together (or not)...Bug #66589 - still not fixed (do not use mysqlhotcopy). One of the last bugs I verified in Oracle...Bug #68514 - beware of memory usage impact with P_S

Bug #68097 - still some details to document in the manual clearly

www.percona.com

PERFORMANCE_SCHEMA - The Ugly

● Names: schema name, long table names… you know whom to blame (and thank for ps_helper). I wish we have public synonyms in MySQL, so I’d create V$SYSSTATS (Bug #68855). You can create views, but that’s overhead, no?

● Why additional views and procedures (ps_helper) are not there by default?● Some important things and storage engines are not properly instrumented yet,

MDL locks, for example… ups, seems fixed in 5.7.3● Heuristics to control size (read Bug #68514 again)… less than obvious and I’d

prefer more control● Make up your mind on what should be in P_S vs I_S vs SHOW statements. Think

about SHOW PROCESSLIST vs I_S.PROCESSLIST vs P_S.THREADS● Want to know what else users care about? Check feature requests for P_S (11

“Verified” as of November 2, 2013), some of them are limitations “by design”:○ Bug #69881 - “Try to compute elapsed time for an event which has not completed” - Todd Farmer○ Bug #66515 - “I have a problem with PS instrumentation that can only be enabled at server start.” -

Mark Callaghan○ Bug #70383 - “Implement a SELECT-syntax/statement as a synonym for SHOW ENGINE

performance_schema STATUS” - Peter Laursen

www.percona.com

Do we have a hope?

www.percona.com

Do you still care about Bug #68079?

“mysql-trunk (a.k.a 5.7.3): Scales up to 32 threads reasonably well, the HW has 48 logical CPUs” - Sunny Bains, October 18, 2013

“There is some confusion here, there are two issues in this bug report. One is the page mutex and the other is the page re-lock. For this query we acquire it in S mode all the time. If we remove the page mutex overhead then this S mode rwlock becomes the next bottleneck. However, this doesn't mean that the original problem is not fixed, it simply means that the fix requires further work that is more involved, that's all.“ - Sunny Bains, October 30, 2013

“These same numbers can be achieved by using "PARTITION BY KEY() PARTITIONS N". By using partitions you can reduce the contention on the block mutex.” - Sunny Bains, October 30, 2013

● Does Oracle care about MySQL?● Was PERFORMANCE_SCHEMA used to find the bottlenecks?

www.percona.com

Thank you!

Questions and Answers

Please, report bugs to:http://bugs.mysql.comUse “Affects Me” button!