mysql monitoring 101
DESCRIPTION
In this presentation I’ll be discussing the following beginner points to understanding and creating monitoring. * Why Monitor? * What’s the minimum to Monitor? * How to monitor? * Monitoring Software Options. * How to use the most basic of monitoring to help * The basics of graphing results * The rule of Everything * The important on Application metrics and timings For a very little investment in time, simple monitoring can be in place, and I can guarantee it will be of benefit to any system. The basis of monitoring are metrics that combined with application measurements can provide trending insights, bottleneck understanding and provide valuable feedback about your growing site.TRANSCRIPT
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Monitoring 101Simple stuff to save your bacon
Ronald Bradford
Principal - 42SQL
MySQL Users Conference
Santa Clara - April 2009Version 1.1 22.Apr.2009
http://ronaldbradford.com
MySQL Monitoring 101
What is Your Situation?How do you know your website is down?
http://ronaldbradford.com
MySQL Monitoring 101
How do you know your website is down?
❖The users will let me know
❖That's somebody else's problem❖Our site is never down❖ Email/SMS/Pager
Is this you?
http://ronaldbradford.com
MySQL Monitoring 101
You have a performance problem now?
❖How long has it been happening?
❖ Is it a new problem or a re-occurring problem?❖Has it gradually become worse over time?
Is this you?
http://ronaldbradford.com
MySQL Monitoring 101
When did you last recover from a backup?
❖We need to think about doing backups
❖We use MySQL replication❖That's somebody else's problem❖We verify our backups worked (not recovery)
Is this you?
http://ronaldbradford.com
MySQL Monitoring 101
Goal
http://ronaldbradford.com
MySQL Monitoring 101
Four Goals to take away
Monitoring a MySQL Server(s)
❖Know what to monitor❖Know how you can monitor
❖ Learn practices to diagnose problems❖Have a foundation of historical information
http://ronaldbradford.com
MySQL Monitoring 101
Overview
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Overview
❖Hardware
❖ Software
❖MySQL
http://ronaldbradford.com
MySQL Monitoring 101
Hardware
http://ronaldbradford.com
MySQL Monitoring 101
Know Your Physical Resources
❖ CPU
❖ Memory
❖ Disk
❖ Network
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Hardware Resources
Bare Essentials
❖ vmstat❖ iostat❖ ps❖ netstat❖ ifconfig❖ top
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Hardware Resources
Very Valuable
❖ sar❖ mpstat❖ dstat❖ ethtool
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Hardware Resources
Very Valuable
❖ /proc❖ /cpuinfo❖ /meminfo❖ /loadavg
http://ronaldbradford.com
MySQL Monitoring 101
Identifying Resource Bottlenecks
❖ Which physical resource is being stressed?
❖ Why?
http://ronaldbradford.com
MySQL Monitoring 101
Identifying Resource Bottlenecks
$ vmstat 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s2 s4 in sy cs us sy id 1 0 0 17114496 2962832 63 211 5180 71 71 0 0 0 0 71 0 4763 4625 3183 72 2 26 1 0 0 17115252 2963184 4 6 439 0 0 0 0 0 0 7 0 3251 4021 2795 79 2 20 1 0 0 17115252 2963144 2 1 0 0 0 0 0 0 0 16 0 3748 4427 3049 89 2 10 1 0 0 17115252 2962912 11 11 1360 0 0 0 0 0 0 47 0 4083 4210 2752 79 2 19 0 0 0 17115248 2962744 4 9 1428 0 0 0 0 0 0 10 0 1072 1015 754 17 1 82 1 0 0 17115248 2962664 4 0 0 0 0 0 0 0 0 42 0 3755 3818 2549 69 2 29 1 0 0 17115240 2962520 2 4 246 0 0 0 0 0 0 3 0 3231 3992 2833 79 1 19 1 0 0 17115228 2962400 3 8 1347 0 0 0 0 0 0 10 0 3706 4339 3063 88 2 10 1 0 0 17115220 2962256 16 11 853 0 0 0 0 0 0 52 0 4275 4201 2672 79 2 19 1 0 0 17115220 2962100 3 5 131 0 0 0 0 0 0 7 0 2742 3275 2385 62 2 36 0 0 0 17115220 2962064 3 0 0 0 0 0 0 0 0 29 0 1887 1247 917 19 1 81 1 0 0 17115216 2961664 7 13 1885 0 0 0 0 0 0 18 0 3597 4674 3270 82 2 17 2 0 0 17114344 2961428 75 281 1953 13 13 0 0 0 0 12 0 3697 4746 3203 87 2 11 1 0 0 17114332 2961396 19 14 211 6 6 0 0 0 0 53 0 4409 4476 2862 77 2 21 1 0 0 17115316 2961544 5 7 426 0 0 0 0 0 0 8 0 2658 3478 2371 64 1 34 0 0 0 17115316 2961500 5 0 0 0 0 0 0 0 0 39 0 2310 1802 1268 26 1 73 1 0 0 17115316 2961428 5 7 823 0 0 0 0 0 0 11 0 3454 4473 3096 82 2 16 1 0 0 17115308 2961224 7 11 920 0 0 0 0 0 0 13 0 3569 4377 3024 86 2 12 1 0 0 17115308 2961016 12 9 870 3 3 0 0 0 0 37 0 4687 5227 3250 82 2 17
Can you spot Memory swapping, Disk I/O and CPU usage changes?
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Single CPU Kernel on 4 Processor Server$ uptime 10:32:22 up 196 days, 11:46, 2 users, load average: 3.86, 4.08, 4.26
$ uname -a Linux db1 2.4.20-6 #1 Thu Feb 27 10:06:59 EST 2003 i686 i686 i386 GNU/Linux [root@db1 mysql]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 10 model name : Pentium III (Cascades) stepping : 1 cpu MHz : 701.636 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1399.19
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Single CPU Kernel - Disabled from boot loader$ more /boot/grub/grub.conf# grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,0)# kernel /vmlinuz-version ro root=/dev/sda3# initrd /initrd-version.img#boot=/dev/sdadefault=0timeout=10splashimage=(hd0,0)/grub/splash.xpm.gz root (hd0,0) kernel /vmlinuz-2.4.20-6 ro root=LABEL=/ initrd /initrd-2.4.20-6.img[root@db1 mysql]# ls /boot/vmlinuvmlinux-2.4.20-6 vmlinux-2.4.20-6smp vmlinuz vmlinuz-2.4.20-6 vmlinuz-2.4.20-6smp
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Excessive Network Load (20GB uncompressed backup)$ ethtool eth0 Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: d Wake-on: d Link detected: yes
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Degraded RAID
❖ Who checks RAID status?❖ How is it checked?
http://ronaldbradford.com
MySQL Monitoring 101
Additional Resources
• Identify Bottlenecks• CPU
• Memory
• Check back for additional articles
http://ronaldbradford.com/blog/identifying-resource-bottlenecks-cpu-2009-03-31/http://ronaldbradford.com/blog/identifying-resource-bottlenecks-memory-2009-04-02/
http://ronaldbradford.com
MySQL Monitoring 101
Software
http://ronaldbradford.com
MySQL Monitoring 101
Know Your Running Software
❖ Operating System
❖ Database
❖ Other
http://ronaldbradford.com
MySQL Monitoring 101
Operating System Specifics
❖Kernel Version❖ 32/64 bit Kernel
❖ 32bit software running on 64bit kernel❖Review all running software❖ Should it be running?
http://ronaldbradford.com
MySQL Monitoring 101
MySQL
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Log Files
❖ Error Log ❖ Could be OS Log
❖ Slow Query Log
1
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Process
❖ Is mysqld running?❖What is the Memory Usage?
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Memory Usage
❖RSS and VSZ
$ ps -eopid,fname,rss,vsz,user,command | grep -e "RSS" -e "mysql" PID COMMAND RSS VSZ USER COMMAND 5463 grep 764 5204 ronald grep -e RSS -e mysql13894 mysqld_s 596 3936 root /bin/sh /usr/bin/mysqld_safe13933 mysqld 4787812 5127208 mysql /usr/sbin/mysqld --basedir=/usr --datadir=/vol/mysql/mysqldata --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock13934 logger 608 3840 root logger -p daemon.err -t mysqld_safe -i -t mysqld
$ ps -eopid,fname,rss,vsz,user,command | grep " mysqld " | grep -v grep | awk '{print $3,$4}'4787820 5127208
http://ronaldbradford.com/blog/are-you-monitoring-rss-vsz-2009-03-08/
2
http://ronaldbradford.com
MySQL Monitoring 101
MySQL System Variables
❖Based on my.cnf configuration❖Customizable in running server
❖ SHOW GLOBAL VARIABLES
http://dev.mysql.com/doc/refman/5.1/en/show-variables.html
3
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Status Variables
❖ Provide valuable clues to the state of your servers
❖GLOBAL and SESSION scope
❖ SHOW [GLOBAL|SESSION] STATUS
http://dev.mysql.com/doc/refman/5.1/en/show-status.html
4
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Connections
❖ SHOW FULL PROCESSLIST
http://dev.mysql.com/doc/refman/5.1/en/show-status.html
5
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Storage Engines
❖ SHOW ENGINE INNODB STATUS❖ Well incorporated in 5.1 Status Variables
http://dev.mysql.com/doc/refman/5.1/en/show-status.html
6
http://ronaldbradford.com
MySQL Monitoring 101
Your MySQL Data
❖Know your data❖ Disk footprint❖ Growth
7
http://ronaldbradford.com
MySQL Monitoring 101
Instance Summary
select table_schema,
sum(data_length+index_length)/1024/1024 as total_mb,
sum(data_length)/1024/1024 as data_mb,
sum(index_length)/1024/1024 as index_mb,
count(*) as tables
from information_schema.tables
group by table_schema
order by 2 desc;
http://ronaldbradford.com
MySQL Monitoring 101
Instance Summary
+----------------------------+----------------+----------------+---------------+--------+| table_schema | total_mb | data_mb | index_mb | tables |+----------------------------+----------------+----------------+---------------+--------+| xxxxxxxxxxxxxxxx_531 | 25301.01871777 | 19678.43473339 | 5622.58398438 | 189 || xxxxxxxxxxxxxxxx_528 | 18983.51311207 | 14739.53361988 | 4243.97949219 | 96 || stats | 14803.18925285 | 5469.20292473 | 9333.98632813 | 1082 || xxxxxxxxxxxxxxxx_4 | 9631.98090553 | 7600.90180397 | 2031.07910156 | 1119 || xxxxxxxxxxxxxxxx_501 | 9141.53529739 | 7263.91127396 | 1877.62402344 | 72 || system_logs | 7973.60180283 | 3237.59008408 | 4736.01171875 | 1553 || stats_cpu | 4544.43988991 | 3317.35981178 | 1227.08007813 | 1243 || xxxxxxxxxxxxxxxx_555 | 4024.49787521 | 3436.00080490 | 588.49707031 | 293 || stats_xxxxxxxxx | 3712.71884155 | 2159.92196655 | 1552.79687500 | 1536 |...
http://ronaldbradford.com
MySQL Monitoring 101
Schema Summary
select table_name,engine,row_format, table_rows, avg_row_length,
(data_length+index_length)/1024/1024 as total_mb,
(data_length)/1024/1024 as data_mb,
(index_length)/1024/1024 as index_mb
from information_schema.tables
where table_schema= DATABASE()
order by 6 desc;
http://ronaldbradford.com
MySQL Monitoring 101
Schema Attributes
❖Collations❖Text/Blob
❖ Storage Engines
http://ronaldbradford.com
MySQL Monitoring 101
Schema Attributes
select table_schema,engine,table_collation,
count(*) as tables
from information_schema.tables
where table_schema=DATABASE()
group by table_schema,engine,table_collation;
select table_schema,table_name,column_name,data_type
from information_schema.columns
where table_schema= DATABASE()
and ( data_type LIKE '%TEXT' OR data_type like '%BLOB');
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring
http://ronaldbradford.com
MySQL Monitoring 101
System Monitoring
❖ dstat$ cat dbc_os.sh
#!/bin/sh
DATETIME=`date +%y%m%d.%H%M`
LOG_DIR="/opt/monitor/log/"INTERVAL=5COUNT=`expr 3600 \/ $INTERVAL`LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.txtCSV_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.csvMYSQL_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.mysql.csv
/usr/bin/dstat --time --cpu --mem --disk --net --proc --page --swap --load --nocolor --noheaders --output ${CSV_LOG_FILE} $INTERVAL $COUNT > $LOG_FILE 2>&1sed -e "1,7d" -e "s/-/,/" -e "s/ /,/" ${CSV_LOG_FILE} > ${MYSQL_LOG_FILE}exit 0
http://ronaldbradford.com
MySQL Monitoring 101
System Monitoring
❖ Example dstat output-----time----- ----total-cpu-usage---- ------memory-usage----- -dsk/total-> date/time |usr sys idl wai hiq siq| used buff cach free| read writ>19-04 17:00:01| 7 2 91 1 0 0|1151M 475M 1850M 480M| 51k 859k>19-04 17:00:06| 5 0 95 0 0 0|1168M 475M 1850M 462M|5734B 134k>19-04 17:00:11| 3 0 96 0 0 0|1175M 475M 1850M 456M| 0 168k>19-04 17:00:16| 4 0 92 3 0 0|1184M 475M 1850M 447M| 20k 625k>19-04 17:00:21| 3 0 97 0 0 0|1142M 475M 1850M 488M| 0 307k>19-04 17:00:26| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 9830B>19-04 17:00:31| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 2458B>19-04 17:00:36| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 34k>19-04 17:00:41| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 4915B>19-04 17:00:46| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 31k>19-04 17:00:51| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 42k>19-04 17:00:56| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 30k>19-04 17:01:01| 1 0 99 0 0 0|1151M 475M 1850M 480M| 0 58k>19-04 17:01:06| 0 0 100 0 0 0|1151M 475M 1850M 480M| 0 9011B>19-04 17:01:11| 8 0 91 0 0 0|1153M 475M 1850M 478M| 0 3277B>19-04 17:01:16| 2 0 99 0 0 0|1152M 475M 1850M 479M| 0 37k>19-04 17:01:21| 0 0 100 0 0 0|1152M 475M 1850M 479M| 0 6554B>
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Monitoring
❖ hourly.sh❖ OS❖ vmstat❖ps❖uptime
❖ MySQL❖Variables❖Status❖Process List❖ Innodb Engine Status❖Table Status
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Analysis
❖ statpack - Human Readable STATUS ❖ Database Activity❖ Statement Activity❖ Prepared Statements❖ Admin Commands❖ Thread Cache❖ Table Cache❖ MyISAM Key Cache❖ .... http://www.markleith.co.uk/?p=21
http://ronaldbradford.com
MySQL Monitoring 101
statpack - Read/Write Ratio==================================================================================================== Uptime: 17 days 17 hours 22 mins Snapshot Period 1: 59 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Statement Activity ====================================================================================================
SELECT: 13,503,876 3,798.56 4,298,170,239 (94.25%) INSERT: 91,101 25.63 25,327,062 (0.56%) UPDATE: 782,004 219.97 220,640,296 (4.84%) DELETE: 9,674 2.72 2,485,643 (0.05%) REPLACE: 0 0.00 4,980 (0.00%) INSERT ... SELECT: 0 0.00 0 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 46,422 13.06 13,700,478 (0.30%) ROLLBACK: 0 0.00 13 (0.00%)
http://ronaldbradford.com
MySQL Monitoring 101
statpack - Unnecessary SQL
==================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Statement Activity ====================================================================================================
SELECT: 16,042 267.37 8,177,050 (46.03%) INSERT: 5,838 97.30 1,826,616 (10.28%) UPDATE: 1,109 18.48 738,546 (4.16%) DELETE: 2,018 33.63 1,374,983 (7.74%) REPLACE: 0 0.00 0 (0.00%) INSERT ... SELECT: 0 0.00 27 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 5,708 95.13 2,161,232 (12.17%) ROLLBACK: 5,746 95.77 3,485,828 (19.62%)
http://ronaldbradford.com
MySQL Monitoring 101
statpack - Ineffective Query Cache
==================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Query Cache ====================================================================================================
QCache Hits / SELECT: 11.11% QCache Hit/Qcache Insert: 31.24% Qcache Hits/Invalidations: 0.00% SELECTs: 16,042 267.37 8,177,050 Query Cache Hits: 1,257 20.95 1,022,301 Query Cache Inserts: 4,181 69.68 2,250,062 Queries Not Cached: 11,864 197.73 5,932,162 Cache Low Memory Prunes: 0 0.00 0 Total Cache Blocks: -421 -7.02 12,051 Queries In Cache: -595 -9.92 4,194 Cache Free Blocks: 770 12.83 3,628
http://ronaldbradford.com
MySQL Monitoring 101
Graphing Options
❖RRDtool❖ gnuplot
❖Google Charts API
❖ vmplot.sh❖ http://www.bigdbahead.com/?p=302❖ http://ronaldbradford.com/blog/extending-
vmplot-2009-03-31/
http://ronaldbradford.com
MySQL Monitoring 101
vmplot.sh output
http://ronaldbradford.com
MySQL Monitoring 101
Always graph your results
40%
30%
50%
60%
70%
43%
http://ronaldbradford.com
MySQL Monitoring 101
Products
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Enterprise Monitor
❖Commercial Product - $$$❖Developed by Sun/MySQL
http://www.mysql.com/products/enterprise/monitor.html
http://ronaldbradford.com
MySQL Monitoring 101
Nagios
❖Network and Application Monitoring❖Open Source
http://www.nagios.org/
http://ronaldbradford.com
MySQL Monitoring 101
Cacti
❖Generic Network Graphing❖Uses RRDtool
❖MySQL specific templates
http://www.cacti.net/http://code.google.com/p/mysql-cacti-templates/
http://ronaldbradford.com
MySQL Monitoring 101
Additional
❖Hyperic❖Ganglia
❖Munin❖Big Brother❖MONyog❖ EM7❖ Zenoss
http://ronaldbradford.com
MySQL Monitoring 101
External Services
❖ Pingdom❖WebPerform
http://ronaldbradford.com
MySQL Monitoring 101
What's missing
❖ Some Standards❖Active Community
❖ Integration for monitoring all stack products
❖Application specific metrics❖ e.g. Number of new customers per hour❖ e.g. Total Sales for day
http://ronaldbradford.com
MySQL Monitoring 101
Professional Help is Available
❖Two decades IT expertise❖ 10 years in MySQL❖ System/Data Architecture❖ Database Performance and Tuning❖ High Availability and Scalability❖ Education and Training
http://ronaldbradford.com