Download - Harrison fisk masteringinnodb-diagnostics
- 1. Mastering InnoDB Diagnostics Harrison Fisk Senior Principal Technical Support Engineer
2. Overview
- Lots of information
-
- Hard to understand
- 3. Easy to get overwhelmed
- Hard to figure out what is useful
4. Two methods for troubleshooting
-
- Symptom based
- What we will be doing today
Data based 5. Information Sources 6. Sources
- MySQL info
-
- SHOW ENGINE INNODB STATUS
- 7. InnoDB monitors
8. SHOW GLOBAL STATUS 9. INFORMATION_SCHEMAtables
-
- SHOW PROCESSLIST
- 10. SHOW MUTEX STATUS
OS info
-
- vmstat / iostator Performance Monitor
- 11. Gdb ( Poor mans profiler (PMP))
12. OS specific tools 13. SHOW INNODB STATUS
- Be careful of truncation
- innodb_status_file
14. Table monitors Ensure enough time has passed 15. Exact format varies
- InnoDB built-in
16. InnoDB plugin 17. XtraDB 18. Data Dictionary 19. Problems
- InnoDB: Error: table test/parent already exists in InnoDB internal InnoDB: data dictionary. Have you deleted the .frm file InnoDB: and not used DROP TABLE? InnoDB: Cannot find table test/child2 from the internal data dictionary InnoDB: of InnoDB though the .frm file for the table exists. Maybe you InnoDB: have deleted and recreated InnoDB data files but have forgotten InnoDB: to delete the corresponding .frm files of InnoDB tables?
- Various errors about not being in sync
20. Can't drop and/or create tables or databases 21. Example errors: 22. Diagnostics
- InnoDB table monitor
-
- Dumps InnoDB data dictionary
- 23. Output to error log
24. CREATE TABLE innodb_table_monitor (id int) ENGINE=innodb; 25. Interpreting
- Ignore system tables
- NamedSYS_
Ignore hidden fields from InnoDB
-
- DB_ROW_ID
- 26. DB_TRX_ID
27. DB_ROLL_PTR
- Ignore hidden indexes
- GEN_CLUST_INDEX
28. Solutions
- Ensure.frmmatches InnoDB
-
- Create.frmas needed
- Create another table with same structure
- 29. Copy.frmto the name of the table
Drop orphaned.frmfiles
- If not in InnoDB
30. Remove them 31. Crashing 32. Problem
- mysqld Crash
-
- Signal 11
- Segfault
Signal 6
- Abort
33. InnoDB loves this one
- May not actually notice immediately
-
- mysqld_safe restarts
34. Diagnostics
- Error log
-
- Hopefully it has info in it
- Core file
-
- Not enabled by default
- 35. OS often restricts setuid cores
- /proc/sys/fs/suid_dumpable
36. coreadm
- General log
-
- When all else fails
37. Interpreting Error log
- Error log
-
- Look for signal type
- Signal 11
- 38. Signal 6
Backtrace
- May or may not be present
- OS
39. Version In 5.0 and before you need to resolve it
- resolve_stack_dump
Search bug system for function names 40. Interpreting Cores
- Must be enabled
41. Memory image produced 42. Requires exactly the same binary and libraries 43. With rpm ensure debuginfo is installed 44. Non-stripped binaries 45. Load with gdb
-
- thread apply all bt
- 46. Use PMP to summarize if needed
- Solaris:pstack
47. Windows: Minidump/DrWatson/Windbg 48. Solutions
- Find cause of the crash
-
- InnoDB doing on purpose
- 49. Real crash
- Find query causing problem
-
- Stop doing it
- 50. Report bug to us
- Upgrade
-
- Lots of bugs fixed each release
51. Locking 52. Problems
- Deadlocks
-
- Real deadlocks
-
- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
- Lock wait timeouts
- ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
General locking slowness 53. Diagnostics
- SHOW INNODB STATUS
-
- TRANSACTIONSsection
- innodb_lock_monitortable
-
- Changes output ofSHOW INNODB STATUS
- INFORMATION_SCHEMA
-
- INNODB_LOCKS
- 54. INNODB_LOCK_WAITS
- SHOW GLOBAL STATUS LIKE 'innodb_row_lock%'
55. Interpreting
- Find what lock is needed
-
- Who is holding conflicting lock
- Look for type of lock
-
- X (exclusive)
- 56. S (shared)
- Relatively rare
Others
- Look for what the lock is on
-
- Table (primary key)
- 57. Index
58. Solutions
- Fix queries to reduce locks
59. Shared locks
-
- Find source
- 60. Eliminate
- Upgrade to 5.1 run in read committed
61. Remove source of lock
- AUTO-INC
-
- Upgrade to 5.1
- innodb_autoinc_lock =2?
- Long running transactions
62. Performance 63. Overview
- Lots of potential performance issues
-
- Disk I/O
- Log files
- 64. Table space
CPU
- Too many rows being read
65. Thread thrashing
- Gather lots of information up front
-
- Big picture, not details first
- Ensure it is during problem time
66. Disk: Log files
- Many small serialized synchronous writes
67. InnoDB redo log writes bottleneck
-
- fdatasync () is expensive in many setups
- innodb_flush_log_at_trx_commit
-
- 1 Safest, sync every commit
- 68. 2 Write every commit
69. 0 Write only every second 70. Diagnostics
- SHOW INNODB STATUS
-
- State of transactions
- COMMITTED IN MEMORY
- 71. PREPARED
FILE I/O
- Pending flushes (fsync) log
72. fsyncs/s
- iostat
-
- Number of write operations ( w/s )
-
- awaitand/orsvctm
73. Solutions
- InnoDB plugin group commit
74. Changeinnodb_flush_log_at_trx_commit 75. Better disks
-
- BBU write cache
- 76. Faster writes
77. Disk: Tablespace
- Reading and writing your actual tables and indexes
78. More common with very large tables
-
- Compare to buffer pool size
- Sometimes caused by read ahead in InnoDB
79. Diagnostics
- SHOW INNODB STATUS
-
- Buffer pool hit rate
- Watch absolute values too
FILE I/O
- Reads/s
- Avg bytes/read shows reada head
Pending reads Read ahead
- Pages read ahead/evicted without access
- iostat
- avgqu-sz
80. await 81. Solutions
- Optimize queries
82. Increaseinnodb_buffer_pool_size 83. New LRU algorithm in InnoDB plugin 84. Tune/disable readahead
- In plugin, can be tuned
85. In non-plugin
- Use gdb hack
86. set srv_startup_is_before_trx_rollback_phase=1 Buy better disks 87. CPU: Lots of Rows
- Bad queries can read lots of rows
-
- Or lots and lots of smaller queries
- Data fully cached
-
- Lots of buffer pool reading
- 88. Often smaller dataset
- Eventually will bottleneck InnoDB on CPU
89. Diagnostics
- SHOW INNODB STATUS
-
- ROW OPERATIONS
- Reads/s
- SHOW GLOBAL STATUS LIKE 'Handler%'
-
- Type of row operations going on
- Slow query log
-
- log-queries-not-using-indexes
90. Solutions
- Fix your queries
-
- Ensure proper index usage
- Do less queries
-
- External caching
- Scale reads
-
- Replication
91. CPU: Thread Thrashing
- InnoDB can have issues at high concurrency
92. Later releases help reduce this dramatically
-
- InnoDB plugin
- 93. Further versions
- XtraDB
94. MySQL 5.5 95. Diagnostics
- SHOW INNODB STATUS
-
- Lots of queued threads
- TRANSACTIONS
- 96. Queries in queue
-
- ROW OPERATIONS
-
-
- reads/s is low
-
- vmstat
-
- Very high CPU usage
- 97. Lots of cs context switches
- SHOW MUTEX STATUS
98. Solutions
- innodb_thread_concurrency
-
- Very high or 0 works best until it doesn't
-
- Sometimes very low works best
- 99. A bit of a black art in tuning
- innodb_concurrency_tickets
-
- Size based on number of rows being touched
- Use newer version
- Lots of work to help this in InnoDB Plugin
100. Questions? 101. The presentation is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle. 102.