basic mysql troubleshooting for oracle dbas

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Basic MySQL Troubleshootingfor Oracle DBAs

Sveta SmirnovaSenior Principal Technical Support EngineerMySQL SupportSeptember 29, 2014


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program Agenda

Introduction

Basic troubleshooting techniques

High concurrency issues

High availability solutions

More information

1

2

3

4

5


Introduction


Click icon to add picture

MySQL architecture

Base - Installation layout - Log filesConnectors - Clients - APIsOptimizerCaches & buffersStorage enginesManagement


Typical installation layout

• datadir

– Schema directory– Table files: *frm, *ibd, *MYD, *MYI, *par, etc.

– Trigger files

– Schema

– …

– InnoDB shared tablespace

– Log files– InnoDB log files

– Binary, relay logs

– Error log

– Slow query log

– General query log

Configurable

You can setup custom path for each component

Including custom path for tables


Connectors

• Clients–MySQL CLI

–MySQL Workbench

–MySQL Enterprise Monitor (MEM)

–Oracle Enterprise Manager with MEM plugin

• APIs

–Exist for most popular programming languages

–C, C++, JDBC, PHP, Python, Net, ODBC

The way you connect to MySQL Server


Plugins

• Storage engines

• Full-text parsers

• Daemon

• INFORMATION_SCHEMA

• Semisynchronous Replication

• Audit

• Authentication

• Password-validation

• Protocol Trace

MySQL Server is highly configurable via plugins

5.7


Storage engines

• Own data

• Own index format

• Own locking model

• Own diagnostic

• Own log files

• CHECK TABLE

From troubleshooting point of view


Basic troubleshooting techniques


MySQL Access Privilege System

• Privileged client cannot connect

• Unprivileged client can connect

• Privileged user cannot perform operation

• Unprivileged user has undesired access

Typical issues



• No roles by default, limited user limits

• All records are in the mysql database (schema)

• Pluggable authentication since version 5.5

• Connections● TCP/IP with login-password

● Socket (Unix)

● Named pipe (Windows)

Overview



select user, host from mysql.user order by user desc, host desc;

• Most descriptive host first, then wildcard

• Socket connection by default on Unix

Where to find out why you have connection issues?



SHOW GRANTS [FOR user@host]

• Grant tables● mysql.db● mysql.tables_priv● mysql.columns_priv● mysql.procs_priv● mysql.proxies_priv

SELECT USER(), CURRENT_USER()

Wrong access



mysql> show grants;

+--------------------------------------------------------------------+

| Grants for root@localhost |

+--------------------------------------------------------------------+

| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION|

| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION |

+--------------------------------------------------------------------+


Wrong access



mysql> show grants for foo@'%';

+-----------------------------------------------------+

| Grants for foo@% |

+-----------------------------------------------------+

| GRANT USAGE ON *.* TO 'foo'@'%' |

| GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%' |

+-----------------------------------------------------+


Wrong access


Error handling

mysql> select max (f1) from t1;

ERROR 1630 (42000): FUNCTION test.max does not exist. Check the

'Function Name Parsing and Resolution' section in the Reference Manual

mysql> select * from t1 where "f1"=1;

Empty set, 1 warning (0.05 sec)

mysql> show warnings;

+-----------+--------+----------------------------------------------------+

| Level | Code | Message |

+-----------+--------+----------------------------------------------------+

| Warning | 1292 | Truncated incorrect DOUBLE value: 'f1' |

+-----------+--------+----------------------------------------------------+

1 row in set (0.00 sec)

Errors vs warnings



• Error information● mysql_errno● mysql_error

• Warnings● mysql_info● mysql_sqlstate● mysql_warning_count

Application (C API)


Error handling

[sveta@delly ~]$ perror 1630

MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):

FUNCTION %s does not exist. Check the 'Function Name Parsing and

Resolution' section in the Reference Manual


MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated

incorrect %.32s value: '%.128s'


OS error code 2: No such file or directory


MySQL error code 150: Foreign key constraint is incorrectly formed

perror


Error handling

● GET DIAGNOSTICS● GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;● GET DIAGNOSTICS CONDITION 1 code = RETURNED_SQLSTATE, msg = MESSAGE_TEXT;

● http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html

In stored routines

http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html


What can affect query execution?

• Startup options or system variables

• How optimizer creates query plan

• Storage engine used

• Parallel execution next section

You run a query, it does not return an error, but still behaves not as expected


System variables and startup options

• Global● Control parameters, necessary for all server processes

– Location of server files: datadir etc.

– Shared buffers

– More

• Session● Control connection-specific parameters

• http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html

Scope

http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html



• SET [GLOBAL] var_name = NEW_VAL

• Command line option

• Configuration file● In default location

– http://dev.mysql.com/doc/refman/5.6/en/option-files.html

● Specified by option --defaults-file

How to set

http://dev.mysql.com/doc/refman/5.6/en/option-files.html



• Global options and few session options● A user with privilege SUPER

• Session options● Anybody

• There is no limits!

Who can change



• Those which control behavior of whole server● Once at server startup

● Can start with low values, then grow to specified

• Connection options● For every connection when connection opens

• Operation-specific● For every operation when needed

When allocated



• SHOW [GLOBAL] STATUS

• GLOBAL● Since server start

• SESSION● For operations in current session

● Can be reset

– FLUSH STATUS

How to control



mysql> show global status like 'Handler_read_rnd_next'\G

*************************** 1. row ***************************

Variable_name: Handler_read_rnd_next

Value: 27


mysql> show status like 'Handler_read_rnd_next'\G

*************************** 1. row ***************************

Variable_name: Handler_read_rnd_next

Value: 7


How to control



• Record currently used variables● SHOW [GLOBAL] VARIABLES

• Make change dynamically if possible● SET [GLOBAL] var_name=NEW_VAL

• Test in one session first

• Then change global variable

• Change configuration file after you are happy with results

Troubleshooting best practices



• Record currently used variables● SHOW [GLOBAL] VARIABLES

• Start mysqld with option --no-defaults● This option must be first one!

• Check if problem is solved

• Change variable values one-by-one until you find one which leads to the problem

When affecting option is not known


Diagnostic tool: INFORMATION_SCHEMA

• Contain metadata information● Tables

● Indexes

● Other

• Allows to create plugins● InnoDB plugins

– We will discuss them later

• Oracle analog● Data Dictionary Views

When affecting option is not known


MySQL Optimizer

• EXPLAIN is less powerful if compare with Oracle● It is improved in version 5.7.3

● Graphic EXPLAIN in MySQL Workbench 6.0

• EXPLAIN EXTENDED• EXPLAIN PARTITIONS• EXPLAIN FORMAT=JSON• INFORMATION_SCHEMA.TRACE

• Status variables 'Handler_%'

Overview

Default in 5.7


EXPLAIN in Oracle

EXPLAIN PLAN SET statement_id = 'example_plan4' FOR

SELECT h.order_number, l.revenue_amount, l.ordered_quantity

FROM so_headers_all h, so_lines_all l

WHERE h.customer_id = :b1

AND h.date_ordered > SYSDATE30

AND l.header_id = h.header_id ;

Plan

--------------------------------------------------

SELECT STATEMENT

NESTED LOOPS

TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL

INDEX RANGE SCAN SO_HEADERS_N1

TABLE ACCESS BY INDEX ROWID SO_LINES_ALL

INDEX RANGE SCAN SO_LINES_N1

http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm

http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm


EXPLAIN in MySQL

mysql> EXPLAIN SELECT user, host FROM mysql.user\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: user

type: index

possible_keys: NULL

key: PRIMARY

key_len: 228

ref: NULL

rows: 4

Extra: Using index



EXPLAIN in MySQL

mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.user\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: user

type: index

possible_keys: NULL

key: PRIMARY

key_len: 228

ref: NULL

rows: 4

filtered: 100.00

Extra: Using index

1 row in set, 1 warning (0.00 sec)

Variant: EXTENDED

mysql> SHOW WARNINGS\G******* 1. row ******* Level: Note Code: 1003Message: /* select#1 */ select `mysql`.ùser`.Ùser` AS ùser`,`mysql`.ùser`.`Host` AS `host` from `mysql`.ùser`1 row in set (0.00 sec)


EXPLAIN in MySQL

mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\G

************** 1. row **************

EXPLAIN: {

"query_block": {

"select_id": 1,

"table": {

"table_name": "user",

"access_type": "index",

"key": "PRIMARY",

...

Variant: FORMAT=JSON

...

"used_key_parts": [

"Host",

"User"

],

"key_length": "228",

"rows": 4,

"filtered": 100,

"using_index": true

}

}

}

1 row in set, 1 warning (0.03 sec)


EXPLAIN in MySQL

• http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen

• http://dev.mysql.com/doc/refman/5.6/en/explain-output.html

• MySQL EXPLAIN in Practice [HOL9232] (past)

• MySQL 5.7: What’s New in the Parser and the Optimizer? [CON2830] (past)

• How to Analyze and Tune MySQL Queries for Better Performance [TUT3157] (past)

• Using MySQL Workbench Performance Tools [HOL9237] (past)

More information

http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen

http://dev.mysql.com/doc/refman/5.6/en/explain-output.html


MySQL Optimizer

mysql> flush status;

Query OK, 0 rows affected (0.00 sec)

mysql> SHOW STATUS LIKE 'Handler_read_%';

+----------------------------+-------+

| Variable_name | Value |

+----------------------------+-------+

| Handler_read_first | 0 |

| Handler_read_key | 0 |

| Handler_read_last | 0 |

| Handler_read_next | 0 |

| Handler_read_prev | 0 |

| Handler_read_rnd | 0 |

| Handler_read_rnd_next | 0 |

+----------------------------+-------+


Handler_% status variables


MySQL Optimizer

• INFORMATION_SCHEMA.TRACE● http://dev.mysql.com/doc/internals/en/optimizer-tracing.html

● join_preparation, join_optimization, join_execution– considered_execution_plans, refine_plan, more

• Query Analyzer in MEM● http://dev.mysql.com/doc/mysql-monitor/3.0/en/mem-quan-using.html

• Graphic EXPLAIN in MySQL Workbench 6.0

Other tools

http://dev.mysql.com/doc/internals/en/optimizer-tracing.html

http://dev.mysql.com/doc/mysql-monitor/3.0/en/mem-quan-using.html


MySQL Storage Engines

• Own way to handle● Corruption

● Index statistics

● CHECK TABLE to check for errors

• They care about physical data, so all data information are on their level

• Options usually start from engine name● myisam_*, innodb_*, custom_*

Specifics


MySQL Storage Engines

• Transactional storage engine

• Physical layout● *frm file – table definition

● Shared tablespace

● *ibd file – tablespace for individual table

– Optional: --innodb_file_per_table– Recommended and default since 5.6

● Redo log files

• OPTIMIZE TABLE = ALTER + ANALYZE• Automatic startup check

InnoDB



• Query or transaction waits a lock, held by another one

• Fight for system resources

• Resource overload

• Resource underload

Common problems


Transactions and their relation to locks

• Lock types

• Levels

–MDL

– Table-level

– Row-level

• What do they lock–Read locks

– Block writes

–Write locks

– Block both reads and writes

● Transactions● Server-level

– MDL locks● Engine level

– Table locks

– Row locks● AUTOCOMMIT

● Supported


Locking issues

• SHOW [FULL] PROCESSLIST● Universal tool which lists all currently running connections

• SHOW ENGINE INNODB STATUS• INFORMATION SCHEMA

● PROCESSLIST● InnoDB tables

• PERFORMANCE SCHEMA● MDL locks

● Table locks

● Server-level transactions

Diagnostic tools


Storage engine specifics

• Transactions and engine-level locks are done at the engine level

• Storage engines have own diagnostic tools

• Use them when hit an issue with such a lock

Diagnostic tools



• SHOW ENGINE INNODB STATUS• InnoDB monitors

● Pseudo-tables

– Turn periodical logging into error log● CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;

– Lock monitor● Changes output format of InnoDB Monitor● CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;

● Options

– innodb_status_output– innodb_status_output_locks

InnoDB



• Tables in INFORMATION_SCHEMA● INNODB_TRX● INNODB_LOCKS● INNODB_LOCK_WAITS● INNODB_METRICS

– Options innodb_monitor_*● Option innodb_print_all_deadlocks

InnoDB


Diagnostic tool: performance_schema

• Monitors internal operations● Events

● Waits

● Mutexes

● Statements

● Stages

● MDL, table-level locks, transactions

● Memory

● Replication

• Similar to Oracle wait interface

For whole server

5.7+


Diagnosting locks

• Table-level locks● PROCESSLIST: “Waiting for table lock”

● P_S.TABLE_HANDLES

• Row-level locks● InnoDB monitors, SHOW ENGINE INNODB STATUS● Tables in INFORMATION_SCHEMA● Option --innodb_print_all_deadlocks

• MDL locks● PROCESSLIST: “Waiting for metadata lock”

● P_S.METADATA_LOCKS

Summary

5.7+

5.7+


High availability solutions


MySQL Cluster

• Special storage engine: NDB

• Stores data on two or more physical machines

• Two or more copies

• General troubleshooting techniques● Applicable

• Specific NDB storage engine techniques


MySQL Replication

• Always available, but you must setup it before use

• Asynchronous master-slave

• Master● Keeps all updates in separate file: binary log

• Slave● IO thread read updates from master and stores in relay log file

● SQL thread executes updates

Overview


Replication IO thread

• Access error● Check slave error log

● SHOW SLAVE STATUS● P_S.replication_connection_status● Try to connect using normal MySQL client using slave credentials

– SHOW GRANTS● Fix privileges on master

● Restart slave

Communication issues


Replication SQL thread

• Simple master-slave● Data is different on master and slave

– Replication event can not be applied● Different errors on master and slave

● Slave lags far behind the master

• Circular replication or other writes in addition to slave SQL thread● Data is different on master and slave

Typical issues



• Was the table modified besides the SQL thread?● How?

● Can it affect content of the table in the wrong way?

• Are the table definitions same on master and slave?● MySQL Utilities

– mysqlrplsync, mysqldbcompare, mysqldiff

• Is it possible that master events was applied in wrong order?● Use mysqlbinlog to find queries which caused the issue

● Check master's application to find what is wrong

Data is different on master and slave



• Lock issues● InnoDB

• Triggers● SET GLOBAL slave_skip_counter● Synchronize tables!

• Different options● Start slave with master's options, then check

Events from master were applied in wrong order



• Threads● Master runs in multiple update threads

● Slave uses single

• Seconds_behind_master is growing

• Tune slave performance● Buffers

● Indexes (for statement-based replication)

Slave lags far behind the master


More information


OpenWorld sessions

• Practical MySQL Optimization [CON5522]● Peter Zaitsev, 3:30 PM, Moscone South - 262

• What’s New in MySQL 5.7 Security [CON1985] ● Georgi Kodinov, 4:45 PM, Moscone South - 262

• MySQL Cost Model [CON3163]● Olav Sandstå, 5:30 PM, Moscone South - 250

October, 01


OpenWorld sessions

• MySQL Troubleshooting with the Performance Schema [HOL9215]● Sveta Smirnova, Lig Isler-turmelle

● 8:30 AM, Hotel Nikko - Monterey

• MySQL Enterprise Edition Features in Practice [HOL9216]● Matt Lord, 10:00 AM, Hotel Nikko - Monterey

• MySQL Performance: Demystified Tuning and Best Practices [CON5097] ● Dimitri Kravtchuk, 10:45 AM, Moscone South - 252

October, 02


Further reading

• MySQL Troubleshooting book● http://shop.oreilly.com/product/0636920021964.do

• Marc Alff's Performance Schema blog● http://marcalff.blogspot.ru/

• Planet MySQL● http://planet.mysql.com/

● http://dev.mysql.com/support/blogs/

● http://mysqlserverteam.com/

http://shop.oreilly.com/product/0636920021964.do

http://marcalff.blogspot.ru/

http://planet.mysql.com/

http://dev.mysql.com/support/blogs/

http://mysqlserverteam.com/


References

• MySQL User Reference Manual● http://dev.mysql.com/doc/refman/5.6/en/index.html

• Knowledge Management Database

• Forums● http://forums.mysql.com

• Bug trackers● http://bugs.mysql.com

● Oracle Internal Bugs database

• My Oracle Support● https://support.oracle.com

http://dev.mysql.com/doc/refman/5.6/en/index.html

http://forums.mysql.com/

http://bugs.mysql.com/

https://support.oracle.com/


You will find me at

• https://twitter.com/svetsmirnova

• https://blogs.oracle.com/svetasmirnova/

• http://www.slideshare.net/SvetaSmirnova/

https://twitter.com/svetsmirnova

https://blogs.oracle.com/svetasmirnova/

http://www.slideshare.net/SvetaSmirnova/


?


Thank you!

basic mysql troubleshooting for oracle dbas

Software

oracle andor

priv mysql

user host

db mysql

sole discretion of oracle

user desc

user order

mysql architectureclick