basic mysql troubleshooting for oracle database administrators

Basic MySQL Troubleshootingfor Oracle Database Administrators

Sveta SmirnovaPrincipal Support Engineer

October, 29, 2015

Table of Contents

•MySQL Server overview

•Basic troubleshooting techniques

•High concurrency issues

•Replication

2 www.percona.com

MySQL Server overview

3 www.percona.com

MySQL architecture

Connectors: C, JDBC, ODBC, Python, ...

Connection Pool: Authentication, Caches

SQL interface

Parser

Optimizer

Caches and Buffers:Global

Engine-specific

Storage engines: InnoDB, TokuDB, ...

File system: Data, Index, logs, other files

• Base• Installation layout• Log files

• Connectors• Clients, APIs

• Optimizer• Cache and buffers• Storage engines• Management

4 www.percona.com

Typical installation layout

• datadir• Schema directory

• Table and trigger files: *.frm, *.ibd, *.MYD,*.par, etc.

• Schema• ...• InnoDB shared tablespace• Log files

• InnoDB redo log files• Binary, relay log files• Error log• Slow query log• General query log

• Configurable• You can setup

custom path foreach component

• Including custompaths for tables

5 www.percona.com

Connectors for MySQL server

• Clients• MySQL CLI• MySQL Workbench• Many other graphical and web-based

• APIs• Exist for most popular programming languages• C, C++, JDBC, PHP, Python, Net, ODBC, more

6 www.percona.com

Monitoring solutions

• Command-line• Percona Toolkit• MySQL Utilities

• With GUI• MySQL Enterprise Monitor (MEM)• MEM plugin for Oracle Enterprise Manager• VividCortex• Many others

7 www.percona.com

Plugins

• Storage engines

• Authentication

• Audit

• Query rewrite

• More

8 www.percona.com

Storage engines

• Own data

• Own index format

• Own locking model

• Own diagnostic

• Own log files

• CHECK TABLE

9 www.percona.com

Basic troubleshootingtechniques

10 www.percona.com

Error processing

• Warnings, errors, and notesmysql> select max (f1) from t1;

ERROR 1630 (42000): FUNCTION test.max does not exist.

Check the ’Function Name Parsing and Resolution’ section

in the Reference Manual

mysql> select * from t1 where "f1"=1;

Empty set, 1 warning (0.05 sec)

mysql> show warnings;

+-----------+--------+----------------------------------------------------+

| Level | Code | Message |

+-----------+--------+----------------------------------------------------+

| Warning | 1292 | Truncated incorrect DOUBLE value: ’f1’ |

+-----------+--------+----------------------------------------------------+

1 row in set (0.00 sec)

11 www.percona.com

Error handling in applications

• Error information• mysql error• mysql errno

• Warnings and notes• mysql info• mysql sqlstate• mysql warning count

12 www.percona.com

Error handling in stored routines

• GET DIAGNOSTICS

• GET DIAGNOSTICS rows = ROW COUNT,conditions = NUMBER;

• GET DIAGNOSTICS CONDITION 1 code =RETURNED SQLSTATE, msg = MESSAGE TEXT;

• SIGNAL/RESIGNAL

• Can be used for custom errors

13 www.percona.com

More information about errors

[sveta@delly ~]$ perror 1630

MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):

FUNCTION %s does not exist. Check the ’Function Name Parsing and Resolution’

section in the Reference Manual


MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE):

Truncated incorrect %.32s value: ’%.128s’


OS error code 2: No such file or directory


MySQL error code 150: Foreign key constraint is incorrectly formed

14 www.percona.com

MySQL Access Privilege System

• No roles by default, limited user limits• All records are in the mysql database (schema)• Pluggable authentication since version 5.5• Connections

• TCP/IP with login-password• Socket (Unix)• Named pipe (Windows)

15 www.percona.com

Common user access issues

• Privileged client cannot connect

• Unprivileged client can connect

• Privileged user cannot perform operation

• Unprivileged user has undesired access

16 www.percona.com

Wrong access checklist

• SHOW GRANTS [FOR user@host]• Grant tables

• mysql.db• mysql.tables priv• mysql.columns priv• mysql.procs priv• mysql.proxies priv

• SELECT USER(), CURRENT USER()

18 www.percona.com

Wrong access examplemysql> show grants;

+--------------------------------------------------------------------+

| Grants for root@localhost |

+--------------------------------------------------------------------+

| GRANT ALL PRIVILEGES ON *.* TO ’root’@’localhost’ WITH GRANT OPTION|

| GRANT PROXY ON ’’@’’ TO ’root’@’localhost’ WITH GRANT OPTION |

+--------------------------------------------------------------------+


mysql> show grants for foo@’%’;

+-----------------------------------------------------+

| Grants for foo@% |

+-----------------------------------------------------+

| GRANT USAGE ON *.* TO ’foo’@’%’ |

| GRANT ALL PRIVILEGES ON ‘test‘.* TO ’foo’@’%’ |

+-----------------------------------------------------+


19 www.percona.com

Connection statistics in P S

• Tables accounts, users, hostsmysql> select user, host, current_connections as cur,

-> total_connections as total from performance_schema.accounts;

+------+-----------+-----+-------+

| user | host | cur | total |

+------+-----------+-----+-------+

| foo | localhost | 0 | 3 |

| root | localhost | 1 | 3 |

| NULL | NULL | 14 | 17 |

+------+-----------+-----+-------+


• HOST CACHE

20 www.percona.com

Connection statistics in P S

• Tables accounts, users, hosts• HOST CACHE

• Content of DNS cache• Errors from:

• Name Server

• Connection

• Authentication

• max connect errors, max user errors, etc.

• Your first assistant in case of connection issue

20 www.percona.com

Performance Schema• Monitors interval operations

• Statements• Stages• Locks• Memory• Variables• Replication• IO• Mutexes and waits

• Similar to Oracle wait interface21 www.percona.com

What can affect query execution?

• You run a query, it does not return an error, but stillbehaves not as expected

• It can be:• Startup options or system variables• How optimizer creates query plan• Storage engine used• Parallel execution - next big section

22 www.percona.com

System variables and options: scope

• Global• Control parameters, necessary for all server

processes• Location of server files: datadir etc.• Shared buffers• More

• Session• Control connection-specific parameters

• MySQL option tables

23 www.percona.com

http://dev.mysql.com/doc/refman/5.7/en/mysqld-option-tables.html

System Variables: how to set

• SET [GLOBAL] var name = NEW VAL

• Command-line option

• Configuration file

• In default location

• Specified by option –defaults-file

24 www.percona.com

http://dev.mysql.com/doc/refman/5.7/en/option-files.html

System Variables: who can change

• Global options and few session options

• A user with privilege SUPER

• Session options

• Anybody

• There are no limits!

25 www.percona.com

Buffers: when allocated

• Those which control behavior of whole server• Once at server startup• Can start with low values, then grow to specified

• Connection options• For every connection when connection opens

• Operation-specific• For every operation when needed• Can be allocated more than once per query

26 www.percona.com

System variables: control before 5.7• SHOW [GLOBAL] STATUS• Tables in Information Schema

• GLOBAL ,SESSION VARIABLES• GLOBAL ,SESSION STATUS

• GLOBAL• Since server start

• SESSION• For operations in current session• Can be reset

• FLUSH STATUS

27 www.percona.com

System status variables: example

mysql> show global status like ’Handler_read_rnd_next’\G

*************************** 1. row ***************************

Variable_name: Handler_read_rnd_next

Value: 27


mysql> show status like ’Handler_read_rnd_next’\G

*************************** 1. row ***************************

Variable_name: Handler_read_rnd_next

Value: 7


28 www.percona.com

Information Schema

• Contains metadata information• Tables• Indexes• Other

• Allows to create plugins• InnoDB plugins

• Similar to Data Dictionary Views in Oracle

29 www.percona.com

System variables: control in 5.7

• Performance Schema tables• variabes by*• user variables by*• status by*

• Statistics grouped by• Global• Session• Thread• Account/Host/User

30 www.percona.com

System variables: best practices

• Record currently used variables• SHOW [GLOBAL] VARIABLES

• Make change dynamically if possible• SET [GLOBAL] var name=NEW VAL

• Test in one session first• Then change global variable• Change configuration file after you are happy with

results31 www.percona.com

When affecting option is not known

• Record currently used variables• SHOW [GLOBAL] VARIABLES

• Start mysqld with option –no-defaults• This option must be first one!

• Check if problem is solved• Change variable values one-by-one until you find one

which leads to the problem

32 www.percona.com

MySQL Optimizer

• EXPLAIN is less powerful if compare with Oracle• Visual EXPLAIN in MySQL Workbench

• EXPLAIN EXTENDED• Should be followed by SHOW WARNINGS

• EXPLAIN PARTITIONS• EXPLAIN FORMAT=JSON• INFORMATION SCHEMA.TRACE• Status variables ’Handler %’

33 www.percona.com

EXPLAIN in Oracle

• http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htmEXPLAIN PLAN SET statement_id = ’example_plan4’ FOR

SELECT h.order_number, l.revenue_amount, l.ordered_quantity

FROM so_headers_all h, so_lines_all l

WHERE h.customer_id = :b1

AND h.date_ordered > SYSDATE30

AND l.header_id = h.header_id ;

Plan

--------------------------------------------------

SELECT STATEMENT

NESTED LOOPS

TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL

INDEX RANGE SCAN SO_HEADERS_N1

TABLE ACCESS BY INDEX ROWID SO_LINES_ALL

INDEX RANGE SCAN SO_LINES_N1

34 www.percona.com

http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm

http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm

EXPLAIN in MySQLmysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.user\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: user

type: index

possible_keys: NULL

key: PRIMARY

key_len: 228

ref: NULL

rows: 4

filtered: 100.00

Extra: Using index

1 row in set, 1 warning (0.00 sec)

mysql> SHOW WARNINGS\G

******* 1. row *******

Level: Note

Code: 1003

Message: /* select#1 */ select ‘mysql‘.‘user‘.‘User‘ AS ‘user‘, ‘mysql‘.‘user‘.‘Host‘ AS ‘host‘ from ‘mysql‘.‘user‘

35 www.percona.com

EXPLAIN: overview

mysql> explain extended select * from t1 join t2 where t1.int_key=1;

+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+

| id | select_type | table | type | possible_keys | key | key_len | ref | rows | f... | Extra |

+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+

| 1 | SIMPLE | t1 | ref | int_key,ik | int_key | 5 | const | 4 | 100. | NULL |

| 1 | SIMPLE | t2 | index | NULL | pk | 9 | NULL | 6 | 100. | Using index; |

Using join buffer |

(Block Nested Loop) |

+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+

2 rows in set, 1 warning (0.00 sec)

Note (Code 1003): /* select#1 */ select ‘test‘.‘t1‘.‘pk‘ AS ‘pk‘,‘test‘.‘t1‘.‘int_key‘ AS ‘int_key‘,‘test‘.‘t2‘.‘pk‘

AS ‘pk‘,‘test‘.‘t2‘.‘int_key‘ AS ‘int_key‘ from ‘test‘.‘t1‘ join ‘test‘.‘t2‘ where (‘test‘.‘t1‘.‘int_key‘ = 1)

Number of select

Select type

Tables, for which information is printed

How data is accessed

Possible keys

Key, which was actually used

Length of the key

Which columns were compared with the index

Number of examined rows

% of filtered rowsrows x filtered / 100 — number of rows,which will be joined with another table

Additional information

Table, for which information is printedProduct of rows here: how many rows in all tables will be accessed

For this example estimated value is 4*6 = 24

Actual (optimized) query as executed by MySQL Server

36 www.percona.com

EXPLAIN FORMAT=JSONmysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\G

*************************** 1. row ***************************

EXPLAIN: {

"query_block": {

"select_id": 1,

"table": {

"table_name": "user",

"access_type": "index",

"key": "PRIMARY",

"used_key_parts": [

"Host",

"User"

],

"key_length": "228",

"rows": 8,

"filtered": 100,

"using_index": true

}

}

}37 www.percona.com

MySQL Optimizer: other tools

• INFORMATION SCHEMA.OPTIMIZER TRACE

• join preparation, join optimization, join execution• considered execution plans, refine plan, more

• Query Analyzer in MEM

• Visual EXPLAIN in MySQL Workbench

39 www.percona.com

http://dev.mysql.com/doc/internals/en/optimizer-tracing.html

http://dev.mysql.com/doc/mysql-monitor/3.0/en/mem-quan-using.html

MySQL Storage Engines

• Own way to handle• Corruption• Index statistics• CHECK TABLE to check for errors

• They care about physical data, so all data information ison their level

• Options usually start from engine name• myisam *, innodb *, custom *

40 www.percona.com

Storage Engines explained

• InnoDB• TokuDB• Popular non-transactional engines

• MyISAM - Previous default engine• Memory• Federated• CSV• Blackhole

41 www.percona.com


• InnoDB

• Most used storage engine• Physical layout

• *frm file – table definition

• Shared tablespace

• *ibd file – tablespace for individual table - –innodb file per table = 1

• Redo log files

• Automatic startup check

41 www.percona.com


• InnoDB

• Own locking model: row-based• Troubleshooting tools

• InnoDB Monitors (SHOW ENGINE InnoDB STATUS)

• Tables in Information Schema

• Instrumentation in Performance Schema

41 www.percona.com


• TokuDB

• Fractal tree• Optimized for compression and writes• Troubleshooting tools

• SHOW ENGINE tokudb STATUS

• Information schema tables

• Status variables

41 www.percona.com


• Popular non-transactional engines

• Server provides locking model• Table-level locks• Can have own instrumentation• MyISAM

• Status variables

• Performance Schema

• Set of utilities, names start from myisam *

41 www.percona.com

High concurrency issues

42 www.percona.com

Common concurrency issues

• Query or transaction waits a lock, held by another one

• Fight for system resources

• Resource overload

• Resource underload

43 www.percona.com

Lock types and transactions

• Lock types• Levels

• MDL• Table-level• Row-level

• What do they lock• Read locks

• Block writes

• Write locks• Block reads and writes

• Transactions• Server-level

• MDL locks• Table locks

• Engine-level• Table locks• Row locks

• AUTOCOMMIT• supported

44 www.percona.com

Locks diagnostic• SHOW [FULL] PROCESSLIST• SHOW ENGINE INNODB STATUS• INFORMATION SCHEMA

• PROCESSLIST• InnoDB table

• Performance Schema• METADATA LOCKS• TABLE HANDLES• EVENTS TRANSACTIONS *

• Both server-level and engine-level• Contain GTID information

45 www.percona.com

Locks diagnostic

• SHOW [FULL] PROCESSLISTmysql> select id, state, info from information_schema.processlist\G

******************** 1. row ********************

id: 5

state:

info: NULL

******************** 2. row ********************

id: 4

state: Waiting for table metadata lock

info: alter table titles add column f1 int

******************** 3. row ********************

id: 2

state: executing

info: select id, state, info from information_schema.processlist

45 www.percona.com

Locks diagnostic

• Performance Schemamysql> select processlist_id, object_type, lock_type, lock_status, source

-> from metadata_locks join threads on (owner_thread_id=thread_id)

-> where object_schema=’employees’ and object_name=’titles’\G

*************************** 1. row ***************************

processlist_id: 4

object_type: TABLE

lock_type: EXCLUSIVE

lock_status: PENDING -- waits

source: mdl.cc:3263

*************************** 2. row ***************************

processlist_id: 5

object_type: TABLE

lock_type: SHARED_READ

lock_status: GRANTED -- holds

source: sql_parse.cc:5707

45 www.percona.com

Locks diagnostic

• SHOW ENGINE INNODB STATUS

-------------

TRANSACTIONS

-------------

Trx id counter 0 26243837

Purge done for trx’s n:o < 0 26243834 undo n:o < 0 0

History list length 2

LIST OF TRANSACTIONS FOR EACH SESSION:

---TRANSACTION 0 26243836, ACTIVE 4 sec, OS thread id 101514240 starting index read

mysql tables in use 1, locked 1

LOCK WAIT 2 lock struct(s), heap size 320, 1 row lock(s)

MySQL thread id 4485, query id 25022137 localhost root Updating

update t set a=36 where a=6

45 www.percona.com

InnoDB Monitors


• Permanent output

• innodb status output

• innodb status output locks

• innodb-status-file - Deleted on normal shutdown!

46 www.percona.com

Other InnoDB diagnostic

• INFORMATION SCHEMA• INNODB TRX• INNODB LOCKS• INNODB LOCK WAITS• INNODB METRICS

• Options innodb monitor *

• Option innodb print all deadlocks

47 www.percona.com

Locks diagnostic summary• Table-level

• PROCESSLIST: ”Waiting for table lock”• P S.TABLE HANDLES

• Row-level• InnoDB monitors


• Tables in INFORMATION SCHEMA• Option –innodb print all deadlocks

• MDL• PROCESSLIST

• ”Waiting for metadata lock”

• P S.METADATA LOCKS48 www.percona.com

Replication

49 www.percona.com

MySQL Replication Overview• Always available, needs to be setup• Asynchronous master-slave• Master

• Keeps all updates in binary log file• Two formats: ROW and STATEMENT

• Slave• IO thread read updates from master and stores in

relay log file• SQL thread executes updates

• Multiple SQL threads since 5.6• Multiple masters since 5.7

• GTIDs since 5.650 www.percona.com

Main troubleshooting tools

• SHOW SLAVE STATUS

• Tables in Performance Schema and system database

• Multi-threaded slave (MTS) has additional output foreach worker

• Error log file

51 www.percona.com

Main troubleshooting tools• SHOW SLAVE STATUS

mysql> show slave status \G

************************ 1. row ************************

Slave_IO_State: Queueing master event to the relay log

...

Master_Log_File: master-bin.000003

Read_Master_Log_Pos: 191

Relay_Log_File: slave-relay-bin.000006

Relay_Log_Pos: 4

Relay_Master_Log_File: master-bin.000002

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

...

Last_Errno: 0

Last_Error:

...

Exec_Master_Log_Pos: 319

...

51 www.percona.com

Main troubleshooting tools• Tables in Performance Schema and system database

• replication connection configuration• replication applier configuration• replication connection status• replication applier status• replication applier status by coordinator• replication applier status by worker - MTS only• mysql.slave master info• mysql.slave relay log info• mysql.slave worker info

51 www.percona.com

Main troubleshooting tools• Multi-threaded slave (MTS)

mysql> select * from replication_applier_status_by_worker\G

*************************** 1. row ***************************

CHANNEL_NAME:

WORKER_ID: 1

THREAD_ID: 25

SERVICE_STATE: ON

LAST_SEEN_TRANSACTION:

LAST_ERROR_NUMBER: 0

LAST_ERROR_MESSAGE:

LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00

*************************** 2. row ***************************

CHANNEL_NAME:

WORKER_ID: 2

THREAD_ID: 26

SERVICE_STATE: ON

LAST_SEEN_TRANSACTION: d0753e78-14ec-11e5-b3fb-28b2bd7442fd:770

...

51 www.percona.com

IO thread: communication issues

• Check slave error log• SHOW SLAVE STATUS• P S.replication connection status• Access error

• Try to connect using normal MySQL client usingslave credentials

• SHOW GRANTS

• Fix privileges on master• Restart slave

52 www.percona.com

SQL thread: typical issues

• Simple master-slave• Data is different on master and slave

• Replication event can not be applied

• Different errors on master and slave• Slave lags far behind the master

• Circular replication or other writes in addition to slaveSQL thread

• Data is different on master and slave

53 www.percona.com

Data is different on master and slave

• Was the table modified besides the SQL thread?• How?• Can it affect content of the table in the wrong way?

• Are the table definitions same on master and slave?• MySQL Utilities

• mysqlrplsync, mysqldbcompare, mysqldiff

• Maybe master events were applied in wrong order?• Use mysqlbinlog to find queries caused the issue• Check application, running on master

54 www.percona.com

Events applied in different order

• Happens only with STATEMENT format• Row lock issues• Triggers

• SET GLOBAL slave skip counter - No GTIDs!• Skip transaction - GTIDs• Synchronize tables!

• Different options• Start slave with master’s options, then check• Happens in older versions

55 www.percona.com

Slave lags behind master

• Threads• Master runs in multiple update threads• Slave uses single

• Seconds behind master is growing - Not 100 % reliable!• Tune slave performance

• Multi-threaded slave• One thread per database in 5.6• Affected by concurrency issues

• Indexes on slave only• For statement-based replication

56 www.percona.com

More information

57 www.percona.com

Further reading

• MySQL Troubleshooting book

• Planet MySQL

• MySQL User Reference Manual

• Bug trackers

• http://bugs.mysql.com• https://bugs.launchpad.net/percona-server/

58 www.percona.com

http://shop.oreilly.com/product/0636920021964.do

http://planet.mysql.com/

http://dev.mysql.com/doc/refman/5.6/en/index.html

http://bugs.mysql.com

https://bugs.launchpad.net/percona-server/

Place for your questions

???

59 www.percona.com

Thank you!

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

60 www.percona.com

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

basic mysql troubleshooting for oracle database administrators

Technology