why use explain format=json?
TRANSCRIPT
EXPLAIN FORMAT=JSONWhy to use?
April, 19, 2016Sveta Smirnova
•The appearance
•Innovations•Grouping and ordering
•Subqueries and unions
•Troubleshooting sugar
Table of Contents
2
• Has nice structured view
• Easily machine-readable
• Allows MySQL Workbench draw nicegraphs
EXPLAIN FORMAT=JSON was introduced in 5.6
3
• It has more information than regularEXPLAIN
• Can replace OPTIMIZER TRACE for somecases
Trendy? - Not only!
4
Innovations
5
• According to user manual• The percentage of rows that were actually
needed, against the equal or bigger numberof resolved rows.
• Is this good or bad?
• Example query• Regular EXPLAIN• EXPLAIN FORMAT=JSON has the answer
What number of ”filtered” rows mean?
6
• According to user manual• Example query
mysql> select * from Country where Name=’Russian Federation’\G
*************************** 1. row ***************************
Code: RUS
Name: Russian Federation
Continent: Europe
...
HeadOfState: Vladimir Putin
Capital: 3580
Code2: RU
1 row in set (0.00 sec)
• Regular EXPLAIN• EXPLAIN FORMAT=JSON has the answer
What number of ”filtered” rows mean?
6
• According to user manual• Example query• Regular EXPLAIN
mysql> explain select * from Country where Name=’Russian Federation’;
+-----+------+----------+-------------+
| ... | rows | filtered | Extra |
+-----+------+----------+-------------+
| ... | 239 | 10.00 | Using where |
+-----+------+----------+-------------+
• What happened with rows?• Why they were filtered?
• EXPLAIN FORMAT=JSON has the answer
What number of ”filtered” rows mean?
6
• According to user manual• Example query• Regular EXPLAIN• EXPLAIN FORMAT=JSON has the answer
"table": {
"table_name": "Country",
"access_type": "ALL",
"rows_examined_per_scan": 239,
"rows_produced_per_join": 23,
"filtered": "10.00",
What number of ”filtered” rows mean?
6
• Covered indexes• Usually use more than one column• Contain all fields, necessary to resolve the
query
• Example• EXPLAIN FORMAT=JSON has the answer• We only need Continent and Population
columns for the index
When to use covered indexes?
7
• Covered indexes• Example
mysql> select count(*) from Country
-> where Continent=’Africa’ and Population > 1000000;
+----------+
| count(*) |
+----------+
| 47 |
+----------+
1 row in set (0,00 sec)
• Will the query use Primary Key?• Or * will look through all rows?
• EXPLAIN FORMAT=JSON has the answer• We only need Continent and Population
columns for the index
When to use covered indexes?
7
• Covered indexes• Example• EXPLAIN FORMAT=JSON has the answer
mysql> explain format=json select count(*) from Country
-> where Continent=’Africa’ and Population > 1000000\G
*************************** 1. row ***************************
...
"used_columns": [
"Continent",
"Population"
],
...
• We only need Continent and Populationcolumns for the index
When to use covered indexes?
7
• Covered indexes
• Example
• EXPLAIN FORMAT=JSON has the answer
• We only need Continent and Populationcolumns for the index
When to use covered indexes?
7
• Example queriesSELECT first_name, last_name FROM employees WHERE
first_name=’Steve’;
...first_name=’Steve’ and last_name like ’V%’;
...first_name=’Steve’ and last_name like ’V%’ and hire_date>’1990-01-01’;
• Candidate indexes• Index effectiveness• Column hire date is not used!• Most effective index would be on
(first name, last name)
Which part of index used?
8
• Example queries• Candidate indexes
CREATE INDEX comp1(first_name);
CREATE INDEX comp2(first_name, last_name);
CREATE INDEX comp3(first_name, last_name, hire_date);
• Last one seem to fit all the queries
• Index effectiveness• Column hire date is not used!• Most effective index would be on
(first name, last name)
Which part of index used?
8
• Example queries• Candidate indexes• Index effectiveness
mysql> explain format=json SELECT first_name, last_name FROM employees
-> WHERE first_name=’Steve’ and last_name like ’V%’
-> and hire_date > ’1990-01-01’\G
*************************** 1. row ***************************
EXPLAIN: {
...
"used_key_parts": [
"first_name",
"last_name"
],
• Column hire date is not used!• Most effective index would be on
(first name, last name)
Which part of index used?
8
• Example queries• Candidate indexes• Index effectiveness• Column hire date is not used!• Most effective index would be on
(first name, last name)
Which part of index used?
8
• Use case• Table has two or more indexes• All can be used to resolve the query• But only one is chosen• Why?
• Example• Index emp no has been chosen• The reason
Why index chosen?
9
• Use case• Example
• Table ‘titles‘ in ‘employees‘ database• The query:
select distinct title from titles where year(from_date) > ’1990’;
• Two indexes:PRIMARY KEY (‘emp no‘,‘title‘,‘from date‘)KEY ‘emp no‘ (‘emp no‘)
• Index emp no has been chosen• The reason
Why index chosen?
9
• Use case• Example• Index emp no has been chosen
mysql> explain select distinct title from titles
-> where year(from_date) > ’1990’\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: titles
partitions: NULL
type: index
possible_keys: PRIMARY,emp_no
key: emp_no
...
• The reason
Why index chosen?
9
• Use case• Example• Index emp no has been chosen• The reason
• Cost statistics for emp nomysql> explain format=json select distinct title
-> from titles where year(from_date) > ’1990’\G
...
"cost_info": {
"query_cost": "89796.80"
},
...
Why index chosen?
9
• Use case• Example• Index emp no has been chosen• The reason
"duplicates_removal": {
...
"cost_info": {
"read_cost": "1252.00",
"eval_cost": "88544.80",
"prefix_cost": "89796.80",
"data_read_per_join": "27M"
},
Why index chosen?
9
• Use case• Example• Index emp no has been chosen• The reason
• Cost statistics for PRIMARY KEYmysql> explain format=json select distinct title from titles
-> force index(primary) where year(from_date) > ’1990’\G
...
"cost_info": {
"query_cost": "531269.80"
},
...
Why index chosen?
9
• Use case• Example• Index emp no has been chosen• The reason
• Access by PRIMARY KEY is 531269.80 /89796.80 = 6 times more expensive!
Why index chosen?
9
Grouping and ordering
10
• Hierarchical structure"grouping_operation": {
"using_filesort": false,
"table": {
...
• Allows to distinguish operations• Separate member for DISTINCT
How GROUP BY proceed
11
• Hierarchical structure• Allows to distinguish operations
"grouping_operation": {
"using_temporary_table": true,
"using_filesort": true,
"cost_info": {
"sort_cost": "26.41"
},
"nested_loop": [
{
...
• Separate member for DISTINCT
How GROUP BY proceed
11
• Hierarchical structure• Allows to distinguish operations• Separate member for DISTINCT
"duplicates_removal": {
"using_temporary_table": true,
"using_filesort": false,
"grouping_operation": {
"using_temporary_table": true,
"using_filesort": true,
...
How GROUP BY proceed
11
• Clearly shows which job required• Is temporary table for ORDER BY?
mysql> explain select distinct last_name
-> from employees order by last_name asc\G
*************************** 1. row ***************************
...
type: ALL
...
rows: 299379
filtered: 100.00
Extra: Using temporary; Using filesort
1 row in set, 1 warning (0.00 sec)
• Lets check what really happened:• Confirmation
Ordering operations
12
• Clearly shows which job required• Is temporary table for ORDER BY?• Lets check what really happened:
mysql> explain format=json select distinct last_name
-> from employees order by last_name asc\G
...
"ordering_operation": {
"using_filesort": false, - No temporary table here!
"duplicates_removal": {
"using_temporary_table": true,
"using_filesort": true,
...
• Confirmation
Ordering operations
12
• Clearly shows which job required• Is temporary table for ORDER BY?• Lets check what really happened:• Confirmation
mysql> explain format=json select last_name
-> from employees order by last_name asc\G
...
"ordering_operation": {
"using_filesort": true,
"cost_info": {
"sort_cost": "299379.00"
...
Ordering operations
12
Subqueries and unions
13
• attached subqueries• Subqueries which are not converted to JOIN
mysql> explain format=json select emp_no from salaries
-> where salary > (select avg(salary) from salaries)\G
*************************** 1. row ***************************
...
"attached_subqueries": [
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
...
• optimized away subqueries• materialized from subquery
Everything about subqueries
14
• attached subqueries• optimized away subqueries
• Subqueries that were executed only once andwere replaced by their result.mysql> explain format=json select emp_no, salary from salaries
-> order by (select max(salary) from salaries)\G
...
"optimized_away_subqueries": [
{
"dependent": false,
"cacheable": true,
...
• materialized from subquery
Everything about subqueries
14
• attached subqueries• optimized away subqueries
• Compare with regular EXPLAINmysql> explain select emp_no, salary from salaries
-> order by (select max(salary) from salaries)\G
...
*************************** 2. row ***************************
id: 2 | key: NULL
select_type: SUBQUERY | key_len: NULL
table: salaries | ref: NULL
partitions: NULL | rows: 2838525
type: ALL | filtered: 100.00
possible_keys: NULL | Extra: NULL
• materialized from subquery
Everything about subqueries
14
• attached subqueries• optimized away subqueries• materialized from subquery• Materialized subquery
mysql> explain format=json select dept_name from departments where
-> dept_no in (select dept_no from dept_manager
-> where to_date is not null)\G
...
"table": { | "materialized_from_subquery": {
"table_name": "<subquery2>",| "using_temporary_table": true,
"access_type": "eq_ref", | "query_block": {
...
Everything about subqueries
14
• order by subqueries• group by subqueries• having subqueries
• Dependent or not• Cacheable or not• If was optimized away• Information about nested subqueries
Subqueries and GROUP/ORDER BY
15
• All members in parent object union result• Each query exists in query specifications
arraymysql> explain format=json select Name from City
-> union select Capital from Country\G
*************************** 1. row ***************************
EXPLAIN: { | "table": {
"query_block": { | "table_name": "City",
"union_result": { | ...
... | "table": {
"query_specifications": [ | "table_name": "Country",
{
...
UNION details
16
Troubleshooting sugar
17
• The issue: temporary table• SQL BUFFER RESULT always uses it• But how to ensure if original query does not?
mysql> explain select sql_buffer_result emp_no, salary/avg(salary)
-> from salaries group by emp_no, salary\G
*************************** 1. row ***************************
id: 1 | key_len: NULL
select_type: SIMPLE | ref: NULL
table: salaries | rows: 2557022
partitions: NULL | filtered: 100.00
type: ALL | Extra: Using temporary; Using filesort
possible_keys: NULL
key: NULL
• EXPLAIN FORMAT=JSON distinguishbuffer and original query
SQL BUFFER RESULT is not hidden
18
• The issue: temporary table• EXPLAIN FORMAT=JSON distinguish
buffer and original querymysql> explain format=json select sql_buffer_result emp_no,
-> salary/avg(salary) from salaries group by emp_no, salary\G
...
"grouping_operation": {
"using_temporary_table": true,
"using_filesort": true,
...
"buffer_result": {
"using_temporary_table": true,
...
SQL BUFFER RESULT is not hidden
18
• Issue: complicated queries• Regular EXPLAIN lists tables in rows
mysql> explain select * from employees join dept_manager using
(emp_no) where emp_no in (select emp_no from (select ...
*************************** 1. row ***************************
table: <subquery2>
*************************** 2. row ***************************
table: dept_manager
*************************** 3. row ***************************
table: employees
*************************** 4. row ***************************
table: <derived3>
...
• Solution: hierarchical structure
nested loop and JOIN hierarchy
19
• Issue: complicated queries• Solution: hierarchical structure
mysql> explain format=json select * from employees join dept_manager
-> using (emp_no) where emp_no in (select emp_no...
...
"nested_loop": [
{
"table": { "table_name": "<subquery2>",
...
"nested_loop": [ { "table": { "table_name": "titles",
...
{
"table": { "table_name": "dept_manager", ...
nested loop and JOIN hierarchy
19
• Provides more details than regularEXPLAIN
• However only necessary information forDBA
• Elegant way to show it• Machine-readable• Trace details in easy-to-view format
EXPLAIN FORMAT=JSON is Cool!
20
• EXPLAIN in the official user manual
• EXPLAIN FORMAT=JSON is Cool! series
More information
21
???
Place for your questions
22
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
Thank you!
23