data dictionary pl17
TRANSCRIPT
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0:An Internal Server ComponentThat Matters
Ståle Deraas, Senior Development ManagerOracle, MySQL27 Apr 2017
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
What is a Data Dictionary?
Data Dictionary before MySQL 8.0
Transactional Data Dictionary in MySQL 8.0
What’s in it for you?
1
2
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What is a Data Dictionary?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Data Dictionary
• Metadata is information about data in an RDBMS
– Column definitions, Index definitions, Foreign key definitions
...
• Data Dictionary is a collection of metadata for all data in an RDBMS
ID NAME WEIGHT HEIGHT GENDER
3 Bob 80 185 M
5 Liz 55 165 F
Metadata
Data
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Server
Data Dictionary
6
Role in Server
Query Executor
Optimizer
StorageEngine
SQLstatement
ClientParser
Result DataDictionary
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
CREATE PROCEDURE p1(v INT)SQL SECURITY INVOKERBEGIN...END
Data DictionaryTypes of Metadata
Data Dictionary
Table Definitions SP Definitions
View Definitions Schemas
ACL
CREATE TABLE customers(id INT AUTO_INCREMENT...PRIMARY KEY (id),INDEX ...FOREIGN KEY ...
)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Data Dictionary before MySQL 8.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
• Metadata stored in a mix of files and tables :
– File based• FRM, TRN, TRG, OPT, PAR
...
– Table based: non-transactional• mysql.proc
...
– Table based: transactional• Innodb.SYS
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
10
Data Dictionary
Files
FRM TRG OPT
System Tables (mysql.)
user procevents
InnoDB System Tables
MyISAM
File system
InnoDB
SQL
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
• INFORMATION_SCHEMA queries are slow
• Inconsistencies due to non-transactional storage of metadata
• Inconsistencies between InnoDB metadata and Server metadata
• Showstopper for crash-safe / transactional DDL
• Replication is challenging, as DDLs are not atomic
• Difficult to extend
• No uniform API
11
Problems
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary
MySQL 8.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
13
Data Dictionary
InnoDBSQL DD TableDD TableDD Table
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
• All metadata stored in tables
• Single metadata repository for all MySQL server subsystems
• Reliable, crash-safe InnoDB tables
• INFORMATION_SCHEMA implemented as views over DD tables
– Queries can be optimized
– Improved performance
– Simpler, uniform implementation, easier to maintain
Main features
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
15
Overview
InnoDB
Data Dictionary
DD Table User Table
INFORMATIONSCHEMA
Views
Archive
User Table
CSV
User Table
Optimizer
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
• The data dictionary schema is based on SQL standard definitions
• Designed for automated upgrade of metadata
• The data dictionary is designed to be easily extended for new requirements
• Designed to allow plugins/components to add INFORMATION_SCHEMA views and PERFORMANCE_SCHEMA tables
Main features, cont’ed
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
InnoDB
Transactional Data Dictionary in MySQL 8.0
17
Architecture
Query Executor
Parser Optimizer
Data Dictionary Tablespace
Data Dictionary API
Storage Engine
PluginUser TableUser Table
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary API in MySQL 8.0
• A single way to deal with Data Dictionary
– For the server core
– For Storage Engines
• A uniform API for all types of metadata
• Provide a way to handle Storage Engine private data– Storage Engines no longer have to store their own metadata
– The API enables SEs to store key-value pairs
18
Design goals
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
SDI: Serialized Dictionary Information
- Copy of metadata from the Data Dictionary in JSON format
- Stored in data tablespaces(InnoDB) or .sdi file (MyISAM)
- Used for data migration and redundancy
The InnoDB Data Dictionary tablespace is the metadata storage
19
Reliability and Redundancy
InnoDB User Tablespaces
User Table
InnoDB Data Dictionary tablespace
mysql.tables
SDI
ID Name
1 User Table 1
2 User Table 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
What’s in it for you?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary Benefits
• INFORMATION SCHEMA
– Improved Performance and Scalability
– Solves longstanding issues
• Reliability– Atomic DDL
– Serialized Dictionary Information
– Automated Data Dictionary upgrade
• +++
21
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 22
MySQLClient
I_S Query Results
MySQL Server
Create temporary table.
Heuristic optimization
Read metadata from File system or
from MyISAM/InnoDB engine.
.
TEMP TABLE
Return rows to user. File system / MyISAM / InnoDB engine
INFORMATION SCHEMA MySQL 5.7: Multiple sources, heuristics, creation of temp tables
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 23
MySQL Client
I_S Query Results
MySQL Server
Optimizer prepares
execution plan
Executor reads metadata from
data dictionary tables
InnoDB storage engine
Return rows to user
INFORMATION SCHEMA MySQL 8.0: Uniform, simple and using server standard features
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA Performance
Confidential – Oracle Internal/Restricted/Highly Restricted 24
Uniform, simpler implemention makes it a lot faster
MySQL Client
I_S Query Results
MySQL Server
Optimizer prepares
execution plan.
Executor reads metadata from
data dictionary tables.
InnoDB storage engine
Return rows to user.
INFORMATION_SCHEMA in 8.0
MySQL Client
I_S Query Results
MySQL Server
Create temporary table.
Heuristic optimization.
Read metadata from File system or
from MyISAM/InnoDB engine.
.
TEMP TABLE
Return rows to user.
INFORMATION_SCHEMA in 5.7
File system / MyISAM
/ InnoDB engine
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA Performance and Scalability
• Typically 30X performance improvements over MySQL 5.7
• More than 100X for some queries like: List all InnoDB table columns
25
I_S queries scale, both with database size and query load
0 20 40 60 80 100 120 140 160
List all InnoDB tables columns 5k tables
List all InnoDB tables columns 10k tables
MySQL 8.0
MySQL 5.7
Time in Seconds (Lower is better)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 26
100 schemas times 50 tables (5000 tables)
Information Schema Performance
0 0.5 1 1.5 2 2.5 3 3.5 4
Count All Schemas
Schema aggregate size stats
All Dynamic Table Info
All Static Table Info
Auto Increments Near Limit
Count All Columns
Count All Indexes
MySQL 8.0
MySQL 5.7
Time in Seconds (Lower is better)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA in MySQL 8.0: Dynamic values
• TABLES.TABLE_ROWS, TABLES.DATA_FREE, …
– Handled by opening table and retrieving data from SE in MySQL 5.7 – expensive!!!
• Two approaches in MySQL 8.0
– information_schema_stats=cached• Default
• Stored in auxiliary tables
• ANALYZE TABLE
– information_schema_stats=latest• Native functions in the I_S views
• Uses special SE API (InnoDB) or falls back to opening tables (MyISAM)
27
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA longstanding issues solved in 8.0
– Bug#34921 comparison with information schema tables don’t honor collation
– Bug#48445 Inconsistency with SHOW and SELECT FROM I_S
– Bug#61846 SHOW FULL COLUMNS displays incorrect privileges for table
– Bug#65121 Inconsistent result for select on INFORMATION_SCHEMA.STATISTICS
– Bug#75532 Join between I_S schema tables is case insensitive/returns wrong value
– Bug#81347 unnecessary scanned all databases for information schema
– .....
28
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability – Atomic DDL
• All metadata stored in InnoDB (transactional SE)
• DDL code changed to avoid intermediate commits
– At server level
– At InnoDB level
– At replication level
29
Transactional behavior
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability – Atomic DDL : Use Case - DROP DATABASE
30
MySQL 5.7
• Delete tables
– Metadata, TRN/TRG/FRM files
– Data, InnoDB tables
• Delete stored programs– Metadata, rows in MyISAM (non-
transactional)
• Delete schema
– Metadata, DB.OPT file
Mix of filesystem, non-transactional/transactional storage and multiple commits
MySQL 8.0
• Delete tables
– Metadata, rows in InnoDB
– Data, InnoDB tables
• Delete stored programs– Metadata, rows in InnoDB
• Delete schema
– Metadata, rows in InnoDB
Updates to transactional storage, one commit
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability – Disaster Recovery
• MySQL 8.0, metadata and data stored in InnoDB (transactional SE)
– Metadata copy stored in SDI embedded in .ibd
– Data stored in .ibd
• MySQL before 8.0, metadata in files, data in InnoDB – Metadata stored in .FRM
– Data stored in .ibd
31
Use case – IMPORT using Serialized Dictionary Information, InnoDB
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL before 8.0
• Moving .FRM and data files around (MyISAM)
• Create tables and do import using SQL (InnoDB)
CREATE TABLE t1 (...)
ALTER TABLE t1 DISCARD TABLESPACE
ALTER TABLE t1 IMPORT TABLESPACE ...
• Easy to do mistakes
MySQL 8.0
• Self-descriptive tablespaces (SDI)
• New IMPORT statement will import the tablespace:
– Read metadata from the SDI, and create the tables
– Import the data into these tables
• Worst case, if SDI is corrupt, you can extract it (JSON format) and edit
32
Reliability – Disaster Recovery• Use case – IMPORT using Serialized Dictionary Information
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability and Ease of Use – Automated Dictionary Upgrade
• The Data Dictionary is versioned
• The MySQL Server will support upgrading dictionary tables automatically
– Done directly by the executable and not in a script
– Reduces user and privilege issues for the process doing the upgrade
• The Data Dictionary will be upgraded atomically
33
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Extending Dictionary Tables from Plugins
• API for plugins to extend INFORMATION_SCHEMA & PERFORMANCE_SCHEMA
– Add new virtual tables/views
• API for plugins to store their specific data in the Data Dictionary
34
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Try for yourself!
• Downloadable 8.0.1 DMR
– dev.mysql.com
• A «hot» labs release right off the press: 8.0.1 + Unified Data Dictionary
• Enjoy and give us your feedback!
• Thank you for listening
• http://mysqlserverteam.com
35