1 / 71 - percona · percona university, ghent, belgium june 2017 mysql document store how to...
TRANSCRIPT
1 / 71
2 / 71
Percona University, Ghent, Belgium June2017
MySQL Document Store
How to replace a NoSQL database by MySQL without effort but with a lot ofgains ?
Frédéric Descamps - MySQL Community Manager - Oracle
3 / 71
Safe Harbor StatementThe following is intended to outline our general product direction. It is intended forinformation purpose only, and may not be incorporated into any contract. It is not acommitment to deliver any material, code, or functionality, and should not be relied up inmaking purchasing decisions. The development, release and timing of any features orfunctionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
4 / 71
about.me/lefred
Who am I ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
5 / 71
Frédéric Descamps@lefredMySQL EvangelistHacking MySQL since 3.23devops believerMySQL Community Manager since May2016living in Belgium 🇧🇪
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
6 / 71
7 / 71
RDBMS & MySQL
Relational Databases
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
8 / 71
Relational DatabasesData Integrity
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
9 / 71
Relational DatabasesData Integrity
normalization
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
10 / 71
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
11 / 71
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, Durability
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
12 / 71
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID compliant
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
13 / 71
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID complianttransactions
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
14 / 71
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID complianttransactions
SQL
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
15 / 71
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID complianttransactions
SQLpowerfull query language
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
16 / 71
NoSQL & MySQL
NoSQL Databases
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
17 / 71
NoSQL or Document StoreSchemaless
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
18 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
19 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
20 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structure
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
21 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objects
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
22 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodel
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
23 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodelobjects persistance without the use of any ORM - mapping oobject-oriented
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
24 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodelobjects persistance without the use of any ORM - mapping oobject-oriented
JSON
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
25 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodelobjects persistance without the use of any ORM - mapping oobject-oriented
JSONclose to frontend
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
26 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodelobjects persistance without the use of any ORM - mapping oobject-oriented
JSONclose to frontendnative in JS
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
27 / 71
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodelobjects persistance without the use of any ORM - mapping oobject-oriented
JSONclose to frontendnative in JSeasy to learn
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
28 / 71
How DBAs see data How Developers see data{ "GNP" : 249704, "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : 10239000, "LifeExpectancy" : 77.8000030517578 }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" }}
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
29 / 71
DMBS or NoSQL ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
30 / 71
DMBS or NoSQL ?
Why not both ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
31 / 71
MySQL as Document Store
Requirements
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
32 / 71
MySQL as Document Store: requirements
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
33 / 71
MySQL as Document Store: requirements1. support of JSON datatype
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
34 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
35 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
3. X-Plugin
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
36 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
3. X-Plugin
4. X-Protocol
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
37 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
3. X-Plugin
4. X-Protocol
extends and replaces the traditional MySQL protocol
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
38 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
3. X-Plugin
4. X-Protocol
extends and replaces the traditional MySQL protocolsupports asynchronous communication to the server
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
39 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
3. X-Plugin
4. X-Protocol
extends and replaces the traditional MySQL protocolsupports asynchronous communication to the serversupports multiple API calls like Javascript, Python, SQL...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
40 / 71
MySQL as Document Store: requirements1. support of JSON datatype
2. CRUD Operations
3. X-Plugin
4. X-Protocol
extends and replaces the traditional MySQL protocolsupports asynchronous communication to the serversupports multiple API calls like Javascript, Python, SQL...
5. MySQL Shell
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
41 / 71
Starting using MySQL DS in few minutes
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
42 / 71
Starting using MySQL DS in few minutes
This is the new MySQL shell as it will be in 8 ;-)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
43 / 71
Migration from MongoDB to MySQL DSFor this example, I will use the well known restaurants collection:
We have two options:
1. We need to dump the data to a file and we will use MySQL Shell with the Pythoninterpreter to load the data.
1. We retrieve the data from MongoDB and load it to MySQL via the Shell in javascript
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
44 / 71
Dump and load using MySQL Shell & Pythonthis example is the courtesy of @datacharmer:https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=300; var all=db.restaurants.�nd() ; all' | perl -pe \ 's/(?:ObjectId|ISODate)\(("[^"]+")\)/ $1/g' > all_recs.json
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
45 / 71
Dump and load using bash and MySQL Shell
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; var all=db.restaurants.�nd() ; all' \| perl -pe 's/(?:ObjectId|ISODate)\(("[^"]+")\)/ $1/g'\| head -n -1 | while read line do mysqlsh [email protected] --password='' -e "rec=$line; session.getSchema('docstore').getCollection('restaurants').add(rec).execute()" done
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
46 / 71
Let's query
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
47 / 71
Let's query
That's too much records to show it here... let's limit it
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
48 / 71
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
49 / 71
Some more examples
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
50 / 71
Some more examples
Let's add a selection criteria:
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
51 / 71
Syntax slightly different than MongoDB
> db.restaurants.�nd({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2){ "borough" : "Queens", "cuisine" : "French", "name" : "La Baraka Restaurant" }{ "borough" : "Queens", "cuisine" : "French", "name" : "Air France Lounge" }
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
52 / 71
CRUD operations
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
53 / 71
what about my old SQL ?
The hidden part of the iceberg
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
54 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
55 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
56 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON values
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
57 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
58 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)Generated/virtual columns
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
59 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)Generated/virtual columns
Indexing JSON data
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
60 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)Generated/virtual columns
Indexing JSON dataForeign Keys to JSON data
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
61 / 71
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)Generated/virtual columns
Indexing JSON dataForeign Keys to JSON dataSQL Views to JSON data
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
62 / 71
Example
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
63 / 71
Example
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
64 / 71
Example (2)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
65 / 71
Example (2)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
66 / 71
Example (3): explain
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
67 / 71
Example (4): add index
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
68 / 71
what's do I gain ?
Conclusion
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
69 / 71
schemalessflexible data structureeasy to start (CRUD)
ConclusionThis is the best of the two worlds in one product !
Data integrityACID CompliantSQL
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
70 / 71
Thank you !
Any Questions ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
71 / 71