1 / 71 - percona · percona university, ghent, belgium june 2017 mysql document store how to...

1 / 71

2 / 71

Percona University, Ghent, Belgium June2017

MySQL Document Store

How to replace a NoSQL database by MySQL without effort but with a lot ofgains ?

Frédéric Descamps - MySQL Community Manager - Oracle

3 / 71

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended forinformation purpose only, and may not be incorporated into any contract. It is not acommitment to deliver any material, code, or functionality, and should not be relied up inmaking purchasing decisions. The development, release and timing of any features orfunctionality described for Oracle´s product remains at the sole discretion of Oracle.

4 / 71

about.me/lefred

Who am I ?

5 / 71

Frédéric Descamps@lefredMySQL EvangelistHacking MySQL since 3.23devops believerMySQL Community Manager since May2016living in Belgium 🇧🇪

6 / 71

7 / 71

RDBMS & MySQL

Relational Databases

8 / 71

Relational DatabasesData Integrity

9 / 71

normalization

10 / 71

normalizationconstraints (foreign keys, ...)

11 / 71

Atomicity, Consistency, Isolation, Durability

12 / 71

Atomicity, Consistency, Isolation, DurabilityACID compliant

13 / 71

Atomicity, Consistency, Isolation, DurabilityACID complianttransactions

14 / 71

15 / 71

SQLpowerfull query language

16 / 71

NoSQL & MySQL

NoSQL Databases

17 / 71

NoSQL or Document StoreSchemaless

18 / 71

no schema design, no normalization, no foreign keys, no data types, ...

19 / 71

no schema design, no normalization, no foreign keys, no data types, ...very quick initial development

20 / 71

Flexible data structure

21 / 71

Flexible data structureembedded arrays or objects

22 / 71

Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodel

23 / 71

Flexible data structureembedded arrays or objectsvalid solution when natural data can't be modelized optimaly into a relationalmodelobjects persistance without the use of any ORM - mapping oobject-oriented

24 / 71

25 / 71

JSONclose to frontend

26 / 71

JSONclose to frontendnative in JS

27 / 71

JSONclose to frontendnative in JSeasy to learn

28 / 71

How DBAs see data How Developers see data{ "GNP" : 249704, "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : 10239000, "LifeExpectancy" : 77.8000030517578 }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" }}

29 / 71

DMBS or NoSQL ?

30 / 71

DMBS or NoSQL ?

Why not both ?

31 / 71

MySQL as Document Store

Requirements

32 / 71

MySQL as Document Store: requirements

33 / 71

MySQL as Document Store: requirements1. support of JSON datatype

34 / 71

2. CRUD Operations

35 / 71

2. CRUD Operations

3. X-Plugin

36 / 71

2. CRUD Operations

3. X-Plugin

4. X-Protocol

37 / 71

2. CRUD Operations

3. X-Plugin

4. X-Protocol

extends and replaces the traditional MySQL protocol

38 / 71

2. CRUD Operations

3. X-Plugin

4. X-Protocol

extends and replaces the traditional MySQL protocolsupports asynchronous communication to the server

39 / 71

2. CRUD Operations

3. X-Plugin

4. X-Protocol

extends and replaces the traditional MySQL protocolsupports asynchronous communication to the serversupports multiple API calls like Javascript, Python, SQL...

40 / 71

2. CRUD Operations

3. X-Plugin

4. X-Protocol

extends and replaces the traditional MySQL protocolsupports asynchronous communication to the serversupports multiple API calls like Javascript, Python, SQL...

5. MySQL Shell

41 / 71

Starting using MySQL DS in few minutes

42 / 71

Starting using MySQL DS in few minutes

This is the new MySQL shell as it will be in 8 ;-)

43 / 71

Migration from MongoDB to MySQL DSFor this example, I will use the well known restaurants collection:

We have two options:

1. We need to dump the data to a file and we will use MySQL Shell with the Pythoninterpreter to load the data.

1. We retrieve the data from MongoDB and load it to MySQL via the Shell in javascript

44 / 71

Dump and load using MySQL Shell & Pythonthis example is the courtesy of @datacharmer:https://www.slideshare.net/datacharmer/mysql-documentstore

$ mongo --quiet --eval 'DBQuery.shellBatchSize=300; var all=db.restaurants.�nd() ; all' | perl -pe \ 's/(?:ObjectId|ISODate)$("[^"]+")$/ $1/g' > all_recs.json

45 / 71

Dump and load using bash and MySQL Shell

$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; var all=db.restaurants.�nd() ; all' \| perl -pe 's/(?:ObjectId|ISODate)$("[^"]+")$/ $1/g'\| head -n -1 | while read line do mysqlsh root@127.0.0.1 --password='' -e "rec=$line; session.getSchema('docstore').getCollection('restaurants').add(rec).execute()" done

46 / 71

Let's query

47 / 71

Let's query

That's too much records to show it here... let's limit it

48 / 71

49 / 71

Some more examples

50 / 71

Some more examples

Let's add a selection criteria:

51 / 71

Syntax slightly different than MongoDB

> db.restaurants.�nd({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2){ "borough" : "Queens", "cuisine" : "French", "name" : "La Baraka Restaurant" }{ "borough" : "Queens", "cuisine" : "French", "name" : "Air France Lounge" }

52 / 71

CRUD operations

53 / 71

what about my old SQL ?

The hidden part of the iceberg

54 / 71

JSON datatype is behind the scenenative datatype (since 5.7.8)

55 / 71

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4

56 / 71

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON values

57 / 71

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)

58 / 71

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS,JSON_SEARCH,...)Generated/virtual columns

59 / 71

Indexing JSON data

60 / 71

Indexing JSON dataForeign Keys to JSON data

61 / 71

Indexing JSON dataForeign Keys to JSON dataSQL Views to JSON data

62 / 71

Example

63 / 71

Example

64 / 71

Example (2)

65 / 71

Example (2)

66 / 71

Example (3): explain

67 / 71

Example (4): add index

68 / 71

what's do I gain ?

Conclusion

69 / 71

schemalessflexible data structureeasy to start (CRUD)

ConclusionThis is the best of the two worlds in one product !

Data integrityACID CompliantSQL

70 / 71

Thank you !

Any Questions ?

71 / 71

1 / 71 - percona · percona university, ghent, belgium june 2017 mysql document store how to...

Documents

percona toolkit for effective mysql administration

best practices for mysql scalability - percona

mysql ha percona cluster @ mysql meetup mumbai

mysql 5.6 and percona server 5 -...

monitoring mysql with prometheus & grafana - percona ·...

percona tool kit for mysql dba's

Эффективная отладка репликации...

transacciones distribuidas en mysql - percona...distribuidas...

the mysql server ecosystem in 2016 webminar nov 2016 ·...

mysql für oracle dbas · 6 / 35 branches und forks dec...

drupal mysql webinar percona

percona xtrabackup - mysql meetup @ mumbai

percona - architecture and design of mysql powered...

percona software & servicesupdate · percona server for...

percona server 5.6: enterprise-grade mysql / plmce 2014

turbocharge your mysql - percona€¦ · turbocharge your...

mysql replication options - percona · mysql replication...

mysql for mongodb dbas - percona€¦ · mysql for mongodb...

mysql monitoring - percona · mysql monitoring finding...

percona live 2014 - scaling mysql in aws