mysql innodb cluster and group replication in a nutshell - hands-on tutorial with mysql enterprise...
TRANSCRIPT
![Page 1: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/1.jpg)
1 / 168
![Page 2: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/2.jpg)
2 / 168
![Page 3: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/3.jpg)
Safe Harbor StatementThe following is intended to outline our general product direction. It is intended forinformation purpose only, and may not be incorporated into any contract. It is not acommitment to deliver any material, code, or functionality, and should not be relied up inmaking purchasing decisions. The development, release and timing of any features orfunctionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
3 / 168
![Page 4: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/4.jpg)
Who I am ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
4 / 168
![Page 5: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/5.jpg)
Frédéric Descamps@lefredMySQL EvangelistManaging MySQL since 3.23devops believerhttp://about.me/lefred
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
5 / 168
![Page 6: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/6.jpg)
get more online
MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
6 / 168
![Page 7: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/7.jpg)
Blogshttp://lefred.be/
http://mysqlhighavailability.com/
https://thesubtlepath.com/blog/mysql/
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
7 / 168
![Page 8: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/8.jpg)
AgendaMySQL InnoDB Cluster & Group Replication concepts
Migration from Master-Slave to GR
How to monitor ?
Application interaction
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
8 / 168
![Page 9: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/9.jpg)
LAB1: Current situation
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
9 / 168
![Page 10: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/10.jpg)
launchrun_app.sh
on mysql1 intoa screensessionverify thatmysql2 is arunning slave
LAB1: Current situation
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
10 / 168
![Page 11: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/11.jpg)
Summary
+--------+----------+--------------+-----------------+| | ROLE | SSH PORT | INTERNAL IP |+--------+----------+--------------+-----------------+ | | | | | | mysql1 | master | 8821 | 192.168.56.11 | | | | | | | mysql2 | slave | 8822 | 192.168.56.12 | | | | | | | mysql3 | n/a | 8823 | 192.168.56.13 | | | | | | | mysql4 | n/a | 8824 | 192.168.56.14 | | | | | | +--------+----------+--------------+-----------------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
11 / 168
![Page 12: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/12.jpg)
Easy High Availability
MySQL InnoDB Cluster
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
12 / 168
![Page 13: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/13.jpg)
InnoDB
cluster
Ease-of-Use
Extreme Scale-Out
Out-of-Box Solution
Built-in HA
High P erformance
Everything Integr ated
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
13 / 168
![Page 14: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/14.jpg)
Our vision in 4 steps
MySQL Document Store
Relational & Document Models
MySQL HA
Out-Of-Box HA
Read Scale-Out
Async Replication + A uto F ailover
Write Scale-Out
Sharding
E1
E3
E2 E4
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
14 / 168
![Page 15: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/15.jpg)
Step 2´s Architecture
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
M
y
S
Q
L
S
h
e
l
l
I
n
n
o
D
B
c
l
u
s
t
e
r
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
M
p
M
M
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
15 / 168
![Page 16: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/16.jpg)
Step 3´s ArchitectureA
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
M
y
S
Q
L
S
h
e
l
l
I
n
n
o
D
B
c
l
u
s
t
e
r
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
M
p
M
M
S1 S2 S3 S4 S...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
16 / 168
![Page 17: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/17.jpg)
Step 4´s ArchitectureA
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
M
y
S
Q
L
S
h
e
l
l
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
I
n
n
o
D
B
c
l
u
s
t
e
r
M
p
M
M
S2 S3 S4 S...S1
I
n
n
o
D
B
c
l
u
s
t
e
r
M
p
M
M
S1 S2 S3 S4 S...
I
n
n
o
D
B
c
l
u
s
t
e
r
M
p
M
M
S1 S2 S3 S4 S...
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
A
p
p
l
i
c
a
t
i
o
n
M
y
S
Q
L
C
o
n
n
e
c
t
o
r
M
y
S
Q
L
R
o
u
t
e
r
replicaset 1
replicaset 2
replicaset 3
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
17 / 168
![Page 18: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/18.jpg)
the magic explained
Group Replication Concept
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
18 / 168
![Page 19: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/19.jpg)
Group Replication: heart of MySQL InnoDBCluster
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
19 / 168
![Page 20: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/20.jpg)
Group Replication: heart of MySQL InnoDBCluster
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
20 / 168
![Page 21: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/21.jpg)
MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQLGR is an implementation of Replicated Database State Machine theoryGR allows to write on all Group Members (cluster nodes) simultaneously whileretaining consistencyGR implements conflict detection and resolutionGR allows automatic distributed recoverySupported on all MySQL platforms !!
Linux, Windows, Solaris, OSX, FreeBSD
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
21 / 168
![Page 22: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/22.jpg)
MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocol (similar to Mencius)Its task: Deliver messages accross the distributed system:
Atomicallyin Total Order
MySQL Group Replication receives the Ordered 'tickets' from this GCS subsystem.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
22 / 168
![Page 23: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/23.jpg)
And for users ?not longer necessary to handle server fail-over manually or with a complicated scriptGR provides fault toleranceGR enables update-everywhere setupsGR handles crashes, failures, re-connects automaticallyAllows an easy setup of a highly available MySQL service!
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
23 / 168
![Page 24: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/24.jpg)
OK, but how does it work ?
it´s just ...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
24 / 168
![Page 25: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/25.jpg)
OK, but how does it work ?
it´s just ...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
25 / 168
![Page 26: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/26.jpg)
OK, but how does it work ?
it´s just ...
... no, in fact the writeset replication is synchronous and then certification and apply ofthe changes are local to each nodes and happen asynchronous.
not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustratethis...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
26 / 168
![Page 27: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/27.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
27 / 168
![Page 28: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/28.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
28 / 168
![Page 29: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/29.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
29 / 168
![Page 30: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/30.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
30 / 168
![Page 31: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/31.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
31 / 168
![Page 32: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/32.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
32 / 168
![Page 33: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/33.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
33 / 168
![Page 34: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/34.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
34 / 168
![Page 35: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/35.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
35 / 168
![Page 36: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/36.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
36 / 168
![Page 37: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/37.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
37 / 168
![Page 38: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/38.jpg)
MySQL Group Replication (autocommit)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
38 / 168
![Page 39: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/39.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
39 / 168
![Page 40: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/40.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
40 / 168
![Page 41: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/41.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
41 / 168
![Page 42: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/42.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
42 / 168
![Page 43: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/43.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
43 / 168
![Page 44: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/44.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
44 / 168
![Page 45: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/45.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
45 / 168
![Page 46: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/46.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
46 / 168
![Page 47: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/47.jpg)
MySQL Group Replication (full transaction)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
47 / 168
![Page 48: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/48.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
48 / 168
![Page 49: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/49.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
49 / 168
![Page 50: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/50.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
50 / 168
![Page 51: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/51.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
51 / 168
![Page 52: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/52.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
52 / 168
![Page 53: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/53.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
53 / 168
![Page 54: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/54.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
54 / 168
![Page 55: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/55.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
55 / 168
![Page 56: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/56.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
56 / 168
![Page 57: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/57.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
57 / 168
![Page 58: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/58.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
58 / 168
![Page 59: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/59.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
59 / 168
![Page 60: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/60.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
60 / 168
![Page 61: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/61.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
61 / 168
![Page 62: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/62.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
62 / 168
![Page 63: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/63.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
63 / 168
![Page 64: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/64.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
64 / 168
![Page 65: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/65.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
65 / 168
![Page 66: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/66.jpg)
Group Replication : Total Order Delivery - GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
66 / 168
![Page 67: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/67.jpg)
Group Replication: return commitAsynchronous Replication:
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
67 / 168
![Page 68: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/68.jpg)
Group Replication: return from commit (2)Semi-Sync Replication:
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
68 / 168
![Page 69: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/69.jpg)
Group Replication: return from commit (3)Group Replication:
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
69 / 168
![Page 70: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/70.jpg)
Group Replication : Optimistic LockingGroup Replication uses optimistic locking
during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification
Let´s first have a look at the traditional locking to compare.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
70 / 168
![Page 71: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/71.jpg)
Traditional locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
71 / 168
![Page 72: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/72.jpg)
Traditional locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
72 / 168
![Page 73: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/73.jpg)
Traditional locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
73 / 168
![Page 74: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/74.jpg)
Traditional locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
74 / 168
![Page 75: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/75.jpg)
Traditional locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
75 / 168
![Page 76: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/76.jpg)
Traditional locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
76 / 168
![Page 77: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/77.jpg)
Optimistic Locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
77 / 168
![Page 78: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/78.jpg)
Optimistic Locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
78 / 168
![Page 79: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/79.jpg)
Optimistic Locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
79 / 168
![Page 80: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/80.jpg)
Optimistic Locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
80 / 168
![Page 81: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/81.jpg)
Optimistic Locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
81 / 168
![Page 82: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/82.jpg)
Optimistic Locking
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
82 / 168
![Page 83: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/83.jpg)
Optimistic Locking
The system returns error 149 as certification failed:
ERROR 1180 (HY000): Got error 149 during COMMIT
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
83 / 168
![Page 84: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/84.jpg)
CertificationCertification is the process that only needs to answer the following unique question:
can the write (transaction) be applied ?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
84 / 168
![Page 85: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/85.jpg)
Certification (2)certification is a deterministic operationcertification individually happens on every member/nodecommunication with other members is not needed for certification
pass: enter in the apply queuefail: drop the transaction
serialized by the total order in GCS/Xcom + GTIDfirst committer wins rule
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
85 / 168
![Page 86: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/86.jpg)
Drawbacks of optimistic lockinghaving a first-committer-wins system means conflicts will more likely happen with:
large transactions
long running transactions
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
86 / 168
![Page 87: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/87.jpg)
GTIDGTIDs are the same as those used by asynchronous replication.
mysql> SELECT * FROM performance_schema.replication_connection_status\G ************************** 1. row *************************** CHANNEL_NAME: group_replication_applier GROUP_NAME: afb80f36-2bff-11e6-84e0-0800277dd3bf SOURCE_UUID: afb80f36-2bff-11e6-84e0-0800277dd3bf THREAD_ID: NULL SERVICE_STATE: ON COUNT_RECEIVED_HEARTBEATS: 0 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00 RECEIVED_TRANSACTION_SET: afb80f36-2bff-11e6-84e0-0800277dd3bf:1-57, f037578b-46b1-11e6-8005-08002774c31b:1-48937 LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
87 / 168
![Page 88: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/88.jpg)
GTIDbut transactions use the Group´s GTID
mysql> show master status\G ************************** 1. row *************************** File: mysql4-bin.000001 Position: 1501 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: afb80f36-2bff-11e6-84e0-0800277dd3bf:1-57, f037578b-46b1-11e6-8005-08002774c31b:1-48937
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
88 / 168
![Page 89: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/89.jpg)
Requirementsexclusively works with InnoDB tablesevery table must have a PK definedonly IPV4 is supporteda good network with low latency is importantmaximum of 9 members per grouplog-bin must be enabled and only binlog_format=ROW is supported
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
89 / 168
![Page 90: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/90.jpg)
Requirements (2)enable GTIDsreplication meta-data must be stored in system tables
--master-info-repository=TABLE --relay-log-info-repository=TABLE
writesets extraction must be enabled TODO: I need to check on 5.7.19 and 8.0.1
--transaction-write-set-extraction=XXHASH64
log-slave-updates must also be enabled
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
90 / 168
![Page 91: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/91.jpg)
Limitationsbinlog checksum is not supported
--binlog-checksum=NONE
savepoints were not supported before 5.7.19 & 8.0.1SERIALIZABLE is not supported as transaction isolation levelhttp://lefred.be/content/mysql-group-replication-limitations-savepoints/http://lefred.be/content/mysql-group-replication-and-table-design/
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
91 / 168
![Page 92: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/92.jpg)
Is my workload ready for Group Replication ?As the writesets (transactions) are replicated to all available nodes on commit, and asthey are certified on every node, a very large writeset could increase the amount ofcertification errors.
Additionally, changing the same record on all the nodes (hotspot) concurrently will alsocause problems.
And finally, the certification uses the primary key of the tables, a table without PK is alsoa problem.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
92 / 168
![Page 93: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/93.jpg)
Is my workload ready for Group Replication ?Therefore, when using Group Replication, we should pay attention to these points:
PK is mandatory (and a good one is better)
avoid large transactions
avoid hotspot
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
93 / 168
![Page 94: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/94.jpg)
ready ?
Migration from Master-Slave to GR
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
94 / 168
![Page 95: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/95.jpg)
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
95 / 168
![Page 96: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/96.jpg)
1) We install andsetup MySQL InnoDBCluster on one of thenew servers
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
96 / 168
![Page 97: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/97.jpg)
2) We restore abackup
3) setupasynchronousreplication on the newserver.
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
97 / 168
![Page 98: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/98.jpg)
4) We add a newinstance to our group
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
98 / 168
![Page 99: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/99.jpg)
5) We point theapplication to one ofour new nodes.
6) We wait and checkthat asynchronousreplication is caughtup
7) we stop thoseasynchronous slaves
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
99 / 168
![Page 100: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/100.jpg)
8) We attach themysql2 slave to thegroup
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
100 / 168
![Page 101: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/101.jpg)
9) Use MySQL Routerfor directing traffic
The plan
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
101 / 168
![Page 102: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/102.jpg)
Latest MySQL 5.7 is already installed onmysql3.
Let´s take a backup on mysql1 using meb:
[mysql1 ~]# mysqlbackup \ --host=127.0.0.1 \ --backup-dir=/tmp/backup \ --user=root --password=X \ backup-and-apply-log
LAB2: Prepare mysql3Asynchronous slave
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
102 / 168
![Page 103: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/103.jpg)
LAB2: Prepare mysql3 (2)Asynchronous slave
Copy the backup from mysql1 to mysql3:
[mysql1 ~]# scp -r /tmp/backup mysql3:/tmp
And restore it:
[mysql3 ~]# mysqlbackup --backup-dir=/tmp/backup --force copy-back[mysql3 ~]# rm /var/lib/mysql/mysql*-bin* # just some cleanup [mysql3 ~]# chown -R mysql. /var/lib/mysql
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
103 / 168
![Page 104: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/104.jpg)
LAB3: mysql3 as asynchronous slave (2)Asynchronous slave
Configure /etc/my.cnf with the minimal requirements:
[mysqld]...server_id=3enforce_gtid_consistency = ongtid_mode = onlog_bin log_slave_updates
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
104 / 168
![Page 105: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/105.jpg)
LAB2: Prepare mysql3 (3)Asynchronous slave
Let´s start MySQL on mysql3:
[mysql3 ~]# systemctl start mysqld
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
105 / 168
![Page 106: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/106.jpg)
find the GTIDs purgedchange MASTERset the purged GTIDsstart replication
LAB3: mysql3 as asynchronous slave (1)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
106 / 168
![Page 107: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/107.jpg)
LAB3: mysql3 as asynchronous slave (2)Find the latest purged GTIDs:
[mysql3 ~]# cat /tmp/backup/meta/backup_gtid_executed.sql SET @@GLOBAL.GTID_PURGED='33351000-3fe8-11e7-80b3-08002718d305:1-1002';
Connect to mysql3 and setup replication:
mysql> CHANGE MASTER TO MASTER_HOST="mysql1", MASTER_USER="repl_async", MASTER_PASSWORD='Xslave', MASTER_AUTO_POSITION=1;
mysql> RESET MASTER;mysql> SET global gtid_purged="VALUE FOUND PREVIOUSLY";mysql> START SLAVE;
Check that you receive the application´s traffic
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
107 / 168
![Page 108: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/108.jpg)
Administration made easy and more...
MySQL-Shell
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
108 / 168
![Page 109: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/109.jpg)
MySQL ShellThe MySQL Shell is an interactive Javascript, Python, or SQL interface supportingdevelopment and administration for the MySQL Server and is a component of the MySQLServer. You can use the MySQL Shell to perform data queries and updates as well asvarious administration operations.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
109 / 168
![Page 110: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/110.jpg)
MySQL Shell (2)The MySQL Shell provides:
Both Interactive and Batch operationsDocument and Relational ModelsCRUD Document and Relational APIs via scriptingTraditional Table, JSON, Tab Separated output results formatsMySQL Standard and X Protocolsand more...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
110 / 168
![Page 111: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/111.jpg)
LAB4: MySQL InnoDB ClusterCreate a single instance cluster
Time to use the new MySQL Shell !
[mysql3 ~]# mysqlsh
Let´s verify if our server is ready to become a member of a new cluster:
mysql-js> dba.checkInstanceCon�guration('root@mysql3:3306')
Change the configuration !
mysql-js> dba.con�gureLocalInstance()
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
111 / 168
![Page 112: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/112.jpg)
LAB4: MySQL InnoDB Cluster (2)Restart mysqld to use the new configuration:
[mysql3 ~]# systemctl restart mysqld
Create a single instance cluster
[mysql3 ~]# mysqlsh
mysql-js> dba.checkInstanceCon�guration('root@mysql3:3306')
mysql-js> \c root@mysql3:3306
mysql-js> cluster = dba.createCluster('MyInnoDBCluster')
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
112 / 168
![Page 113: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/113.jpg)
LAB4: Cluster Statusmysql-js> cluster.status(){ "clusterName": "MyInnoDBCluster", "defaultReplicaSet": { "name": "default", "primary": "mysql3:3306", "status": "OK_NO_TOLERANCE", "statusText": "Cluster is NOT tolerant to any failures.", "topology": { "mysql3:3306": { "address": "mysql3:3306", "mode": "R/W", "readReplicas": {}, "role": "HA", "status": "ONLINE" } } }}
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
113 / 168
![Page 114: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/114.jpg)
Add mysql4 to the Group:
restore the backupset the purged GTIDsuse MySQL shell
LAB5: add mysql4 to the cluster (1)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
114 / 168
![Page 115: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/115.jpg)
LAB5: add mysql4 to the cluster (2)Copy the backup from mysql1 to mysql4:
[mysql1 ~]# scp -r /tmp/backup mysql4:/tmp
And restore it:
[mysql4 ~]# mysqlbackup --backup-dir=/tmp/backup --force copy-back[mysql4 ~]# rm /var/lib/mysql/mysql*-bin* # just some cleanup [mysql4 ~]# chown -R mysql. /var/lib/mysql
Start MySQL on mysql4:
[mysql4 ~]# systemctl start mysqld
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
115 / 168
![Page 116: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/116.jpg)
LAB5: MySQL shell to add an instance (3)[mysql4 ~]# mysqlsh
Let´s verify the config:
mysql-js> dba.checkInstanceCon�guration('root@mysql4:3306')
And change the configuration:
mysql-js> dba.con�gureLocalInstance()
Restart the service to enable the changes:
[mysql4 ~]# systemctl restart mysqld
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
116 / 168
![Page 117: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/117.jpg)
LAB5: MySQL InnoDB Cluster (4)Group of 2 instances
Find the latest purged GTIDs:
[mysql4 ~]# cat /tmp/backup/meta/backup_gtid_executed.sql SET @@GLOBAL.GTID_PURGED='33351000-3fe8-11e7-80b3-08002718d305:1-1002';...
Connect to mysql4 and set GTID_PURGED
[mysql4 ~]# mysqlsh
mysql-js> \c root@mysql4:3306mysql-js> \sqlmysql-sql> RESET MASTER;mysql-sql> SET global gtid_purged="VALUE FOUND PREVIOUSLY";
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
117 / 168
![Page 118: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/118.jpg)
LAB5: MySQL InnoDB Cluster (5)mysql-sql> \js
mysql-js> dba.checkInstanceCon�guration('root@mysql4:3306')
mysql-js> \c root@mysql3:3306
mysql-js> cluster = dba.getCluster()
mysql-js> cluster.checkInstanceState('root@mysql4:3306')
mysql-js> cluster.addInstance("root@mysql4:3306")
mysql-js> cluster.status()
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
118 / 168
![Page 119: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/119.jpg)
Cluster Statusmysql-js> cluster.status(){ "clusterName": "MyInnoDBCluster", "defaultReplicaSet": { "status": "Cluster is NOT tolerant to any failures.", "topology": { "mysql3:3306": { "address": "mysql3:3306", "status": "ONLINE", "role": "HA", "mode": "R/W", "leaves": { "mysql4:3306": { "address": "mysql4:3306", "status": "RECOVERING", "role": "HA", "mode": "R/O", "leaves": {} } } } } }}
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
119 / 168
![Page 120: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/120.jpg)
Recovering progressOn standard MySQL, monitor the group_replication_recovery channel to seethe progress:
mysql> show slave status for channel 'group_replication_recovery'\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: mysql3 Master_User: mysql_innodb_cluster_rpl_user ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... Retrieved_Gtid_Set: 6e7d7848-860f-11e6-92e4-08002718d305:1-6,7c1f0c2d-860d-11e6-9df7-08002718d305:1-15,b346474c-8601-11e6-9b39-08002718d305:1964-77177,e8c524df-860d-11e6-9df7-08002718d305:1-2 Executed_Gtid_Set: 7c1f0c2d-860d-11e6-9df7-08002718d305:1-7,b346474c-8601-11e6-9b39-08002718d305:1-45408,e8c524df-860d-11e6-9df7-08002718d305:1-2 ...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
120 / 168
![Page 121: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/121.jpg)
point the applicationto the cluster
Migrate the application
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
121 / 168
![Page 122: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/122.jpg)
LAB6: Migrate the applicationNow we need to point the application to mysql3, this is the only downtime !
...[ 21257s] threads: 4, tps: 12.00, reads: 167.94, writes: 47.98, response time: 18[ 21258s] threads: 4, tps: 6.00, reads: 83.96, writes: 23.99, response time: 14[ 21259s] threads: 4, tps: 7.00, reads: 98.05, writes: 28.01, response time: 16[ 31250s] threads: 4, tps: 8.00, reads: 111.95, writes: 31.99, response time: 30[ 31251s] threads: 4, tps: 11.00, reads: 154.01, writes: 44.00, response time: 13[ 31252s] threads: 4, tps: 11.00, reads: 153.94, writes: 43.98, response time: 12[ 31253s] threads: 4, tps: 10.01, reads: 140.07, writes: 40.02, response time: 17^C[mysql1 ~]# run_app.sh mysql3
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
122 / 168
![Page 123: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/123.jpg)
LAB6: Migrate the applicationStop asynchronous replication on mysql2 and mysql3:
mysql2> stop slave;mysql3> stop slave;
Make sure gtid_executed range on mysql2 is lower or equal than on mysql3
mysql[2-3]> show global variables like 'gtid_executed'\Gmysql[2-3]> reset slave all;
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
123 / 168
![Page 124: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/124.jpg)
previous slave(mysql2) can nowbe part of the cluster
Add a third instance
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
124 / 168
![Page 125: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/125.jpg)
LAB7: Add mysql2 to the groupWe first validate the instance using MySQL Shell and we configure it.
[mysql2 ~]# mysqlsh
mysql-js> dba.checkInstanceCon�guration('root@mysql2:3306')
mysql-js> dba.con�gureLocalInstance()
We also need to remove super_read_only from my.cnf to be able to use the shell to addthe node to the cluster.
[mysql2 ~]# systemctl restart mysqld
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
125 / 168
![Page 126: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/126.jpg)
LAB7: Add mysql2 to the group (2)Back in MySQL shell we add the new instance:
[mysql2 ~]# mysqlsh
mysql-js> dba.checkInstanceCon�guration('root@mysql2:3306')
mysql-js> \c root@mysql3:3306
mysql-js> cluster = dba.getCluster()
mysql-js> cluster.addInstance("root@mysql2:3306")
mysql-js> cluster.status()
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
126 / 168
![Page 127: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/127.jpg)
LAB7: Add mysql2 to the group (3){ "clusterName": "MyInnoDBCluster", "defaultReplicaSet": { "name": "default", "primary": "mysql3:3306", "status": "OK", "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", "topology": { "mysql2:3306": { "address": "mysql2:3306", "mode": "R/O", "readReplicas": {}, "role": "HA", "status": "ONLINE" }, "mysql3:3306": { "address": "mysql3:3306", "mode": "R/W", "readReplicas": {}, "role": "HA", "status": "ONLINE" }, "mysql4:3306": { "address": "mysql4:3306", "mode": "R/O", "readReplicas": {}, Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
127 / 168
![Page 128: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/128.jpg)
writing to a single server
Single Primary Mode
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
128 / 168
![Page 129: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/129.jpg)
Default = Single Primary ModeBy default, MySQL InnoDB Cluster enables Single Primary Mode.
mysql> show global variables like 'group_replication_single_primary_mode';+---------------------------------------+-------+| Variable_name | Value |+---------------------------------------+-------+| group_replication_single_primary_mode | ON |+---------------------------------------+-------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
129 / 168
![Page 130: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/130.jpg)
Default = Single Primary ModeBy default, MySQL InnoDB Cluster enables Single Primary Mode.
mysql> show global variables like 'group_replication_single_primary_mode';+---------------------------------------+-------+| Variable_name | Value |+---------------------------------------+-------+| group_replication_single_primary_mode | ON |+---------------------------------------+-------+
In Single Primary Mode, a single member acts as the writable master (PRIMARY) and therest of the members act as hot-standbys (SECONDARY).
The group itself coordinates and configures itself automatically to determine whichmember will act as the PRIMARY, through a leader election mechanism.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
130 / 168
![Page 131: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/131.jpg)
Who´s the Primary Master ?As the Primary Master is elected, all nodes part of the group knows which one waselected. This value is exposed in status variables:
mysql> show status like 'group_replication_primary_member';+----------------------------------+--------------------------------------+| Variable_name | Value |+----------------------------------+--------------------------------------+| group_replication_primary_member | 28a4e51f-860e-11e6-bdc4-08002718d305 |+----------------------------------+--------------------------------------+
mysql> select member_host as "primary master" from performance_schema.global_status join performance_schema.replication_group_members where variable_name = 'group_replication_primary_member' and member_id=variable_value;+---------------+| primary master|+---------------+| mysql3 |+---------------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
131 / 168
![Page 132: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/132.jpg)
Create a Multi-Primary Cluster:It´s also possible to create a Multi-Primary Cluster using the Shell:
mysql-js> cluster=dba.createCluster('perconalive',{multiMaster: true})
A new InnoDB cluster will be created on instance 'root@mysql3:3306'.
The MySQL InnoDB cluster is going to be setup in advanced Multi-Master Mode.Before continuing you have to con�rm that you understand the requirements andlimitations of Multi-Master Mode. Please read the manual before proceeding.
I have read the MySQL InnoDB cluster manual and I understand the requirementsand limitations of advanced Multi-Master Mode.Con�rm [y|N]:
Or you can force it to avoid interaction (for automation) :
js> cluster=dba.createCluster('perconalive',{multiMaster: true, force: true})
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
132 / 168
![Page 133: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/133.jpg)
get more info
Monitoring
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
133 / 168
![Page 134: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/134.jpg)
Performance SchemaGroup Replication uses Performance_Schema to expose status
mysql3> SELECT * FROM performance_schema.replication_group_members\G *************************** 1. row *************************** CHANNEL_NAME: group_replication_applier MEMBER_ID: 00db47c7-3e23-11e6-afd4-08002774c31b MEMBER_HOST: mysql3.localdomain MEMBER_PORT: 3306 MEMBER_STATE: ONLINE
mysql3> SELECT * FROM performance_schema.replication_connection_status\G *************************** 1. row *************************** CHANNEL_NAME: group_replication_applier GROUP_NAME: afb80f36-2bff-11e6-84e0-0800277dd3bf SOURCE_UUID: afb80f36-2bff-11e6-84e0-0800277dd3bf THREAD_ID: NULL SERVICE_STATE: ON COUNT_RECEIVED_HEARTBEATS: 0 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00 RECEIVED_TRANSACTION_SET: afb80f36-2bff-11e6-84e0-0800277dd3bf:1-2 LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
134 / 168
![Page 135: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/135.jpg)
Member StateThese are the different possible state for a node member:
ONLINE
OFFLINE
RECOVERING
ERROR: when a node is leaving but the plugin was not instructed to stopUNREACHABLE
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
135 / 168
![Page 136: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/136.jpg)
Status information & metrics
Membersmysql> SELECT * FROM performance_schema.replication_group_members\G
*************************** 1. row *************************** CHANNEL_NAME: group_replication_applier MEMBER_ID: 00db47c7-3e23-11e6-afd4-08002774c31b MEMBER_HOST: mysql3.localdomain MEMBER_PORT: 3306 MEMBER_STATE: ONLINE *************************** 2. row *************************** CHANNEL_NAME: group_replication_applier MEMBER_ID: e1544c9d-4451-11e6-9f5a-08002774c31b MEMBER_HOST: mysql4.localdomain.localdomain MEMBER_PORT: 3306 MEMBER_STATE: ONLINE
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
136 / 168
![Page 137: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/137.jpg)
Status information & metrics
Connectionsmysql> SELECT * FROM performance_schema.replication_connection_status\G
*************************** 1. row *************************** CHANNEL_NAME: group_replication_applier GROUP_NAME: afb80f36-2bff-11e6-84e0-0800277dd3bf SOURCE_UUID: afb80f36-2bff-11e6-84e0-0800277dd3bf THREAD_ID: NULL SERVICE_STATE: ON COUNT_RECEIVED_HEARTBEATS: 0 LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00 RECEIVED_TRANSACTION_SET: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089, afb80f36-2bff-11e6-84e0-0800277dd3bf:1-2834 LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row *************************** CHANNEL_NAME: group_replication_recovery GROUP_NAME: SOURCE_UUID: THREAD_ID: NULL SERVICE_STATE: OFF COUNT_RECEIVED_HEARTBEATS: 0
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
137 / 168
![Page 138: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/138.jpg)
Status information & metrics
Local node status mysql> select * from performance_schema.replication_group_member_stats\G
*************************** 1. row *************************** CHANNEL_NAME: group_replication_applier VIEW_ID: 14679667214442885:4 MEMBER_ID: e1544c9d-4451-11e6-9f5a-08002774c31b COUNT_TRANSACTIONS_IN_QUEUE: 0 COUNT_TRANSACTIONS_CHECKED: 5961 COUNT_CONFLICTS_DETECTED: 0 COUNT_TRANSACTIONS_ROWS_VALIDATING: 0 TRANSACTIONS_COMMITTED_ALL_MEMBERS: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089 afb80f36-2bff-11e6-84e0-0800277dd3bf:1-5718 LAST_CONFLICT_FREE_TRANSACTION: afb80f36-2bff-11e6-84e0-0800277dd3bf:5718
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
138 / 168
![Page 139: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/139.jpg)
Performance_SchemaYou can find GR information in the following Performance_Schema tables:
replication_applier_con�guration
replication_applier_status
replication_applier_status_by_worker
replication_connection_con�guration
replication_connection_status
replication_group_member_stats
replication_group_members
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
139 / 168
![Page 140: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/140.jpg)
Status during recovery mysql> SHOW SLAVE STATUS FOR CHANNEL 'group_replication_recovery'\G
*************************** 1. row *************************** Slave_IO_State: Master_Host: <NULL> Master_User: gr_repl Master_Port: 0 ... Relay_Log_File: mysql4-relay-bin-group_replication_recovery.000001 ... Slave_IO_Running: No Slave_SQL_Running: No ... Executed_Gtid_Set: 5de4400b-3dd7-11e6-8a71-08002774c31b:1-814089, afb80f36-2bff-11e6-84e0-0800277dd3bf:1-5718 ... Channel_Name: group_replication_recovery
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
140 / 168
![Page 141: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/141.jpg)
Sys SchemaThe easiest way to detect if a node is a member of the primary component (when thereare partitioning of your nodes due to network issues for example) and therefore a validcandidate for routing queries to it, is to use the sys table.
Additional information for sys can be downloaded athttps://github.com/lefred/mysql_gr_routing_check/blob/master/addition_to_sys.sql
On the primary node:
[mysql? ~]# mysql < /tmp/addition_to_sys.sql
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
141 / 168
![Page 142: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/142.jpg)
Sys SchemaIs this node part of PRIMARY Partition:
mysql3> SELECT sys.gr_member_in_primary_partition();+------------------------------------+| sys.gr_node_in_primary_partition() |+------------------------------------+| YES |+------------------------------------+
To use as healthcheck:
mysql3> SELECT * FROM sys.gr_member_routing_candidate_status;+------------------+-----------+---------------------+----------------------+| viable_candidate | read_only | transactions_behind | transactions_to_cert |+------------------+-----------+---------------------+----------------------+| YES | YES | 0 | 0 |+------------------+-----------+---------------------+----------------------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
142 / 168
![Page 143: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/143.jpg)
LAB8: Sys Schema - Health CheckOn one of the non Primary nodes, run the following command:
mysql-sql> �ush tables with read lock;
Now you can verify what the healthcheck exposes to you:
mysql-sql> SELECT * FROM sys.gr_member_routing_candidate_status;+------------------+-----------+---------------------+----------------------+| viable_candidate | read_only | transactions_behind | transactions_to_cert |+------------------+-----------+---------------------+----------------------+| YES | YES | 950 | 0 |+------------------+-----------+---------------------+----------------------+
mysql-sql> UNLOCK TABLES;
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
143 / 168
![Page 144: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/144.jpg)
application interaction
MySQL Router
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
144 / 168
![Page 145: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/145.jpg)
MySQL RouterMySQL Router is lightweight middleware that provides transparent routing between yourapplication and backend MySQL Servers. It can be used for a wide variety of use cases,such as providing high availability and scalability by effectively routing database traffic toappropriate backend MySQL Servers.
MySQL Router doesn´t require any specific configuration. It configures itself automatically(bootstrap) using MySQL InnoDB Cluster´s metadata.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
145 / 168
![Page 146: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/146.jpg)
LAB9: MySQL RouterWe will now use mysqlrouter between our application and the cluster.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
146 / 168
![Page 147: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/147.jpg)
LAB9: MySQL Router (2)Configure MySQL Router that will run on the app server (mysql1). We bootstrap it usingthe Primary-Master:
[root@mysql1 ~]# mysqlrouter --bootstrap mysql3:3306 --user mysqlrouterPlease enter MySQL password for root: WARNING: The MySQL server does not have SSL con�gured and metadata used by the router may be transmitted unencrypted.
Bootstrapping system MySQL Router instance...MySQL Router has now been con�gured for the InnoDB cluster 'MyInnoDBCluster'.
The following connection information can be used to connect to the cluster.
Classic MySQL protocol connections to cluster 'MyInnoDBCluster':- Read/Write Connections: localhost:6446- Read/Only Connections: localhost:6447
X protocol connections to cluster 'MyInnoDBCluster':- Read/Write Connections: localhost:64460- Read/Only Connections: localhost:64470
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
147 / 168
![Page 148: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/148.jpg)
LAB9: MySQL Router (3)Now let´s modify the configuration file to listen on port 3306:
in /etc/mysqlrouter/mysqlrouter.conf:
[routing:MyInnoDBCluster_default_rw]-bind_port=6446+bind_port=3306
We can stop mysqld on mysql1 and start mysqlrouter into a screen session:
[mysql1 ~]# systemctl stop mysqld[mysql1 ~]# systemctl start mysqlrouter
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
148 / 168
![Page 149: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/149.jpg)
LAB9: MySQL Router (4)Before killing a member we will change systemd´s default behavior that restartsmysqld immediately:
in /usr/lib/systemd/system/mysqld.service add the following under[Service]
RestartSec=30
[mysql3 ~]# systemctl daemon-reload
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
149 / 168
![Page 150: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/150.jpg)
LAB9: MySQL Router (5)Now we can point the application to the router (back to mysql1):
[mysql1 ~]# run_app.sh
Check app and kill mysqld on mysql3 (the Primary Master R/W node) !
[mysql3 ~]# kill -9 $(pidof mysqld)
mysql> select member_host as "primary" from performance_schema.global_status join performance_schema.replication_group_members where variable_name = 'group_replication_primary_member' and member_id=variable_value;+---------+| primary |+---------+| mysql4 |+---------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
150 / 168
![Page 151: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/151.jpg)
ProxySQL / HA Proxy / F5 / ...
3rd party router/proxy
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
151 / 168
![Page 152: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/152.jpg)
ProxySQL also has native support for GroupReplication which makes it maybe the bestchoice for advanced users.
3rd party router/proxyMySQL InnoDB Cluster can also work with third party router / proxy.
If you need some specific features that are not yet available in MySQL Router, liketransparent R/W splitting, then you can use your software of choice.
The important part of such implementation is to use a good health check to verify if theMySQL server you plan to route the traffic is in a valid state.
MySQL Router implements that natively, and it´s very easy to deploy.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
152 / 168
![Page 153: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/153.jpg)
operational tasks
Recovering Node
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
153 / 168
![Page 154: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/154.jpg)
Recovering Nodes/MembersThe old master (mysql3) got killed.MySQL got restarted automatically by systemdLet´s add mysql3 back to the clsuter
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
154 / 168
![Page 155: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/155.jpg)
LAB10: Recovering Nodes/Members[mysql3 ~]# mysqlsh
mysql-js> \c root@mysql4:3306 # The current master
mysql-js> cluster = dba.getCluster()
mysql-js> cluster.status()
mysql-js> cluster.rejoinInstance("root@mysql3:3306")
Rejoining the instance to the InnoDB cluster. Depending on the originalproblem that made the instance unavailable, the rejoin operation might not besuccessful and further manual steps will be needed to �x the underlyingproblem.
Please monitor the output of the rejoin operation and take necessary action ifthe instance cannot rejoin.
Please provide the password for 'root@mysql3:3306': Rejoining instance to the cluster ...
The instance 'root@mysql3:3306' was successfully rejoined on the cluster.
The instance 'mysql3:3306' was successfully added to the MySQL Cluster.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
155 / 168
![Page 156: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/156.jpg)
mysql-js> cluster.status(){ "clusterName": "MyInnoDBCluster", "defaultReplicaSet": { "name": "default", "primary": "mysql4:3306", "status": "OK", "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", "topology": { "mysql2:3306": { "address": "mysql2:3306", "mode": "R/O", "readReplicas": {}, "role": "HA", "status": "ONLINE" }, "mysql3:3306": { "address": "mysql3:3306", "mode": "R/O", "readReplicas": {}, "role": "HA", "status": "ONLINE" }, "mysql4:3306": { "address": "mysql4:3306", "mode": "R/W", "readReplicas": {}, "role": "HA", "status": "ONLINE" } } }}
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
156 / 168
![Page 157: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/157.jpg)
Recovering Nodes/Members (automatically)This time before killing a member of the group, we will persist the configuration on disk inmy.cnf.
We will use again the same MySQL command as previouslydba.con�gureLocalInstance() but this time when all nodes are alreadypart of the Group.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
157 / 168
![Page 158: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/158.jpg)
LAB10: Recovering Nodes/Members (2)Verify that all nodes are ONLINE.
...mysql-js> cluster.status()
Then on all nodes run:
mysql-js> dba.con�gureLocalInstance()
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
158 / 168
![Page 159: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/159.jpg)
LAB10: Recovering Nodes/Members (3)Kill one node again:
[mysql3 ~]# kill -9 $(pidof mysqld)
systemd will restart mysqld and verify if the node joined.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
159 / 168
![Page 160: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/160.jpg)
understanding
Flow Control
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
160 / 168
![Page 161: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/161.jpg)
Flow ControlWhen using MySQL Group Replication, it´s possible that some members are lagging behindthe group. Due to load, hardware limitation, etc... This lag can become problematic to keepgood certification performance and keep the possible certification failures as low aspossible.
More problems can occur in multi-primary/write clusters when the applying queue grows,the risk to have conflicts with those not yet applied transactions increases.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
161 / 168
![Page 162: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/162.jpg)
Flow Control (2)Within MySQL Group Replication´s FC implementation :
the Group is never totally stalled
the node having issues doesn´t send flow control messages to the rest of the groupasking to slow down
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
162 / 168
![Page 163: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/163.jpg)
Flow Control (3)Every member of the Group send some statistics about its queues (applier queue andcertification queue) to the other members.
Then every node decide to slow down or not if they realize that one node reached thethreshold for one of the queue:
group_replication_�ow_control_applier_threshold (defaultis 25000)
group_replication_�ow_control_certi�er_threshold
(default is 25000)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
163 / 168
![Page 164: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/164.jpg)
Flow Control (4)So when group_replication_�ow_control_mode is set to QUOTA on thenode seeing that one of the other members of the cluster is lagging behind (thresholdreached), it will throttle the write operations to the the minimum quota.
This quota is calculated based on the number of transactions applied in the last second,and then it is reduced below that by subtracting the "over the quota" messages from thelast period.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
164 / 168
![Page 165: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/165.jpg)
LAB10: Flow ControlDuring this last lab, we will reduce the flow control threshold on the Primary Master:
mysql> show global variables like '%�ow%';+----------------------------------------------------+-------+| Variable_name | Value |+----------------------------------------------------+-------+| group_replication_�ow_control_applier_threshold | 25000 || group_replication_�ow_control_certi�er_threshold | 25000 || group_replication_�ow_control_mode | QUOTA |+----------------------------------------------------+-------+3 rows in set (0.08 sec)
mysql> set global group_replication_�ow_control_applier_threshold=100;
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
165 / 168
![Page 166: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/166.jpg)
LAB10: Flow Control (1)And now we block all writes on one of the Secondary-Masters:
mysql> �ush tables with read lock;
And we check how the queue is growing:
mysql> SELECT * FROM sys.gr_member_routing_candidate_status;+------------------+-----------+---------------------+----------------------+| viable_candidate | read_only | transactions_behind | transactions_to_cert |+------------------+-----------+---------------------+----------------------+| YES | NO | 487 | 0 |+------------------+-----------+---------------------+----------------------+
Did you notice something on the application when the threshold was reached ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
166 / 168
![Page 167: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/167.jpg)
LAB10: Flow Control (2)If nothing happened, please increase the trx rate:
[root@mysql1 ~]# run_app.sh mysql1 --tx-rate=500
When the application's writes are low, you can just remove the lock and see the queue andthe effect on the application:
mysql> UNLOCK TABLES;
Create flow control again and when you see the application writing just a fewtransactions, on the Primary-Master, disable the flow control mode:
mysql> set global group_replication_�ow_control_mode='DISABLED';
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
167 / 168
![Page 168: MySQL innodb cluster and Group Replication in a nutshell - hands-on tutorial with MySQL Enterprise Backup](https://reader030.vdocuments.net/reader030/viewer/2022021418/5a647c9c7f8b9a36568b4b45/html5/thumbnails/168.jpg)
Thank you !
Any Questions ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
168 / 168