my sql fabric webinar tw2
DESCRIPTION
TRANSCRIPT
![Page 1: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/1.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
MySQL Fabric:Easy Management of MySQL Servers
杜修文 [email protected] Principal Sales ConsultantOracle LLC
July 17, 2014
![Page 2: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/2.jpg)
2
Insert Picture Here
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |
An extensible and easy-to-use framework for managing a farm of MySQL server supporting high-availability and sharding
MySQL Fabric
![Page 3: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/3.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |3
MySQL Fabric: Goals & Features
Connector API Extensions Support Transactions Support full SQL
Decision logic in connector Reducing network load
Load Balancing Read-Write Split Distribute transactions
Global Updates Global tables Schema updates
Shard Multiple Tables Using same key
Sharding Functions Range (Consistent) Hash
Shard Operations Using built-in executor Shard move Shard split
![Page 4: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/4.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |4
Transaction Handling
![Page 5: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/5.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |5
Routing Transactions Pre-declare properties of transactions Detecting transaction boundaries Push as much as possible to server
Managing Session State Move session state between servers
− Easy to use− Expensive and error prone
Reset state after each transaction− Transactions start with default session state
Transaction Handling
![Page 6: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/6.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |6
Routing Transactions
Shard #2
Shard #1
Shard #3
Switch
StateStore Executor
QUERYKEY
KEY
QUERY
Contain decision logicfor routing queries
Contain informationabout location of shards
SHARD#
![Page 7: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/7.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |7
Birds-eye View
High Availability Groups (Shards)
MySQL FabricNode
Application
XML-RPC
SQL
Connector
Connector
Connector
Operator
![Page 8: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/8.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |8
MySQL Fabric: Prerequisites
MySQL Servers (version 5.6.10 or later) Server for meta-data backing store Servers being managed
Python 2.6 or 2.7 No support for 3.x yet
MySQL Utilities 1.4.0 Available at http://labs.mysql.com/
![Page 9: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/9.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |9
MySQL Fabric: Configuration
Backing Store MySQL server Persistent storage for state Storage engine-agnostic
Protocol Address where node will be Currently only XML-RPC
Logging Chatty: INFO (default) Moderate: WARNING URL for rotating log
[storage]address = localhost:3306user = fabricpassword =database = fabricconnection_timeout = 6
[protocol.xmlrpc]address = localhost:8080threads = 5
[logging]level = INFOurl = file:///var/log/fabric.log
![Page 10: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/10.jpg)
10
Insert Picture Here
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |
Connecting to aMySQL Fabric Farm
![Page 11: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/11.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |11
Fabric-aware Connector API
Connector API Extensions Support Transactions Support full SQL
Decision logic in connector Reducing network load
Load Balancing Read-Write Split Distribute transactions
Fabric-aware Connectors Connector/J Connector/Python Connector/PHP
Fabric-aware Frameworks Doctrine Hibernate
Focus on Connector/Python
![Page 12: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/12.jpg)
12
Insert Picture Here
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |
Architecture forSharding
![Page 13: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/13.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |13
Benefits of Sharding
Write scalability Can handle more writes
Large data set Database too large Does not fit on single server
Improved performance Smaller index size Smaller working set Improve performance
UID 10000-20000 UID 20001-40000
![Page 14: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/14.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |14
MySQL Fabric: Sharding Goals & Features
Connector API Extensions Support Transactions Support full SQL
Decision logic in connector Reducing network load
Shard Multiple Tables Using same key
Global Updates Global tables Schema updates
Sharding Functions Range (Consistent) Hash
Shard Operations Using built-in executor Shard move Shard split
![Page 15: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/15.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |15
Mapping the Sharding Key
What is a sharding key? Single column Multi column
− Same table?− Different tables?
How is the key transformed? Hash Range User-defined
ComputeShard#
Key
Shard#
(X)(X,Y,...)
RANGEHASHSomething else
![Page 16: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/16.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |16
Sharded Tables
In desperate needof sharding!
Foreign keys
![Page 17: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/17.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |17
Multi-table Query with Sharded Tables
SELECT first_name, last_name, salaryFROM salaries JOIN employees USING (emp_no)WHERE emp_no = 21012 AND CURRENT_DATE BETWEEN from_date AND to_date;
Referential Integrity Constraint Example query joining salaries and employees Same key, same shard: co-locate rows for same user
JOIN normally based on equality Using non-equality defeats purpose of foreign key
![Page 18: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/18.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |18
Global Tables
Does not needto be sharded
![Page 19: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/19.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |19
Multi-table Query with Global Tables
SELECT first_name, last_name, GROUP_CONCAT(dept_name) FROM employees JOIN dept_emp USING (emp_no) JOIN departments USING (dept_no)WHERE emp_no = 21012 GROUP BY emp_no;
JOIN with departments table Has no employee number, hence no sharding key Table need to be present on all shards
How do we update global tables?
![Page 20: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/20.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |20
Sharding Architecture
Shards
MySQL Fabric Node
Application
GlobalGroup
Global Updates
ShardUpdates
Replication
Support global updatefor off-line shards
![Page 21: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/21.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |21
Demo
![Page 22: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/22.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |22
Demo Server Side – Startup Instances
3306
3606
FabricRepository
3506
Fabric Node
32274
3308 3314 3408 3406
![Page 23: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/23.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |23
Demo Server Side – Configuration of Fabric
/etc/mysql/fabric.cfg[DEFAULT]prefix = sysconfdir = /etclogdir = /var/log[storage]address = localhost:3506user = fabricpassword = welcome1database = fabricauth_plugin = mysql_native_passwordconnection_timeout = 6connection_attempts = 6connection_delay = 1
[servers]user = fabricpassword = welcome1[protocol.xmlrpc]address = localhost:32274threads = 5user = adminpassword =welcome1 disable_authentication = norealm = MySQL Fabricssl_ca = ssl_cert = ssl_key = [executor]executors = 5[logging]level = INFOurl = file:///var/log/fabric.log
![Page 24: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/24.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |24
Startup Fabric
•Fabric help command mysqlfabric help commands•Instantiate Fabricmysqlfabric manage setup•Bring up Fabricmysqlfabric manage start --deamonize
![Page 25: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/25.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |25
Demo – Configure by Fabric
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314 3408 3406
![Page 26: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/26.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |26
Fabric Group Operation
Create 1 global group, 2 shard groupmysqlfabric group create gp-globalmysqlfabric group create gp1mysqlfabric group create gp2
Assign Servers to groupsmysqlfabric group add gp-global 127.0.0.1:3606mysqlfabric group add gp1 127.0.0.1:3306mysqlfabric group add gp1 127.0.0.1:3308mysqlfabric group add gp1 127.0.0.1:3314mysqlfabric group add gp2 127.0.0.1:3406mysqlfabric group add gp2 127.0.0.1:3408
Pick the master server by Fabricmysqlfabric group promote gp1mysqlfabric group promote gp2
![Page 27: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/27.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |27
Fabric Group Operations
• Let Fabric manage auto-failover for youmysqlfabric group activate gp-globalmysqlfabric group activate gp1mysqlfabric group activate gp2
• Define Range shard on to global groupmysqlfabric sharding create_definition RANGE gp-global
• Add shard table – employees to the shard created (shard map id = 1) shard by id columnmysqlfabric sharding add_table 1 employees.employees id
• Assign rows with id value 1~999 to group gp1, 1000 above to group gp2mysqlfabric sharding add_shard 1 gp1/1,gp2/1000 --state=ENABLED
![Page 28: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/28.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |28
Demo – Databases is Ready
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
![Page 29: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/29.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |29
Demo Application Side– Need Connector/J 5.1.30 or later
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
![Page 30: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/30.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |30
Demo Application Side– Cache Fabric Configuration on creating DB Connection
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/JConnection
Object
![Page 31: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/31.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |31
Java Application- Global tables and DDLs are accessed by property fabricServerGroup
import com.mysql.fabric.jdbc.FabricMySQLConnection;…rawConnection = DriverManager.getConnection( baseUrl + database + "?fabricServerGroup=gp-global“,user, password);statement = rawConnection.createStatement();…statement.executeUpdate("create table employees.employees (emp_no int not null,first_name varchar(50),last_name varchar(50), department_id int,primary key (emp_no))");…statement.executeUpdate("insert into employees.departments values (1,'Sales'),(2,'Financial')");
![Page 32: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/32.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |32
Demo Environment – Dispatch Request Based On Cached Configuration with Connection
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/JConnection
Object – Shard Table
Write Sharded Table
Write Global Table
![Page 33: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/33.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |33
Demo Environment – Dispatch Request Based On Cached Configuration with Connection
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/JConnection
Object – Shard Table
Write Sharded Table
Write Global Table
![Page 34: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/34.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |34
Demo Environment – Dispatch Request Based On Cached Configuration with Connection
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/JConnection
Object – Shard Table
Write Sharded Table
Write Global Table
![Page 35: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/35.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |35
Shared tables are accessed by property fabricShardTable and fabericShardKey
rawConnection = DriverManager.getConnection( "jdbc:mysql:fabric://127.0.0.1:32274/mysql?fabricShardTable=employees.employees“, user, password);
…for (int i = 0; i < 4; ++i) { connection.setShardKey(ids[i].toString()); ps.setInt(1, ids[i]); ps.setString(2, firstNames[i]); ps.setString(3, lastNames[i]); ps.setString(4, departmentIds[i]); ps.executeUpdate();}
![Page 36: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/36.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |36
Demo Environment – Read/Write Split within Shard
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/JConnection
Object – Shard Table
Read Only Operations
Read Global Table
![Page 37: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/37.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |37
Read Only Operations
FabricMySQLConnection newConn = (FabricMySQLConnection) rawConnection;
newConn.setReadOnly(true);ps = newConn.prepareStatement( "select emp_no, first_name, last_name, d.name, i.variable_value from employees.employees e, information_schema.global_variables i, employees.departments d where e.department_id = d.department_id and i.variable_name='port' and emp_no = ?");
![Page 38: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/38.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |38
Demo Environment – Failure Detected
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/JConnection
Object
Write Global Table
Change Master to 3606
Change Master to 3308
X
![Page 39: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/39.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |39
Demo Environment – Failover within Shard
GP1
3306
GP-Global
3606
GP2
FabricRepository
3506
Fabric Node
32274
3308-M 3314-M 3408-M 3406
JavaConnector/JConnection
Object
Write Shard Table
Write Global Table
X
![Page 40: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/40.jpg)
40
Insert Picture Here
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |
Closing Remarks
![Page 41: My sql fabric webinar tw2](https://reader033.vdocuments.net/reader033/viewer/2022051818/54b746324a795955768b4602/html5/thumbnails/41.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |41
Thoughts for the Future
Connector multi-cast Scatter-gather UNION of result sets More complex operations?
Internal interfaces Improve extension support Improve procedures
support
Command-line interface Improving usability Focus on ease-of-use
More protocols MySQL-RPC Protocol? AMQP?
More frameworks?
More HA group types DRBD MySQL Cluster
Fabric-unaware connectors?