maximizing mysql availability, data integrity, performance, and scalability · maximizing mysql...
TRANSCRIPT
. © 2011 Schooner Information Technology, p1
Maximizing MySQL Availability, Data Integrity, Performance, and Scalability
Schooner Information Technology. October 25, 2011
Dr. John R. Busch Founder and CTO Schooner Information Technology [email protected]
. © 2011 Schooner Information Technology, p2
Maximizing MySQL Availability, Data Integrity, Performance, and Scalability
Abstract: First, we discuss the use of synchronous replication of InnoDB to eliminate data loss and to provide cluster-wide data consistency. We then discuss instant automatic fail-over providing five 9s to maximize service availability. Next we discuss the use of high parallelism and granular concurrency control with synchronous replication to fully exploit commodity servers and storage to maximize performance. Next we discuss extending base synchronous clusters with tightly-coupled, parallel asynchronous replication to achieve auto-failover and high consistency across geographically dispersed data centers. Bio: Dr. John R. Busch is Founder and CTO of Schooner Information technology, focusing on high performance and highly available databases for high volume web facing, cloud, and enterprise services. Prior to Schooner, John was director of computer system research at Sun Microsystems Laboratories, VP of Engineering with Diba, founder of Clarity Software, and R&D director of computer systems at Hewlett Packard. John holds a Ph.D. in Computer Science from UCLA, M.S. in Mathematics from UCLA, M.S. in Computer Science from Stanford University, and attended the Stanford Sloan program.
. © 2011 Schooner Information Technology, p3
Data
§ Most important and valuable component of modern applications and websites § Driving revolutionary changes in computing and the internet
Ø New opportunities for generating revenue Ø More efficient use of current business processes and infrastructure
§ Data access downtime or poor performance has a major cost to a business’ bottom line
Business’ Most Valuable Asset: Its Data
. © 2011 Schooner Information Technology, p4
“Let me tell you the difference between Facebook and everyone else, we don't crash EVER! If our service is down for even a minute, our entire reputation is irreversibly destroyed Facebook and Google invest hundreds of millions of dollars every year on custom software and hardware infrastructure to optimize availability, performance, administration, and cost
The Mission-Critical Imperative
. © 2011 Schooner Information Technology, p5
§ Providing high data availability, excellent response time is critical for key classes of businesses
Ø Web 2.0 Ø eCommerce Ø High-volume websites Ø Telecommunications
§ They require a mission critical database
The Mission-Critical Imperative
. © 2011 Schooner Information Technology, p6
Mission-Critical Database Requirements
High Availability
High Performance
and Scalability
Simple and Powerful
Administra<on Data Integrity Cost Effec<ve
Standards and
Compa<bility
. © 2011 Schooner Information Technology, p7
Mission-Critical Database Goals and Metrics
§ Service unavailability (minutes/year) from failures, disaster recovery, or during planned administration
§ Probability of data loss or corruption; data consistency levels
§ Transaction throughput, response time;
performance scalability; performance stability § Ease of cluster administration; fail-over
automation; monitoring and optimization tools
§ Total cost of ownership (TCO); return on investment (ROI)
§ Level of standards compliance and certification
§ High Availability
§ High Data Integrity
§ High Performance and Scalability
§ Simple and powerful administration § Cost effective
§ Standards and Compatibility
Goals Metrics
. © 2011 Schooner Information Technology, p8
Loosely-Coupled Asynchronous and Semi-Synchronous Replication
Master mysqld Slave mysqld
Relay log
InnoDB Tx log DB MySQL
Bin log InnoDB Tx log
DB
Last tx=100 Last tx=100 Last tx=70 (asynch Last tx=100 (synch)
Last tx=50
Log events pulled by Slave
Tx.Commit(101) Repl.apply(51)
tx=101 tx=101 tx=51 tx=51
Read version based on tx=50
Example Products : MySQL Enterprise 5.1 Asynchronous and 5.5/5.6 Semi-Synchronous Replication
. © 2011 Schooner Information Technology, p9
Loosely-Coupled Asynchronous and Semi-Synchronous Replication
Master mysqld Slave mysqld
Relay log
InnoDB Tx log DB MySQL
Bin log InnoDB Tx log
DB
Last tx=100 Last tx=100 Last tx=70 (asynch Last tx=100 (synch)
Last tx=50
Log events pulled by Slave
Tx.Commit(101) Repl.apply(51)
tx=101 tx=101 tx=51 tx=51
Read version based on tx=50
Inconsistent (Stale) Data
Slow Recovery Complex Management
Slow ExecuCon
PotenCal Data Loss
Example Products : MySQL Enterprise 5.1 Asynchronous and 5.5/5.6 Semi-Synchronous Replication
. © 2011 Schooner Information Technology, p10
Loosely-Coupled Asynchronous and Semi-Synchronous Replication
Limited Service Availability • Master fail-over, re-synch of slaves
Limited Data Integrity • Lost data; inconsistent Data
Limited Performance and Utilization
• Low throughput and low utilization Complex Administration
• Manual processes, slave re-synch
High Cost of Ownership • High capital expense from server sprawl • Increased operating expense from power, space, admin • Reduced revenue and customer satisfaction from service downtime
. © 2011 Schooner Information Technology, p11
Tight Coupling and Fully Synchronous Replication
• Slaves in lock-step with Master • At master transaction commit, all Slaves guaranteed to have received and committed the changes
SchoonerSQL (Master)
Cluster Admin
MySQL
Op<mized Parallel Execu<on Threads in
SchoonerSQL
SchoonerSQL (Read Master)
Parallel Synchronous Replica<on During Transac<on Execu<on
MySQL clients
MySQL clients
Concurrently Execu<ng
Transac<ons Parallel
SchonerSQL Replica<on Threads
InnoDB
Concurrently Execu<ng
Transac<ons
Red Hat, CentOS Linux
Standard X86 Server
Parallel SchoonerSQLReplica<on Threads
InnoDB
Red Hat, CentOS Linux
Standard X86 Server
Op<mized Parallel Execu<on Threads in
SchoonerSQL
. © 2011 Schooner Information Technology, p12
Tight-Coupling and Synchronous Replication
SchoonerSQL Master
SchoonerSQL ReadMaster
InnoDB Tx log DB InnoDB
Tx log DB
Last tx=100 Last tx=100 Last tx=100 Last tx=100
Log for tx=100 pushed to Slave
Tx.Commit(101) Repl.apply(100)
tx=101 tx=101
Slave ACK for tx=100
Eliminates Service InterrupCons § Fast , Transparent Fail-‐Over (VIPs) § Fast incremental recovery
Easy Management High Performance High UClizaCon
No Data Loss Cluster-‐Wide Consistent Reads
. © 2011 Schooner Information Technology, p13
Tightly-coupled MySQL synchronous replication with fast, automated application transparent fail-over and can provide much higher service availability than that achievable with asynchronous or semi-synchronous replication
Availability Improvement from Synchronous ReplicaCon (% CumulaCve Down Time ReducCon)
Tight Coupling and Synchronous Replication with Transparent Automatic Fail-over Improves Service Availability by 90%
0
10
20
30
40
50
60
70
80
90
100
MySQL 5.5 SchoonerSQL Auto-‐Failover
Schooner SQL On-‐Line Upgrades
. © 2011 Schooner Information Technology, p14
Tight Coupling and Synchronous Replication Provides Much Higher Performance Throughput per Server
DBT2 open-source OLTP version of TPC-C 1000 warehouses, 32 connections 0 think-time Result metric: TPM (new order)
Measurement Configuration 2 node Master-Slave configuration 2 socket Westmere 72GB DRAM
Synchronous TransacCon Throughput per Server can be Much greater Than Asynchronous or Semi-‐Synchronous
(with hard disc drives (HDDs))
0 2 4 6 8 10 12 14
MySQL 5.5 asynch
MySQL 5.5 semi-‐synch
SchoonerSQL
Transac<on Throughput with Hard Drives (kTPM)
. © 2011 Schooner Information Technology, p15
Tight Coupling and Synchronous Replication Enables Vertical Scaling with Commodity Flash Memory, Cores
DBT2 open-source OLTP version of TPC-C 1000 warehouses, 32 connections 0 think-time Result metric: TPM (new order)
Measurement Configuration 2 node Master-Slave configuration 2 socket Westmere 72GB DRAM
TransacCon Throughput with Hard Disc Drives
0 10 20
MySQL 5.5 asynch
MySQL 5.5 semi-‐synch
SchoonerSQL
0
20
40
60
80
100
120
MySQL 5.5 asynch MySQL 5.5 semi-‐synch
SchoonerSQL
TransacCon Throughput with Flash Drives
. © 2011 Schooner Information Technology, p16
Response Time (ms)
Tight Coupling and Synchronous Replication Lowers Response Times
0
1
2
3
4
5
6
7
MySQL 5.5 asynch MySQL 5.5 semi-‐synch
SchoonerSQL
Performance : Transac<on Response Time
. © 2011 Schooner Information Technology, p17
Tight Coupling and Synchronous Replication Provides High Performance Stability
SchoonerSQL MySQL 5.5
Asynchronous MySQL 5.5
Semi-synchronous
. © 2011 Schooner Information Technology, p18
Tight Coupling and Synchronous Replication Lowers Total Cost of Ownership
Lower Cost § Reduced capital and operating costs through reduction in servers, power, space, admin § Savings from increased service availability and associated revenue and customer retention
§ TCO and ROI models are customer and workload specific § Function (throughput/server; server, rack, and network costs, software license and support costs, admin costs; space and power costs; cost of downtime)
0
1
2
3
4
MySQL 5.5 SchoonerSQL
Total Cost of Ownership (rela<ve)
. © 2011 Schooner Information Technology, p19
Tight Coupling and Synchronous Replication with Transpaent Automated Failover and Admin GUI Greatly Simplifies Administration
§ Fail-over can be completely automatic and instant § requiring no administrator intervention or service interruption
§ Cluster Administrator GUI and CLI can provide a single point for cluster-wide management
§ single click slave creation and database migration
. © 2011 Schooner Information Technology, p20
Powerful Administration : Monitoring, Trouble-shooting, Tuning, Alerts
. © 2011 Schooner Information Technology, p21
Extending Synchronous Replication with Parallel Asynch and Auto-Failover in WAN Geographic Distribution/DR
• WAN/geographically dispersed data centers – Requires Asynchronous replication
– Can’t add additional ~100ms with high potential variance to query response time for synchronous replication
• Data Integrity Requirement : Remote consistency lag and recovery time should be ~ WAN latency
– Maximize WAN data consistency – Minimize disaster recovery time
• Requires high performance asynchronous replication – Must have multi-threaded asynchronous parallelizing updates
. © 2011 Schooner Information Technology, p22
Extending Synchronous Replication with Parallel Asynch and Auto-Failover in WAN Geographic Distribution/DR • HA Requirement: Must automatically fail-over when synchronous
master fails-over or asynch slave fail-over occurs, requires WAN asynchronous replication tight coupling with synch replication group
. © 2011 Schooner Information Technology, p23
Unlimited Query Scaling
0 50000
100000 150000 200000 250000 300000
1 2 4 8
Requ
ests/s
Nodes in Cluster
SchoonerSQL Throughput (sysbench OLTP)
Reads/s Writes/s
Query Scaling in a Synchronous Replication Group
§ Fully replicated Master/Slave cluster § No cluster overhead for adding queries to a slave § Can add synchronous query read master nodes linearly § Unlimited query scaling extension with parallel asynchronous replication (no data loss, auto-failover, high data integrity)
§ With shared nothing architectures, there is potentially severe cross node query degradation
. © 2011 Schooner Information Technology, p24
Scaling Updates
• Database Update Scalability – Vertically scale with commodity : flash memory, more cores,
higher frequency
• Compelling option exploiting low cost, high performance commodity technology
0 50
100 150
Schooner Synch Cluster: Hard Drives
Schooner Synch Cluster : Flash
. © 2011 Schooner Information Technology, p25
Unlimited Update Scaling : Optimized Transparent Sharding
• Database Update Scalability
…After Optimal Vertical Scaling: Unlimited Query Scaling : Horizontally Scale Through Transparent Workload Aware Database Partitioning and Replication
• Database workload aware
– Administrator analysis and configuration tools – Allows optimal layout, including replicated structures, for query data access
optimization
• Application Transparent – High performance, highly available dynamic query execution across shards
. © 2011 Schooner Information Technology, p26
MySQL 5.5
MySQL NDB Cluster
Clustrix
Linux DRBD
ConCnuent Tungsten
Golden Gate
Schooner SQL
Fail-‐Over Down<me Minutes-‐hours
seconds seconds minutes seconds Minutes-‐hours
seconds
Automated Fail-‐over No Yes Yes No Yes No Yes
Data Loss Yes No yes No Yes Yes No
Data Consistency No Yes Yes No No No Yes
Performance Med Med/Low Med/Low Med Low Low High
Scalability Low Med/Low Med/Low Low Low Low High
Ease of Management
Low High High Low Med Med High
WAN perf and auto-‐ fail-‐over
No No No No No No Yes
InnoDB Compa<ble High Med/Low Med/Low High High High High
Custom Hardware No No Yes No No No No
Cost (TCO) High Med High High High High Low
Heterogeneous Databases
No No No No Yes Yes No
MySQL Database Alternatives for Mission-Critical Deployments
. © 2011 Schooner Information Technology, p27
SchoonerSQL Advantage
• 4-‐20x more throughput/server vs. MySQL 5.5 • High performance synchronous and asynchronous replica<on
Highest Performance and Scalability
• No service interrup<on for planned or unplanned database down<me • Instant automa<c fail-‐over • On-‐line upgrade and migra<on • 90% less down<me vs. MySQL 5.5 • Full WAN support with master auto-‐failover
Highest Availability
Visibility and Control • Easy cluster administra<on • No error-‐prone manual processes • Monitoring and Op<miza<on
• No lost data • Cluster-‐wide data consistency
Highest Data Integrity
• Full MySQL + InnoDB: not a toolkit • Free your staff to build your business, not a custom database
• Cut server capex (consolida<on) • Cut opex (power, pipe, DBA <me) • Increase revenue (eliminate service interrup<ons) • TCO 70% cheaper than MySQL 5.5
Compelling Economics
Out-‐of-‐the-‐box Product
100% MySQL Enterprise InnoDB CompaCble
Broad Industry Deployment • eCommerce, Social Media, Telco,
Financial Services, Education • High volume web sites • Geographically distributed websites
!
. © 2011 Schooner Information Technology, p28
Try SchoonerSQL! We guarantee SchoonerSQL will increase your Availability, Performance, and Scalability Contact us: Schooner Information Technology, Inc. 501 Macara Avenue, Suite 101 Sunnyvale, CA 94085 USA Tel: +1 408-773-7500 www.schoonerinfotech.com Email: [email protected]
Evaluating the Options and Trade-offs for Your Data Center? Let Schooner Help!
. © 2011 Schooner Information Technology, p29
Thank You!