continuent tungsten: proxies on steroids for ha and performance!
DESCRIPTION
TRANSCRIPT
© Continuent4/14/08
Continuent Tungsten:Proxies on Steroids for HAand Performance!
Technology Preview
Robert Hodges, CTO of Continuent
April 15, 2008
2 © Continuent
What We Will Cover
/ Defining the Scale-Out Problem
/ A Hard Look at MySQL Replication for Scale-Out
/ Can We Do Better?
/ Replication Reloaded: Tungsten Scale-Out Stack
/ How to Get Software and Get Started Today
3 © Continuent
A Word from Our Sponsor: Continuent
/ Company• Leading provider of database availability solutions for open source
databases
/ Products• uni/cluster – multi-master database clustering that replicates data across
multiple databases and load balances reads• Works with off-the-shelf database servers• Uses “database virtualization” to provide a seamless client interface
/ Value• Low-cost open source business critical solutions• Highly available data• Raise performance and hardware utilization through load balancing• Remove chance of data loss
4 © Continuent ConfidentialConfidential4 © Continuent
Defining the Scale-Out Problem
5 © Continuent
Requirement: Availability and Performance
/ Data Availability• Bring up alternate database after local failure• Bring up alternate database after site loss• Keep both sites on-line for immediate failover• Maintain/upgrade/migrate without disruption• Restart/restore following data loss or outage due to human error• Ensure commits to multiple databases and/or sites• Always accept transactions no matter what• Stay available in grid environments
/ Performance• Scale reads transparently• Scale writes transparently• Spread transaction load across multiple active sites• Rapid capacity scaling up/down in grid environments
/ Nirvana: increased data availability andperformance scalability by adding commodityhardware
6 © Continuent
The Dream - Flexible Scale-OutS
cala
bil
ity
Availability
Db
App
Db
Db
App
App
7 © Continuent ConfidentialConfidential7 © Continuent
A Hard Look at MySQL Replication
8 © Continuent
MySQL Replication for Scale-Out
/ MySQL database replication has set the high bar forhorizontal scale-out on large websites for many years
/ Replication is “baked-in” to MySQL• Initial set-up takes only a few minutes• Handy SQL commands to configure masters and slaves
/ Flexibly copies all or some databases
/ Offers a choice of statement or row replication• Row replication only in MySQL 5.0 and previous releases
/ Works across WANs, hence can be used to linkmultiple sites
/ In short, it’s a key reason why MySQL is so popular
9 © Continuent
…But Is It Really That Easy?
/ Replication is hard -- no single approach solves allproblems
/ Master is the weak point in the MySQL design• Replacing a crashed master when there are multiple slaves• Sync between binlog and database• Loss of committed transactions
/ Latency due to single threaded update on slaves
/ Administration is harder than it looks• Parameter settings: innodb_flush_log_at_trx_commit=2• Provisioning, resynchronizing, monitoring• Automating failover
/ Read scaling requires application changes
/ Not a write scaling solution
10 © Continuent
A Hard Problem -- Master Election after a Crash
Slave
Master
Slave
Slave
NewNewMasterMaster
WhereWhere was I??was I??
WhereWhere was I??was I??
11 © Continuent
Customer Feedback from Our Clustering Work
/ Use cases for data base clustering driven byHA/scaling/both
/ Analysis of US/Europe sales shows followingpatterns:
• Almost 100% of customers are looking for HA solution• Scalability was a requirement for ~30%
/ What Customers Need• Transparent, easy-to-deploy HA• Protection from data loss/corruption• Higher database performance with fewer application
changes
12 © Continuent
MySQL vs. Customer Requirements
/ MySQL replication is excellent for implementing readscaling on custom applications
• For the do-it-yourself set, it’s the best thing out there
/ BUT--It does not suit everyone:
/ Users who need easy-to-deploy availability
/ Users who cannot afford to lose data
/ Users who cannot afford intrusive applicationchanges
/ Users who are not MySQL experts
13 © Continuent ConfidentialConfidential13 © Continuent
Can We Do Better?New Technologies for Scale-Out
14 © Continuent
Continuent Database Clustering History
/ Continuent implemented m/cluster for MySQL -- thefirst successful commercial cluster based on groupcommunications
/ Our current uni/cluster product implements clustersusing middleware and off-the-shelf databases
/ We have supported MySQL and PostgreSQL for years
/ Strong experience in commercial/open sourcesolutions
• Sequoia -- Multi-master state-machine clustering• Myosotis -- Fast native-to-JDBC proxies• Hedera -- Generic group communications adapters• Bristlecone -- Testing tools for scale-out architectures
15 © Continuent
What Have We Learned?
/ Replication works for building clusters - It’s the onlyapproach that really scales
/ Database virtualization using middleware is essentialfor encapsulating location of copies
/ Group communications are hard to configure buthighly effective for controlling clusters
/ Biggest challenges are simple management andsimple application semantics
/ WAN support is the killer application of databaseclustering
/ There is no one-size-fits-all solution
16 © Continuent
What Are the Up-And-Coming Technologies?
/ CAP-aware databases (Consistent, Available,Partition-Tolerant)
• Now appearing in non-SQL databases near you: AmazonDynamo and SimpleDB
/ Pluggable replication -- Different replicationmechanisms for different problems
• GORDA research project has pioneered approach
/ Partitioning -- Dividing data and automatically routingqueries
• MySQL community is driving solutions
/ Open source databases -- This stuff is going to be bigsome day!
17 © Continuent ConfidentialConfidential17 © Continuent
Replication Reloaded:The Tungsten Scale-Out Stack
18 © Continuent
Our Goal
Highly reliable, transparent, easy todeploy solutions for data availability
and performance scalability that workwith unaltered, stock RDBMS and
economical, off-the-shelf hardware
TungstenTungsten: An architecture for: An architecture forscale-out that meets the goalscale-out that meets the goal
19 © Continuent
Tungsten Overview
/ Data Services integrate multiple copies of data withinand across sites into a single highly available,performant database
/ Database virtualization with controlled data latencyand consistency
/ Database-neutral replication designed for highavailability
/ Cluster-aware management framework backed byexcellent graphical tools
/ “Stack” organization of technology to allowsubstitution and extensions
20 © Continuent
An Open, Database-Neutral Scale-Out Stack
Tungsten ConnectorTungsten Connector Native Wire Protocol Native Wire Protocol ProxyingProxying;; Connection semanticsConnection semantics
SequoiaSequoiaSynchrononous Synchrononous
Multi-Master ReplicationMulti-Master Replication
Tungsten ReplicatorTungsten ReplicatorAsynchronousAsynchronous
Master/SlaveMaster/Slave ReplicationReplication
Tungsten ManagerTungsten ManagerCluster-awareCluster-aware service management for LAN/WANservice management for LAN/WAN
HederaHederaGroup Group Comm Comm AdaptersAdapters
BristleconeBristleconeScale-Out Scale-Out Perf Perf TestingTesting
Group CommunicationsGroup CommunicationsJGroups, JGroups, AppiaAppia, Spread, Spread
DBMS ServersDBMS ServersMySQLMySQL, , PostreSQLPostreSQL, Oracle, Oracle
Application StacksApplication StacksPHP, PHP, PerlPerl, Java, Ruby, etc., Java, Ruby, etc.
21 © Continuent
Why Are Stacks So Important?
/ No single solution can solve all data availability andscaling use cases
/ Stack approach allows you to reuse components ondifferent scale-out problems
/ The stack is an extensible framework
Tungsten Connector (Tungsten Connector (MyosotisMyosotis))
SequoiaSequoiaTungstenTungsten ReplicatorReplicator
Tungsten ManagerTungsten Manager
HederaHederaBristleconeBristlecone
CachingCaching
PartioningPartioning
Certification-Certification-BasedBased
ReplicationReplication
22 © Continuent
Virtualized SQL Connection Semantics
Application StacksApplication StacksPHP, PHP, PerlPerl, Java, Ruby, etc., Java, Ruby, etc.
NodeNode NodeNode NodeNode NodeNode
All UpdatesVisible
Own UpdatesVisible
UpdatesEventually
Visible
Tungsten Data Service
23 © Continuent
High-Availability Replication
Master-master replication
/ No master failover headaches
/ Automatic load balancing
Master-slave replication
/ Wider range of supported SQL
/ Faster performance
/ Works over WAN
NodeNode
NodeNodeNodeNode
NodeNode
NodeNodeNodeNode
SequoiaSequoia Tungsten ReplicatorTungsten Replicator
24 © Continuent
Managing within and across Sites
HostHost
Service Service ManagerManager
ReplicatorReplicatorNodeNode
HostHost
Service Service ManagerManager
ReplicatorReplicatorNodeNode
HostHost
Service Service ManagerManager
ConnectorConnector
ConnectorConnector
HostHost
Service Service ManagerManager
ConnectorConnector
ConnectorConnector
Site 1Site 1
Administrative Administrative ClientClient
JMXJMX
Group CommunicationGroup Communication
Site 2Site 2
HostHostHostHostHostHost
Event BridgeEvent Bridge
25 © Continuent
GUI Tools Demo
26 © Continuent
Now -- Multi-Master Data Services
/ Ensure constant availability and avoid master failurescenarios
/ Focus: Scalable and easily manageable multi-masterreplication
• High quality management tools• Automatic load balancing of read requests• 3+ master nodes• Seamless connections from native clients and Java• Support for latest MySQL versions and table types• Solaris and Linux platforms
27 © Continuent
Java Clients
Now -- uni/cluster 2008
Client ApplicationsLAMP Stack Clients(PHP, Perl, C, Ruby, etc.)
uni/cluster connector
uni/cluster JDBC Driver
DBServer
DBServer
DBServer
DBServer
Clients
uni/cluster Driver
Controller
JDBC Driver
Controller
JDBC Driver
Controller
JDBC Driver
Controller
JDBC Driver
28 © Continuent
Phase 1 - Master/Slave Data Services
/ Constant availability with high transparency andupdate performance
/ Focus: Seamless upgrade from existing standalonedatabase
• One-step software installation• Rapid data service configuration• Lightweight, multi-platform management tools• No application code changes
/ High availability replication• Automatic and manual promotion of slave to master• Bounded latency between slave and master• Transparent proxying with read distribution to idle slaves
29 © Continuent
Phase 1 - Master/Slave Data Service
ConnectorsFailover, Connection Semantics
Client ClientT
un
gs
ten
Ma
ma
ge
me
nt
To
ols
Service Management
Master DBServer
Slave DBServer
ReplicationNode Mgr
Slave DBServer
ReplicationNode Mgr
ReplicationNode Mgr
30 © Continuent
Phase 2 - Cross-Site Data Services
/ Ensure availability and scaling across sites
/ Focus: Cross-site clustering with efficient failover• Replication of data to secondary site databases• Minimal, understandable application changes• Accept transactions across active systems on multiple sites
/ WAN clustering features• Tools and admin model for cross-site management• Handle WAN quality of service (latency, transient failures)• SQL partitioning
/ Emphasis on WAN solutions that work without cornercases or data inconsistencies
31 © Continuent
Phase 2 - Cross-Site Data Services
Data Service Connectors
Read/WriteApplications
Slave DBServer
Master DBServer
Slave DBServer
Connectors
Site 2Site 2SiteSite 11
Read/WriteApplications
Read-OnlyApplications,
Reports
Read-OnlyApplications,
Reports
DNS Load Balancing
32 © Continuent ConfidentialConfidential32 © Continuent
How to Get Software and GetStarted Today
33 © Continuent
Technology Roadmap
/ Uni/cluster 2007.1 is available today!
/ Uni/cluster 2008 available in late Q2• GUI administrative tools• Tungsten management framework• Multiple controllers
/ Tungsten scale-out stack projects will be publishedsoon
/ Tungsten master/slave services will be available earlyQ4
34 © Continuent
Getting Software
/ For access to commercial and open source products,visit us at the following web addresses
www.continuent.com
www.continuent.org
/ Visit us at the Continuent booth in the exhibit hall
/ Watch for more information on Tungsten very soon!
35 © Continuent
For More on Scale-Out Performance…
/ Come hear about scale-out testing using Bristlecone
Portable Scale-OutBenchmarks for MySQL
Wednesday 10:50-11:50am
Ballroom D
36 © Continuent ConfidentialConfidential36 © Continuent
Thank you!
Questions?