high availability perl dbi + mysql
DESCRIPTION
A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.TRANSCRIPT
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQLYAPC::NA 2005
High AvailabilityDBI & MySQL
Steve Purkis
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
Slide 3
High Availability?
• What is High Availability?• Ever have an application fail because a
mission critical server crashed?
• High Availability is an attempt to provide zero application downtime.
Slide 4
High Availability?
• How do I achieve HA?:• Analyze your apps• Most important components?
• Remove single points of failure (SPOFs):• Add redundant servers• Replicate data (backups, realtime)• Load balancing• Failover when things go pear-shaped
Slide 5
High Availability?
• This talk…• Introduces one approach to building HA Perl
apps that rely on MySQL.
• We’ve used similar techniques with Oracle
• They work for us…• They may not be right for you!
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
Slide 7
Setting Up MySQL
• Some typical architectures…
Slide 8
Setting Up MySQL
• Ye olde basic setup• Not HA.
Application Server
MySQLServer
Slide 9
Setting Up MySQL
• One master, one read-only slave• Good for load balancing• Still not really HA!
Application Server
MySQLMaster
MySQL Slave Data
Replication
reads writes
Load Balancer
Slide 10
Setting Up MySQL
• Two masters• It’s HA, but…• Flaky, tricky to work with!
Application Server
MySQLMasterTwo-way
Replication
MySQLMaster
Load Balancer
Slide 11
MySQL Slave
Setting Up MySQL
• Two masters, multiple slaves• Only one master active at a time• HA
Application Server
MySQL Slaves Data
Replication
reads writes
Primary Master
2nd aryMaster
Master Failover
Load Balancer
• HA (unless your hosting center goes down)
Slide 12
Setting Up MySQL
• Your HA MySQL architecture will depend on:• SLAs• Application requirements• Load• Physical spread of servers• Budget
• Caveats:• Are your apps read or write heavy?• Beware of replication lag time
• Common in high-latency networks• Do your apps need the latest data?
• Transactions?
Slide 13
Setting Up MySQL
• Load balancing• Hardware?• Software?
• Master Failover• On Slaves:
CHANGE MASTER TO …• Manual?• Automate?
• Cron? Application? SLB?
• On App servers:• Connect to Secondary
• Not required if using a hot spare w/same ip
Slide 14
Setting Up MySQL
• Recommended reading:• High Performance MySQL
- Jeremey Zawodny & Derek Balling
• MySQL Replication docs• MySQL Cluster white paper
• For Linux:• Linux Virtual Server Project• Linux-HA
Slide 15
Setting Up MySQL
• If you have any trouble:• MySQL mailing list
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
Slide 17
Our Approach
• Our requirements:• Read-heavy apps• Over 750 clients, many with SLAs• Reliability:
• Read availability most important• Must work if a server farm goes down
• Speed is of essence:• Millions of requests a day
• Don’t need transactions• Minimize costs
Slide 18
Our Approach
Application Server
Application Server
Application Server
MySQL Slave
MySQL Slaves
MySQL Slaves
Application Server
MySQL Master
MySQL2nd ary
Master F
ailover
Data Replication
95% 5%
readswrites
Slide 19
Farm 2
Farm 3Farm 1
Our Approach
Application Server
Application Server
MySQL Slave
Application Server
MySQL Master
MySQL 2nd ary
Master F
ailoverIn-Farm Data Replication
Application Server
MySQL SlaveMySQL Slaves
MySQL SlaveMySQL Slaves
Query caching
Slide 20
Our Approach
• DBI Framework:• Load balancing• Server selection• Read / Write query?
• Failover
Application
DBI Framework
----------------------------------- snip: how to tell reads from writes? ---------------------------------
my ($action, $dbh) = $query =~ /\Aselect|\A\(select|\Ashow|\Adesc/i ? ("read", $dbh_read) : ("write", $dbh_write);
----------------------------------------------------------------------------------------------------------
Slide 21
Our Approach
• Load balancing• On connect• Weighted server selection
• Availability, number of processes, user weights, which farm
• MySQL idle timeout
• Failover• Connect to next slave• Automatic query retry
Application
DBI Framework
MySQL Slave
MySQL Slave
reads
Slide 22
Our Approach
• Failover• Persistent connection to
all masters• Automatic query retry• Automatic fallback
Application Server
DBI Framework
Primary Master
2nd ary Master
Master F
ailover
writes
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
Slide 24
DBI Framework
• Two wrapper functions:• dbconnect( 'read' | 'write' )• Read: Select slave to connect to• Write: Connect to all masters
• $sth = sql( $query )• Read / write dbh selection• Failover & fallback
• Pseudo code?
Slide 25
DBI Framework
• dbconnect( 'read' | 'write' )• Read:
For each slaveweight = 0, next if can’t pingcheck number of processes with mysqladminweight = no. processes / user weighting
Connect to slave with lowest weightSanity check: run a simple queryTry next slave if that failed
• Write:For each master
ConnectSanity check: run a simple query
Set $write_dbh to primary master
Slide 26
DBI Framework
• $sth = sql( $query )• Determine query type…• Read:
Execute query.Failover to next slave on error, and retry query.
• Write:Fallback?
If not using primary, and we failed over X seconds ago, try reconnecting to master.
Execute query.Failover to next master on error, and retry query.
Slide 27
DBI Framework
• Major Drawbacks:• Using DBI-based CPAN modules is hard!
Slide 28
DBI Framework
• Looking Ahead…
• Push HA logic into the DBI layer• Write DBD::MysqlHA ?• DBIx::HA ?• DBD::Multiplex ?• DBIx::DBCluster ?• SQL Relay ?
• MySQL Cluster?
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
Slide 30
DBI Frameworks on CPAN
• If there’s time…
Slide 31
DBI Frameworks on CPAN
• The ones I know a bit about:• DBIx::HA• DBD::Multiplex• DBIx::DBCluster
Slide 32
DBI Frameworks on CPAN
• DBIx::HA• Generic HA solution
(written & tested for Sybase)• Configurable by db name (%DATABASE::conf)• Looks well thought out
Slide 33
DBIx::HA
Pros• Does Failover
• On query failure & dbh disconnected
• Does timeouts:• connect, query execute• Safe signals
• Connect all dsns on init• Supports Apache::DBI
Cons• No read / write distinction• No way to choose which
dbh to use on failover• (want to use a mysql-
specific algorithm)
• Timeouts with SIGALRM• (but how else, really?)
• No ping checks for non-Apache::DBI• (uses $dbh->ping anyways -
we prefer ICMP ping)
Slide 34
DBIx::HA
• Also: written & tested for Sybase!
• Potential DBD::Mysql problems:• auto_reconnect
we may reconnect to a db when we should be failing over
Slide 35
DBI Frameworks on CPAN
• DBD::Multiplex• send requests to multiple dsn's• Configure servers to use in $dsn
(pipe-separated list)
Slide 36
DBD::Multiplex
Pros• Supports master/slave
setup:• differentiates between
reads / writes• Connects to all dsns
(good for fast master failover)
• Does failover:• default behaviour
reads: first_successwrites: first_error
Cons• Can only specify one
master• (though if you specify none,
writes can go to all with 'first_error' exit mode)
• Connects to all dsns• (don't want to connect to all
slaves)
• No customizable slave failover algorithm
• No reconnects• No fallback!
Slide 37
DBI Frameworks on CPAN
• How could we re-use CPAN modules?
• DBIx::HA• Sub-class to introduce MySQL specific functionality?• Introduce a callback for server selection on failover?• Use in conjunction with DBD::Multiplex for read/write dbh
selection?• There could be problems though…
• In general:• Backwards compat• sql() wrapper (for backwards compat)• Custom logging?• Our db wrappers do other things too…
Slide 38
DBI Frameworks on CPAN
• Maybe the best way to reuse them is to nick their ideas?
• DBD::MysqlHA ?• Combination of DBIx::HA and DBD::Multiplex• MySQL specific:
• Customizeable server selection algorithm• Persistent connections