yet another replication tool : rubyrep
DESCRIPTION
Denish PatelOne of the key features any Enterprise considers when choosing a database technology for their architecture solution stack is that of replication. Oracle and MySQL both have built in replication solutions, but as of yet PostgreSQL doesn't support a built in replication solution. There are many replication solutions available however, and different companies are using different solutions customized for their needs.Among all of the solutions, Slony is probably the most widely tested and deployed within organizations, although it does have the following limitations:* Replicated tables must have a unique or primary key* It does not support replication of large objects* Schema changes are not propagated (though they can be coordinated)* It does not support synchronizing databases outside of replication* There are limitations on version compatability; you can not replicate from PostgreSQL 8.2 to PostgreSQL 8.4 for example* It is more difficult to set up than many other replication solutionsOne new alternative to Slony is a project known as RubyRep, which is designed to avoid some of the limitations of Slony. RubyRep has full stack solution from comparing two databases to start replicating them.In the talk, I will explore the RubyRep usage with examples for following actions:- Compare- Sync- ReplicateTRANSCRIPT
/
Yet Another Replication Tool RubyRep
Denish PatelDatabase Architect
1Friday, March 26, 2010
Who am I?
• With OmniTi for more than 3 years
• Manage high traffic database systems
• Replication database system deployments
• Not a core hacker of RubyRep
• “Oh, We are hiring!!”
• Contact : [email protected]
2Friday, March 26, 2010
Next 30 minutes ..
• Replication
• Various Tools
• Slony? Why another tool ?
• RubyRep
• Install
• Features
• Examples
• Tweaking replication policies
3Friday, March 26, 2010
Replication
• Types
• Synchronous
• Semi-synchronous (O My!)
• Asynchronous
• Methods
• Master - Slave
• Master - Master
• Needs
• Horizontal Scaling
• Standby Database
• Backup Database
4Friday, March 26, 2010
Tools
Program Type Method Based on
PgCluster -II Synchronous M-M Shared Disk
Slony-I Asynchronous M-S Trigger
Bucardo Asynchronous M-M, M-S Trigger
Londiste Asynchronous M-S Trigger
Mammoth Asynchronous M-S Log
RubyRep Asynchronous M-M, M-S Trigger
5Friday, March 26, 2010
Why not Slony?
• Replicated tables MUST need PK or UK
• Doesn’t support large objects
• Doesn’t support synchronizing tables outside of replication
• Limitations of version compatibility
• Difficult to setup
• Doesn’t support Master - Master
• Difficult to monitor and manage
6Friday, March 26, 2010
RubyRep Toolset
• Scan
• Sync
• Replicate
7Friday, March 26, 2010
RubyRep Key Features
• Easy to install, setup, configure and monitor
• Platform independent
• Table design independent
• Simple PK
• Combined PK
• No PK
• Supports large objects i.e bytea, tsvector
• Database Version Independent
• Database independent
• PostgreSQL
• MySQL
8Friday, March 26, 2010
Install
• Standard Ruby Version
• Install Ruby Environment , Install DB drivers , Install RubyRep
• Difficult to install
• Slower performance
• JRuby Version
• Install JVM, Download and Extract gzip file
• Easy to install
• Faster performance
9Friday, March 26, 2010
Scan
• Features
• Compare “Left” and “Right” Databases/Tables
• Customize batch size
• Load on client server
• Steps:
• Generate config file
• ./rubyrep generate omniti.conf
• Configure
• Run the Scan
• ./rubyrep scan -c omniti.conf
• Production Results
• Users table with 60M rows and 45 columns
• 3 hours
• Running from monitor server
10Friday, March 26, 2010
Scan
• To scan all tables starting with ‘c’:
• rubyrep scan -c omniti.conf /^c/
• Scan with default options will produce output like this:
• clients 100% ......................... 0
• clients_details 100% ......................... 5
• Options
• -s, --summary[=detailed]
• -d, --detailed[=mode]
• full , keys, diff
• -b, --progress-bar[=length]
• -c, --config=CONFIG_FILE
• --help
11Friday, March 26, 2010
Config
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484'
}
config.include_tables 'users,users_backup'
config.include_tables /^o/ # regexp matches all tables starting with o
end
12Friday, March 26, 2010
Sync
• Compare and Sync
• :sync_conflict_handling - :ignore (default), :left_wins, :right_wins
• :sync_record_handling
• :left_record_handling , :right_record_handling
• :ignore, :delete, :insert(default)
• Logging
• :ignored_changes , :all_changes ,
• :ignored_conflicts , :all_conflicts
• Others
• :row_buffer_size , :commit_frequency
• http://www.rubyrep.org/configuration.html
13Friday, March 26, 2010
Sync RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti’ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
# To match all tables except those ending with ‘backup’:
config.include_tables /./
config.exclude_tables /backup$/ #exclude tables ending with backup
end
14Friday, March 26, 2010
Replicate
• Replicate Database/Tables
• Global vs Table specific settings
• rubyrep replicate -c omniti.conf
• Setup necessary infrastructure tables
• Setup necessary triggers for the replicated tables
• Initial scan is executed and synched
• Tweak sequences
• Left -Odd , Right -Even for M-M replication
• Default M-M replication setting
• :replicate_record _handling = :replicate
• Replication Conflict handling
• :later_wins, :earlier_wins
• Automatically Sync newly created table(Restart required)
• Uninstall
• rubyrep uninstall -c omniti.conf
15Friday, March 26, 2010
ReplicateRR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti‘ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
config.options[:sync_conflict_handling] = :left_wins
config.options[:replication_conflict_handling] = :left_wins
config.add_table_options 'admin',
:sync_conflict_handling => :right_wins,
:replication_conflict_handling => :right_wins
config.include_tables /./
end
16Friday, March 26, 2010
Master-Slave Config
config.include_tables /./ # regexp matching all tables in the database
config.options[:auto_key_limit] = 40
config.options[:adjust_sequences] = false
config.options[:sequence_increment] = 1
config.options[:replication_conflict_handling] = :left_wins
config.options[:logged_replication_events] = [
:ignored_changes,
:ignored_conflicts
]
end
17Friday, March 26, 2010
Customize as needed ...
• Tables without PK
• config.options[:auto_key_limit] = 2
• config.add_table_options /_/, :auto_key_limit => 2
• config.include_tables 'articles_tags', :key => ['article_id', 'tag_id']
• Checks before and after
• :before_table_sync => "SET foreign_key_checks = 0"
• :after_table_sync => "SET foreign_key_checks = 1"
• Low bandwidth with RubyRep Proxy
• Benefits only Scan and Compare ; only difference is transferred
• :proxy_host => '172.16.1.5', , :proxy_port => '9876'
• Cascading replication
• :rep_prefix to use separate triggers and tables
18Friday, March 26, 2010
References
• Installation Notes
• http://rubyforge.org/frs/?group_id=7932
• User Guide
• http://www.rubyrep.org/documentation.html
• API Documentation
• http://rubyrep.rubyforge.org/
• Tutorials
• http://www.rubyrep.org/tutorial.html
• Blogs
• http://denishjpatel.blogspot.com/2009/08/yet-another-postgresql-replication-tool.html
19Friday, March 26, 2010
Thanks
• Arndt Lehmann
• Thank you!!
20Friday, March 26, 2010
Questions ?
21Friday, March 26, 2010