yet another replication tool: rubyrep

/

Yet Another Replication Tool RubyRep

Denish PatelDatabase Architect

1Friday, March 26, 2010

Who am I?

• With OmniTi for more than 3 years

• Manage high traffic database systems

• Replication database system deployments

• Not a core hacker of RubyRep

• “Oh, We are hiring!!”

• Contact : [email protected]


mailto:[email protected]

mailto:[email protected]

Next 30 minutes ..

• Replication

• Various Tools

• Slony? Why another tool ?

• RubyRep

• Install

• Features

• Examples

• Tweaking replication policies


Replication

• Types

• Synchronous

• Semi-synchronous (O My!)

• Asynchronous

• Methods

• Master - Slave

• Master - Master

• Needs

• Horizontal Scaling

• Standby Database

• Backup Database


Tools

Program Type Method Based on

PgCluster -II Synchronous M-M Shared Disk

Slony-I Asynchronous M-S Trigger

Bucardo Asynchronous M-M, M-S Trigger

Londiste Asynchronous M-S Trigger

Mammoth Asynchronous M-S Log

RubyRep Asynchronous M-M, M-S Trigger


Why not Slony?

• Replicated tables MUST need PK or UK

• Doesn’t support large objects

• Doesn’t support synchronizing tables outside of replication

• Limitations of version compatibility

• Difficult to setup

• Doesn’t support Master - Master

• Difficult to monitor and manage


RubyRep Toolset

• Scan

• Sync

• Replicate


RubyRep Key Features

• Easy to install, setup, configure and monitor

• Platform independent

• Table design independent

• Simple PK

• Combined PK

• No PK

• Supports large objects i.e bytea, tsvector

• Database Version Independent

• Database independent

• PostgreSQL

• MySQL


Install

• Standard Ruby Version

• Install Ruby Environment , Install DB drivers , Install RubyRep

• Difficult to install

• Slower performance

• JRuby Version

• Install JVM, Download and Extract gzip file

• Easy to install

• Faster performance


Scan

• Features

• Compare “Left” and “Right” Databases/Tables

• Customize batch size

• Load on client server

• Steps:

• Generate config file

• ./rubyrep generate omniti.conf

• Configure

• Run the Scan

• ./rubyrep scan -c omniti.conf

• Production Results

• Users table with 60M rows and 45 columns

• 3 hours

• Running from monitor server


Scan

• To scan all tables starting with ‘c’:

• rubyrep scan -c omniti.conf /^c/

• Scan with default options will produce output like this:

• clients 100% ......................... 0

• clients_details 100% ......................... 5

• Options

• -s, --summary[=detailed]

• -d, --detailed[=mode]

• full , keys, diff

• -b, --progress-bar[=length]

• -c, --config=CONFIG_FILE

• --help


Config

RR::Initializer::run do |config|config.left = {

:adapter => 'postgresql', # or 'mysql'

:database => 'omniti',

:username => 'dba',

:password => 'securepasswd',

:host => '127.0.0.1',

:schema_search_path => 'omniti’ }

config.right = {

:adapter => 'postgresql',


:username => 'dba',


:host => '127.0.0.1',

:port => '5484'

}

config.include_tables 'users,users_backup'

config.include_tables /^o/ # regexp matches all tables starting with o

end


Sync

• Compare and Sync

• :sync_conflict_handling - :ignore (default), :left_wins, :right_wins

• :sync_record_handling

• :left_record_handling , :right_record_handling

• :ignore, :delete, :insert(default)

• Logging

• :ignored_changes , :all_changes ,

• :ignored_conflicts , :all_conflicts

• Others

• :row_buffer_size , :commit_frequency

• http://www.rubyrep.org/configuration.html


http://www.rubyrep.org/configuration.html

http://www.rubyrep.org/configuration.html

Sync RR::Initializer::run do |config|config.left = {



:username => 'dba',


:host => '127.0.0.1',

:schema_search_path => 'omniti’ }

config.right = {



:username => 'dba',


:host => '127.0.0.1',

:port => '5484' }

# To match all tables except those ending with ‘backup’:

config.include_tables /./

config.exclude_tables /backup$/ #exclude tables ending with backup

end


Replicate

• Replicate Database/Tables

• Global vs Table specific settings

• rubyrep replicate -c omniti.conf

• Setup necessary infrastructure tables

• Setup necessary triggers for the replicated tables

• Initial scan is executed and synched

• Tweak sequences

• Left -Odd , Right -Even for M-M replication

• Default M-M replication setting

• :replicate_record _handling = :replicate

• Replication Conflict handling

• :later_wins, :earlier_wins

• Automatically Sync newly created table(Restart required)

• Uninstall

• rubyrep uninstall -c omniti.conf


ReplicateRR::Initializer::run do |config|config.left = {



:username => 'dba',


:host => '127.0.0.1',

:schema_search_path => 'omniti‘ }

config.right = {



:username => 'dba',


:host => '127.0.0.1',

:port => '5484' }

config.options[:sync_conflict_handling] = :left_wins

config.options[:replication_conflict_handling] = :left_wins

config.add_table_options 'admin',

:sync_conflict_handling => :right_wins,

:replication_conflict_handling => :right_wins

config.include_tables /./

end


Master-Slave Config

config.include_tables /./ # regexp matching all tables in the database

config.options[:auto_key_limit] = 40

config.options[:adjust_sequences] = false

config.options[:sequence_increment] = 1

config.options[:replication_conflict_handling] = :left_wins

config.options[:logged_replication_events] = [

:ignored_changes,

:ignored_conflicts

]

end


Customize as needed ...

• Tables without PK

• config.options[:auto_key_limit] = 2

• config.add_table_options /_/, :auto_key_limit => 2

• config.include_tables 'articles_tags', :key => ['article_id', 'tag_id']

• Checks before and after

• :before_table_sync => "SET foreign_key_checks = 0"

• :after_table_sync => "SET foreign_key_checks = 1"

• Low bandwidth with RubyRep Proxy

• Benefits only Scan and Compare ; only difference is transferred

• :proxy_host => '172.16.1.5', , :proxy_port => '9876'

• Cascading replication

• :rep_prefix to use separate triggers and tables


References

• Installation Notes

• http://rubyforge.org/frs/?group_id=7932

• User Guide

• http://www.rubyrep.org/documentation.html

• API Documentation

• http://rubyrep.rubyforge.org/

• Tutorials

• http://www.rubyrep.org/tutorial.html

• Blogs

• http://denishjpatel.blogspot.com/2009/08/yet-another-postgresql-replication-tool.html


http://rubyforge.org/frs/?group_id=7932

http://rubyforge.org/frs/?group_id=7932

http://www.rubyrep.org/documentation.html

http://www.rubyrep.org/documentation.html

http://www.rubyrep.org/tutorial.html

http://www.rubyrep.org/tutorial.html

Thanks

• Arndt Lehmann

• Thank you!!


Questions ?


yet another replication tool: rubyrep

Education