replicating postgresql databases using slony-i
TRANSCRIPT
![Page 1: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/1.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 1
Asynchronous Replication for PostgreSQL
Slony-I
Slony-I on Microsoft Windows
Dave Page
![Page 2: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/2.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 2
� Asynchronous Replication for PostgreSQL
� Developed by Jan Wieck of Afilias
� Now a community project
Slony-I
![Page 3: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/3.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 3
Uses
� Load balancing
� Redundancy/Failover
� Remote/distributed servers
� Upgrades
![Page 4: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/4.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 4
Architecture
![Page 5: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/5.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 5
Flexible Topology
![Page 6: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/6.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 6
Object types
� Data
� Sequence values
� Schema changes
![Page 7: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/7.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 7
Porting team
� Hiroshi Saito� pgAdmin, psqlODBC, Npgsql, pgInstaller
� Dave Page� pgAdmin, PostgreSQL, psqlODBC, pgInstaller, Npgsql, pgWeb
� Magnus Hagander� PostgreSQL Server, pgInstaller, pgWeb
� Andreas Pflug� pgAdmin, PostgreSQL Server
![Page 8: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/8.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 8
Hiroshi
� Provided the initial ‘quick n dirty’ port.
� Organised the project
![Page 9: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/9.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 9
Dave
� ‘Ducttape’ test suite
� New regression test suite
� Build system/Makefiles
� Patch/CVS management
![Page 10: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/10.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 10
Magnus
� Code port
� slonik – based on Hiroshi’s work
� slon� Reuse pgpipe from PostgreSQL� Service control code� Event logging
![Page 11: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/11.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 11
Andreas
� GUI Management using pgAdmin
![Page 12: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/12.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 12
Asynchronous Replication for PostgreSQL
Slony-I
Porting Slony-I
Magnus Hagander
![Page 13: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/13.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 13
Porting overview
� The bad� Designed for Unix� Relied on Unix tools and architecture
� The good� Portable between Unixes� Based on PostgreSQL build system
![Page 14: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/14.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 14
Porting Slonik - easy
� No shell utils available� Slonik 1.1 uses SED� Hiroshi already fixed
� Path issues� Find the ”share” directory� Windows compatible paths
![Page 15: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/15.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 15
Porting Slon - easy
� Pthreads� Find library to link with
� Winsock� Simple initialization issue
� Pipes � Steal from PostgreSQL
� Signals� Ignore!
![Page 16: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/16.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 16
Porting slon – a bit more work
� Eventlog integration� Centralised logging already� Eventlog when service, stdout when
console� Create message library
� Versioning metadata� Steal most from PostgreSQL� Decimal version number in config.h
![Page 17: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/17.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 17
Porting slon – most work
� fork()� Service integration� Two problems, one solution
![Page 18: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/18.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 18
Slon – Unix architecture
init
rc.slony rc.xyz
slon watchdog
slon engine
slon watchdog
slon engine
fork() fork()
![Page 19: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/19.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 19
CreateProcess() CreateProcess()
slon win32 service handler
initService Control Manager
Slon – Windows architecture
rc.slony rc.xyz
slon watchdog
slon engine
slon watchdog
slon engine
fork() fork()
![Page 20: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/20.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 20
Porting – end result
� Slon runs on the commandline� Only for testing/debugging!
� Single service, multiple engines� One config file per engine� Paths stored in registry, add/remove with
slon commandline
� Multiple services, multiple engines� Different versions of slon
![Page 21: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/21.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 21
Simple slony replication
DEMO
![Page 22: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/22.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 22
Base table creation
db1db1db1db1
CREATE TABLE t (
name text NOT NULL PRIMARY KEY)
INSERT INTO t VALUES (’Dave’)
INSERT INTO t VALUES (’Magnus’)
db2db2db2db2
CREATE TABLE t (
name text NOT NULL PRIMARY KEY)
![Page 23: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/23.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 23
Installing Slony
slon –regservice
slon –addengine c:\slony\db1.conf
slon –addengine c:\slony\db2.conf
slon -listengines
db1.confdb1.confdb1.confdb1.conf
log_level=1
log_timestamp=false
cluster_name='test'
conn_info='host=127.0.0.1 user=postgresdbname=db1'
![Page 24: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/24.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 24
Slon setup script
# Create slony cluster
cluster name = test;
node 1 admin conninfo = ’host=127.0.0.1 user=postgres database=db1’;
node 2 admin conninfo = ’host=127.0.0.1 user=postgres database=db2’;
init cluster (id=1, comment=’Node 1’)
# Create set of tables with one table
create set (id=1, origin=1)
set add table (set id=1, origin=1, id=1, fullyqualified name = ’public.t’)
![Page 25: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/25.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 25
Slon setup script (contd)
# Create node for second enginestore node (id=2, comment=’Node 2’);# Create paths between the two nodesstore path (server=1,client=2,
conninfo=’host=127.0.0.1 user=postgresdbname=db1’);
store path (server=2,client=1,conninfo=’host=127.0.0.1 user=postgresdbname=db2’);
store listen (origin=1, provider=1, receiver=2);store listen (origin=2, provider=2, receiver=1);
# Subscribe the slavesubscribe set (id=1, provider=1, receiver=2,
forward=no)
![Page 26: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/26.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 26
Asynchronous Replication for PostgreSQL
Slony-I
Graphical management of Slony-I
Andreas Pflug
![Page 27: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/27.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 27
pgAdmin III architecture
� C++� wxWidgets 2.6� Native libpq PostgreSQL connection� Native Windows and GTK2 look and feel� For PostgreSQL 7.3 and above� Some helper programs and modules
![Page 28: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/28.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 28
Slony-I installation: modules
� Performed by Windows installer� Performed by make;make install from
source� Use identical Slony-I versions on all
servers!� PostgreSQL servers may have different
versions and run on different operating systems
![Page 29: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/29.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 29
Slony-I installation: Create cluster
� First node in cluster� Node: database with
installed cluster and running slon process
� Uses Slony-I creation scripts
� See pgAdmin's slony path option
![Page 30: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/30.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 30
Slony-I installation: Join cluster
� Create node and copy replication configuration from existing node
� Installs software in current database from existing cluster
![Page 31: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/31.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 31
Slony-I installation: paths
� Path: describes how a slon process connects to other nodes
� Libpq connect string
![Page 32: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/32.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 32
Slony-I installation: listens
� Listen: instructs a node to poll events from other nodes
![Page 33: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/33.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 33
Slony-I cluster status
� See node statistics
![Page 34: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/34.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 34
Slony-I replication sets
� Set: collection of tables and sequences
� All table and sequence data originating on one node
![Page 35: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/35.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 35
Slony-I Tables
� Table needs unique index, PK preferred
� pgAdmin doesn‘t offer tables without unique index
� Select triggers on the table that Slony-I should disable on slave nodes
![Page 36: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/36.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 36
Slony-I subscriptions
� Table and sequences must be present in slave node before subscribing!
� Subscribed sets can‘t be modified; use merge set instead.
![Page 37: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/37.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 37
Slony-I DDL script replication
� Use replication to execute changes to subscribed tables to insure master and slave have identical definitions
� May replicate any DDL script
![Page 38: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/38.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 38
Slony-I switch over
� Gracefully exchange master and slave role to a set between two nodes
� Both nodes must be fully functional� Function "move set"
![Page 39: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/39.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 39
Slony-I fail over
� Master node has failed� Failover tries to restore as much data
from slave nodes as possible� Designate a new master out of the
previous slaves� Not yet supported in pgAdmin III V1.4
![Page 40: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/40.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 40
pgAdmin future
� Coming in V1.6:� Slony-I failover support� Set creation wizard� Health analysis improvements
![Page 41: Replicating PostgreSQL Databases Using Slony-I](https://reader033.vdocuments.net/reader033/viewer/2022052504/554a1557b4c90507558b50b4/html5/thumbnails/41.jpg)
28th October 2005 Dave Page - Magnus Hagander - Andreas Pflug 41
Slony-I and pgAdmin Conclusion
� Most Slony-I functions accessible through easy-to-use GUI
� At-a-glance view on cluster health� Integrated with other administrative tasks
PostgreSQL has integrated replication!