Transcript
Page 1: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Configuration Managementwith Subversion and RPM

www.immobilienscout24.de

Berlin | 19.05.2011 | Schlomo SchapiroSystemarchitekt, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

The Linux Way of Doing Things

Page 2: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 2 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Agenda

www.immobilienscout24.de: Behind the scenes

The Linux Way of Doing Things

RPM Deployment

Configuration Management in Subversion

YADT – an Augmented Deployment Tool

Page 3: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 3 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

www.immobilienscout24.de

1.7 billion page impressions per monthhttp://www.immobilienscout24.de/de/anbieten/kennzahlen.jsp

Linux and standard open source toolchain

RHEL

LAMP server

Tomcat, Spring, Hibernate …

MySQL and Oracle

2 data centres with async WAN, >600 VM, SAN, NAS …

24/7 operations

Agile development, now going toward „Continuous Delivery“

Automate, Automate, Automate

Page 4: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 4 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Page 5: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 5 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

What problem do we solve?

Web

Container

Java VM

Linux OS

Physical/Virtual HW

Virtualization Layer

PHP/Perl

Apache

PHP AppStandalone

MySQL

Syslog SNMP

Postfix

Standalone

Manage entire platform above (virtual) hardware layer

Page 6: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 6 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

The UNIX Way of Thinking

Tomcat

Apache Java VM

AppWeb Tomcat

Web

PHP/Perl

Web

MySQL Syslog Net-SNMP

PostfixSSHStandalone

Linux OS

physical/virtual HW

On the Linux level

If we look very closely

Everything is just a file!

And think UNIX

Programs

Libraries

Software configuration

Operational configuration

Environmental configuration

Even Startup dependencies

And also what to run and when

Reboot is the final test!

Page 7: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 7 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Linux tools we like to use

Software Management:

RPM Packages

Versioning & Transactions

Validation and Rollback

Deploy & run actions: post-install, pre-remove, pre-install …

YUM Repositories

Proven to work with millions of systems

Release Upgrades, Repo Mirrors, Downgrades, Provisioning …

Bash & Python scripts:

/etc/init.d scripts for service management

YADT for multi-server system management

Automate, Automate, Automate :-)

Page 8: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 8 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Configuration = Software

Same tools usable for software and configuration

One tool to rule them all

Software & Configuration

Dependencies between software and configuration

RPM file conflicts enforce separation of concerns→

Doing this with a 100% coverage is probably a new idea!

Page 9: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 9 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Configuration Management in SVN

ConfigSVN

post-commitYUM repos

Page 10: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 10 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Big Picture – Static Structure

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

Page 11: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 11 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Static Structure – Pros and Cons

Variables follow samestructure as config data

Only one tool (SVN) needed

GUI talks only to SVN

Versioning and changetracking for variables

Variables and config files change together (atomic) easy rollback→

Is this simple overlaying structure sufficient?

Simplify the world so that it is good enough!

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

KISS!

Page 12: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 12 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Big Picture – Group Assignment

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creationPost-commit

hookcreates

is24-config-$hostnameRPM

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

config├── all│ └── etc│ └── is24│ └── system.properties├── groups│ ├── tst│ │ └── etc│ │ └── is24│ │ ├── web.properties│ │ └── db.properties│ ├── apache│ │ └── etc│ │ └── is24│ │ └── httpd.conf│ └── webapp│ └── etc│ └── is24│ └── web.properties└── host ├── testserver-263 └── tstweb02

meta├── hosts│ ├── ttestserver-263│ │ ├── SPECIAL_KEY=value│ │ └── memberOf=apache│ └── tstweb02│ ├── memberOf=webapp│ └── memberOf=tst└── groups ├── webapp │ ├── member=tstweb02 │ ├── memberOf=tomcat │ └── RPM_REQUIRES=is24-webapp-env, is24-jcr └── tst ├── member=tstweb02 ├── DB_USER=tuv-db-user └── DB_HOST=dbserver-20

svn ci

Page 13: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 13 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Group Assignment – Pros and Cons

Variables follow samestructure as config data

Only one tool (SVN) needed

GUI talks only to SVN

Versioning and changetracking for variables

Variables and config files change together (atomic) easy rollback→

Free assignment of hosts to groups, nested groups

Groups define function and pull in configuration and software

Final result is combination of all groups (in nesting order)

Not any more so much KEEP IT SIMPLE

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creationPost-commit

hookcreates

is24-config-$hostnameRPM

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

config├── all│ └── etc│ └── is24│ └── system.properties├── groups│ ├── tuv│ │ └── etc│ │ └── is24│ │ ├── web.properties│ │ └── db.properties│ ├── apache│ │ └── etc│ │ └── is24│ │ └── httpd.conf│ └── webapp│ └── etc│ └── is24│ └── web.properties└── host ├── testserver-263 └── tstweb02

meta├── hosts│ ├── srv001│ │ ├── SPECIAL_KEY=value│ │ └── memberOf=apache│ └── testserver-263│ ├── memberOf=webapp│ └── memberOf=tuv└── groups ├── webapp │ ├── member=testserver-263 │ ├── memberOf=tomcat │ └── RPM_REQUIRES=is24-webapp-env, is24-jcr └── tuv ├── member=testserver-263 ├── DB_USER=tuv-db-user └── DB_HOST=dbserver-20

svn ci

Page 14: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 14 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

System Personalisation: RPM Dependencies

is24-tomcat is24-base-env is24-appl-env

is24-tomcat-data

is24-jcr-base is24-sitemapassembler

is24-jtools is24-sitemapassembler-config

is24-contentrules

jdk

is24-mod_jkhttpd

Requiresis24-config-tstweb02

is24-jcr

Page 15: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 15 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

YADT – an Augmented Deployment Tool

Meta Control Layer

Open Source – GNU General Public License

100% Python / Bash

Local file management through RPM

Global service managent through YADT

Model datacenter dependencies:

RPM packages

Servers and services

Between services on different servers

Stateless design – current status is the state, define target states

All actions restartable by following the dependency model

SVN RPM post-commit hook: Maintained as part of YADT→

Page 16: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 16 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Immobilien Scout GmbH

http://bit.ly/is24greatplace

Operate Germanys Leading Real Estate Listing Portal: www.immobilienscout24.de

Linux, XEN, Apache, Tomcat, Java, LAMP, Oracle, MySQL, Nagios, DRBD, RPM

Open Source & DevOps Company

Always seeking to hire new skill:

Admins

System Developers

System Engineers

Architects

Java & PHP Developers

Mobile (iOS & Android) Developers

Page 17: Subversion Day Berlin 2011 Configuration Management With Subversion And Rpm

Slide 17 | Configuration Management with SVN and RPM | Schlomo Schapiro

http://code.google.com/p/yadt/

Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: [email protected]: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.


Top Related