batou - multi-(host|component|environment|version|platform) deployment
DESCRIPTION
Batou ist ein Open-Source Werkzeug um mit Python das Deployment von komplexen Anwendungen zu beschreiben und zu automatisieren. Das Konzept besteht darin eine passende allgemeine Modellsprache in Form einer API zu definieren. Dabei stehen Wiederverwendung, Toleranz gegenüber unbekannten Zuständen und Ausdrucksstärke im Vordergrund. Der Vortrag zeigt den Werdegang und unsere Motivation zur Entwicklung von batou auf und zeigt die konzeptionellen Stärken anhand praktischer Beispiele. Wir haben in den letzten Jahren eine Reihe großer und kleiner Projekte beim Deployment betreut. Im Rahmen unserer Plattform "gocept.net" bereiten wir unsere Erfahrungen in Dokumentation und offenen Werkzeugen auf und haben daher batou entwickelt. batou ist sowohl in Python geschrieben und nutzt Python um Deployment-Strukturen zu beschreiben. Wir möchten Batou gerne vorstellen um unsere Erfahrungen weiterzugeben und (auch auf den Sprints) anzuregen sich mit dem Werkzeug auseinanderzusetzen, Feedback zu bekommen, und daran weiterzuarbeiten. Siehe auch: http://bitbucket.org/gocept/batouTRANSCRIPT
BATOUMulti-(Component|Environment|Host|Platform) deployment
Christian Theune <[email protected]>
Wednesday, October 31, 12
Wednesday, October 31, 12
Wednesday, October 31, 12
INHALT
•Motivation
• Hello World
• Tiefer in den Kaninchenbau
• Ausblick
Wednesday, October 31, 12
WARUM?
Wednesday, October 31, 12
Wednesday, October 31, 12
Wednesday, October 31, 12
Wednesday, October 31, 12
Wednesday, October 31, 12
Wednesday, October 31, 12
PARTY LIKE ITS ...
• 1999: PHP FTP-Uploads
• 2001: Shell-Skripte um Zope-Instanzen zu verwalten
• 2003: Zope 2.4 “mkinstance”
• 2006: zc.buildout
• 2009: Fabric
Wednesday, October 31, 12
BATOUVS
Puppet, Chef Fabric, Salt JuJu
Systemkonfiguration versus Service deployment
Modellierung Laufzeitumgebung
Wednesday, October 31, 12
HELLO WORLD
http://www.flickr.com/photos/oskay/472097903
Wednesday, October 31, 12
./.hg
./buildout.cfg
./bootstrap.py
./components
./components/hello
./components/hello/component.py
./environments
./environments/dev.cfg
./environments/prod.cfg
$ find .Wednesday, October 31, 12
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', content='Hello world')
components/hello/component.pyWednesday, October 31, 12
[hosts]localhost = hello
environments/dev.cfgWednesday, October 31, 12
$ bin/batou-local dev localhostUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)
Wednesday, October 31, 12
./
./components
./environments
...
./work
./work/hello
./work/hello/hello
$ find .Wednesday, October 31, 12
$ bin/batou-local dev localhost
Wednesday, October 31, 12
$ cat "foo" > work/hello/hello$ bin/batou-local dev localhostUpdating Hello > File(hello) > Content(hello)
Wednesday, October 31, 12
[environment]service_user = testhost_domain = gocept.net
[hosts]test01 = hellotest02 = hello
environments/prod.cfgWednesday, October 31, 12
$ bin/batou-remote prodtest02.gocept.net: connectingtest01.gocept.net: connectingtest01.gocept.net: bootstrappingtest02.gocept.net: bootstrappingOKOKDeploying test01.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OKDeploying test02.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OK
Wednesday, October 31, 12
$ bin/batou-remote prodtest02.gocept.net: connectingtest01.gocept.net: connectingtest01.gocept.net: bootstrappingtest02.gocept.net: bootstrappingOKOKDeploying test01.gocept.net/helloOKDeploying test02.gocept.net/helloOK
Wednesday, October 31, 12
HELLO WORLD
• Komponenten und Environments
• batou-local im Konfiguration eines Hosts lokal zu realisieren
• batou-remote um eine ganze Umgebung zu deployen
• Konvergenz und Idempotenz
Wednesday, October 31, 12
IN DEN KANINCHENBAU
• ein Programm
• verwaltet mit Supervisor
Wednesday, October 31, 12
DAS PROGRAMM
#!/usr/bin/env pythonimport timewhile True: print 'Hello, world!' time.sleep(1)
Wednesday, October 31, 12
KOMPONENTE
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', source='hello.py', mode=0o755)
components/hello/component.pyWednesday, October 31, 12
[hosts]localhost = hello
environments/dev.cfg
ENVIRONMENT
Wednesday, October 31, 12
INSTALLATION
$ bin/batou-local dev localhostUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)Updating Hello > File(hello) > Mode(hello)
Wednesday, October 31, 12
LÄUFT?
$ ./work/hello/helloHello, world!Hello, world!^C
Wednesday, October 31, 12
Wednesday, October 31, 12
BUILDOUT
[buildout]parts = supervisor
[supervisor]recipe = collective.recipe.supervisor
components/supervisor/buildout.cfgWednesday, October 31, 12
BUILDOUT
from batou.component import Componentfrom batou.lib.buildout import Buildout
class Supervisor(Component):
def configure(self): self += Buildout('supervisor', python='2.7')
components/supervisor/component.pyWednesday, October 31, 12
[buildout]parts = supervisor
[supervisor]recipe = collective.recipe.supervisorport = {{component.address.listen}}serverurl = http://{{component.address.listen}}
SERVER-ADRESSE
components/supervisor/buildout.cfgWednesday, October 31, 12
SERVER-ADRESSE
components/supervisor/component.py
from batou.component import Componentfrom batou.lib.buildout import Buildoutfrom batou.utils import Address
class Supervisor(Component): port = 9000 def configure(self): self.address = Address( self.host.fqdn, self.port) self += Buildout('supervisor', python='2.7')
Wednesday, October 31, 12
NETZWERKADRESSEN
a = Address('localhost:631')a = Address('localhost', '631')
str(a.connect) == 'localhost:631'a.connect.host == 'localhost'a.connect.port == '631'
str(a.listen) == '127.0.0.1:631'a.listen.host == '127.0.0.1'a.listen.port == '631'
Wednesday, October 31, 12
SERVER-ADRESSEfrom batou.component import Componentfrom batou.lib.buildout import Buildoutfrom batou.utils import Address
class Supervisor(Component): port = 9000 def configure(self): self.address = Address( self.host.fqdn, self.port) self += Buildout('supervisor', python='2.7')
components/supervisor/component.pyWednesday, October 31, 12
[hosts]localhost = program, supervisor
[component:supervisor]port = 6000
environments/dev.cfg
SERVER-ADRESSE
Wednesday, October 31, 12
WELCHE PROGRAMME STARTEN?
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', source='hello.py', mode=0o755) self.provide('program', '{}/hello'.format(self.workdir))
components/hello/component.pyWednesday, October 31, 12
w
WELCHE PROGRAMME STARTEN?
class Supervisor(Component):
def configure(self): self.programs = self.require( 'program', host=self.host) self += Buildout('supervisor', python='2.7')
components/supervisor/component.pyWednesday, October 31, 12
[buildout]parts = supervisor[supervisor]recipe = collective.recipe.supervisorport = {{component.address.listen}}serverurl = ...programs ={%- for program in component.programs %} {{loop.index0}} prog{{loop.index}} {{program}} true{% endfor -%}
WELCHE PROGRAMME STARTEN?
components/supervisor/buildout.cfgWednesday, October 31, 12
RESSOURCEN UND ABHÄNGIGKEITEN
•provide(key, value)
•require(key, host=None) => [x, y, z]
•require_one(key, host=None) => x
Wednesday, October 31, 12
RESSOURCEN UND ABHÄNGIGKEITEN
• provide ohne require
• require ohne provide
• require_one mit != 1 Element
• Flapping durch Sortieren vermeiden.
•Dependencies werden automatisch gefunden
Wednesday, October 31, 12
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', source='hello.py', mode=0o755) self += Program('hello', command='hello')
HOOK KOMPONENTEN
components/hello/component.pyWednesday, October 31, 12
class Program(HookComponent): namevar = 'name' key = 'supervisor:Program'
command = None program_template = '{name} {command} true'
def configure(self): super(Program, self).configure() self.command = os.path.normpath( os.path.join(self.workdir, self.command))
def format(self, supervisor): return self.program_template.format( name=self.name, command=self.command)
Wednesday, October 31, 12
class Program(HookComponent): key = 'supervisor:Program' ...
class Supervisor(Component):
def configure(self): self.programs = self.require( Program.key, host=self.host) self += Buildout('supervisor', python='2.7')
HOOK COMPONENTS
components/supervisor/component.py
Kann eigentlich nicht hier leben - muss importierbar sein!
Wednesday, October 31, 12
[buildout]parts = supervisor[supervisor]recipe = collective.recipe.supervisorport = {{component.address.listen}}serverurl = ...programs ={%- for program in component.programs %} {{loop.index0}} {{program.format}}{% endfor -%}
WELCHE PROGRAMME STARTEN
components/supervisor/buildout.cfgWednesday, October 31, 12
class Supervisor(Component): def configure(self): ... def verify(self): self.assert_file_is_current( 'var/supervisord.pid', ['.batou.buildout.success']) self.assert_no_subcomponent_changes()
def update(self): out, err = self.cmd('bin/supervisorctl pid') try: int(out) except ValueError: self.cmd('bin/supervisord') else: self.cmd('bin/supervisorctl reload')
SUPERVISOR STARTEN
Wednesday, October 31, 12
@REBOOT
Nur ein "bag".class Service(Component): namevar = 'executable' pidfile = None
class Supervisor(Component): def configure(self): ... self += Service('bin/supervisord', pidfile='var/supervisord.pid')
Wednesday, October 31, 12
PLATTFORMEN@platform('flyingcircus.io', Service)class UserInit(Component):
def configure(self): target = '/var/spool/init.d/{}/{}'.format( self.environment.service_user, basename(self.parent.executable)) self += File(target, source='init.sh', is_template=True, mode=0o755, leading=True)
Wednesday, October 31, 12
[environment]host_domain = flyingcircus.ioplatform = flyingcircus.io
$ bin/batou-local --platform= \ production test01 # keine Plattform!
PLATTFORMEN
Wednesday, October 31, 12
REVIEW• File taucht häufig auf
• Address
• Jinja2-Templates
• Environment overrides
• Ressourcen + HookComponents
• verify/update-Cycle
• Plattform-Spezifika
Wednesday, October 31, 12
TIPPS
• Komponenten unabhängig von Host
• Gemeinsamkeiten in Komponenten bündeln
• verify/update so kurz wie nur möglich
• je abstrakter eine Komponente desto unwarscheinlicher trifft man verify/update
Wednesday, October 31, 12
VORDEFINIERTE KOMPONENTEN
File VirtualEnv Buildout zip
Secrets Supervisor Download CMMI
git svn tar cron
Wednesday, October 31, 12
FAZIT
•modell-getrieben aber pragmatisch
• Konvergenz und Idempotenz
•Der Teufel liegt im Detail. Immer wieder.
Wednesday, October 31, 12
AUSSICHT
•Dokumentation
• Bessere Fehlermeldungen
• Besserer Status-Output
• Alternativen zu SSH/HG
Wednesday, October 31, 12
FRAGEN?
• http://bitbucket.org/gocept/batou
• http://packages.python.org/batou
• Bitte gebt mir Bierdeckelfeedback!
• http://mobro.co/theuni
Wednesday, October 31, 12