osdc 2014: jochen lillich - dynamic infrastructure orchestration
DESCRIPTION
Getting Configuration Management in place is a big step in the direction of infrastructure automation. Chef, Puppet and Co. replace error-prone manual changes with periodic system convergence runs controlled by a central database. Even with Puppet’s exported resources and Chef’s search capabilities, the weakness of this approach is that it is rather static. In situations where we need to propagate information quickly, handle failure detection, or tolerate network partitions, other tools might offer better solutions. In this talk, I’m going to present some of these alternatives (e.g. serf, etcd) and how they can be used to allow for more dynamic configuration changes.TRANSCRIPT
![Page 1: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/1.jpg)
freistil.it
Dynamic Infrastructure Orchestration
Jochen Lillich
![Page 2: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/2.jpg)
Jochen Lillich
@geewiz
Sysadmin since 1993
CEO of freistil IT
freistilbox: PaaS for Drupal & WordPress
![Page 3: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/3.jpg)
Config Management
Elimination of manual changes
Consistent configuration
Single source of truth
DRY
![Page 4: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/4.jpg)
Weaknesses
Periodic convergence runs
Central database
![Page 5: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/5.jpg)
Orchestration
Perform changes quickly
Handle failure
Tolerate network partitions
![Page 6: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/6.jpg)
serf
![Page 7: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/7.jpg)
Basics
Cluster communication tool
Developed by Hashicorp
Simple binary, written in Go
Gossip protocol
![Page 8: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/8.jpg)
serf node 1$ serf agent -node=node1 -bind=srv1
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'srv1'
Bind addr: '192.168.0.11:7946'
…
==> Log data will now stream in as it occurs:
2014/04/09 16:05:51 [INFO] agent: Serf agent starting
2014/04/09 16:05:51 [INFO] serf: EventMemberJoin: node1 192.168.0.11
2014/04/09 16:05:52 [INFO] agent: Received event: member-join
![Page 9: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/9.jpg)
serf node 2
$ serf agent -node=node2 -bind=srv2
…
$ serf join srv1
Successfully joined cluster by contacting 1 nodes.
$ serf members
node2 192.168.0.12:7946 alive
node1 192.168.0.11:7946 alive
![Page 10: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/10.jpg)
Membership
Cluster management
Service configuration
![Page 11: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/11.jpg)
Failure detection
Detects failure quickly
Notifies other nodes
Executes handler scripts
![Page 12: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/12.jpg)
Eventsmember-join
member-leave
member-failed
member-update
member-reap
user
query
![Page 13: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/13.jpg)
User events
Deploy application
Perform convergence run
Query information from the cluster
![Page 14: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/14.jpg)
Example handler
#!/bin/bash
!
echo
echo "New event: ${SERF_EVENT}. Data follows..."
while read line; do
printf "${line}\n"
done
![Page 15: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/15.jpg)
User event
$ serf event deploy-app
!
2014/04/09 17:06:32 [INFO] agent: Received event: user-event: deploy-app
![Page 16: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/16.jpg)
etcd
![Page 17: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/17.jpg)
Basics
HA key-value store
Developed by CoreOS
Written in Go
Raft consensus algorithm
Pure HTTP API
![Page 18: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/18.jpg)
Key-value store
/release = 9607bcfeb48905d26db8f
/cluster
/cluster/node1 = node1.example.com
/cluster/node2 = node2.example.com
![Page 19: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/19.jpg)
Quorum
Available
![Page 20: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/20.jpg)
Quorum
Available
![Page 21: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/21.jpg)
Quorum
Available
![Page 22: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/22.jpg)
Quorum
Unavailable
![Page 23: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/23.jpg)
Writing$ etcdctl set /message Hello
Hello
$ curl -L -X PUT \ http://127.0.0.1:4001/v2/keys/message \ -d value=“Test"
{"action":"set","node":{"key":"/message","value":"Test","modifiedIndex":15,"createdIndex":15},"prevNode":{"key":"/message","value":"Hello","modifiedIndex":2,"createdIndex":2}}
![Page 24: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/24.jpg)
Reading
$ etcdctl get /message
Hello
$ curl -L \ http://127.0.0.1:4001/v2/keys/message
{"action":"get","node":{"key":"/message","value":"Test","modifiedIndex":15,"createdIndex":15}
![Page 25: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/25.jpg)
Deleting
$ etcdctl rm /message
$ curl -L -X DELETE \ http://127.0.0.1:4001/v2/keys/message
![Page 26: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/26.jpg)
Waiting
$ curl -L http://127.0.0.1:4001/v2/keys/release\?wait\=true
![Page 27: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/27.jpg)
Waiting
$ curl -L http://127.0.0.1:4001/v2/keys/release\?wait\=true
{"action":"set","node":{"key":"/release","value":"9607bcfe","modifiedIndex":16,"createdIndex":16},"prevNode":{"key":"/release","value":"18512199","modifiedIndex":14,"createdIndex":14}}
![Page 28: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/28.jpg)
Applications
Locking
Leader election
Database master
Elastic IP
![Page 29: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/29.jpg)
Leader election
etcd
App
![Page 30: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/30.jpg)
Leader election
etcd
AppDB?
![Page 31: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/31.jpg)
Leader election
etcd
AppDB?
Nope.
![Page 32: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/32.jpg)
Leader election
etcd
AppDB?
Nope.
I’ll wait.
![Page 33: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/33.jpg)
Leader election
etcd
App
![Page 34: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/34.jpg)
Leader election
etcd
App DB X
![Page 35: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/35.jpg)
Leader election
etcd
App DB XMaster me!
![Page 36: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/36.jpg)
Leader election
etcd
App DB XMaster me!
Ok!
![Page 37: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/37.jpg)
Leader election
etcd
App DB X
DB X is master.
![Page 38: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/38.jpg)
Leader election
etcd
App DB X
DB X is master.
![Page 39: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/39.jpg)
Leader election
etcd
App DB X
DB X is master.
![Page 40: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/40.jpg)
Leader election
etcd
App DB X
![Page 41: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/41.jpg)
Leader election
etcd
App DB X
DB Y
![Page 42: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/42.jpg)
Leader election
etcd
App DB X
DB Y
Master me!
![Page 43: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/43.jpg)
Leader election
etcd
App DB X
DB Y
Master me!
No, DB X is.
![Page 44: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/44.jpg)
Leader election
etcd
App DB X
DB Y
![Page 45: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/45.jpg)
Leader election
etcd
App
DB Y
![Page 46: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/46.jpg)
Leader election
etcd
App
DB Y
![Page 47: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/47.jpg)
Leader election
etcd
App
DB Y
![Page 48: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/48.jpg)
Leader election
etcd
App
DB Y
You da master!
![Page 49: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/49.jpg)
Leader election
etcd
App
DB YDB Y is master now.
![Page 50: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/50.jpg)
Leader election
etcd
App
DB Y
![Page 51: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration](https://reader034.vdocuments.net/reader034/viewer/2022051314/554a080fb4c905507a8b565d/html5/thumbnails/51.jpg)
Summary