Beyond static configuration managementAn overview of current developments !Dr. Stefan Schimanski

About meOpenSource developer since mid of 90ies

Freelance Developer & Consultant

1st DevOpsDays

Ex-head of Linux Competence Center @ German Air Traffic Controlpushing Puppet, DevOps, CI, OpenStack & agile

Before spent 5 years @ LMU Munich to come up with essentially this:

!

!

@the1stein, github.com/sttts, sts@1stein.org, schimmi@freenode

Long long time ago….

Long long time ago, in 2011:

The DevOps world was simple. !

Essentially, choose between: !

Cfengine/Puppet/Chef Ubuntu/Debian/Red Hat/Suse

implement monitoring, metrics, orchestration, log-management and deploy apps, preferably from

some CI/CD pipeline, usually on bare metal or IaaS

– In some OpenSpace @ DevOpsDays Berlin May 2013

„Who thinks that Docker changes everything?“

16 months later

Claim: DevOps tool chains are being disrupted

Claim: System engineering is being disrupted

Claim: Linux/Unix userland is being disrupted

Exciting times

Development Linux Distributions change,

become very special-purpose oriented. !A container doesn’t need - multi-user/tenant support - an init system - initrd, kernel, boot loader, installer,

hardware detection - application separation - a strict file system hierarchy standard - a lot of daemons.

Development Linux Distributions change,

become very special-purpose oriented. !A host for containers doesn’t need - multi-user/tenant support - a lot of user customizations - an extensive packaging system - an in-place&time update system - a strict file system hierarchy standard - a lot of traditional daemons - a desktop environment.

CoreOS an OS for cluster nodes running containers

based on systemd

fault-tolerant distributed key-value store: etcd

cluster layer above systemd: fleet and fleedctl

clear separation of state and OS

update = reboot

under heavy development

Container Orchestration

figFast, isolated development environments using Docker

like a vagrant for docker

single-host

fig.yml describing multiple containers, with links, environment variables, etc.

maestro-ng Container orchestration for Docker environments

multi-host supportwith static host assignment

powerful container dependencies and linking features

service discovery via env variables and integration with zookeeper

fleet in CoreOS

Normal systemd primitives for dependencies

libswarm by Docker A minimalist toolkit to compose network services

libswarm, libchan

Development static to dynamic system configuration

zookeeperserf

consul

etcd

SkyDock

ZookeeperDemo

etcdInitialization • app1: etcd -peer-addr 192.168.0.1:7001 -addr 192.168.0.1:4001 -data-dir /tmp/etcd -name app1 -bind-addr 0.0.0.0 • app2: etcd -peer-addr 192.168.0.2:7001 -addr 192.168.0.2:4001 -data-dir /tmp/etcd -name app2 -bind-addr 0.0.0.0

-peers 192.168.0.1:7001, 192.168.0.3:7001 • app3: etcd -peer-addr 192.168.0.3:7001 -addr 192.168.0.3:4001 -data-dir /tmp/etcd -name app3 -bind-addr 0.0.0.0

-pears 192.168.0.1:7001, 192.168.0.2:7001 !Distributed Key/Value store • etcdctl set /message Hello • etcdctl get /message

Hello • curl -L http://127.0.0.1:4001/v2/keys/message

{"action":"get","node":{"key":"/message","value":"Hello","modifiedIndex":4,"createdIndex":4}} !Watches • app1: etcdctl watch /foo-service —recursive • app2: etcdctl set /foo-service/container2 localhost:2222

app1: localhost:2222 !TTLs • etcdctl set /foo "Expiring Soon" --ttl 20 # gone after 20 seconds

confd

/etc/confd/conf.d/myapp-nginx.toml /etc/confd/templates/nginx.tmpl

Key/Values in etcd (or consul):

confd -verbose -onetime -node 'http://127.0.0.1:4001' -confdir /etc/confd/conf.d

Serfserf agent & serf join app2:7946 app3:7946 serf event user:deploy !

!

!

!

!

!

serf members -tag role=load-balancer app1.devops-meetup.com 10.131.251.53:7946 alive role=db

deploy.sh

Consulapp1:# consul agent -server -data-dir /tmp/consul -bootstrap-expect 1

app2:# consul agent -server -data-dir /tmp/consul

app3:# consul agent -server -data-dir /tmp/consul

consul join app2 app3

consul members

Consul – service discoveryapp1: /root/consul.d/web.json

consul agent -server -data-dir /tmp/consul -config-dir /root/consul.d

dig @127.0.0.1 -p 8600 web.service.consul

apt-get install dnsmasq echo server=/consul/127.0.0.1#8600 >> /etc/dnsmasq reload dnsmasqdocker run —dns 172.17.42.1 ubuntu ping -c 1 web.service.consul

Consul – dynamic service announcement

Early integration of Mesos/Marathon and Consul – not todayhttps://github.com/keenlabs/marathon/commit/290036e34337dcd6483550b7ab7d723bc4378d5f

curl -i localhost:8500/v1/agent/services

!

curl -i -X PUT localhost:8500/v1/agent/service/register -d ‚{"ID":"web2","Name":"web2","Service":"web","Tags":["prod"],"Port":31774}'

curl -i -X DELETE localhost:8500/v1/agent/service/deregister/web2

Consul – service announcement from dockerconsul agent -server -data-dir /tmp/consul -config-dir /root/consul.d -client 0.0.0.0 # so docker can access consul API

Dockerfilewith run_consul_service.sh code:

Consul – discovery in action

app1: docker run -e CONSUL_SERVICE='{"Name":"web2","Service":"web","Tags":["prod"],"Port":1080}' -e RUN="nodejs /webserver.js" -p 1080:80 -i -t consul-ubuntu

app3: curl web2.service.consul:1080

!

!

What about dynamic ports assigned by Docker?

Vulcand – web proxy with native etcd supportChanges to configuration take effect immediately without restarting the service

An operation system for a computer - manages hardware resources - offers services to programs - isolates running programs

for a cluster of machines

Development

Mesos a cluster meta operation system

„Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.“

Fault-tolerant replicated master using ZooKeeper

Scalability to 10,000s of nodes

Isolation between tasks with Linux Containers

Quite OS agnostic

developed since 2010, OpenSource

Marathon = init daemon on Mesos

Chronos = cron daemon on Mesos

Deimos = Docker integration

– Abdur Chowdhury Former Chief Scientist, Twitter Inc. Claim:

Mesosphere brings Google-scale compute to everybody.

Scaling

Fault Tolerance

Mesos/MarathonDemo

Google Kubernetes also a cluster operation system„Kubernetes is a system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.“

released into OpenSource in June

very early stage

IBM, Mesosphere, Red Hat promise to support it

Mesos promise support as Framework ontop of Mesos

„Main contribution: abstractions for system engineering which are proven within Google“

runs on CoreOS with etcd

runs Docker containers

Development App operation will split into - platform operation - app operation

Platform will be more&more off-the-shelf or PaaS

Deis – Heroku-style PaaS on top of CoreOS, prototype on Mesos

• Still in (quite) early development • depends a lot on CoreOS,

e.g. on Fleet’s non-existing scheduling • similar project: Flynn

Where are !- classical configuration management, - IaaS solutions like OpenStack, - monolithic PaaS like OpenShift and CloudFoundry, - classical Linux distributions?

Exciting times