deploying dockerized applications with salt

Hello.Nice to meet you!Deploying Dockerized apps with Salt

1

Who am I?

Senior Developer @ SOON_

I'm @krak3n on GitHub, Twitter etc

Hi, I’m Chris

2

What is SOON_?

We make brands successful by combining insightful strategy, seamless technology, persuasive design and relevant content

We can help you be more digital** we don’t like the word “digital”, but it’s the word people use. Ask us and we’ll tell you why.

3

What is Do-It?

Do-It is the UK’s biggest volunteering network

Listing volunteering opportunities from thousands of charities and social action groups throughout the UK

Many of the opportunities on Do-it come from physical Volunteer Centres around the UK

4

The approach

Complete re-platform of the Java application. Start from scratch

Use modern technologies and frameworks

Separate Backend and Frontend builds for an API driven application

5

Docker 101

Wrapper around LXC* - method of running multiple isolated Linux systems

Run single processes

Ship your application with your OS

docker run --rm -it ubuntu:14.04 /bin/bashroot@7623d871ac08:/#

Easy to build and run applications anywhere (that supports docker)

* Linux Containers - Released 2006

Version control for LXC

$ docker run ubuntu:14.04 apt-get update$ docker ps -lCONTAINER ID IMAGE 9a0fa929a3c7 ubuntu:14.04 $ docker commit 9a0fa929a3c7 ubuntu:14.04

$ docker run apt-get install -y curl$ docker ps -lCONTAINER ID IMAGE97957263ea5c ubuntu:14.04$ docker commit 97957263ea5c ubuntu:14.04

$ docker run ubuntu:14.04 curl http://google.com

Build Images with Dockerfile$ cat ./DockerfileFROM ubuntu:14.04RUN apt-get update && apt-get install -y curl

$ docker run --rm ubuntu_with_curl curl http://google.com

$ docker build -t ubuntu_with_curl .

Push images to a central repository

$ docker login -e you@somewhere.com -u you -p 123$ docker push ubuntu_with_curl

$ docker login -e you@somewhere.com -u you -p 123 http://your.index.com$ docker push your.index.com/ubuntu_with_curl

Pull Images from a central repository.

$ docker pull ubuntu_with_curl$ docker run --rm ubuntu_with_curl curl http://google.com

Pass environment variables to containers$ docker run --rm -it -e FOO=foo ubuntu:14.04 bashroot@2281fb4f13a4:/# echo $FOOfoo

Run containers in detached mode

$ docker run -d foo ubuntu:14.04 foo

You can name containers

$ docker run -d --name foo ubuntu:14.04 some_process

Then you can stop, start and restart your contains like processes

$ docker stop foo$ docker start foo$ docker restart foo

View the logs of your containers

$ docker logs --follow foo

Or attach* to running containers$ docker attach foo$ docker start -a foo* does not allow you to execute commands inside the container

Execute commands in running containers - useful for debugging$ docker run -d --name foo ubuntu:14.04 foo$ docker exec -it foo /bin/bashroot@4cc94bc02b5f:/# ps aux

Remove containers

$ docker rm foo$ docker rm -f foo

Also remove images$ docker rmi ubuntu:14.04$ docker rmi -f ubuntu:14.04

Containers can be linked with each other$ docker run -d -e RABBITMQ_USER=chris -e RABBITMQ_PASS=chris --name rabbitmq tutum/rabbitmq

$ docker run --rm -it --link rabbitmq:rabbitmq ubuntu:14.04 /bin/bash

$ root@507079e8b54a:/# echo $RABBITMQ_PORTtcp://172.17.0.19:5672

Sharing data with volumes

$ docker run --rm -it \ -v /host/path:/container/path/ foo ubuntu:14.04 \ /bin/bash

$ docker run --rm -it \ -v /host/path/file.x:/container/path/file.x foo \ ubuntu:14.04 /bin/bash

e.g your PostgreSQL data directory

$ docker run -v /data/psql:/var/lib/postgresql \ --name db orchardup/docker-postgresql

$ docker run --privileged --rm -it \ foo ubuntu:14.04 \ /bin/bash

6

A simple application(we will get to Salt SOON_)

Let's look at a simple Flask application we'll deploy later with Salt and Docker

/demo├── Dockerfile├── setup.py└── foo ├── __init__.py └── app.py

#!/usr/bin/env python# encoding: utf-8

from flask import Flaskapp = Flask(__name__)

@app.route('/')def hello_world(): return 'Hello World!'

if __name__ == '__main__': app.run(host='0.0.0.0')

Tree app.py

And the DockerfileFROM ubuntu:14.04RUN apt-get update && apt-get install -y python python-dev \ && apt-get clean \ && apt-get autoclean \ && apt-get autoremove -y \ && rm -rf /var/lib/{apt,dpkg,cache,log}/ADD . /fooWORKDIR /fooRUN python setup.py installEXPOSE 5000CMD ["python", "/foo/foo/app.py"]

Build it, Push it, Run It!

$ docker build -t soon/foo .$ docker push soon/foo$ docker run --rm -p 5000:5000 foo

7

Salt ♥ Docker

Before Salt can use docker we need to install docker and docker-Py

docker-ppa: pkgrepo.managed: - name: deb https://get.docker.io/ubuntu docker main - keyserver: hkp://keyserver.ubuntu.com:80 - keyid: 36A1D7869245C8950F966E92D8576A8BA88D21E9 - require: - pkg: software-properties-common - pkg: apt-transport-https

lxc-docker: pkg: - installed service.running: - name: docker - sig: /usr/bin/docker - require: - pkg: lxc-docker

docker-py: pip.installed: - reload_modules: True - require: - pkg: python-pip

We have already pushed soon/foo to Docker Hub*, using Salt, we can pull down that image

soon/foo: docker.pulled: - tag: latest - force: True - require: - pip: docker-py - service: lxc-docker

* this is a public repo - anyone can download it: docker pull soon/foo

docker-py must be installed

force: True - Always pull the latest image

Now we need to create the containerfoo-container: docker.installed: - name: foo - image: soon/foo - require: - docker: soon/foo

Then we can run the new containerfoo: docker.running: - container: foo - port_bindings: "5000/tcp": HostIp: "" HostPort: "5000" - require: - docker: foo-container

Simples! But thats not enough...

We also need to be able to stop and remove containers when the image has changed.

We can watch for changes to the image

foo-absent: cmd.wait: - name: docker rm -f foo - watch: - docker: soon/foo

We can’t use docker.absent

Like the private IP address of the container

$ salt-call docker.get_containers$ salt-call docker.get_containers inspect=True$ salt-call docker.inspect_container foo

Getting information about our running containers

"NetworkSettings": { "IPAddress": "10.1.0.13", "Gateway": "10.1.42.1", "Ports": { "5000/tcp": null }}

Now we can set up an Nginx* config to proxy directly to the container* assumes you have Nginx installed via Salt

foo-nginx-config: file.managed: - name: /etc/nginx/conf.d/foo.conf - source: salt://foo.nginx.conf - template: jinja - watch_in: - service: nginx - require: - docker: foo

In our foo nginx config, we can get the IP of our foo container and proxy directly to it without the need for port binding

{% set foo = salt['docker.inspect_container']('foo')['out'] %}{% set ip = foo['NetworkSettings']['IPAddress'] %}

server { listen 80; server_name foo.com; location / { proxy_pass http://{{ ip }}:5000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $remote_addr; }}

We also need to clean up after ourselves

$ docker rmi $(docker images -q -f dangling=true)

This only needs to be run if the image changed and after the container was removed

cleanup: cmd.wait: - name: docker rmi $(docker images -q -f dangling=true) - watch: - cmd: foo-absent

We can also run multiple containers of the same image{% for x in range(1, 5) %} # 4 containers

foo-{{ x }}-absent: cmd.wait: - name: docker rm -f foo-{{ x }} - watch: - docker: soon/foo - watch_in: - cmd: cleanup

foo-{{ x }}-container: docker.installed: - name: foo-{{ x }} - image: soon/foo - require: - docker: soon/foo - watch: - cmd: foo-{{ x }}-absent

foo-{{ x }}: docker.running: - container: foo-{{ x }} - require: - docker: foo-{{ x }}-container - require_in: - file: foo-nginx-config

{% endfor %}

Tell nginx how many containers we run

foo-nginx-config: file.managed: - name: /etc/nginx/conf.d/foo.conf - source: salt://foo3.nginx.conf - template: jinja - context: no_containers: 4 - watch_in: - service: nginx

Add an upstream backend to nginx to proxy to our 4 containersupstream backend {{% for x in range(1, no_containers + 1) -%}{%- set foo = salt['docker.inspect_container']('foo-' + x|string)['out'] -%} server {{ foo['NetworkSettings']['IPAddress'] }}:5000;{% endfor -%}}

server { listen 80; server_name foo.com; location / { proxy_pass http://backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $remote_addr; }}

Be careful with docker logs

They are not rotated and everything from STDOUT and STDERR ends up in them

They are only cleaned out when you remove a container

https://github.com/docker/docker/issues/7333

Set your app to log to a centralised log store, like LogStash

Private docker repositories for your secret source

docker-registries: https://index.docker.io/v1/: email: foo@bar.com password: 124 username: foo

Just add this to your pillars

8

Bonus Round

Continuous delivery with CircleCImachine: services: - docker environment: REPO: soon/foo TAG: $(sed 's/master/latest/;s/\//\-/' <<<$CIRCLE_BRANCH)

dependencies: pre: - sed "s/<EMAIL>/$DOCKER_EMAIL/;s/<AUTH>/$DOCKER_AUTH/" < .dockercfg.template > ~/.dockercfg override: - docker build -t $REPO:$TAG .

test: override: - docker run -it --name test --net=host $REPO:$TAG python setup.py test

deployment: release: branch: master commands: - docker push $REPO:$T AG

Salt REST API to trigger state runsrest_cherrypy: port: 8000 host: 127.0.0.1 disable_ssl: True webhook_disable_auth: True

$ service salt-api start

Add reactor events to react to webhook callsreactor: - 'salt/netapi/hook/circleci/success': - /srv/salt/reactor/deploy.sls

Start the Salt API Service

Create a reactor sls to handle the event

def run(): ret = {} ret['deploy'] = { 'cmd.state.highstate': [ {'tgt': '*'}, ] } return ret

Send the event by hitting the webhook urlcurl -sS -k http://domain.com/hook/circleci/success

How about Slack notifications?import jsonimport urllibimport urllib2

def slack(message, color=None): attachment = { "text": message, } if color: attachment['color'] = color payload = { "channel": "#your-channel", "attachments": [attachment] } payload = urllib.pathname2url(json.dumps(payload)) payload = 'payload=' + payload request = urllib2.Request('https://slack.hook', payload) urllib2.urlopen(request)

... slack('Deploying...')... return ret

And update the CircleCI config

deployment: release: branch: master commands: - docker push $REPO:$TAG - curl -sS -k https://domain.com/hook/circleci/success

9

Questions?

Thank you! :)