Continuous Deployment with Kubernetes, Docker

and GitLab CI

@alexander_kielClojure Berlin 2016

Outline• Continuous Deployment why?

• Docker

• Kubernetes

• Sample Clojure Service

• Deploy with GitLabCI

Continuous Deployment• What do we want?

• Increase responsiveness

• Decrease time to market

• Gain confidence by deploying often in small amounts

• How to achieve that?

• Automate everything

• Always deploy the master into production

• Use feature toggles when needed

Simple Git Workflow

• Works for in-house apps

• not for libs or shipping apps

• No versions, no tags, just SHA’s

• Latest commit on master is always deployed to production

• Feature/fix branches are merged when ready

masterfeature/fixbranches

1ebb95d

be61dda

6e4010d

Docker• Like VM’s but much more light-weight and shippable

• Runs on Linux, executes processes in an isolated environment (resource limitation, filesystem, network)

• Container principle: Can contain everything, but looks the same from the outside

• A container platform can run every container

• Developers have max. freedom what to do

• In contrast: PaaS like Heroku - has to support the language

Kubernetes• Container runtime platform

• Originally designed by Google - now Open Source

• One of the most active projects on GitHub - 20,000 stars, 40,000 commits, 15,000 issues, 200 releases

• Alternatives: Apache Mesos, Docker Swarm (lacks features)

Kubernetes Architecturek8s-master-1

k8s-master-2

k8s-master-3

load-balancer-1

load-balancer-2

DNS RR

k8s-worker-1

proxy

app-1

k8s-worker-2

proxy

app-2

k8s-worker-n

proxy

app-k

etcd clusterquorum

HAProxy

• Runs on VMware ESX

• CoreOS Linux

• Single YAML file as configuration

• Everything in containers

Kubernetes - Pods• A Pod is a deployable unit in

Kubernetes

• Pods can contain multiple containers

• Containers inside a Pod share on port space, can use localhost and can communicate via IPC and shared memory

• Idea: one process per container - many cooperating processes in one Pod

apiVersion: v1 kind: Pod metadata: name: <pod-name> labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>

Kubernetes - Deployments

• A Deployment ensures that certain number of Pods are always running

• It consists of a Pod template and the number of replicas

• It supports hot-redeployments by changing parts of the Pod template

• Horizontal scaling is possible

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: <deployment-name> spec: replicas: 2 template: metadata: labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>

Kubernetes - Services• Kubernetes uses an overlay

network to provide different address spaces (we use flannel)

• Every Pod has an IP address - but it changes every time one is created

• Services provide a stable IP address for groups of Pods

• Service names are resolvable by an internal DNS

• Service selectors are used to match Pods according to there labels

apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP selector: service: clojure-berlin-2016

Kubernetes - External Access• Kubernetes networks are internal

only

• External access through load balancers necessary

• Certain Platforms like Google Compute Engine provide load balancer integration with Kubernetes

• We have our own solution as a combination of HAProxy and Kubernetes NodePort

• Kubernetes Services with type NodePort are exposed on every worker under a certain port

frontend http bind 0.0.0.0:80 mode http option httplog

acl host_clj hdr(host) clj.<domain>

use_backend clj if host_clj

backend clj mode http balance roundrobin option httplog server worker-1 <ip>:32599 check server worker-2 <ip>:32599 check

Deployment Lifecycle

GitLab CI

Source Code

build

test

Kubernetes Test

Cluster

Kubernetes Prod

Cluster

automatic deployment

manualdeployment

git push

Sample Clojure Service• .gitlab-ci.yml

• Like .travis.yml contains instructions for GitLabCI how to test, build and deploy

• Dockerfile

• Instructions for Docker how to build the image of the app

• Artifact of the build is a docker image - not uberjar

• kube-deployment.yml

• Kubernetes deployment instructions

• kube-svc.yml

• Kubernetes service description

https://github.com/alexanderkiel/clojure-berlin-2016

The Core Namespace

(ns clojure-berlin-2016.core (:require [aleph.http :as http] [clojure.core.async :refer [<!! chan]]))

(defn -main [& args] (-> (fn [_] {:status 200 :body "Clojure Berlin 2016"}) (http/start-server {:port 8080})) (<!! (chan)))

• A simple web server returning "Clojure Berlin 2016"

The Leiningen Project File

(defproject clojure-berlin-2016 "<VERSION>" :dependencies [[aleph "0.4.1"] [org.clojure/clojure "1.8.0"] [org.clojure/core.async "0.2.395"]]

:main clojure-berlin-2016.core)

• <VERSION> is replaced at build time by the Git SHA

• :main is for lein run to work

.gitlab-ci.yml - test/buildimage: clojure:lein-2.7.1

stages: - test - build - deploy

test: stage: test tags: - docker script: - lein test

build: stage: build tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" project.clj - docker build -t clojure-berlin-2016:$CI_BUILD_REF . - docker push clojure-berlin-2016:$CI_BUILD_REF

.gitlab-ci.yml - deploy branchdeploy-branch: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml except: - master when: manual

• Used to test a feature/fix branch in a full environment

.gitlab-ci.yml - deploy test

deploy-master: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml only: - master

.gitlab-ci.yml - deploy prod

deploy-prod: stage: deploy environment: prod image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-prod-a - kubectl apply -f kube-deployment.yml only: - master when: manual

Docker fileFROM clojure:lein-2.7.1

COPY src /app/src COPY project.clj /app/

WORKDIR /app

RUN lein with-profile production deps

EXPOSE 80

CMD ["lein", "with-profile", "production", "run"]

• Just copy the sources into the container

• Use Leiningen itself to run in production

kube-deployment.ymlapiVersion: extensions/v1beta1 kind: Deployment metadata: name: clojure-berlin-2016 spec: replicas: 2 template: metadata: labels: app: lens service: clojure-berlin-2016 spec: containers: - name: clojure-berlin-2016 image: dreg.life.uni-leipzig.local/clojure-berlin-2016:<VERSION> ports: - containerPort: 8080 resources: requests: cpu: "125m" memory: "1Gi" limits: cpu: 1 memory: "2Gi"

kube-svc.yml

apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 8080 protocol: TCP selector: service: clojure-berlin-2016

Steps to Follow• Create the Kubernetes Service

• kubectl create -f kube-svc.yml

• Edit HAProxy Config

• add rules and backend for the service

• Push to GitLab

• git push

Pipeline in GitLab CI

Deployment in GitLabCI

Environments in GitLabCI

• Very good visibility of wich commit is deployed in which environment right now

• Manual deployment to prod possible

Environment History

• Easy to see when what commit was deployed

• Rollback possible

Numbers• Our team has 4 developers

• We run 2 Kubernetes clusters (test and prod) with about 96 GB RAM and and 24 vCPU’s each

• We run about 60 pods in production

• We have other services like central log aggregation running using Fluentd and Elasticsearch/Kibana

Thank You• Sample Project on Github

https://github.com/alexanderkiel/clojure-berlin-2016

• Twitter@alexander_kiel

• Mailalexanderkiel@gmx.net