martin höfling tng technology consulting …...© tng technology consulting gmbh 3 / 54...

Martin HöflingJohannes Ebke

From Zero to Webscale:Evolving a Continuous Delivery Pipeline

TNG Technology Consulting

2 / 54© TNG Technology Consulting GmbH

What this talk is about

Introduction

►Context

►Challenges

►Technology

Evolution towards fully automatic deployment

Remarks & Conclusions

Overview


Introduction: Context

TNG Technology Consulting GmbH

~230 regular employees,

►> 50% with a PhD

►Physics, Mathematics, Computer Science

Founded in 2001, grown ever since, CAGR ~ 25%

Place of Business: Munich – Heart of Bavaria

Value-based consulting partnership

Focus on high end IT


Startup Characteristics

Few people

Moving targets

No dedicated operations

Motivated tech addicts

Fast decisions - flat hierarchy

Everybody feels responsible(ideally…)



The Product



Scaling

Introduction: Challenges

…sometimes many, sometimes fewer users


Complexity

Introduction: Challenges

Multiple backend types

Several Databases and Queues

Despite of Complexity: Daily Releases


What Technology are we using?

ReactWebpack

Gulp

TornadoPython3.5 Elasticsearch

MongoDB RedisJenkins

Packer

Docker

SaltStackTerraform

Boto3 AWS

Vagrant

Icinga2 ELK

Backbone

Introduction: Technology



Introduction


►Phase 1: Starting up

►Phase 2: Automating deployment to the cloud

►Phase 3: Fully automated infrastructure setup


Overview


Starting Up

Set up CI pipeline

►Automatic testing

Create deployable artifact

►Deploy „somewhere“:

►Reproducible

Starting Up


Infrastructure at the Beginning

Starting Up

Local Cloud

developers

CI / Jenkinstest instance

production single EC2

SSH


What is SaltStack?

Starting Up

Quoting saltstack.com:

… „scalable and flexible configuration management“

… „event-driven automation of CloudOps, ITOps and DevOps“


Why SaltStack?

Key features:

Declarative configuration state management

Scalability to thousands of managed machines

Event driven reaction and reconfiguration:

►Predictive Orchestration

Nice to have:

Provider agnostic (Cloud-) Ops

Implemented / extensible in Python

Large, friendly community

Starting Up


Declarative Configuration: Salt States

Starting Up

/etc/secrets: file.managed: - mode: 600 - contents: | root:secret

nginx: service.running: - reload: True - watch: - file: /etc/secrets

/etc/secrets: file.managed: - mode: 600 - contents: | root:secret

nginx: service.running: - reload: True - watch: - file: /etc/secrets

ResourceState Function

Parameter

Dependency


Event Driven Orchestration with Salt

Starting Up

Salt Master Salt Minion loadbalancer-12

Salt Minion loadbalancer-12



subscribe

Salt Minion app-42

subscribe

Salt Minion monitoring

subscribe

Master Event Bus

Master - Minion Event Bus

Salt Minion elasticsearch-11



Starting Up





subscribe

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

subscribe


subscribe

Master Event Bus



From: AWS type: new machine launched machine: elasticsearch-12




Starting Up





subscribe

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

subscribe


subscribe

Master Event Bus



From: AWS type: new machine launched machine: elasticsearch-12

To: elasticsearch-12 Function: install_es Arguments: -version: 2.1.3


publish job



Starting Up





subscribe

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

Salt Minion app-42

subscribe


subscribe

Master Event Bus



From: elasticsearch-12 JobId: 3032 Result: OK, installed

To: elasticsearch-12 Function: install_es Arguments: -version: 2.1.3




Starting Up





subscribe

Salt Minion app-42

subscribe


subscribe

Master Event Bus



From: elasticsearch-12 JobId: 3032 Result: OK, installed

To: app-* Function: add_ES_to_config Arguments: -server: elasticsearch-36


publish job



Starting Up





subscribe

Salt Minion app-42

subscribe


subscribe

Master Event Bus



From: app-[1..42] JobId: 3034 Result: OK, config

To: app-* Function: add_ES_to_config Arguments: -server: elasticsearch-36



Automating deployment with Salt

Local Clouddevelopers

test instances

production single EC2CI / Jenkins

Salt master

Salt / ØMQ Artifact

Starting Up


Starting Up - Lessons Learned

Keep testing and production in sync

Automate deployment early

►Select and get familiar with your deployment tool

Make the deployment accessible for the team

► Infrastructure as Code from the beginning

Starting Up



Introduction



►Phase 2: Automating deployment to the cloud



Overview


Automating Deployment into the Cloud

Rationale:

Cloud Machines are Cattle, not Pets ...

► ... they must be replaceable quickly

(Bill Baker, Microsoft)

Approach:

Automate dynamic resource configuration (e.g. deployment):

► resource allocation, software installation, …

► distributed systems with many moving parts

Manual configuration of static resources:

►VPC, VPN, Firewall, Routing and DNS

Modularize CI/CD

Automating Deployment


Dockerized Jenkins Pipeline

Reasons:

Flexibility

►Versioned and adapting CD pipeline

►Deployability in case of hardware failure

Transparency

►Docker test cluster also runs locally

Rapid feedback

►Scale beyond single machine

►Parallel integration tests



Dockerized Jenkins Pipeline – Build



test instances

production instance

CI

Test / Build

Artifact


Dockerized Jenkins Pipeline – Deploy



test instances

production instance

Salt / ØMQ CI

Salt master


Salt Cloud – Machine Lifecyle made easy

Cloud Profile

►Different providers possible

Cloud Map

►List instances for each profile

Salt-cloud

►Creates / destroys machines

► Installs Salt

►Attaches machine to salt-master


appserver: provider: aws image: ami-bdc9dad1 size: t2.large

loadbalancer: provider: gce image: centos-6 size: n1-standard-1

appserver: provider: aws image: ami-bdc9dad1 size: t2.large

loadbalancer: provider: gce image: centos-6 size: n1-standard-1

loadbalancer: - lb1 - lb2

appserver: - app1 - app2 - app3

loadbalancer: - lb1 - lb2

appserver: - app1 - app2 - app3


Salt Cloud from Dockerized Salt Container



test instances

production instance

Salt / ØMQ CI

Salt master

Salt Cloud new instance


Distribute your Application

Reasons:

Improved availability

Horizontal scaling

Problems:

Adds complexity to your setup

►Deployment often requires complex orchestration




How to Create a Distributed Application?


Create Distributed Applications with Salt Cloud



all-in-one instances

distributed instance

Salt / ØMQ

CI

Salt master

new instance master

new instance


Create Distributed Applications with Salt Cloud




Salt / ØMQ

CI

Salt master

instance master

trigger salt-cloud

salt cloud


Deployment of a Distributed Application

Separate non critical and critical steps

Non-critical:

►Build and Push Frontend / Backend Package

►Update Instance Master

Critical step

►Apply configuration to (critical) systems

► Independent of local CI







Salt / ØMQ

CI

Salt master

instance master






Salt / ØMQ

CI

Salt master

instance master ELBs

trigger activation

ØMQ


Autoscaling

Goals:

Only use computing resources required at the moment

Automatically replace dead or disconnected instances

Approaches:

Autoscaling solely with Salt

AWS Autoscaling technology



Deployment with Autoscaled Application Tier




CI

Salt master

instance master

Salt / ØMQ app template

trigger activation


Deployment with Autoscaled Application Tier




CI

Salt master

instance master

ELBs


app AMI

ASGs

trigger activation

Python Salt / Boto3


Deployment – Third Party Services




CI

Salt master

instance master

ELBs


app AMI

ASGsElasticache

Python Salt / Boto3trigger activation


Automating Deployment – Lessons Learned

Know the capabilities and limits of the different services

►what scales and what does not scale

Integration of Ops tools in CI not trivial

►Salt Return Codes are “surprising”

Frequent deployments

►… are key to Reliability

Reliability

►… is key to frequent deployments




Introduction



►Phase 2: Automating deployment to the Cloud



Overview


Fully automated Infrastructure Setup

„Click here for New Datacenter“

Trigger:

Separate testing and production into two accounts

►Restricting access to production to a team subset

Goal:

Define static infrastructure as code



Salt for Infrastructure – and its limits

Use and extend Salt fordynamic configuration ofinfrastructure:

DNS, ELB, ASG

Orchestration Scripts:

Deployment, Backup &Restore

Problem:

►Static Infrastructure


#!python

import boto3#!python import boto3

#!python

import boto3


Static Infrastructure (~200 Resources per App Instance)


Local CloudDevOps team

CDN distributions

Route53 DNS Zones

S3 Storage

multiple Acounts

multiple VPCsinternet / NAT / VPN

gateways

VPN connections

routing / subnets

SEIPs, Security Groups


Terraforming the Static Infrastructure


Local CloudDevOps team

CDN distributions

Route53 DNS Zones

S3 Storage

multiple Acounts

multiple VPCs

AWS API

internet / NAT / VPN gateways

VPN connections

routing / subnets

SEIPs, Security Groups


Terraform in Action


$ terraform plan

$ terraform apply

Differences to SaltStack:

Keeps track of state

Implicit dependencies by referencing resources

Plan changes beforehand to avoid disruption

resource "aws_vpc" "main" { cidr_block = "10.1.0.0/16" tags { Name = "Testing" }}

resource "aws_internet_gateway" "gw" { vpc_id = "${aws_vpc.main.id}"}

resource "aws_vpc" "main" { cidr_block = "10.1.0.0/16" tags { Name = "Testing" }}

resource "aws_internet_gateway" "gw" { vpc_id = "${aws_vpc.main.id}"}


Terraform – Lessons Learned

Modularize: Keep testing andproduction as similar as possible


Make sure you check the documentationthat the features you require are there.

Terraform enabled us to easily manage~200 resources and ~500 dependencies.



Introduction



Overview


Distribute Knowledge in the Team

Not all need in depth knowledge

►Two of us focus on CI/CD and infrastructure

Make deployment visible and accessible

► Jenkins pipelines everybody can use

►Check in deployment code into git

Simplify deployment wherever possible



Why Automating and DefiningInfrastructure as Code?

In software development...

… you test your code – right?

►Automatic & reproducible

… you review your code?

►Quality and Maintainability

So why is infrastructure often…

… not reviewed?

… not automatically tested?

… so broken?



We have the tools!

Let‘s go and fix this!

Automate!



Thank you for listening!

Questions?


Contact

Martin Höfling

[email protected]

@martoss13

martinhoefling

Johannes Ebke

[email protected]

JohannesEbke

mailto:[email protected]

mailto:[email protected]

martin höfling tng technology consulting …...© tng technology consulting gmbh 3 / 54...

Documents