using containers for continuous integration and continuous delivery

42
USING CONTAINERS FOR CONTINUOUS INTEGRATION AND CONTINUOUS DELIVERY / Carlos Sanchez csanchez.org @csanchez Watch online at carlossg.github.io/presentations

Upload: carlos-sanchez

Post on 21-Jan-2018

185 views

Category:

Software


7 download

TRANSCRIPT

Page 1: Using Containers for Continuous Integration and Continuous Delivery

USING CONTAINERS FOR

CONTINUOUS INTEGRATION ANDCONTINUOUS DELIVERY

/

Carlos Sanchez

csanchez.org @csanchez

Watch online at carlossg.github.io/presentations

Page 2: Using Containers for Continuous Integration and Continuous Delivery

ABOUT MEEngineer, Scaling Jenkins

Author of Jenkins Kubernetes plugin

Contributor to Jenkins, Maven official Docker images

OSS contributor at Apache Maven, Eclipse, Puppet,…

Page 3: Using Containers for Continuous Integration and Continuous Delivery

OUR USE CASE

Scaling JenkinsYour mileage may vary

Page 4: Using Containers for Continuous Integration and Continuous Delivery
Page 5: Using Containers for Continuous Integration and Continuous Delivery

Isolated Jenkins masters

Isolated agents and jobs

Memory and CPU limits

Page 6: Using Containers for Continuous Integration and Continuous Delivery

BUT IT IS NOT TRIVIAL

Page 7: Using Containers for Continuous Integration and Continuous Delivery

SCALING JENKINSTwo options:

More agents per masterMore masters

Page 8: Using Containers for Continuous Integration and Continuous Delivery

SCALING JENKINS: MORE AGENTS

Pros

Multiple plugins to add more agents, evendynamically

Cons

The master is still a SPOFHandling multiple configurations, pluginversions,...There is a limit on how many agents can beattached

Page 9: Using Containers for Continuous Integration and Continuous Delivery

SCALING JENKINS: MORE MASTERS

Pros

Different sub-organizations can self service andoperate independently

Cons

Single Sign-OnCentralized configuration and operation

Page 10: Using Containers for Continuous Integration and Continuous Delivery

The best of both worlds

CloudBees Jenkins Operations Center with multiplemasters

Dynamic agent creation in each master

CLOUDBEES JENKINS ENTERPRISE

Page 11: Using Containers for Continuous Integration and Continuous Delivery

FIRST IMPLEMENTATION

Apache MesosMesosphere MarathonTerraformPacker

Page 12: Using Containers for Continuous Integration and Continuous Delivery

&

Page 13: Using Containers for Continuous Integration and Continuous Delivery

We can run both Jenkins masters and agents inKubernetes

Page 14: Using Containers for Continuous Integration and Continuous Delivery

STORAGEHandling distributed storage

Masters can move when they are restarted

Jenkins masters need persistent storage, agents(typically) don't

Using PersistentVolumeClaim so you canprovide any implementation

Page 15: Using Containers for Continuous Integration and Continuous Delivery

CAVEATSPerformance

Lack of multi-AZ for block storage, ie. EBS AWS

"Worst-case scenario for just about anycommonly used network-based

storage"

Page 16: Using Containers for Continuous Integration and Continuous Delivery

NETWORKINGJenkins masters open several ports

HTTPJNLP agent

Page 17: Using Containers for Continuous Integration and Continuous Delivery

NETWORKING: HTTPWe use ingress rules and nginx ingress controller

Operations CenterJenkins masters

Path based routing cje.example.com/master1

Page 18: Using Containers for Continuous Integration and Continuous Delivery

NETWORKING: JNLPAgents started dynamically in cluster can connect tomasters internallyAgents manually started outside cluster connectdirectly

Using NodePort

Page 19: Using Containers for Continuous Integration and Continuous Delivery

AGENTS WITH INFINITE* SCALE!

Dynamic Jenkins agents, running as PodsMulti-container support

One Jenkins agent image, others customJenkins Pipeline support for both agent Poddefinition and executionPersistent workspaceAuto configured

Jenkins Kubernetes Plugin

Page 20: Using Containers for Continuous Integration and Continuous Delivery

ON DEMAND JENKINS AGENTSpodTemplate(label: 'mypod') { node('mypod') { sh 'Hello world!' } }

Page 21: Using Containers for Continuous Integration and Continuous Delivery

GROUPING CONTAINERS (PODS)podTemplate(label: 'maven', containers: [ containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-a ttyEnabled: true, command: 'cat') ]) {

node('maven') { stage('Get a Maven project') { git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage('Build a Maven project') { sh 'mvn -B clean package' } } } } }

Page 22: Using Containers for Continuous Integration and Continuous Delivery

USING DECLARATIVE PIPELINEpipeline { agent { kubernetes { label 'mypod' containerTemplate { name 'maven' image 'maven:3.3.9-jdk-8-alpine' ttyEnabled true command 'cat' }}} stages { stage('Run maven') { steps { container('maven') { sh 'mvn -version' } }}} }

Page 23: Using Containers for Continuous Integration and Continuous Delivery

MULTI-LANGUAGE PIPELINEpodTemplate(label: 'maven-golang', containers: [ containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-a ttyEnabled: true, command: 'cat'), containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')]) {

node('maven-golang') { stage('Build a Maven project') { git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { sh 'mvn -B clean package' }}

stage('Build a Golang project') { git url: 'https://github.com/hashicorp/terraform.git' container('golang') { sh """ mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform cd /go/src/github.com/hashicorp/terraform && make core """ }}}}

Page 24: Using Containers for Continuous Integration and Continuous Delivery

PODS: SELENIUMExample:

Jenkins agentMaven buildSelenium Hub with

FirefoxChrome

5 containers

Page 25: Using Containers for Continuous Integration and Continuous Delivery

podTemplate(label: 'maven-selenium', containers: [

containerTemplate(name:'maven-firefox', image:'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),

containerTemplate(name:'maven-chrome', image:'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),

containerTemplate(name: 'selenium-hub', image: 'selenium/hub:3.4.0'),

Page 26: Using Containers for Continuous Integration and Continuous Delivery

// because containers run in the same network space, we need // make sure there are no port conflicts // we also need to adapt the selenium images because they we // designed to work with the --link option

containerTemplate(name: 'selenium-chrome', image: 'selenium/node-chrome:3.4.0', envVars: [ containerEnvVar(key: 'HUB_PORT_4444_TCP_ADDR', value: 'loc containerEnvVar(key: 'HUB_PORT_4444_TCP_PORT', value: '444 containerEnvVar(key: 'DISPLAY', value: ':99.0'), containerEnvVar(key: 'SE_OPTS', value: '-port 5556'), ]),

Page 27: Using Containers for Continuous Integration and Continuous Delivery

containerTemplate(name: 'selenium-firefox', image: 'selenium/node-firefox:3.4.0', envVars: [ containerEnvVar(key: 'HUB_PORT_4444_TCP_ADDR', value: 'loc containerEnvVar(key: 'HUB_PORT_4444_TCP_PORT', value: '444 containerEnvVar(key: 'DISPLAY', value: ':98.0'), containerEnvVar(key: 'SE_OPTS', value: '-port 5557'), ])

Page 28: Using Containers for Continuous Integration and Continuous Delivery

node('maven-selenium') { stage('Checkout') { git 'https://github.com/carlossg/selenium-example.git' parallel (

Page 29: Using Containers for Continuous Integration and Continuous Delivery

firefox: { container('maven-firefox') { stage('Test firefox') { sh """ mvn -B clean test -Dselenium.browser=firefox \ -Dsurefire.rerunFailingTestsCount=5 -Dsleep=0 """ } } },

Page 30: Using Containers for Continuous Integration and Continuous Delivery

chrome: { container('maven-chrome') { stage('Test chrome') { sh """ mvn -B clean test -Dselenium.browser=chrome \ -Dsurefire.rerunFailingTestsCount=5 -Dsleep=0 """ } } }

Page 31: Using Containers for Continuous Integration and Continuous Delivery

STORAGE

GCE disksGlusterFSNFSEBSetc

Persistent volumes

Page 32: Using Containers for Continuous Integration and Continuous Delivery

USING PERSISTENT VOLUMESapiVersion: "v1" kind: "PersistentVolumeClaim" metadata: name: "maven-repo" namespace: "kubernetes-plugin" spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi

Page 33: Using Containers for Continuous Integration and Continuous Delivery

podTemplate(label: 'maven', containers: [ containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-a ttyEnabled: true, command: 'cat') ], volumes: [ persistentVolumeClaim(mountPath: '/root/.m2/repository', claimName: 'maven-repo', readOnly: false) ]) {

node('maven') { stage('Build a Maven project') { git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { sh 'mvn -B clean package' } } } }

Page 34: Using Containers for Continuous Integration and Continuous Delivery

RESOURCE REQUESTS ANDLIMITS

podTemplate(label: 'mypod', containers: [ containerTemplate( name: 'maven', image: 'maven', ttyEnabled: true, resourceRequestCpu: '50m', resourceLimitCpu: '100m', resourceRequestMemory: '100Mi', resourceLimitMemory: '200Mi')]) { ... }

Page 35: Using Containers for Continuous Integration and Continuous Delivery

DEPLOYING TOKUBERNETES

Page 36: Using Containers for Continuous Integration and Continuous Delivery
Page 37: Using Containers for Continuous Integration and Continuous Delivery

If you haven't automatically destroyedsomething by mistake, you are not

automating enough

Page 38: Using Containers for Continuous Integration and Continuous Delivery

podTemplate(label: 'deployer', serviceAccount: 'deployer', containers: [ containerTemplate(name: 'kubectl', image: 'lachlanevenson/k8s-kubectl:v1.7.8', command: 'cat', ttyEnabled: true) ]){ node('deployer') { container('kubectl') { sh "kubectl apply -f my-kubernetes.yaml" } } }

Page 39: Using Containers for Continuous Integration and Continuous Delivery

kubernetes-pipeline-pluginpodTemplate(label: 'deploy', serviceAccount: 'deployer') {

stage('deployment') { node('deploy') { checkout scm kubernetesApply(environment: 'hello-world', file: readFile('kubernetes-hello-world-service.yaml')) kubernetesApply(environment: 'hello-world', file: readFile('kubernetes-hello-world-v1.yaml')) }}

stage('upgrade') { timeout(time:1, unit:'DAYS') { input id: 'approve', message:'Approve upgrade?' } node('deploy') { checkout scm kubernetesApply(environment: 'hello-world', file: readFile('kubernetes-hello-world-v2.yaml')) }} }

Page 40: Using Containers for Continuous Integration and Continuous Delivery

Or Azure kubernetes-cd-pluginkubernetesDeploy( credentialsType: 'KubeConfig', kubeConfig: [path: '$HOME/.kube/config'],

configs: '*.yaml', enableConfigSubstitution: false, )

Page 41: Using Containers for Continuous Integration and Continuous Delivery

THE FUTUREPeriodic snapshotting, ie. EBS volumes in AWSAffinityDedicated workers for agents, infra autoscalingMulti-regionNamespaces per team: quotas, isolation