continuous integration using docker & jenkins

Continuous Integration using Docker &JenkinsLinuxCon Europe 2014 October 13-15, 2014

Mattias GieseSolutions ArchitectB1 Systems GmbH

[email protected]

B1 Systems GmbH - Linux/Open Source Consulting, Training, Support & Development

mailto:[email protected]

Introducing B1 Systems

founded in 2004operating both nationally and internationallymore than 60 employees; low employee turnoverProvider for IBM, SUSE, Oracle & HPvendor-independent (hardware and software)Focus:

ConsultingSupportDevelopmentTrainingOperationsSolutions

B1 Systems GmbHContinuous Integration using Docker &

Jenkins 2 / 43

http://www.b1-systems.de

Areas of Expertise

Virtualization (XEN, KVM & RHEV)Systems management (Spacewalk, Red Hat Satellite, SUSEManager)Configuration management (Puppet & Chef)Monitoring (Nagios & Icinga)IaaS Cloud (OpenStack & SUSE Cloud)High availability (Pacemaker)Shared Storage (GPFS, OCFS2, DRBD & CEPH)File Sharing (ownCloud)Packaging (Open Build Service)Providing on-site systems administration and/or development


Jenkins 3 / 43


Partners


Jenkins 4 / 43


Deployment Stack


Jenkins 5 / 43


Deployment Stack – Technologies Used

Docker an open platform for developers and sysadmins to build,ship, and run distributed applications

Fig a Docker orchestration toolGitlab an Open Source software to collaborate on code

Jenkins an Open Source continuous integration systemPuppet/r10k an Open Source configuration management system to

define the state of an IT infrastructure and toautomatically enforce this state


Jenkins 6 / 43


Docker – Build, Ship andRun Applications


Jenkins 7 / 43


What is Docker?

an open platform for developers and sysadminsOpen Source Engine to standardize LXCbuild, ship and run (distributed) applicationseasy to usecreate and share imageschroot on steroidsall you need is inside the containernot a Virtual Server – less overhead


Jenkins 8 / 43


Technologies Used

Linux Containers (LXC)chrootuse of Linux Kernel features:

cgroupskernel namespaces. . .


Jenkins 9 / 43


Features

can run any distribution“if it will run on Linux it will run in Docker”limited to the same architecture as the hostall you need is inside the container

librariesdependencies. . .


Jenkins 10 / 43


Fig – A Docker Orchestration Tool

simple orchestration tool for Dockereasy to deploy and usehelps to define and control a multi-container service


Jenkins 11 / 43


GitLab – Collaboration on Code


Jenkins 12 / 43


Features

completely free and Open Sourcemanage and browse Git repositorieskeep your code secure on own servermanage access permissionsperform code review and merge requestshooksmuch more!


Jenkins 13 / 43


Gitlab 1/3


Jenkins 14 / 43


Gitlab 2/3


Jenkins 15 / 43


Gitlab 3/3


Jenkins 16 / 43


Jenkins – Continuous Integration


Jenkins 17 / 43


What is Jenkins?

Jenkins is a server-based system for continuous integrationrunning in a servlet container (like Apache Tomcat).Jenkins supports SCM tools including AccuRev, CVS,Subversion, Git, Mercurial, Perforce, Clearcase and RTC, andcan execute Apache Ant and Apache Maven.Jenkins is free software, released under the MIT License.Builds can be started by various means, including beingtriggered by commit in a version control system.Jenkins monitors executions of repeated jobs, such as building asoftware project or jobs run by cron.Jenkins is written in Java and released under the MIT License.


Jenkins 18 / 43


Jenkins 1/3


Jenkins 19 / 43


Jenkins 2/3


Jenkins 20 / 43


Jenkins 3/3


Jenkins 21 / 43


Jenkins PluginsDocker Buildstep allows to add various docker commands into a job

as a build step.Docker publish provides the ability to build projects with a Dockerfile

and publish them to the docker registry.Git allows the use of Git as a build SCM.

Gitlab a build trigger that makes GitLab think Jenkins is aGitLab CI.

Build Pipeline Plugin provides a Build Pipeline View of upstream anddownstream connected jobs that typically form a buildpipeline.

Downstream-Ext Plugin supports extended configuration fortriggering downstream builds.

Publish Over SSH Plugin transfers files and data secured by SSH.Chuck Norris ;) . . .


Jenkins 22 / 43


Jenkins – Build Pipeline Plugin

provides a Build Pipeline View of upstream and downstreamconnected jobs that typically form a build pipeline.offers the ability to define manual triggers for jobs that requireintervention prior to execution, e.g. an approval process outsideof Jenkins.


Jenkins 23 / 43


Jenkins – Downstream-Ext Plugin

This plugin supports extended configuration for triggeringdownstream builds:

triggers build only if downstream job has SCM changestriggers build if upstream build result is better/equal/worse thanany given result (SUCCESS, UNSTABLE, FAILURE,ABORTED)for Matrix (alias multi-configuration) jobs, you can decide whichpart of the job should trigger the downstream job: parent only,configurations only, or both


Jenkins 24 / 43


Jenkins – Publish Over SSH PluginSCP – send files over SSH (SFTP)execute commands on a remote serverusername and password or public key authenticationpasswords & -phrases encryption in configuration files and UISSH SFTP/SSH Exec as a build step during the build processSSH before a (maven) project build, or to run after a buildwhether the build was successful or notsend files directly from the artifacts directory of the build that isbeing promoted ("promotion aware")optionally override the authentication credentials for each serverin the job configurationoptionally retry if the transfer of files failsenable the command/script to be executed in a pseudo TTY


Jenkins 25 / 43


Puppet – Configuration Management


Jenkins 26 / 43


What is Puppet?a configuration management system to define a certain state ofan IT infrastructuredeveloped since 2005 by Puppet Labsdescribes resources and their state in manifestsuses its own declarative languagedistributes these manifests through a server program calledmasterAgents on the target systems enforce the desired state.System specific information will be discovered using facter fora dynamic configuration.Agents also send a report on the taken actions back to thePuppet master.Puppet’s open API can send and receive data to/fromthird-party tools.


Jenkins 27 / 43


Puppet – r10k

deployment helper for Puppet modules (internal/Puppet Forge)uses a cache directory to preserve spacemay use a so called Puppetfile for complex deployment needs(⇒ Gemfile)


Jenkins 28 / 43


Use Case 1: Automatic Testing of a PuppetModule


Jenkins 29 / 43


Use Case 1 – Automatic Testing of a puppetmodule


Jenkins 30 / 43


Use Case 1 – Prerequesites

Requirements:base Docker container for every supported OS of a moduleare being built as seperate jobs

Preparation:1 git push to development branch2 Gitlab triggers Jenkins Webhook3 Jenkins merges dev with test branch


Jenkins 31 / 43


Use Case 1 – puppet-memcached-units

1 Jenkins creates a new container.2 r10k deploys all puppet code.3 simple syntax and style(lint) checks4 rspec-puppet is run.


Jenkins 32 / 43


puppet-memcached-units


Jenkins 33 / 43


Use Case 1 – Job:puppet-memcached-acceptance

1 Jenkins starts a fresh container from a puppet-enabled baseimage

2 r10k deploys all needed Puppet code3 puppet apply is run with the specified module:

PUPPETENV=/puppet/environments/dev/puppet apply –debug –modulepath $PUPPETENV/modules$PUPPETENV/modules/$PUPPETMODULE/test/init.pp


Jenkins 34 / 43


puppet-memcached-acceptance


Jenkins 35 / 43


Use Case 2: Integration/Acceptance Testingof a Simple Webapp


Jenkins 36 / 43


Use Case 2: Integration/Acceptance Testingof a Simple Webapp

simple use case for a multi-tier apphttpd + webapp and mysqlneed to be linked togethershould be automatically deployed if possibletwo containers: owncloud and mysqlautomatic rebuild on changeafter changes, integration testing should be doneif integration tests succeed, deploy app on staging host


Jenkins 37 / 43


Jenkins Job Overview


Jenkins 38 / 43


Docker Build Pipeline 1


Jenkins 39 / 43


Docker Build Pipeline 2


Jenkins 40 / 43


Next Steps


Jenkins 41 / 43


Next StepsMost Docker image jobs are identical:

implement flow/job dsl plugincreate a dynamic test matrix for Puppet modules

multi-configuration jobs may help with thatuse Packer for building base Docker images (Puppet Provisioner)

integrate with CoreOS/Project Atomic/OpenStackimplement a better global orchestration scheme (TerraForm,Ansible, SaltStack maybe?)integrate with clustered configuration through Serf/Consul orothersautomatic handoff to QA, tests with real world data in stagingenvironmentseasy push button tagging of Docker images to ’production’

may be implemented through ’promoted builds’


Jenkins 42 / 43


Thank you for your attention!For further information, please contact:

[email protected] or +49 (0)8457 - 931096

B1 Systems GmbH - Linux/Open Source Consulting, Training, Support & Development

mailto:[email protected]

continuous integration using docker & jenkins

Technology