Jean-Paul van Deursen

Wiebe de Roos

ABN-AMRO

Dockerizing the enterprise –fast and secure

Wiebe de Roos

CI/CD Consultant / IT Wizard

• Studied Communication & Multimedia Design and Master of Management & ICT

• 12+ years of IT expertise • CI/CD Consultant / Engineer implementing

Jenkins Enterprise in AWS at ABN AMRO• Lots of expertise about Docker (security) topics• Speaker at ABN AMRO and other industry

conferences

Who are we?

Jean-Paul van Deursen

IT Wizard

• Studied Electrotechnical Engineering @TU Delft

• 20+ years of experience in IT in various roles• Currently active as Wizard in the Center of

Expertise Software Development and Control• Mission: make ABN AMRO the leading digital

bank

What’s on the menu?

• Docker Use Cases @ ABN AMRO• ABN-AMRO – current status of CI/CD• The existing CI platform• Challenges and limitations• Vision of the future• The new & improved CI platform• Docker containers everywhere• Pipelines to fit all use cases• Security• What’s next?• Questions and answers / discussion

Docker Use Cases @ABN AMRO

• PR like Dev provisioning – Shift Left• Mocking dependencies• Encapsulate technical debt• Checkpointing and versioning• CICD Pipeline components (masters/agents)

Use cases for CI/CD

5

Produce automated builds and detect errors as soon as possible, by integrating and testing all changes on a regular (daily) basis.

High frequency delivery of a tested functional piece of software that can be deployed to production rapidly.

Fully automated process including deployment to production without human interaction.

Continuous Integration

Continuous Deployment

Continuous Delivery

Many manual handovers and approvals

Long lead time for software delivery

Software quality issues found at a late stage

Code merging happening at a late stage

Inefficient cooperation between DEV and OPS

Big non-frequent releases to Production

It is not only about tooling but mainly mindset & behavior, a changed Way of Working and process improvements.

• Increase maturity of teams • Set up the conditions (tooling, pipelines, generic building blocks) for the teams to get

working.• Train the blocks on applying the right mindset, knowledge and appropriate tooling

We know other large companies which need 3 - 8 years, and changed their approach along the way.

Therefore we keep the overall stages in mind, but plan for the coming three months. Focus on learning and improving instead of long term planning.

CI/CD pipeline orchestration midrange

7

Dependency scan

Check out project from

SCM

Developer triggers build

Build project and execute unit

tests

Code quality scan

Secure coding scan

Publish Deployable

artifact

N

Y

ABN AMRO has introduced a set of quality gates and build breakers. The principle is that the Jenkins build is broken once therequired quality or security is not met and the developer needs to fix the defect in order to proceed. The developer has access to software quality in his IDE so defects can be detected and fixed in an early stage

Standard CI pipelines and buildbreakers

Existing CI platform – Jenkins on VMs• Statistics:

• +/-1500 users• 350+ projects• 10000+ Jenkins jobs

• 1 Jenkins Operation Centre• 10 Jenkins Masters

• 30+ Linux build slaves• 30+ Windows build slaves• 4 OSX build slaves• 25+ HP-fortify (secure coding) slaves

70+ (!!!) VMs in on-prem datacentre…and GROWING…

Challenges and limitations – how to…?• Hard to handle growth of DEV teams.• A lot of static VMs, constantly upscaling needed. • Hard to maintain all the servers.• Server configuration out of sync.• No Docker container support.• No true team autonomy.• A mix of tools and versions on

each build slave.• Innovation is slow.

Five major improvements

1. Empower the CI/CD teams: decentralized maintenance.

2. Docker containers instead of static VMs

3. Support flexibility of tech stacks and configuration.

4. Infrastructure as Code & Configuration as Code.

5. Cloudbees Jenkins Enterprise is critical to the CI/CD program

The new and improved CI platform

AWS

CMS CI

Master Slave

CD

CI

The new and improved CI platform - architecture

Teams can create their own Jenkins master and run their own pipelines.This solution prevents interference of teams with each other. Reduces conflicts.

Context of containers in Jenkins Enterprise

1. Platform2. Running Jenkins jobs3. Build containers4. Application containers

Specific

Generic

Use case: Jenkins Build agents (containers)

Have a proper

solution for the configuration

difficulties

A never ending story…

Pipelines - overview• Q1 2017: Birth of the standard pipelines (STPLs)• Lots of benefits but also challenges• Q1 2018: Birth of the new (Dockerized) pipelines:

• A pipeline for Docker images• A Dockerized pipeline for Java applications

• Easy to use, easy to implement & extend• Security is build-in• A reference for other technologies

Docker image pipeline – main building blocks

16

Smoke test

Jenkinsfile + Dockerfile from

SCM

Developer triggers Docker

image build

Build Docker image

Docker lint syntax check

Docker container dependencies

check

Sign + Publish Docker image in trusted registry

N

YDocker container configuration

check

Apply security profiles

Pipelines – Docker image pipeline

• A pipeline which creates Docker images• That are secure• That are versioned and tested• Which are “official” and “approved”• Ready to re-use by DEV teams

Pipelines – Java pipeline Dockerized

• A pipeline which uses Docker images (building blocks from previous pipeline)

• Create Java artefact• Package in Docker image• Security stages in place• Push to registry• Ready to deploy to (Xlrelease/Xldeploy, AWS, Kubernetes)

Docker Security topics on all levels

Security is needed on every level

Security – why all this?

To avoid compromised containers where-ever they are used: secure business continuity

Security (1): Syntax check

v1.6.2-6-gcfb547a: Pulling from hadolint/hadolintStatus: Downloaded newer image for hadolint/hadolint:v1.6.2-6-gcfb547a/dev/stdin:3 DL3005 Do not use apt-get upgrade or dist-upgrade/dev/stdin:3 DL3009 Delete the apt-get lists after installing something/dev/stdin:4 DL3008 Pin versions in apt get install. Instead of `apt-get install <package>` use `apt-get install <package>=<version>`/dev/stdin:4 DL3015 Avoid additional packages by specifying `--no-install-recommends`Docker lint syntax check (just like SonarQube

Security (2): Anchore

Security (3): Sonatype - Nexus Lifecycle

Security (4): Docker benchmark (OSS)

What’s next - roadmap

CJE to PR

Finish Dockerizedpipelines

Onboard 50 teams this year

Docker runtime scanning

Choose a container runtime on AWS

PoC for a small number of innovative teams

Enterprise based solution for all DEV teams

Questions and Answers

Thank you!

Wiebe de Roos – wiebe.de.roos@nl.abnamro.comJean Paul van Deursen – jean.paul.van.deursen@nl.abnamro.com