CI/CD @ bol.com

What I’ll be talking about

1. About me & bol.com2. The CI/CD story @ bol.com3. Current setup4. Mayfly5. The future in the cloud

About me

● Maarten Dirkse● @mdirkse on Twitter● In IT since 2007 (5 years @ bol.com)● Java developer -> CI/CD engineer● Bitten by the container bug in 2014● Hobby: local politics

About bol.com

● Largest online retailer in the Netherlands and Belgium (5.8 million customers, 10+ million products)

● 55 (and growing) multi-disciplinary teams of 5-8 people

● Strong Scrum culture (introduced in 2009)

● 200+ services and apps (SOA, mostly Java + DB backend)

● Mix of fixed sprint rhythm and CD

The Developer Freedom index

Once upon a build-time.....

The situation ca. 2014

● 4 week release cycle● Big-bang release

○ Shop went offline!

● Scrum -> 200 stories per cycle to production● Jenkins -> DeployIT -> Schuberg Phillis

○ Software was “thrown over the wall” to ops

● Every team had admin rights on their jenkins jobs

Freedom index: not free

● Developers couldn’t do their own releases at the time of their choosing● Releases had to be coordinated with SBP

○ Even some property changes

● Stack on which apps ran was tightly locked down● Developers could go crazy on TST, but could do almost nothing on PRO

○ Endless requests for SSH access to servers which were inevitably denied

● 2 levels of gatekeeping: ops and SBP

● On the plus side, they could configure their Jenkins jobs...

Current pipeline

our CD story “Man on the Moon” to give teams autonomy

How things get to production

Build Store Orchestrate Deploy

Run

Key aspects

● TAXP system: custom abstraction over Jenkins jobs○ No more job admin rights for teams

● Teams can deploy to PRO at will (have to send notification)● TST, ACC environments (ACC is “production-like”, used for performance

tests)● No change management process● SRT gatekeeper of deploy functionality and new services

Mayfly

Genesis of Mayfly

Test

Acc

Pro

<master> <master>

<master>

<master>

Mayfly idea

Pro

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

<branch>

.......... .....

Mayfly provides per user story:

● Feature branch in SCM (currently git via Stash)● Continuous integration jobs (Jenkins)● Isolated, production-like runtime environment (Docker cluster)● Automated Definition of Done check● Logs & metrics (Logstash, Graphite, Prometheus)● Optional user story-specific database (Oracle, PostgreSQL, Mongo)

30% of all commitsdone via Mayfly

Freedom index: partially free

● Developers control their releases● Developers don’t control CI or CD● Mayfly offers lots of freedom, until TST

Building and deploying in the cloud

The challenge

Build a container-centric, cloud-native CI/CD pipeline that:

● Is easy to use and get started with● Makes it easy to deploy small changes● Is fully customizable● Can scale to thousands of deploys per

day

What about the current stack?

Will the CI/CD stack that we use at the moment suffice in the cloud?

The current stack

JenkinsDo builds

Artifactory RundeckStore artifacts Orchestration of RPM

builds and rolling out of artifacts

PuppetActually install the new artifact on an existing

machine

Maintenance nightmare

Very expensive

docker registry

*Not* a deployment

tool

Want immutable

infrastructure

The current stack would work, but we can do better by using cloud- native tools

The new stack

Gitlab CIBuild docker images

Google Container Registry

Spinnaker

Store docker imagesDeploy dockerized apps

KubernetesRun docker containers

More developer control & less DPI

maintenanceGoogle’s concern Actual deployment

toolFacilitates immutable

infrastructure

Opt-outis an option!

Convention over configuration

Convincing over compulsion

CI/CD is a product that needs to appeal

Iterate on a vision, don’t crowdsource the design

Freedom index: free

● Developers have full control over CI● Developers have full control over CD● Developers have full control over the stack

○ Well, at least from the kernel up

● Constraints that do exist are, as much as possible, handled transparently● And if opt-out is an option, but comes with many responsibilities

Maarten Dirkse

Thanks!