elastic build environment

©2013 CloudBees, Inc. All Rights Reserved 1©2013 CloudBees, Inc. All Rights Reserved

Next Step in Automation:Elastic Build EnvironmentKohsuke Kawaguchi / CloudBees, [email protected] / @kohsukekawa

©2013 CloudBees, Inc. All Rights Reserved 2

Have You Met Jenkins? http://jenkins-ci.org/

http://jenkins-ci.org/


32%


18%


My Jenkins around 2006


Workload

http://www.flickr.com/photos/gbyrnes/912576883/


If only we had more computers…

• Just building & testing them all…• Running tests more frequently• Testing individual commits


What you don’t wanthttp://www.flickr.com/photos/drocpsu/8546730021/


Just enough computersjust in time

Elasticity


My Jenkins around 2007


18

But not this, either


Just enough computersof the right kind

just in time

Elasticity!


Correct answer

• Test assumes a fixture running on port 8080– Doesn’t check if it’s already being used

• If another test runs at the same time…?


Correct answer

• Because of “pkill -f -9 tomcat” cleanup


Isolation

http://www.flickr.com/photos/jumilla/8667648797/


Isolation

• At odds with large multi-core systems

• x86 virtual machines• User isolation• Kernel containers


!


Correct answer

• Same Maven ID, two different jars

• Different projects designate different ones• Local cache gets cleaned up periodically

• Whichever first runs after cache cleanup “wins”


Correct answer

• Test script leaves background daemon process behind

• Over time it’ll slowly choke slaves


Throw away & create new

Elasticity!


Ladder to Cloud

Single

MultipleElastic


Parallel Testing

Test Group #1 Test Group #2 Test Group #3


Parallel Testing

foo #10 Test Group #1

foo #11 Test Group #2

foo #12 Test Goup #3


Validated Merge: Jenkins Enterprise by CloudBees

upstreamrepo

gate repo


Recap: Elasticity Benefits

• Just-in-time capacity• Diversity without overhead• Isolation• Productivity gain– parallel testing– validated merge– keep failed tests on the side


Solid OSS Elasticity Plugins

• EC2 plugin• Jclouds plugin– OpenStack, CloudStack

• Launch and tear down slaves on demand

©2013 CloudBees, Inc. All Rights Reserved 40Image © http://sfcitizen.com/blog/wp-content/uploads/2011/11/6302790910_c4eb865892_o-copy.jpg

©2013 CloudBees, Inc. All Rights Reserved

Demo


• Master– Serves HTTP requests– Stores all important info

• Slaves– 280KB single jar– Assumed to be

unreliable– Scale to 100s

Distributed builds with Jenkins

Master

slave1

slave2

slave3

slave4

slave5

slave6

slave7

slave8


Lots More Way To Deploy Slaves

• Good old shell scripting

$ java -jar slave.jar -jnlpUrl URL


How Flexible Does It Get?

• Slave agent is a little Java program– Runs locally on the machine that it controls– Access files, fork processes, etc., on behalf

of master

• Communication with master– Just needs a bi-di byte stream

• No shared file system, no network topology constraints, etc


Static Virtual Machines As Build Slaves

• Allocate lots of identical VMs– Better consistency, lower overhead

• Curve up 1 big machine to small VMs• Throw away > maintain


Labels

• Refer to slavesbut not by name

Name: alphaLabel: windows 32bit

Name: bravoLabel: linux 32bit

Name: charlieLabel: linux 64bit


Labels




This job needs to run on “linux”


Labels




This job needs to run on “linux && 32bit”


More Label Benefits

• Labels can be anything– Geographic locations– Availability of proprietary software

installed

• Better resource utilization– Jenkins have maximum freedom

• Resilient to node addition/removal– Treat servers like livestocks, not like pets


Load Statistics Monitoring

49


“Cloud” plugin


EC2 Plugin

• Label → Amazon Machine Image (AMI)• Integrates with various AWS features– Spot Instance– Identity and Access Management (IAM)– Virtual Private Cloud (VPC)


jclouds plugin

• CloudStack, OpenStack, and many more

• “One-time use” support

Jenkins jclouds cloud


Then Adjust Capacity based on it


Jenkins Enterprise by CloudBees

• VMWare auto-scaling plugin– Snapshot– Power on-off management– Hypervisor-aware scheduling– Folder based pooling– VMWare tools integration– One-time use support


Mansion

Slave Slave

Slave Slave

CloudBees DEV@cloud


Maven

Git

Ant

Mercurial

Gradle

Subversion

Linux Kernel

Hardware

Linux Container = zero cost virtualization


For OS X

Maven

Git

XCode

Git

XCode

Subversion

OS X OS X OS X

QEMU QEMU QEMU

Linux Kernel

Apple Hardware


Kernel Same-page Merging

OS X OS X OS X


Mansion

Slave

Slave

Slave

Workspace 1

Workspace 2

Workspace 3

Workspace 4


Slave

Slave

Workspace

Workspace’’

Workspace’


Slave

Slave

~/.m2/repository

~/.m2/repository

~/.m2/repository


この苦しみから解脱する方法はないかと

• お釈迦様の絵

• 徐々に蓄積してきたノウハウ・ツールを大公開

Elasticity Nirvana?


Testing things in piecesTraditional Approach


Testing the whole is hard

• Database• Load balancer• Messaging layer• Identity service• …


Substitute vs Mock


What you really want = real deal!

Test and production environment should be identical

… which means automating:– Database– Load balancer–Messaging layer– Identity service–…


The Challenge

Infrastructure as Code


The Real Challenge

Elasticity


Single

MultipleElastic

The same evolution ladder

aka PaaS

elastic build environment

Technology