tap into the power of slaves with jenkins by kohsuke kawaguchi
TRANSCRIPT
![Page 1: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/1.jpg)
©2010 CloudBees, Inc. All Rights Reserved
©2010 CloudBees, Inc. All Rights Reserved
Tap into the Power of Slaves with Jenkins
Kohsuke Kawaguchi
![Page 2: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/2.jpg)
©2010 CloudBees, Inc. All Rights Reserved
2 ©2010 CloudBees, Inc. All Rights Reserved
Have You Met Jenkins? h"p://jenkins-‐ci.org/
![Page 3: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/3.jpg)
©2010 CloudBees, Inc. All Rights Reserved
3 ©2010 CloudBees, Inc. All Rights Reserved
x 64,000
![Page 4: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/4.jpg)
©2010 CloudBees, Inc. All Rights Reserved
4 ©2010 CloudBees, Inc. All Rights Reserved
![Page 5: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/5.jpg)
©2010 CloudBees, Inc. All Rights Reserved
5 ©2010 CloudBees, Inc. All Rights Reserved
x 700
![Page 6: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/6.jpg)
©2010 CloudBees, Inc. All Rights Reserved
6 ©2010 CloudBees, Inc. All Rights Reserved
![Page 7: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/7.jpg)
©2010 CloudBees, Inc. All Rights Reserved
7 ©2010 CloudBees, Inc. All Rights Reserved
![Page 8: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/8.jpg)
©2010 CloudBees, Inc. All Rights Reserved
8 ©2010 CloudBees, Inc. All Rights Reserved
![Page 9: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/9.jpg)
©2010 CloudBees, Inc. All Rights Reserved
9 ©2010 CloudBees, Inc. All Rights Reserved
![Page 10: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/10.jpg)
©2010 CloudBees, Inc. All Rights Reserved
10 ©2010 CloudBees, Inc. All Rights Reserved
![Page 11: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/11.jpg)
©2010 CloudBees, Inc. All Rights Reserved
11 ©2010 CloudBees, Inc. All Rights Reserved
© Kurt Löwenstein EducaConal Center InternaConal Team h"p://commons.wikimedia.org/wiki/File:DiscriminaCon-‐Diversity-‐03.jpg
CreaCve Commons A"ribuCon 2.0 Generic
![Page 12: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/12.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Demo Really, We Made It Easy!
![Page 13: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/13.jpg)
©2010 CloudBees, Inc. All Rights Reserved
13 ©2010 CloudBees, Inc. All Rights Reserved
• Master – Serves HTTP requests – Stores all important
info
• Slaves – 280KB single jar – Assumed to be
unreliable – Scale to at least 100
Distributed builds with Jenkins
Master
slave1
slave2
slave3
slave4
slave5
slave6
slave7
slave8
![Page 14: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/14.jpg)
©2010 CloudBees, Inc. All Rights Reserved
14 ©2010 CloudBees, Inc. All Rights Reserved
Lots More Ways to Deploy Slaves
• Via DCOM
![Page 15: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/15.jpg)
©2010 CloudBees, Inc. All Rights Reserved
15 ©2010 CloudBees, Inc. All Rights Reserved
Lots More Way To Deploy Slaves
• Good old shell scripting
$ java -jar slave.jar -jnlpUrl URL
![Page 16: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/16.jpg)
©2010 CloudBees, Inc. All Rights Reserved
16 ©2010 CloudBees, Inc. All Rights Reserved
How Flexible Does It Get?
• Slave agent is a little Java program – Runs locally on the machine that it controls – Access files, fork processes, etc., on behalf of
master
• Communication with master – Just needs a bi-di byte stream
• No shared file system, no network topology constraints, etc
![Page 17: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/17.jpg)
©2010 CloudBees, Inc. All Rights Reserved
17 ©2010 CloudBees, Inc. All Rights Reserved
Static Virtual Machines As Build Slaves
• Allocate lots of identical VMs – Better consistency, lower overhead
• Curve up 1 big machine to small VMs • Throw away > maintain
![Page 18: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/18.jpg)
©2010 CloudBees, Inc. All Rights Reserved
18 ©2010 CloudBees, Inc. All Rights Reserved
Labels
• Refer to slaves but not by name
Name: alpha Label: windows 32bit
Name: bravo Label: linux 32bit
Name: charlie Label: linux 64bit
![Page 19: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/19.jpg)
©2010 CloudBees, Inc. All Rights Reserved
19 ©2010 CloudBees, Inc. All Rights Reserved
Labels
Name: alpha Label: windows 32bit
Name: bravo Label: linux 32bit
Name: charlie Label: linux 64bit
This job needs to run on “linux”
![Page 20: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/20.jpg)
©2010 CloudBees, Inc. All Rights Reserved
20 ©2010 CloudBees, Inc. All Rights Reserved
Labels
Name: alpha Label: windows 32bit
Name: bravo Label: linux 32bit
Name: charlie Label: linux 64bit
This job needs to run on “linux && 32bit”
![Page 21: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/21.jpg)
©2010 CloudBees, Inc. All Rights Reserved
21 ©2010 CloudBees, Inc. All Rights Reserved
More Label Benefits
• Labels can be anything – Geographic locations – Availability of proprietary software installed
• Better resource utilization – Jenkins have maximum freedom
• Resilient to node addition/removal – Treat servers like livestocks, not like pets
![Page 22: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/22.jpg)
©2010 CloudBees, Inc. All Rights Reserved
22 ©2010 CloudBees, Inc. All Rights Reserved
Labels ↓
ElasCc Slaves
![Page 23: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/23.jpg)
©2010 CloudBees, Inc. All Rights Reserved
23 ©2010 CloudBees, Inc. All Rights Reserved
Tracking demand for specific label
![Page 24: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/24.jpg)
©2010 CloudBees, Inc. All Rights Reserved
24 ©2010 CloudBees, Inc. All Rights Reserved
Then Adjust Capacity based on it
![Page 25: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/25.jpg)
©2010 CloudBees, Inc. All Rights Reserved
25 ©2010 CloudBees, Inc. All Rights Reserved
How?
• EC2 Plugin – Longest history, most popular
• JClouds plugin – For OpenStack & CloudStack users, mainly – Unfortunate name, if you ask me
• Common to both – Multiple image types
![Page 26: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/26.jpg)
©2010 CloudBees, Inc. All Rights Reserved
26 ©2010 CloudBees, Inc. All Rights Reserved
Interesting Use Cases Worth Noting
• Single-use slaves (JClouds) – Used once & destroyed
• Static but utility-priced slaves (EC2) – Jenkins stops idle slaves to reduce $$$ – Great for infrequent use
• Coming soon: spot instances (EC2)
![Page 27: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/27.jpg)
©2010 CloudBees, Inc. All Rights Reserved
27 ©2010 CloudBees, Inc. All Rights Reserved
CloudBees DEV@cloud
• Completely elastic slaves • Keeps your warm workspace • Per-minute pricing, not per-hour
![Page 28: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/28.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Now that we have more slaves…
#1: parallel test execution
![Page 29: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/29.jpg)
©2010 CloudBees, Inc. All Rights Reserved
29 ©2010 CloudBees, Inc. All Rights Reserved
Running Tests in Parallel
Why?
![Page 30: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/30.jpg)
©2010 CloudBees, Inc. All Rights Reserved
30 ©2010 CloudBees, Inc. All Rights Reserved
Pre-requisites
Build script that supports test exclusion file
![Page 31: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/31.jpg)
©2010 CloudBees, Inc. All Rights Reserved
31 ©2010 CloudBees, Inc. All Rights Reserved
How It Works
Test Group #1 Test Group #2 Test Group #3
![Page 32: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/32.jpg)
©2010 CloudBees, Inc. All Rights Reserved
32 ©2010 CloudBees, Inc. All Rights Reserved
How It Works
foo #10 Test Group #1
foo #11 Test Group #2
foo #12 Test Goup #3
![Page 33: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/33.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Demo
![Page 34: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/34.jpg)
©2010 CloudBees, Inc. All Rights Reserved
34 ©2010 CloudBees, Inc. All Rights Reserved
For Maven
• Trick – Turn a build into a Maven repository – Maven profiles for controlling what to run
when
Build Parallel Test
Test Group #1
Test Group #2
Test Group #3
![Page 35: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/35.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Demo
![Page 36: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/36.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Now that we have more slaves…
#2: activities than span multiple computers
![Page 37: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/37.jpg)
©2010 CloudBees, Inc. All Rights Reserved
37 ©2010 CloudBees, Inc. All Rights Reserved
Build Flow Plugin
• Groovy DSL for coordinating job executions – High-level primitives – Ability to define abstractions
b = build(“acme-‐build”) guard {
parallel ( { build(“acme-‐test1”, param1:b.number) }, { build(“acme-‐test2”, param1:b.number) } )
} rescue { build(“acme-‐teardown”)
}
![Page 38: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/38.jpg)
©2010 CloudBees, Inc. All Rights Reserved
38 ©2010 CloudBees, Inc. All Rights Reserved
• Embed BPMN workflow engine in Jenkins – Timeout, fork, join, …
• Workflow is version controlled in Git – Push to Jenkins to load them up
Jenkow Plugin
![Page 39: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/39.jpg)
©2010 CloudBees, Inc. All Rights Reserved
39 ©2010 CloudBees, Inc. All Rights Reserved
Lease and SSH access
• Lease Jenkins executors for interactive SSH access
Remote terminal access plugin
![Page 40: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/40.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Demo
![Page 41: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/41.jpg)
©2010 CloudBees, Inc. All Rights Reserved
41 ©2010 CloudBees, Inc. All Rights Reserved
Lease and SSH access: characteristics
• Easy coordination between nodes • Jenkins scheduling aware • No dependency to individual slaves
![Page 42: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/42.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Now that we have more slaves…
#3: validated merge
![Page 43: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/43.jpg)
©2010 CloudBees, Inc. All Rights Reserved
43 ©2010 CloudBees, Inc. All Rights Reserved
Did this ever happen to you?
![Page 44: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/44.jpg)
©2010 CloudBees, Inc. All Rights Reserved
44 ©2010 CloudBees, Inc. All Rights Reserved
Commit ↓ Push ↓ Test
![Page 45: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/45.jpg)
©2010 CloudBees, Inc. All Rights Reserved
45 ©2010 CloudBees, Inc. All Rights Reserved
Commit ↓ Test ↓ Push
![Page 46: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/46.jpg)
©2010 CloudBees, Inc. All Rights Reserved
46 ©2010 CloudBees, Inc. All Rights Reserved
More concretely
• You push to Jenkins • Jenkins merges it with upstream • Jenkins tests it • If good, Jenkins pushes it upstream
upstream repo
gate repo
![Page 47: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/47.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Demo
![Page 48: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/48.jpg)
©2010 CloudBees, Inc. All Rights Reserved
48 ©2010 CloudBees, Inc. All Rights Reserved
Conclusions
• Tame more slaves • Lots of interesting things you can do
![Page 49: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/49.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Now that we have more slaves…
#4: matrix projects
![Page 50: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/50.jpg)
©2010 CloudBees, Inc. All Rights Reserved
50 ©2010 CloudBees, Inc. All Rights Reserved
Motivation
• You often do the same thing with slight variations – Compile C++ code for different platforms – Test with different browsers – Or more generally, think of it as
for (x in [a,b,c]) { for (y in [d,e,f]) { doSomethingWith(x,y,z,..); } }
![Page 51: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/51.jpg)
©2010 CloudBees, Inc. All Rights Reserved
51 ©2010 CloudBees, Inc. All Rights Reserved
Model
• Define axes – One axis ≈ one for loop
• Choose from pre-defined types of Axis – Slave axis: pick slaves by their names or their
labels • e.g., linux, solaris, and windows
– Generic axis: arbitrary values exposed as environment variables
![Page 52: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/52.jpg)
©2010 CloudBees, Inc. All Rights Reserved
52 ©2010 CloudBees, Inc. All Rights Reserved
Matrix Project Gimmicks
• Filtering – Otherwise combinations increase exponentially – Not all combinations make sense – Use boolean expression to trim down the size
– Or tell Jenkins to cut the workload to N% • Jenkins will thin out the combinations by itself
(label==“windows”).implies(browser==“iexplore”) && (label==“mac”).implies(browser==“safari”)
![Page 53: Tap into the power of slaves with Jenkins by Kohsuke Kawaguchi](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f3e43b4c90572088b514d/html5/thumbnails/53.jpg)
©2010 CloudBees, Inc. All Rights Reserved
Demo