managing jenkins with jenkins (jenkins user conference palo alto, 2013)
DESCRIPTION
How we use system Groovy scripts to automate housekeeping and monitoring of our Jenkins instances.TRANSCRIPT
Tweet @garethbowles with feedback!
Managing Jenkins with Jenkins
Monday, October 28, 13Hi, I’m Gareth from Netflix. It’s great to be back for my third JUC !I’m going to talk about how we use Jenkins’ System Groovy Scripts feature to automate the management of our Jenkins infrastructure.I hope you’ll get inspired by some of these ideas to try them out yourselves. As this is a short talk there may not be much time for questions, but find me afterwards or tweet me if you want to follow up.
Tweet @garethbowles with feedback!
• 3200 jobs
• 3TB build data
• 6 masters
• ~100 slaves / ~250 executors
• 15% build failure rate
• All 700 engineers can set up / run jobs
@
Monday, October 28, 13We have a pretty big Jenkins setup, it’s the centrepiece of the pipeline that builds our streaming service.Our main master has 3200 builds, 3TB of build data dataWe have a growing number of masters as we split jobs between different masters for speed and reliabilityHousekeeping and monitoring has to be automated due to this scale. We need to make sure our slaves are online, disk space for builds is kept under control, builds run on the right slaves, and be able to update the configuration of many jobs or slaves at a time.And because all our engineers have job configuration access, we need to make sure that some basic conventions are followed and nothing gets too out of control.
Tweet @garethbowles with feedback!
How can we automate all the Jenkins things ?
Monday, October 28, 13Jenkins has quite a few options for automation.
Tweet @garethbowles with feedback!
How can we automate all the Jenkins things ?
• Remote API - nice, but limited update functions
Monday, October 28, 13Jenkins has quite a few options for automation.
Tweet @garethbowles with feedback!
How can we automate all the Jenkins things ?
• Remote API - nice, but limited update functions
• Jenkins CLI - ditto
Monday, October 28, 13Jenkins has quite a few options for automation.
Tweet @garethbowles with feedback!
How can we automate all the Jenkins things ?
• Remote API - nice, but limited update functions
• Jenkins CLI - ditto
• Custom plugins - powerful but heavyweight
Monday, October 28, 13Jenkins has quite a few options for automation.
Tweet @garethbowles with feedback!
A Better Option
Monday, October 28, 13Here’s a more lightweight but still very powerful way to automate Jenkins tasks.
Tweet @garethbowles with feedback!
System Groovy Scripts
Monday, October 28, 13Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain.You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example).Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
Tweet @garethbowles with feedback!
System Groovy Scripts
• Full access to the Jenkins master’s JVM
Monday, October 28, 13Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain.You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example).Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
Tweet @garethbowles with feedback!
System Groovy Scripts
• Full access to the Jenkins master’s JVM
• Much quicker than writing a plugin
Monday, October 28, 13Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain.You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example).Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
Tweet @garethbowles with feedback!
System Groovy Scripts
• Full access to the Jenkins master’s JVM
• Much quicker than writing a plugin
• Make use of Jenkins' job scheduling, logging, notification and data visualization
Monday, October 28, 13Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain.You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example).Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
Tweet @garethbowles with feedback!
System Groovy Scripts
• Full access to the Jenkins master’s JVM
• Much quicker than writing a plugin
• Make use of Jenkins' job scheduling, logging, notification and data visualization
• Needs the Groovy plugin: https://wiki.jenkins-ci.org/display/JENKINS/Groovy+plugin
Monday, October 28, 13Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain.You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example).Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
Tweet @garethbowles with feedback!
Examples
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
Examples• Disable old or broken jobs
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
Examples• Disable old or broken jobs
• Relabel a set of slaves
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
Examples• Disable old or broken jobs
• Relabel a set of slaves
• Monitor slave uptime
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
Examples• Disable old or broken jobs
• Relabel a set of slaves
• Monitor slave uptime
• Disconnect bad slaves
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
Examples• Disable old or broken jobs
• Relabel a set of slaves
• Monitor slave uptime
• Disconnect bad slaves
• Set number of builds to keep
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
Examples• Disable old or broken jobs
• Relabel a set of slaves
• Monitor slave uptime
• Disconnect bad slaves
• Set number of builds to keep
• Check build labels
Monday, October 28, 13Here are a few examples of the work we do with system Groovy scripts.If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected.We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast).We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
Tweet @garethbowles with feedback!
How do I figure out what to write ?
Monday, October 28, 13Figuring out where to start writing a script can be tricky, but there are lots of resources available.
Tweet @garethbowles with feedback!
Resources
Monday, October 28, 13There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.We wrote a simple object inspector method that shows you what type of object you’re working with.
Tweet @garethbowles with feedback!
Resources • Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler+Plugin) or GitHub repo (https://github.com/jenkinsci/jenkins-scripts)
Monday, October 28, 13There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.We wrote a simple object inspector method that shows you what type of object you’re working with.
Tweet @garethbowles with feedback!
Resources • Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler+Plugin) or GitHub repo (https://github.com/jenkinsci/jenkins-scripts)
• Script console in Jenkins - be careful !
Monday, October 28, 13There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.We wrote a simple object inspector method that shows you what type of object you’re working with.
Tweet @garethbowles with feedback!
Resources • Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler+Plugin) or GitHub repo (https://github.com/jenkinsci/jenkins-scripts)
• Script console in Jenkins - be careful !
• API JavaDoc: http://javadoc.jenkins-ci.org/overview-summary.html
Monday, October 28, 13There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.We wrote a simple object inspector method that shows you what type of object you’re working with.
Tweet @garethbowles with feedback!
Resources • Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler+Plugin) or GitHub repo (https://github.com/jenkinsci/jenkins-scripts)
• Script console in Jenkins - be careful !
• API JavaDoc: http://javadoc.jenkins-ci.org/overview-summary.html
• Object inspector method
Monday, October 28, 13There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.We wrote a simple object inspector method that shows you what type of object you’re working with.
Tweet @garethbowles with feedback!
Resources • Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler+Plugin) or GitHub repo (https://github.com/jenkinsci/jenkins-scripts)
• Script console in Jenkins - be careful !
• API JavaDoc: http://javadoc.jenkins-ci.org/overview-summary.html
• Object inspector method
Monday, October 28, 13There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.We wrote a simple object inspector method that shows you what type of object you’re working with.
Tweet @garethbowles with feedback!
Object Inspector Method
Monday, October 28, 13Here’s the handy object inspector method that display an object’s type, properties and methods. Once yuo know the type you can look in the JavaDoc or search existing code to find out how to do what you want.
Tweet @garethbowles with feedback!
Be careful about testing !
Monday, October 28, 13System Groovy scripts can do some serious damage ! You often want to loop over every job or build or slave, so be careful to test your code.I learned the hard way when I accidentally kicked off every job on our main Jenkins master. We’ve made a few other mistakes too.
Tweet @garethbowles with feedback!
Monday, October 28, 13Here’s another one where we accidentally disabled all the jobs in the system and then had to figure out how to identify the jobs that should really be disabled. We keep this one around as a cautionary tale.
Tweet @garethbowles with feedback!
Developing scripts
Monday, October 28, 13Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets !Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
Tweet @garethbowles with feedback!
Developing scripts
• Use a local Jenkins instance (java -jar jenkins.war)
Monday, October 28, 13Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets !Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
Tweet @garethbowles with feedback!
Developing scripts
• Use a local Jenkins instance (java -jar jenkins.war)
• Use the script console or inline code in the job config. while developing
Monday, October 28, 13Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets !Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
Tweet @garethbowles with feedback!
Developing scripts
• Use a local Jenkins instance (java -jar jenkins.war)
• Use the script console or inline code in the job config. while developing
• Code editor or IDE for longer scripts
Monday, October 28, 13Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets !Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
Tweet @garethbowles with feedback!
Developing scripts
• Use a local Jenkins instance (java -jar jenkins.war)
• Use the script console or inline code in the job config. while developing
• Code editor or IDE for longer scripts
• Check the script into SCM when it’s ready
Monday, October 28, 13Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets !Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
Tweet @garethbowles with feedback!
Awesome ! Where can I get this stuff ?
• https://github.com/jenkinsci/jenkins-scripts
• http://scriptlerweb.appspot.com/catalog/list
Monday, October 28, 13Here’s where you can get lots of System Groovy script goodness. I just added a bunch of our scripts that I mentioned in this talk.
Tweet @garethbowles with feedback!
Jenkins User Conference Palo Alto , Oct 23 2013 #jenkinsconf
Thank You To Our Sponsors Platinum
Gold
Silver
Monday, October 28, 13
Tweet @garethbowles with feedback!
Thank You !Email: gbowles@{gmail,netflix}.com
Twitter: @garethbowles
Linkedin: www.linkedin.com/in/garethbowles
Want to do Jenkins development for your day job ? We’re hiring ! http://jobs.netflix.com
Monday, October 28, 13Here’s how to get in touch.
We’re hiring for lots of positions, but in particular we’re looking for someone to own Jenkins at Netflix - all the way from keeping the infrastructure running to writing custom plugins and fixing the Jenkins core. If you want to take Jenkins in new directions as your full time job - with a heavy dose of cloud thrown in - come and find a Netflixer.
thanks again