how to train your jenkins?
DESCRIPTION
Jenkins User Conference - Effective utilization of Jenkins resources by building a mesos clusterTRANSCRIPT
![Page 1: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/1.jpg)
Jenkins User Conference San Francisco #jenkinsconf
How to train your Jenkins? … to Use Your Resources Effectively and Get More Value for Each Buck Spent on Resources
Manivannan Selvaraj & Soma OrugantieBay Inc.ww.ebay.com
October 23, 2014
#jenkinsconf
![Page 2: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/2.jpg)
Jenkins User Conference San Francisco #jenkinsconf
At eBay Inc. we take CI very seriously
![Page 3: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/3.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Background- CI in PayPal
• One Ginormous Jenkins Instance, customized for PayPal
• 40,000 runnable jobs
• 12,000 builds are run each day
• Java, Node, Python, and C++ stacks, Fortify and Freestyle jobs too
• Unit test and deploy flows
• One Jenkins master with a pool of a hundred Jenkins slaves
• Specialized “Monster Machine” stages build 40,000,000 lines of
code in a half hour
![Page 4: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/4.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Background- CI in eBay Marketplaces
• Individual VMs running Jenkins for each Application
• 2500+ VMs in use
• Each Jenkins would have 2 executor on master to perform builds
Jenkins CIWorkspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
e
Jenkins CIWorkspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
e
Jenkins CIWorkspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
e
Jenkins CIWorkspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
e
Jenkins CIWorkspac
eJenkins CI
Workspac
eJenkins CI
Workspac
eJenkins CI
Workspac
e
![Page 5: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/5.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Individual Jenkins VMsPros
Can add their own plug-ins
Can upgrade to various versions of Jenkins
Issues are isolated to one instance
Cons
Poor resource utilization - < 10%
Cost of maintaining 2500+ VMs – Millions of $ every year
Provisioning VMs takes several minutes
Individual instances have local storage of limited size
CI’s can disappear, losing all data and build history
![Page 6: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/6.jpg)
Jenkins User Conference San Francisco #jenkinsconf
New CI Architecture - Challenges
• Put a smile on Dev-Ops
• Optimal utilization of resources
• Preserve user experience
• High availability
• Self healing infrastructure
![Page 7: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/7.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Light weight Jenkins
…
Apache Mesos Cloud
Light weight Jenkins for each user
Developer 1 Developer 2 Developer n
0.4 CPU 512 MB mem
![Page 8: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/8.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Mesos Architecture
![Page 9: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/9.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Build slaves - Jenkins Mesos Plugin
• Jenkins Mesos plugin community is very active
• We have contributed enhancements/patches to the plugin
![Page 10: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/10.jpg)
Jenkins User Conference San Francisco #jenkinsconf
CIAAS
Marathon
Mesos Master
Mesos Cluster
NFS
Request for CI
Zookeeper
Jenkins Master Jenkins Slave
CI Application life cycle
![Page 11: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/11.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Jenkins Mesos Plugin - our Contributions
• Custom workspace• Multiple flavors of slaves• Access to workspace• Enable Maven jobs• Disconnect of slaves after idle timeout
![Page 12: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/12.jpg)
Jenkins User Conference San Francisco #jenkinsconf
![Page 13: How to train your Jenkins?](https://reader036.vdocuments.net/reader036/viewer/2022062419/55936d321a28ab78138b4760/html5/thumbnails/13.jpg)
Jenkins User Conference San Francisco #jenkinsconf
Thank You To Our Sponsors
Platinum Gold
Silver Corporate