paas automation for mortals - events.static.linuxfound.org · optionality* with mesos invest in a...
TRANSCRIPT
![Page 1: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/1.jpg)
Mesos + Singularity:Mesos + Singularity:PaaS automation for mortalsPaaS automation for mortals
Gregory Chomatas @gchomatas
PaaS team
![Page 2: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/2.jpg)
120 meters: My shortest travel to a Conference120 meters: My shortest travel to a Conference
![Page 3: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/3.jpg)
Thales of Miletus - 624 BC
Those who can, do, the others philosophise...Really?
Miletus
![Page 4: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/4.jpg)
Optionality* with MesosOptionality* with Mesos
Invest in a Mesos-powered PaaS and keepdoing what you love most; building your
product
* Optionality is the property of asymmetric upside (preferably unlimited) w
ith corresponding limited downside (preferably tiny)
![Page 5: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/5.jpg)
BloggingSEO
Social MediaCMS
Lead ManagementLanding PagesCalls-to-Action
Marketing AutomationEmail
AnalyticsCRM Sync
![Page 6: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/6.jpg)
The underlying culture & structureThe underlying culture & structure
12-factor apps .net monolith to microservices Small, autonomous teams withend-to-end ownership - no ops
![Page 7: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/7.jpg)
High productivityHigh productivity
~100 engineers 800+ components that can beupdated/scaled independently QA: ~400 small to mediumAWS machinesPROD: ~750 medium to largeAWS machines
Source: Martin Fowlerhttp://martinfowler.com/bliki/MicroservicePremium.html
![Page 8: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/8.jpg)
1. Develop locally
2. Provision QA aws instance3. Deploy via local Python script
4. Provision PROD aws instance5. Deploy via local Python script
6. Repeat 4 & 5 to scale…10. Repeat 4&5 at 4am to replace hw
But for how long...But for how long...
![Page 9: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/9.jpg)
Statically partitioning the cluster is inefficientStatically partitioning the cluster is inefficient
![Page 10: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/10.jpg)
The cost of flexibility & asynchronicityThe cost of flexibility & asynchronicity
High operational overhead
Poor utilisation & elasticity
Higher rate of failures
![Page 11: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/11.jpg)
Redress the balance with a Mesos-based PaaSRedress the balance with a Mesos-based PaaS
Abstract away machines Homogenous environment Scale out in seconds Centralized deployablesregistry
Sept 2013: Our First Mesos Cluster
![Page 12: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/12.jpg)
To Boldly go...to Singularity
![Page 13: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/13.jpg)
Singularity: do more with a single schedulerSingularity: do more with a single scheduler
Great UI & HTTP API Native Docker Support Health Checks Load Balancing API Log Maintenance
Oct 2013: Start building Singularity
![Page 14: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/14.jpg)
Singularity: do even more...Singularity: do even more...
Security / artifact signatureverification Agent & Rack maintenance Webhooks Auto-rollback Email Notifications Executor cleanup
Singularity Components
![Page 15: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/15.jpg)
The PaaS StackThe PaaS Stack
BUILD DEPLOY RUN
Jenkins Orion Singularity
![Page 16: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/16.jpg)
The Build / Deploy cycleThe Build / Deploy cycle
buildpack runner
S3
![Page 17: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/17.jpg)
The Deployer - Dry runThe Deployer - Dry run
![Page 18: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/18.jpg)
The Deployer - DeployingThe Deployer - Deploying
![Page 19: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/19.jpg)
SingularitySingularity
Deployable viewDeployable view
![Page 20: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/20.jpg)
Singularity Singularity Task viewTask view
![Page 21: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/21.jpg)
Singularity fSingularity file tailingile tailing
![Page 22: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/22.jpg)
Singularity Singularity Health Checks & ResourcesHealth Checks & Resources
![Page 23: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/23.jpg)
Singularity Singularity All Deployables viewAll Deployables view
![Page 24: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/24.jpg)
Singularity Singularity
Cluster Status Cluster Status
![Page 25: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/25.jpg)
SingularitySingularity
Cluster Maintenance viewCluster Maintenance view
![Page 26: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/26.jpg)
Migration to Mesos - TimelineMigration to Mesos - Timeline
![Page 27: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/27.jpg)
Manual Server ProvisioningManual Server Provisioning
![Page 28: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/28.jpg)
Server provisioning UI usageServer provisioning UI usage
![Page 29: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/29.jpg)
1800+ deployables1800+ deployables
~300 deploys / day~300 deploys / day
10 minutes from git10 minutes from gitpush to productionpush to production
![Page 30: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/30.jpg)
I want persistent IMAP connections to 200k+I want persistent IMAP connections to 200k+inboxes (Jul 2015)inboxes (Jul 2015)
![Page 31: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/31.jpg)
![Page 32: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/32.jpg)
Stateful Services Single Process services Hard coded stationary hosts Cgroups memory isolation User resistance
Migration IssuesMigration Issues
![Page 33: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/33.jpg)
All eggs in one basket Mesos / Framework issues (pingbackport) Failures (Zookeeper, Mesos, Singularity) Cluster Maintenance Missing features
Operational IssuesOperational Issues
![Page 34: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/34.jpg)
Phased rollout of new Kernel, Instancetypes Rolling upgrade of instance basicSW with puppet vars
MaintenanceMaintenance
Rolling upgrade of master/agent process with ansible Local testing on docker cluster Roll out at infra-QA then product-QA and last to Productioncluster Deploy tools deploy themselves but maintain command linealternative with fabric
![Page 35: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/35.jpg)
Optionality with Mesos@HubSpotOptionality with Mesos@HubSpot
Singularity
Ghidorah - Load Balancers in Mesos
Massive Builds in Mesos
Baragon - Tasks Load Balancer Manager
Mesos Spark Cluster
![Page 36: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/36.jpg)
The sweet spotThe sweet spot
Source: Mark Leslie (http://firstround.com/review/The-Arc-of-Company-Life-and-How-to-Prolong-It/)
![Page 37: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/37.jpg)
Invest early in a deploy & build infrastructureInvest early in a deploy & build infrastructure
Dedicate 1-2 engineers to experiment onDedicate 1-2 engineers to experiment ona Mesos powered PaaSa Mesos powered PaaS
Try Singularity today!Try Singularity today!github.com/HubSpot/Singularity
![Page 38: PaaS automation for mortals - events.static.linuxfound.org · Optionality* with Mesos Invest in a Mesos-powered PaaS and keep doing what you love most; building your product * Optionality](https://reader036.vdocuments.net/reader036/viewer/2022071007/5fc4de16e6587e39cc604c44/html5/thumbnails/38.jpg)
HubSpot Blog: How We Built Our Stack For Shipping at Scale
Blazar: An out-of-this world build system!
Baragon: Load Balancer API
Useful linksUseful links