from 0 to 1000 apps: the first year of cloud foundry at the home depot
TRANSCRIPT
From 0 to 1000 AppsThe first year of Pivotal Cloud Foundry @ The Home
Depot from a platform operator’s perspective.
From 0 to 1000 AppsThe first year of Pivotal Cloud Foundry @ The Home
Depot from a platform operator’s perspective.
3100
introductionssetting expectationsreminisce & learnaha momentquestions
Agenda
Well, hi!
15 years
Anthony McCulleyManager, Application PlatformsThe Home Depot anthonymcculley [email protected]
Startups
Platform operator perspective
Being a platform operator
A year-long retrospective
A surprise reveal at the end…
A surprise reveal at the end…
We ready?
Jun 2015
June 2015
?PCF installed on-prem
Jul 2015
July 2015
?
July 2015
Lesson #1
Be empathetic to the people, processes, and technology already in place.
July 2015
2 teams using Pivotal Cloud Foundry (Pivotal Labs)3 foundations (Sandbox, Nonprod, Prod)? App instances
Aug 2015
August 2015
Dev Forum: Introducing PCF
Lesson #2
Let your community know that a thing exists… and real.
August 2015 – PCF is here but has caveats
Platform requires permission to use itPlatform isn’t self-serviceNo idea how to charge customers for it…. so we don’t (yet)1st team goes to production… takes 4.5 hours!
Sep 2015
September 2015
Gave prod access to devs
Prod
Lesson #3
Use new things as an excuse to question everything and do things differently.
September 2015
Enabled dev teams to be Org Managers in ProdDecided on no chargebacks… PCF is “free” to useSeveral teams using their own Slack channels
Oct 2015
October 2015
October 2015
Not much going on…Data center incident impacts Nonprod for 5 daysGreat blameless retro as a resultAttend our first “Cloud Foundry Workshop” with Pivotal
Nov 2015
November 2015
Dev Forum: GitHub, Slack
Lesson #4
Establish a reliable cadence for communicating to community.
November 2015
Slack doubles overnight to 300 peopleWe pilot self-provisioning in Non ProdPivotal (Josh Kruck) asks me to consider running workshopsI give a videotaped talk at a Cloud Foundry meetup
This spawns internal discussions – “Orange Tape”
Dec 2015
December 2015
Make it accessible
December 2015
Josh asks (again) if we considered running workshopsSelf-provisioning added for ProductionWillItConnect madness beginsRemove need to request Org Manager
Lesson #5
Remove all barriers to entry – human or otherwise.
Jan 2016
January 2016
Pupil becomes the master
January 2016
We start running several PCF workshops2x a week, every week for next 4 months800 developers will eventually do workshop
Lesson #6
Go beyond operating the platform – be an advocate, a teacher, and a thought leader.
Feb 2016
February 2016
February 2016
Workshop tweaked and rebrandedRemoved deeper CF topicsAdded organizational change topicsGit, Slack, Full Stack, Paired Programming
My team adopts 100% paired “programming”
Lesson #7
Establish a scalable customer service and support model.
Customer Support
Encourage pair programmingDevs: Ask your pair -> then slack -> then the internet
We dedicate a pair each iteration to 100% customer serviceConversational support.
No ticketsNo queues
Mar 2016
March 2016
March 2016
250 devs go through Modern Development WorkshopHighest single month
We create #pcf-operators slack channelPublic channel for matrixed teams to engageInfrastructure, networking, Pivotal, platform, security
Apr 2016
April 2016
?
April 2016
Production is running out of capacity just as…
”PCF is the strategic platform of choice”
May 2016
May 2016
May 2016
Prod runs out of capacity
April 2016
We turn off self-provisioning in Production
Lesson #8
Don’t run out of capacity.
Lesson #8 (again)
No. Seriously. Don’t run out of capacity.
Get really good at monitoring capacity.
Lesson #9
Be careful about your choice of words.
“Use PCF” changed to “Build 12 Factor Apps”.
Jun 2016
June 2016
June 2016
We stand up 2nd production foundationWe increase capacity on existing foundationAnd then…
Red Alert: 1st Major Incident
Red Alert: 1st Major Incident
Its okay. We are all learning.
Lesson #10
Don’t over react to first incident. Avoid operational scarring.
Takes great leaders!
Lesson #11
Have a dedicated team… dedicated to PCF.
Lesson #12
Don’t hand “support” of platform off to another team.
Jul 2016
July 2016
Today
By the numbers
3000 Apps4000 App Instances5800 people in Slack – over 1100 of them in #cloudfoundry1300 unique platform users6 foundations (more planned EOY)14000 cf push – 1500 to Production
in June alone> 2 billion log messages a month
Lesson #12
Don’t police developers.
Study them with user analytics.
Lesson #13
Based on analytics, single out people and teams to have discussions with.
What’s next?
Motivate teams to be lean with capacity Showback modelquota vs. utilization ratios
Remove all matrixed dependencies for teamsAutomate network and security requests
Community-built smoke testsAdd app tests to our tests on changes
I promised an aha moment!
We don’t “operate a platform”.We enable developers.We are a product team.Developers are our customers.
Thanks! Any questions?