Effecting a DevOps Transformation at Salesforce.comAgile 2013
Dave MangotArchitect, Infrastructure Engineering
@davemangot
in/dmangot
Karthik RajanVP, Infrastructure Engineering
@kabbyr
in/karthikrajan
Safe Harbor
Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
Karthik Rajan
Responsible for the infrastructure platform for running the salesforce.com service -
“Infrastructure as Code”
5 years @ salesforce.com, in various roles - developer to management
Photographer. Petrolhead.
Collector of many many gadgets
Dave Mangot
• Mainly focused on automation, monitoring, orchestration and lastly...cultural change!
• Systems Engineer 15+ years, a variety of companies big (Cable & Wireless, Sun Microsystems) and small (Tagged.com, Terracotta)
The 5 Why’s:
• Why are we the same? Different?• Why didn’t we transform TechOps on the first try?
• Why are we succeeding?• Why we still have challenges
• Why are you here?
Why are we the same? Why are we different?
http://flic.kr/p/eeWbP8 http://flic.kr/p/eeWbGp
A little story...
It was the year 2000
Number of people
fast innovativesmart
Number of Major Releases per year
7 years later
rapid success
59,300+ Customers
10 B transactions/Quarter
tons of people
it was getting more difficult to deliver
Features Delivered per Team
Days between Major Releases
Number of Major Releases in 2006
2007 – Birth of ADM
Major R&D wide Agile Transformation to ADM
On time delivery?
Last waterfall release
But what about infrastructure?
More importantly, here’s a story of a typical start-up
We’ve got an app…
We’ve got a customer or two…
Web
DB
LogsConf
Data
Stat
Hlth
ID
App
Othr
AppLogsConf
Data
Stat
Hlth
ID
App
Othr
LogsConf
Data
Stat
Hlth
ID
App
Othr
Customer
Admin
I’ve got a few more customers…
DB LogsConf
Data
Stat
Hlth
ID
App
Othr
FW
LBLB
FW
Now we’re making money…
DB LogsConf
Data
Stat
Hlth
ID
App
Othr
FW
LBLB
FWFW
FWFW
FW
LBLB
LBLB
DB LogsConf
Data
Stat
Hlth
ID
App
Othr
Salesforce.com Architecture
Why didn’t we transform TechOps on the first try?
Delivering on business priorities
Scaling through hiring
Still a relatively small team
But thenRapid success continues
100,000+ Customers
1 B transactions/day
tons and tons and tons people
So can we continue to
innovate at scale?
Lack of visibility
http://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Morning_Fog_at_GGB.JPG/448px-Morning_Fog_at_GGB.JPG
Resource Bottlenecks
Lack of responsiveness, lack of team alignment on priorities
Unpredictable completion of projects and initiatives
2012 – Infrastructure Take 2 ADM + DevOps
Take 2a
Developers supporting Operations (part time)
Take 2b
Infrastructure Engineering
TechOps splits into
Operations
Take 2c
Clouds
Embedded teams
How to use DevOps?
No Crisis
The 3 Ways – Gene Kim
The First Way emphasizes the performance of the entire system
The Second Way is about creating the right to left feedback loops.
The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure;
C A M S
Culture
Automation
http://upload.wikimedia.org/wikipedia/commons/6/61/Differential_free.png
Metrics
http://www.aosabook.org/images/graphite/composer-ui.png
Sharing
Why are we succeeding?
The Toyota Way - Kata
http://upload.wikimedia.org/wikipedia/commons/7/70/Iaido2.jpg
ADM for Cultural Change - DevOps Kata
•Daily Standup - #8 “Encourage effective two way communication and other means to drive out fear throughout the organization so that everybody may work effectively and more productively for the company.”
•Sprint Retrospective - # 13 “Institute a vigorous program of education and self-improvement”
•Sprint Demo - #9 “Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.”
Cloud Realignment
http://www.freefoto.com/images/46/01/46_01_49---Clouds_web.jpg
Theory of Constraints as per Goldratt
http://images4.wikia.nocookie.net/__cb20080623095705/starwars/it/images/thumb/d/d4/JawaNEGAS.jpg/331px-JawaNEGAS.jpg
Hack Day
Internal DevOps Mini-Conference
http://www.flickr.com/photos/gazeronly/7645165306/
Infrastructure Development Lifecycle
http://www.freefoto.com/images/46/01/46_01_49---Clouds_web.jpg
Artifact Y Deployed to Environment X from release store
Test Results marked in ReleaseDB
Eligible forpromotionto next environment
Environment specific tests are Run(increasing clock time)
PASS?
Ejected fromprocessFAIL?
Testing and Deployment
Virtualization
Embeds
http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Embedded_Artillery_Shell_at_Fort_Sumter_%287639234388%29.jpg/800px-Embedded_Artillery_Shell_at_Fort_Sumter_%287639234388%29.jpg
Scrum Master Training
Failure? Still?
http://theintentionalwriter.com/wp-content/uploads/2013/01/head_in_hands.jpg
Front Door Process
http://upload.wikimedia.org/wikipedia/commons/thumb/e/ef/Alnwick_Castle_a_frontal_door.JPG/450px-Alnwick_Castle_a_frontal_door.JPG
Bugs
http://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Cotton_Harlequin_Bugs.jpg/757px-Cotton_Harlequin_Bugs.jpg
Why we still have challenges
Why are you here?
Present and Future Challenges
• Bringing Agile into traditional IT Ops
• Bringing IT Ops in with Infra Eng and R&D
• Re-educating workforce
• Recruiting
• Scaling Securely