Real World DevOpsRodrigo Campos (@xinu)
Velocity NY 2014
Hi I’m Rodrigo!
• Proudly crimping cables since 1992
• System Administrator
• Operations Director at walmart.com in Latin America
I’m not going to talk about
• Silver Bullets
• Unicorns
• Tools and Utilities
Instead I’ll focus on
• The challenges we faced
• How DevOps has helped us
• Bumps and bruises along the way
Our challenges
Our challenges
• Deploy a whole new platform before Black Friday
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
• Develop a company-wide agile mindset
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
• Develop a company-wide agile mindset
• Rebuild trust between tech and business
Different companies, same problems
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
• Lots of bureaucracy and red-tape
Some phrases (roughly translated)
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
• “This is operations’ job”
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
• “This is operations’ job”
• “Consider this a favor”
How DevOps was seen
So what is DevOps anyway?
So what is DevOps anyway?
• Being loosely defined can actually be a good thing!
So what is DevOps anyway?
• Being loosely defined can actually be a good thing!
• For us: “DevOps is a culture that fosters communication, transparency and collaboration"
So what is DevOps anyway?
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
• Transparency: I want to know your problems, I’ll make sure you know mine!
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
• Transparency: I want to know your problems, I’ll make sure you know mine!
• Collaboration: We’re in the same boat and I really mean it!
Operations Developers
UsThem InfoSec
Works on my Machine
We needed to change people's mindset
Source: http://youtu.be/Fx8OBeNmaWw
We needed to change people's mindset
We needed to change people's mindset
• Presentations to technology, business and human capital teams
We needed to change people's mindset
• Presentations to technology, business and human capital teams
• Reinforcing an inclusive and welcoming culture was essential
We needed to change people's mindset
• Presentations to technology, business and human capital teams
• Reinforcing an inclusive and welcoming culture was essential
• Everyone felt prepared and willing to change
We needed to change people's mindset
Typical Scenario
Infrastructure
Operations
Developers
Business
{ }
• Things go over the wall
• No alignment between tech and business
• One-way communications
• IT as a necessary evil (aka cost center)
We were already a little bit different…
Infrastructure
Operations
Developers
Business
{ }
DevOps Team
1st step
Infrastructure
Operations
Developers
Business
{ }Production Engineering
DevOps
2nd step
Infrastructure
Production ManagementDevelopers
Business
{ }
Production Engineering + { }
DevOps
InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
• Integrate security tests into continuous integration
InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
• Integrate security tests into continuous integration
• Rugged DevOps: http://slidesha.re/1r87Vs7
InfoSec
• No more FUD
• What is riskier to the business?
• Lack of speed is a risk too!
InfoSec
• ITIL: “You keep using that word, I don’t think it means what you think it means”
• People often blame ITIL as an excuse for badly designed processes
• Surprisingly ITIL compliance was not our biggest problem
Compliance
• Audits are still our biggest challenge
• In our case, we have a sponsor (the CTO)
• Someone (who happens to be me) will be held accountable if something goes wrong
Compliance
The bad apples
• We’ve adopted the “No asshole rule"
• There will be obstructive people that won’t embrace change
• If you can't get rid of them, simply let them fade into oblivion
KPIs
KPIs
• Deployment Frequency
• Speed of Deployment
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
• CultureSource: Puppet Labs Blog (http://bit.ly/Z3gRpu)
Bumps & Bruises
Bumps & Bruises• DO:
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
• Work hard on a sponsor
Bumps & Bruises
Bumps & Bruises• DON'T:
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
• Underestimate the managers' ability to influence behavior
• In the beginning it will be a lonely fight
• Some people will label you as the “hippie-anarchist”
Keep in mind
Results
Results• New eCommerce platform successfully deployed
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
• Increase in deployment frequency and success rate
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
• Increase in deployment frequency and success rate
• Business is incorporating agile methodologies
Any questions?
• Special thanks to: https://www.flickr.com/photos/51321237@N07/ https://www.flickr.com/photos/kh-67/ https://www.flickr.com/photos/jrparis/ https://www.flickr.com/photos/andrewdubock/ https://www.flickr.com/photos/rachelyoung/ !https://bitly.com/bundles/o_4p62vc3lid/5
