gartner starting and scaling dev ops
TRANSCRIPT
Starting and Scaling DevOps at Enterprise — Lessons Learned
Template version: 8/29/2012
‹#›
Tapabrata “Topo” PalDirector, Individual Contributor, DevOps Evangelist
[email protected] @TopoPal
Past: • PhD in Semiconductor Physics• Over 20 years of IT experience as Developer, Architect,
System Engineer• Experience in Retail, Healthcare and Finance industries
Relative to our peers, we are a start-up
Most of you know Capital One is a Credit Card company. We are one of the largest in the US with over 70 million accounts. Many know that we are also one of the nations largest banks. Fewer, however, realize that we are a Founder led 20 years old Technology Company.
Our “youngest” competitor is 108 years oldWe respect our elders…
But we disrupted the credit industry with our leader’s vision….
“Every single person in the US had the same credit card.
It made absolutely no sense.”
Rich FairbankCapital One Founder & CEO
Information Based Strategy (IBS)
Data Technology Data Science
Test & Learn
+ +
How we fulfilled on that vision was truly innovativeWe used data, technology, and data science to execute an Information Based Strategy:… design products based on customer's needs, passions, and life stages – not “one size fits all”… make adjustments to products or presentation and see the impact in the data… predict business results before full-scale market deployment… we were “doing” Big-Data before even the name Big-Data came into the picture.
The Digital Revolution
Today our vision is to change banking for good. We are he nation’s largest Digital Bank. Digital is the new Bank branch. What is striking is the degree at which Mobile has become the preferred channel for Banking.
"Ultimately the winners in banking will have the capabilities of a
world-class software company.“
Rich FairbankCapital One Founder & CEO
As Rich Fairbank said: Ultimately the winners in banking will have the capabilities of a world-class software company.
About Five Years Ago…
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise.. Mostly out sourced.. 100% Waterfall We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, … Build, Deploy, Test was all manual… Our success rate was quite high - even though we lacked the speed.
About Five Years Ago…
• Mostly outsourced
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise.. Mostly out sourced.. 100% Waterfall We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, … Build, Deploy, Test was all manual… Our success rate was quite high - even though we lacked the speed.
About Five Years Ago…
• Mostly outsourced• Waterfall
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise.. Mostly out sourced.. 100% Waterfall We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, … Build, Deploy, Test was all manual… Our success rate was quite high - even though we lacked the speed.
About Five Years Ago…
• Mostly outsourced• Waterfall• Quarterly Releases
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise.. Mostly out sourced.. 100% Waterfall We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, … Build, Deploy, Test was all manual… Our success rate was quite high - even though we lacked the speed.
About Five Years Ago…
• Mostly outsourced• Waterfall• Quarterly Releases• Manual Processes
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise.. Mostly out sourced.. 100% Waterfall We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, … Build, Deploy, Test was all manual… Our success rate was quite high - even though we lacked the speed.
First Proof of Concept
Our first proof of concept was for a Java based project. We used Subversion for source control, Hudson for CI, Maven as build tool and Nexus as binary repository.
Seed of “DevOps”
That Proof of concept was like a small camp fire… Within a few months, we were able to show the improvement - from days to minutes - literally. Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile. You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey. Before moving this forward, we went to the drawing board and started creating a strategy around it.
Seed of “DevOps”
That Proof of concept was like a small camp fire… Within a few months, we were able to show the improvement - from days to minutes - literally. Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile. You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey. Before moving this forward, we went to the drawing board and started creating a strategy around it.
Seed of “DevOps”
That Proof of concept was like a small camp fire… Within a few months, we were able to show the improvement - from days to minutes - literally. Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile. You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey. Before moving this forward, we went to the drawing board and started creating a strategy around it.
Seed of “DevOps”
That Proof of concept was like a small camp fire… Within a few months, we were able to show the improvement - from days to minutes - literally. Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile. You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey. Before moving this forward, we went to the drawing board and started creating a strategy around it.
BYOD
Build Your Own DevOps
‹#›
Development • Architecture • Design • Code • Test
As far as our understanding of DevOps goes.. Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace. But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them, how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are serious about security, we needed to move security also closer to the development - DevOpsSec.
‹#›
Business • Requirements • Feature Request • Roadmap
Development • Architecture • Design • Code • Test
As far as our understanding of DevOps goes.. Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace. But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them, how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are serious about security, we needed to move security also closer to the development - DevOpsSec.
‹#›
Business • Requirements • Feature Request • Roadmap
Development • Architecture • Design • Code • Test
Operations • Infrastructure • Platforms • Environment • Deployment • Incident Mgmt • Change & Release Mgmt.
As far as our understanding of DevOps goes.. Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace. But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them, how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are serious about security, we needed to move security also closer to the development - DevOpsSec.
‹#›
Business • Requirements • Feature Request • Roadmap
Development • Architecture • Design • Code • Test
Operations • Infrastructure • Platforms • Environment • Deployment • Incident Mgmt • Change & Release Mgmt.
Information SecurityApplication Security Security Testing Information Security Infrastructure Security
As far as our understanding of DevOps goes.. Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace. But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them, how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are serious about security, we needed to move security also closer to the development - DevOpsSec.
‹#›
Business • Requirements • Feature Request • Roadmap
Development • Architecture • Design • Code • Test
Operations • Infrastructure • Platforms • Environment • Deployment • Incident Mgmt • Change & Release Mgmt.
Information SecurityApplication Security Security Testing Information Security Infrastructure Security
DevOpsSec
As far as our understanding of DevOps goes.. Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace. But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them, how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are serious about security, we needed to move security also closer to the development - DevOpsSec.
Three pillars of DevOpsSec
We based our DevOpsSec transformation on three major pillars..
Three pillars of DevOpsSec
Automate Everything
We based our DevOpsSec transformation on three major pillars..
Three pillars of DevOpsSec
Automate EverythingShift-Left
We based our DevOpsSec transformation on three major pillars..
Three pillars of DevOpsSec
Automate EverythingShift-LeftDashboard Everything
We based our DevOpsSec transformation on three major pillars..
Automate Everything - a.k.a “Tools Explosion”
This is a huge pain point of DevOps - tools explosion. We started with just three tools and now we have a whole bunch of tools. And this is just going to be like this for a long time to come!
Code Build Release MonitorDeploy + Test Execution
All these tools are to create a continuous delivery pipeline 1. Everything as code 2. Security and compliance at every step 3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?
Code Build Release MonitorDeploy + Test Execution
App
Test
Infra
All these tools are to create a continuous delivery pipeline 1. Everything as code 2. Security and compliance at every step 3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?
Code Build Release MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD
All these tools are to create a continuous delivery pipeline 1. Everything as code 2. Security and compliance at every step 3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?
Code Build Release MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD Infra
App
All these tools are to create a continuous delivery pipeline 1. Everything as code 2. Security and compliance at every step 3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?
Code Build Release MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD Infra
App
Flow
All these tools are to create a continuous delivery pipeline 1. Everything as code 2. Security and compliance at every step 3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?
Code Build Release MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD Infra
App
Flow Feedback
All these tools are to create a continuous delivery pipeline 1. Everything as code 2. Security and compliance at every step 3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?
Building a server in a Datacenter
Steps = 62Days = 60Cost = $25kPain = @#$%^&*Opportunity cost = ?????
Who has not felt the pain in getting a server in a data center? We did! All our developers did, all our Operators did, everyone involved in getting a server felt various levels of pain, frustration… Plus it became a serious bottleneck in the pipeline.
We had been running our Dev and QA Environments on public cloud for a while. This year we have put some of our critical production load on public cloud. We have also put containerized application… and we created a lot of the tooling around that ourselves.
Shift-Left: Mindset Shift
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same team can implement the whole pipeline from the get go.
Shift-Left: Mindset Shift
Test AutomationSecurity Scan
Environment BuildSoftware Build Setup
Deployment SetupProduction Support
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same team can implement the whole pipeline from the get go.
Shift-Left: Mindset Shift
Test AutomationSecurity Scan
Environment BuildSoftware Build Setup
Deployment SetupProduction Support
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same team can implement the whole pipeline from the get go.
Shift-Left: Mindset Shift
Test AutomationSecurity Scan
Environment BuildSoftware Build Setup
Deployment SetupProduction Support
PlanningArchitecture
DesignCoding
Unit Testing
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same team can implement the whole pipeline from the get go.
http://github.com/capitalone/Hygieia
Dashboard Everything
Dashboard everything… Hygieia DevOps Dashboard… We built it ourselves and we Open Sourced it. This product has taken off since its launch last summer and has own the “Open Source Rookies of the Year 2015” award.
Hygieia has a team level dashboard where you can visualize the teams workitems, coding activities, build status, static code analysis, unit test, security scan, functional test results, deployment status across environments - with drill downs to get more details. It also has a “product” level dashboard..
Where you can see a roll up view of all teams dashboards. You can track every commit from commit stage to production and see quality trends. This helps us determining where the bottlenecks are… so that the teams can start the discussion as to how to remove the bottlenecks.
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
23
How do we scale
How do we do this at a large scale? Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site. Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s subject can be anything - tools, process,
24
Our First ever Software Engineering Conference
2 days
1,200+ attendees
13 Learning Tracks
28 Tech Expo Booths
52 Sessions
0 Vendor
Our mission was to promote the culture of sharing, collaborating and building relationship; and also let people learn something new. Attendees came from Richmond and McLean, VA; San Francisco, New York, Chicago, Texas, Seattle, UK.
25
Key Learnings
25
Key Learnings
Start small, prove success, learn from mistakes
25
Key Learnings
Start small, prove success, learn from mistakesExecutive Support is a must
25
Key Learnings
Start small, prove success, learn from mistakesExecutive Support is a mustEmpower your engineers
25
Key Learnings
Start small, prove success, learn from mistakesExecutive Support is a mustEmpower your engineersNo tools discussion
25
Key Learnings
Start small, prove success, learn from mistakesExecutive Support is a mustEmpower your engineersNo tools discussionEmbrace Open Source and Open Technologies
25
Key Learnings
Start small, prove success, learn from mistakesExecutive Support is a mustEmpower your engineersNo tools discussionEmbrace Open Source and Open TechnologiesShare, share, share…
25
Key Learnings
Start small, prove success, learn from mistakesExecutive Support is a mustEmpower your engineersNo tools discussionEmbrace Open Source and Open TechnologiesShare, share, share…It is endless!