a tale of two pipelines: to devops or not to devops
TRANSCRIPT
A Tale of Two Pipelines: To DevOps or Not To DevOps
Brett HoferGlobal DevOpsPractice Lead
25+ years of IT experience
Dynatrace• Recognized thought leader DevOps & Big Data• Featured speaker, presenter, writer - DevOps.com,
DevOpsSummit Journal, InfoWorld and many more• Author eBook – “The Art of DevOps”
Prior to Dynatrace• Senior app manager for 80M+ budget, staff of 70+
including onshore/offshore• Lead designer and developer of multi-platform
apps, multi-channel intranet/internet web • Senior Solutions Architect on 60M+ initiative(s)
Welcome to“A Tale of Two
(very different) Pipelines”
It’s about people,
processes and technologies!
Are you sinking, surfing, or just
surviving?
Our idea here will be to help identify bad situations by
relating to all or part of the two stories …
We’ll aim to ignite ideas of positive
change by sharing experiences through
these stories.
A story about one fictional organization struggling
with all too familiar pipeline
challenges and how it took a corporate catastrophe to
finally decide on transitioning to solid DevOps practices.
Which pipeline will look like yours?Maybe somewhere in the middle?
What can you do to influence change?
What exactly are we moving through the pipeline?
Servicing requests from business ideas, non functional needs, fixes, functionality adjustments
Servicing requests from business ideas, non functional needs, fixes, functionality adjustments
Servicing requests from business ideas, non functional needs, fixes, functionality adjustments
Not every change is
created equal!
Success equals the RIGHT change at the RIGHT time
• Precision in priority ensures all efforts are focused on what team has agreed is success!
• Getting people to agree can be tough.
• Key: invest in “best-of-breed” technology to track, collaborate, and continually monitor FACTS versus OPINIONS when prioritizing
Typical top priority factors
• Value• Cost• Risk• Impact• Time
Add continuous monitoring to deliver continuous improvements
• Find opportunities and improve value
• Identify / manage cost factors
• Reduce and mitigate risk
• Identify and manage impacts
• Find problems fast and reduce time, costs and risk
All the right people know the health status of a change and build as it moves through the pipe
• Test and monitor early and often• Find and fix problems fast and early
To DevOps or not to DevOps
Fingerscrossed…
Welcome to easyTravel
Debute latest online booking system
• Development over a year• Lots of internal jokes
“neverTravel”• Ops “hope they load tested”• Late and over budget• App manager changed twice• Tension between business
and dev stalled budget twice
Mark SmoltenChief Marketing Officer
• 2 years at easyTravel• Other successful roles driving biz up
30%• Challenged by
• channels against fierce competition• convincing exec management to
invest in new, unproven outlets• not enough info on current
platform use and state• not being tied into new system in
dev
• Mark contacts TravelVision and lines up a spot on their wildly popular live TV Show “Book Me Now!”
• Mark offers viewers an amazing deal on a live TV promo code in exchange for debuting the new easyTravel site
• Live show November 7th, 8:00 p.m.
• Yay, Mark!
TravelVision’s BookMeNow!
Tina AntonVP, eCommerce
• 8 years at easyTravel, 15 years industry• Promoted to head of eCommerce
two years ago• More experienced in the travel
business than eCommerce• Oversees the overall vision of every
easyTravel eBusiness function• Maintains online budgeting decisions• Very friendly, but long history of
battling it out with IT
Mark: Enthusiastic | Tina: Concerned
Mark speaks with Tinaabout his Live Promo Deal with TravelVision.
He explains the code should affect pricing only for the duration of the show and only for those who provide the code issued during live broadcasting.
Mark: Enthusiastic | Tina: Concerned
Mark speaks with Tinaabout his Live Promo Deal with TravelVision.
He explains the code should affect pricing only for the duration of the show and only for those who provide the code issued during live broadcasting.
Tina mentions that they don’t have such a feature on the site and wishes he had consulted with her first. She asks him “when is this?”
Mark: Enthusiastic | Tina: Concerned
Mark speaks with Tinaabout his Live Promo Deal with TravelVision.
He explains the code should affect pricing only for the duration of the show and only for those who provide the code issued during live broadcasting.
Tina mentions that they don’t have such a feature on the site and wishes he had consulted with her first. She asks him “when is this?”
“We air on live TV to show off the new site and the promo on November 7th at 8pm! It’s the only time until next fall that they will have this event.”
Mark: Enthusiastic | Tina: Concerned
Mark speaks with Tinaabout his Live Promo Deal with TravelVision.
He explains the code should affect pricing only for the duration of the show and only for those who provide the code issued during live broadcasting.
Tina mentions that they don’t have such a feature on the site and wishes he had consulted with her first. She asks him “when is this?”
“We air on live TV to show off the new site and the promo on November 7th at 8pm! It’s the only time until next fall that they will have this event.”
Tina, very apprehensive, explains “but that’s less than two months away! We don’t even have a promo feature on the site since all our travel packages are already discounted.”
Mark: Enthusiastic | Tina: Concerned
Mark states “This is the business nowadays. We have to adapt quickly to capture market opportunities. The discounts will come from the marketing budget. We need to make this happen!”
Mark: Enthusiastic | Tina: Concerned
Mark states “This is the business nowadays. We have to adapt quickly to capture market opportunities. The discounts will come from the marketing budget. We need to make this happen!”
Tina, continues to voice her concerns trying to explain that there is a big change process for something like this.
Mark: Enthusiastic | Tina: Concerned
Mark states “This is the business nowadays. We have to adapt quickly to capture market opportunities. The discounts will come from the marketing budget. We need to make this happen!”
Tina, continues to voice her concerns trying to explain that there is a big change process for something like this.
“Mark, things are also more complicated these days due to all the channels we support. This would need to be available on Web, Web Mobile and in our native Mobile apps.”
Rick TrentonSenior Project Manager
• Consultant with the company one year• Ex-military very structured and organized• Assigned to the project management of
easyTravel Web and mobile • Brokers budgets and costs between
business and IT for major deliverables • Challenged with producing weekly
project statuses to management in PowerPoint
• Regularly maintains high level tasks in his Microsoft Project Plan
Tim FolkDirector of Digital Services
• 5 years at easyTravel• Former Senior Solutions Architect • Inherited responsibility 4 months ago • Not a fan of his Management Admin
Tasks• Manages all the Development of Web
& mobile channels for easyTravel• Challenged with lack of time, too many
meetings, putting out fires• He manages a local smaller IT staff but
required to use at least 70% offshore staffing for his dev to keep costs down
Mark: Enthusiastic | Tina: Concerned
Tina now contacts both Rick and Tim to explain this new Feature that must be ready and tested in all channels by November 7th. Less than two months away.
Mark: Enthusiastic | Tina: Concerned
Tina now contacts both Rick and Tim to explain this new Feature that must be ready and tested in all channels by November 7th. Less than two months away.
Rick’s PM contract is funded by the business so he is very appeasing to Tina while Tim is drowning in numerous problems. Tim explains “I’m worried about the viability of the entire new site by November never mind this new feature.”
Mark: Enthusiastic | Tina: Concerned
Tina now contacts both Rick and Tim to explain this new Feature that must be ready and tested in all channels by November 7th. Less than two months away.
Rick’s PM contract is funded by the business so he is very appeasing to Tina while Tim is drowning in numerous problems. Tim explains “I’m worried about the viability of the entire new site by November never mind this new feature.”
“Well you guys know the marching orders. I have confidence you will get this done. Keep me posted on the progress and take care.”
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
Tim contacts three key members
critical to getting this
done.
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
Meet Frank BozmanDigital Services Operations LeadeasyTravel
Kelly explains her understanding of the feature
and it’s urgency.
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
Meet Frank BozmanDigital Services Operations LeadeasyTravel
“Ok Kelly, we are going to need to get this fully documented and exactly how it should work.”
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Hey Tim, I’m going to need to put this on a completely
separate code branch and we
will need another environment. It’s going to impact too many things to put in the Oct
release.”
To understand Tim’s concerns,let’s look at the app pipeline he owns …
Fingerscrossed…
Tale of Two Pipelinesby Brett Hofer
Senior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Hey Tim, I’m going to need to put this on a completely
separate branch and we will need
another environment. It’s going to impact too many things
Scott’s Traditional Reasoning on incorporating this change PROMOFEATURECHANGE
Source Control
easyTravel Master CODE Branch v2.0
easyTravel CODE Branch v2.2 (Oct)
easyTravel CODE Branch v2.5 (Jan)
easyTravel CODE Branch v3.0 (Mar)
Managed VM Infrastructure
DEV1vm
DEV2vm
DEV3vm
DEV4 (ER)vm
QA1vm
QA2vm
QA3vm
QA4 (ER)vm
UATvm
PERFvm
PROD Avms
PROD Bvms
mergeHosts 2.2 build
Hosts 2.5 build
Hosts 3.0 build
Hosts Fix Builds
Hosts 2.2 build
Hosts 2.5 build
Hosts 3.0 build
Hosts Fix BuildseasyTravel CODE Branch v2.3 (Nov)
promotions
PROMO Only Dev related Code
Tale of Two Pipelinesby Brett Hofer
FEATURECHANGE
PROMOMeet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Haha, we have about as much
chance of getting new machines provisioned in
that timeframe as I have in getting a decent build from Scott.”
Tale of Two Pipelinesby Brett Hofer
FEATURECHANGE
PROMOMeet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Hey Frank, it might be easier if the machines
you get me weren’t as slow as you. I’m
tempted to have our guys bring in
their home machines and
deploy to those.”
Tale of Two Pipelinesby Brett Hofer
FEATURECHANGE
PROMOMeet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Honestly, if I put in a ticket to provision new VMs with the infrastructure department it
will have mold on it by November.”
Tale of Two Pipelinesby Brett Hofer
Senior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Hey Tim, I’m going to need to put this on a completely
separate branch and we will need
another environment. It’s going to impact too many things
Frank’s Traditional Ops Reasoning on procuring new VMsPROMOFEATURECHANGE
Managed VM Infrastructure
DEV1vm
DEV2vm
DEV3vm
DEV4 (ER)vm
QA1vm
QA2vm
QA3vm
QA4 (ER)vm
UATvm
PERFvm
PROD Avms
PROD Bvms
Hosts 2.2 build
Hosts 2.5 build
Hosts 3.0 build
Hosts Fix Builds
Hosts 2.2 build
Hosts 2.5 build
Hosts 3.0 build
Hosts Fix Builds
promotions
PROMO
Enter a Dev 5 Provisioning Ticket
Enter a QA 5 Provisioning Ticket
Outsourced Company Managing Environments locates frame & Builds Hostaccording to security, OS, CPU, Memory.
Outsourced Company Managing Environments locates frame & Builds Hostaccording to security, OS, CPU, Memory.
Franks staff will manually setupthe DEV environment for deploying Nov
Franks staff will manually setupthe QA environment for deploying Nov
DEV5vm
Hosts Fix Builds
QA5vm
Hosts 2.3 PromoLead Time to identify, provision and get cost contract – 30 to 60 days
Tale of Two Pipelinesby Brett Hofer
FEATURECHANGE
PROMOMeet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Guys, maybe we should just
concentrate on finding a
solution that will get us
there. This needs to get done.
Kelly let us know when you have our requirements.”
Tale of Two Pipelinesby Brett Hofer
FEATURECHANGE
PROMOMeet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Wow, it sounds like you guys have your work cut out and on
that note, thanks Tim, we will have your requirements in a few days”
Now let’s evaluate
this feature as it moves
through their
pipeline.
The good news – the pipeline is somewhat agile
Plan
Design
DevelopTest
Operate
The bad news – all the cycles are long!3 month cycles now require an interim
releasePlan
Design
DevelopTest
Operate
ScoreC-
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
CMO Recognizes Major Opportunity Initiates Need
FEATURECHANGE
• Industry is fast past and always fluid.
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
High Level Business Functionality
• All Channel must change• Actually 3 Pipelines affected
• Mechanics of how it will work
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
Separately Feature tracking ++Risk, +Time, +Cost • Dev uses tracking software, but no
other groups do
• Frank adds the task to his trackingspreadsheet
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
DesignTesting
Strategy
GatherTime &
Cost
Track on Overall
Plan
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
DesignTesting
Strategy
GatherTime &
Cost
Track on Overall
Plan
Meet Raheeb Patel Senior Quality Assurance LeadeasyTravel Contractor
- His company contracted by QA department- Follows his QA dept and corporate guidelines- Needs very specific detailed technical specs
to put his team in motion- Runs a large team of offshore manual testers
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
DesignTesting
Strategy
GatherTime &
Cost
Track on Overall
Plan
Design Feature Impact Assessment - +++RISK, +++TIME, +++COST• Done conversationally among
a few technical resources• Frequent inaccuracy and guessing.
Has major cascading effects on capacity, architecture, scheduling
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
DesignTesting
Strategy
GatherTime &
Cost
Track on Overall
Plan
Architecture of Feature - +++RISK, +++TIME, +++COST• Done conversationally among
architects • Many know the general architecture
but aren’t up to speed on all of the detailed changes over each release.
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emailsBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
StoryBoard
Identify FeatureImpacts
DesignTesting
Strategy
GatherTime &
Cost
Track on Overall
Plan
Capacity Planning - +++RISK, +++TIME, +++COST• Lacks comprehensive understanding
of site traffic and visits• Lacks comprehensive understanding
of scalability concerns
CapacityDesign
Plan
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
PROMO
Test
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
Using Offshore Development - +RISK, ++TIME, +COST • Timezone diff often = long response
times for fix & feedback cycles• Lack of Backend System Support
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
Developers on varying Platforms - +RISK, +TIME • Not developing on the same
technical stack as intended deploytargets - introduces unnecessaryrisks
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
Overuse of Merging - +RISK, +TIME, +COST • Increased manual effort on changes
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
Too many premature Check-Ins - ++RISK, ++TIME, ++COST
• No static policy checking on dev code • No runtime policy checking on dev code• No code review on critical functions • No architectural validation
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
Manual Run Builds - +RISK, +TIME
• Often requires contacting one person• Limited Visibility and Feedback of
build results
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
Slow Build Time - +TIME • Build is taking over 50 minutes
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Diagnose
Dev4
9% Low and Limited Unit Coverage - ++RISK, -VALUE • Limited Visibility and Feedback
• Places high/all reliance on functional testing – Which is manual!!!
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
PROMO
Dev4
Limited Result Visibility - +RISK, +TIME • Only one or two people
• Manual skimming of build logs • Often too much promoted “thrown
over the wall”
Diagnose
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
Manual Changes to Environments - +RISK, -VALUE• Prone to cross environment errors• Manual adjustments forgotten and
often not repeated as necessary• Static environments get dirty• Kills innovation speed
PROMO
CheckResults
Dev4
Diagnose
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
Test
Manual Deployments in Development - +TIME, -VALUE• More manual delays in promotion
PROMO
Dev4
CheckResults
Diagnose
Tale of Two Pipelinesby Brett Hofer
Dev
Communication spreadsheets/meetings/phone/emails
Dev
Dev
Ops
FEATURECHANGE
Code Check-In RunBuild
Manual Dev Env Config
ManualDeploy
CheckResults
ManualMerge
Unit Test
Test Prod
Test
ers
ManuallyPrepare
Test Data
DocumentTest Scripts
Manual Stage Env
Config
Manual Prod Env
Config
PROMO
Test
Dev4Dev4
Diagnose
Scott’s Dev Visibility & Status Sharing
• Few actually know the build health in Dev• Few actually know common Dev problems • Too little screened • Lacks macro level metrics
Tale of Two Pipelinesby Brett Hofer
Dev
spreadsheets/meetings/phone/emails
Dev
Dev
Ops
Test Prod
Test
ers
DetermineTest Suite
Manual DeployQA 5
Prod
FEATURECHANGE
PROMO
Manual Tests
EnterProblems
Manual Deploy
Perform
Manual Tests
ReportProblems
Validate Build
Validate Build
DiagnoseRoot
Cause
Check Infrastruct
ure
Manual Deploy
UAT
Tale of Two Pipelinesby Brett Hofer
Dev
spreadsheets/meetings/phone/emails
Dev
Dev
Ops
Test Prod
Test
ers
DetermineTest Suite
Manual DeployQA 4
Prod
FEATURECHANGE
PROMO
Manual Tests
EnterProblems
Manual Deploy
Perform
Manual Tests
ReportProblems
Validate Build
Validate Build
DiagnoseRoot
Cause
Check Infrastruct
ure
Manual Deploy
UAT
Manual Test Selection - ++TIME, --VALUE, ++COST
• Selecting a lot more manual testing than what the change requires stifles new changes and costs time
• Manual Testing Greatly slows feedback time.
Tale of Two Pipelinesby Brett Hofer
Dev
spreadsheets/meetings/phone/emails
Dev
Dev
Ops
Test Prod
Test
ers
DetermineTest Suite
Manual DeployQA 4
Prod
FEATURECHANGE
PROMO
Manual Tests
EnterProblems
Manual Deploy
Perform
Manual Tests
ReportProblems
Validate Build
DiagnoseRoot
Cause
Check Infrastruct
ure
Manual Deploy
UAT
MTTR SLOW - ++TIME, --VALUE, ++COST • Test usually reported as
just failed with confusing description
• Finding exact issue is hard• Wasted Time - Finger
Pointing• Unnecessary people
involved in failure
Validate Build
Almost 2 months later
. . .
Now it’s prime time
November 7 at 8:10 p.m.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Hi everyone, I’m Lisa Goodheart, your host of
Book Me Now!
Tale of Two Pipelinesby Brett Hofer Before DevOps
Today is our Big Deal day where once a year we
showcase a select travel service live allowing them to put their best foot forward
demonstrating great deals to our viewers.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Here with me is Eric Wallace the VP of Sales
from easyTravel. He has a special treat for you
today.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Thanks Lisa, I’m honored to be here and to show your audience what we
believe is the most exciting travel site on the
market.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Our business negotiates some of the most
incredible packages for the cheapest prices and
then bundles them.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Today we will demonstrate an absolute exclusive for this
show. If your audience jumps on our site either web or on their phone and uses this one time PROMO code.
Your Book Me Bo
Tale of Two Pipelinesby Brett Hofer Before DevOps
Your Book Me Now Exclusive Live Promo
It will randomly discount our packages between 5% to 95% upon searching and
if you begin to book it upon that search you can
book that deal.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Your Book Me Now Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Bo
Thank you Eric. This is really exciting. Our
audience will now notice the Promo Code scrolling
on the bottom of their screen.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Let’s demonstrate
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
Tale of Two Pipelinesby Brett Hofer Before DevOps
We put in the Promo
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Tale of Two Pipelinesby Brett Hofer Before DevOps
Where we want to take our Journey
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
Tale of Two Pipelinesby Brett Hofer Before DevOps
When we want to go…
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Tale of Two Pipelinesby Brett Hofer Before DevOps
Now click search…
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
20 seconds later . . .
Tale of Two Pipelinesby Brett Hofer Before DevOps
Um this is unusually long…
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Tale of Two Pipelinesby Brett Hofer Before DevOps
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Wow, your site must be working pretty
hard on those deals.
Tale of Two Pipelinesby Brett Hofer Before DevOps
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Yes, it’s chugging away to find your
viewers the deals… is it
possible it’s your internet here?
Tale of Two Pipelinesby Brett Hofer Before DevOps
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Tale of Two Pipelinesby Brett Hofer Before DevOps
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
And… that doesn’t look good. Maybe a
commercial?
Tale of Two Pipelinesby Brett Hofer Before DevOps
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Well folks we are sorry but it looks like easyTravel is running
into a technical difficulty. We will be
back after these messages…
Tale of Two Pipelinesby Brett Hofer Before DevOps
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Off Air
Tale of Two Pipelinesby Brett Hofer Before DevOps
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Well that was a complete waste of
time and money. You people need to get your act together.
Off Air
Tale of Two Pipelinesby Brett Hofer Before DevOps
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Off AirYou’re not kidding, heads will roll for this one. That
was embarrassing and expensive.
Tale of Two Pipelinesby Brett Hofer Before DevOps
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
And that had nothing to do with our
internet.
Off Air
Tina’s house . . .
Are you kidding me!
And the War Room begins . . .
• Not only was all the money spent on the feature a loss but it turned out the 60% booking growth was a long term growth strategy for Mark the CMO.
• The actual event traffic was 1600% of peak.
• An estimated $4M in new net bookings was lost.
• Losses due to brand damage unquantifiable.
Now let’s walk through this same scenario in a DevOps culture . . .
Let’s see the story when the promo
feature is sent through their new DevOps
pipeline.
The good news – the pipeline IS Agile
2 Week Sprints Plan
Design
DevelopTest
Operate
The new Digital Services EnvironmentPipeline Transparency, Proximity & Availability Matters
Development Test Production
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Hey Tim, I’m going to need to put this on a completely
separate branch and we will need
another environment. It’s going to impact too many things
The DevOps Reasoning on incorporating this change (Use a feature toggle)
Source Control
easyTravel Release
Ops Infrastructure As Code
Performance and Dashboards
Infrastructure Cloud ServicesAvailable in Dev/Test &
SECURED Prod
PROMO
Dev Source Code
Test Scripts
CI_CD Configs
CLO
UD
API
LOAD BALANCE POLICY
easyTravelInstance
Tech StackImage for 2.0
Tech StackImage for 3.0
Tech StackImage for 3.5
PROMOFEATURECHANGE
Cloud Image Library
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina
Meet Scott DuttonDigital Services Dev LeadeasyTravel
- Works for Tim
Meet Frank BozmanDigital Services Operations LeadeasyTravel
- Works for Tim
“Hey Tim, I’m going to need to put this on a completely
separate branch and we will need
another environment. It’s going to impact too many things
The DevOps Reasoning on incorporating this change (Use a feature toggle)
Source Control
easyTravel Release
Ops Infrastructure As Code
Performance and Dashboards
Infrastructure Cloud ServicesAvailable in Dev/Test &
SECURED Prod
PROMO
Dev Source Code
Test Scripts
CI_CD Configs
CLO
UD
API
LOAD BALANCE POLICY
easyTravelInstance
Tech StackImage for 2.0
Tech StackImage for 3.0
Tech StackImage for 3.5
PROMOFEATURECHANGE
Cloud Image Library
Aligning All Release Source Code, Automation Scripts, Database Scripting,
Performance and Operational Assets in Source Control Is critical.
It is the PIPE!
Build Once, Deploy and Continuously Monitor in any environment
Meet Kelly RochellSenior Business Analyst easyTravel
- Works for Tina“Hey Tim, I’m going to need to put this on a completely
separate branch and we will need
another environment. It’s going to impact too many things
The DevOps General FlowPROMOPROMOFEATURECHANGE
Test EnvironmentDev Environment
DeveloperLocal
Environments
Source Control
Production Environment
Dynatrace in Dev Dynatrace in Stage Dynatrace Prod
Load, Int, Func Tests
Report Results
Env Automation
Deploy Automation
Build ServerPerform Build
Run Unit Tests
Report Results
Environment Setup
Deploy Automation
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
CMO Recognizes Major Opportunity Initiates Need
FEATURECHANGE
• Industry is fast past and always fluid.
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
High Level Business Functionality
• All Channel must change• Actually 3 Pipelines affected
• Mechanics of how it will work
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
Mar
ket
Plan DesignFEATURECHANGE
PROMO
PMO
Desig
n
Creates Need
ValidatesSets
Urgent
IdentifiesBiz Details
High LevelEstimate
InitiatesTracking
InitiatesTracking
Aligned all Release Tracks and Sprint Tasks • BA
• Dev• Ops• Test• Data• Security
• Automation• XA
Tale of Two Pipelinesby Brett Hofer
Dev
Biz
ITQ
A
Plan Design FEATURECHANGE
PROMO
PMO
Identify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomArchitect
Design
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Desig
n
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomArchitect
Design
Design Feature Impact Assessment
• Leveraging Dynatrace runtime Visualizations on current relationships creates high accuracy on Impact decisions.
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomArchitect
Design
Design - Feature Impact Assessment
• Leveraging Dynatrace Actions provides high accuracy on page Impact decisions.
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomArchitect
Design
Architecture of Feature
• Leveraging Dynatrace runtime Visualizations on current relationships creates high accuracy on architectural decisions.
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomArchitect
Design
Architecture of Feature
• Leveraging Dynatrace end-to-end transactional visibility creates high accuracy on architectural decisions.
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Biz
ITQ
A
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomationArchitect
Design
eXperience Assurance Decisions
• Leveraging all of the Dynatrace User Experience Management Data
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomationArchitect
Design
Automation Architecture - ++Time, ++Cost, ++Risk • Role in designing and managing all
automation to move the feature through the pipeline.
• Leveraging Technologies to Automate Processes & Orchestration in CI & CD
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi
zIT
QA
Plan Design FEATURECHANGE
PROMO
PMO
Desig
nIdentify FeatureImpacts
AppArchitect
CapacityDesign
Plan
StoryBoard
Identify FeatureImpacts
GatherTime &
Cost
Track on Overall
Plan
TestDesign
Plan
XADesign
PerfArchitectReview
Identify FeatureImpacts
AutomationArchitect
Design
Performance Architecture • Performance Architect Reviews
Performance & Scalability Concerns of the Feature
• Designs Dashboarding Strategy forMonitoring Feature & Health
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Perf
Eng Create
Perf Dashboard
InstrumentApp
Unit TestCI Server
CheckResults
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Perf
Eng Create
Perf Dashboard
InstrumentApp
Unit TestCI Server
CheckResultsBIG GOALS
• How fast can we determine a check-in will fail and get feedback to the right resource• How far can we progress a healthy build/app without manually touching it• How fast can we identify the problem and turnaround the fix
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI Server
Using Onsite & Localized Development - -RISK, --TIME, -COST • Collaborate Often
• No Black Box Development • Culture
CheckResults
Perf
Eng
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI ServerDevelopers use Cloud Instances - -RISK, -TIME
• Developer Instances match targetdeployment stack.
• Easy to onboard other developers• Allows switching technology stacks
CheckResults
Perf
Eng
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI Server
CheckResults
Architectural Validation - --RISK, --TIME, +Value
• Developers, Leads and Solution Architects use Dynatrace to check transaction validity prior to check-in. Great on critical transactions and faster than code reviews.
Perf
Eng
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
CreatePerf
Dashboard
CreatePerf
Dashboard
CheckResults
Unit TestCI Server
Using a feature toggle - -TIME, -COST • Increased manual effort on changes
Perf
Eng
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
CreatePerf
Dashboard
CreatePerf
Dashboard
Unit Test
CI Server
CheckResults
Perf
Eng
Code Coverage Static Code Checking - -RISK, -TIME, -COST • Manage Testing Code Coverage
• Govern Coding Practices • Code Coverage is on Main
Dashboards
JaCoCo
Test
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit Test
CI Server
Source Control Respository - --RISK, -TIME, -COST • Aligning ALL Release assets
is critical
CheckResults
Test
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI Server
Build Automation - -RISK, -TIME
• Comprehensive Build Scripts• Include Dependency Resources • Leverage Profiles (Build script
is host aware)• Includes Monitoring of Tests
CheckResults
Ant
Test
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI Server
Automated Builds- -RISK, -TIME
• Scheduled off a check-in trigger• Orchestrate tests and deployments• Steps Exec Build Automation Goal
CheckResults
Test
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI Server
Fast Build Time - -TIME • Build takes only 15 minutes
CheckResults
Test
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CI-CDProv Env
CheckResults
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit TestCI Server
Unit & Unit Integration Testing - -RISK, -TIME
• 78% Coverage • All Unit Tests are monitored by
Dynatrace with Results maintainedby Build
CheckResults
Test
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
FEATURECHANGE
Code Check-In CI Server
Setup Release Blueprint
CI-CDProv Env
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Unit Test
CheckResults
• dynaTrace Monitored Unit Results - --RISK, --TIME
•
• Regression Violations generate change actions in tracking software
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
ReleaseCloud
Blueprint
CI-CDProv Env
Unit Test
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4DeveloperCloud
Blueprint
ManualMerge
Code ConfigMgmnt
Perf
Eng Create
Perf Dashboard
CreatePerf
Dashboard
CI Server
CheckResults
Configuration Management --RISK, --TIME
• All ops/infrastructure configs are represented in code and
aligned with the release• EnvForRelease = Cloud/Container Blueprint + ConfigCode
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CI-CDProv Env
Unit Test
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Test
Eng Create
Perf Dashboard
CreatePerf
Dashboard
Automated Cloud/Container Provisioning - --RISK, --TIME
• Establish Instance to Deploy
CI Server
CheckResults
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CI-CDProv Env
Unit Test
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Perf
Eng Create
Perf Dashboard
CreatePerf
Dashboard
CI Server
CheckResults
Automated Test Development- --RISK, --TIME
• All testers are test automation engineers.• All tests are integrated with Dynatrace
APM
Silk Performer
Selenium
Tale of Two Pipelinesby Brett Hofer
Dev
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Ops
FEATURECHANGE
Code Check-In
Setup Release Blueprint
CI-CDProv Env
Unit Test
Test Prod
Test
Eng Manually
PrepareTest Data
CreateAutomation
Scripts
PROMO
Test
Diagnose
Dev4Setup Developer Blueprint
ManualMerge
Code ConfigMgmnt
Perf
Eng Create
Perf Dashboard
CreatePerf
Dashboard
CI Server
CheckResults
Performance Dashboards are Deliverables --RISK, ++VALUE
• Major Features need Visibility• The dashboard indicates if the PROMO feature
is being used, response times, inter-dependency health or any errors related. Watched throughout lifecycle.
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
Test Prod
Test
Eng
Auto RunFull
Test Suite
EnvReady
Prod
FEATURECHANGE
PROMO
Manual Tests
ReportProblems
Validate Build
Validate Results
DiagnoseRoot
Cause
Check Infrastruct
ure
SelectiveLoadTest
Tale of Two Pipelinesby Brett Hofer
FEEDBACK LOOP dashboards/emails/tracking technology/group chat
Dev
Dev
Dev
Ops
Test Prod
Test
Eng
Auto RunFull
Test Suite
EnvReady
Prod
FEATURECHANGE
PROMO
Manual Tests
ReportProblems
Validate Build
Validate Results
DiagnoseRoot
Cause
Check Infrastruct
ure
SelectiveLoadTest
Finding Root Cause –TIME, ++VALUE
The Team under DevOps was able
to do four iterations and a full load test
three weeks ahead of schedule.
The feature has been in.
Tale of Two Pipelinesby Brett Hofer
Now click search…
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
Belize
12/18/2016
1/5/2017
Tale of Two Pipelinesby Brett Hofer
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
11/29/2016
12/5/2016
And there you go an 80% discount
on a trip to Belize…
Tale of Two Pipelinesby Brett Hofer
Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi
ROLLTHEDICE
11/29/2016
12/5/2017
Wow, that is fast and incredible. I’m sure our viewers want to thank easyTravel. I’m being
told this is our highest viewed show to date!
The DevOps Xcelerator
Confidential, Dynatrace LLC
Master Performance
Architect
Confidential, Dynatrace LLC
Assess
• Identify Target Application• Interview Key Teams with Set Topics
BluePrint
• Record Gaps & Recommendations• Written Roadmap to Drive Implementation Plan
Implement
• Integrate into Automation Process• Build Dashboards• Educate on Process
12
3
30 D
ays
60 D
ays
What We Do
Confidential, Dynatrace, LLC
Questions or to Inquire about the DevOps Engagement email: [email protected]
Please put TALES WEBINAR in the Subject Line