![Page 1: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/1.jpg)
© Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved.
The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)
1
Cornelia Davis (@cdavisafc) Director, Platform Engineering, Cloud Foundry
Software Quality in the Devops World
![Page 2: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/2.jpg)
© Copyright 2014 Pivotal. All rights reserved.
“Software is Eating the World”
2
![Page 3: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/3.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Software is Changing Industries
3
$3.5B valuation Financial Services
$10B valuation Travel & Hospitality
$17B valuation Transportation
$3.2B Acquisition by Google Home Automation
$20B valuation Entertainment
$26B valuation Tesla--Automotive
![Page 4: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/4.jpg)
© Copyright 2014 Pivotal. All rights reserved. 4
You are either building a software business… !!
Or losing to someone who is.
![Page 5: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/5.jpg)
© Copyright 2014 Pivotal. All rights reserved. 5
Amazon, a book store in Seattle, deploys code every 11 seconds
![Page 6: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/6.jpg)
© Copyright 2014 Pivotal. All rights reserved.
What is Continuous Delivery?
6
$Busin
ess
Devel
opm
ent
QA
Ope
ratio
nsCu
stom
er
![Page 7: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/7.jpg)
© Copyright 2014 Pivotal. All rights reserved.
What is Continuous Delivery?
7
$
![Page 8: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/8.jpg)
© Copyright 2014 Pivotal. All rights reserved. 8
Continuous Delivery - How?
![Page 9: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/9.jpg)
© Copyright 2014 Pivotal. All rights reserved. 9
Warner Music: Software FactoriesWarner Software Factory Platform • New applications and major updates
- Before: 6 months, team of 10 developers - After: 6 weeks, same team - Speed/Agility: 400% faster on new platform - HR Hard Savings: $1.1M per application update
delivered
![Page 10: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/10.jpg)
© Copyright 2014 Pivotal. All rights reserved. 10
Developers are Responsible for Quality• Test Driven Development
!!!
•Unit Tests
!
• Integration Tests
!!
•Automation
Foo
Foo Bar
![Page 11: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/11.jpg)
© Copyright 2014 Pivotal. All rights reserved.
“Operations is the secret sauce”
11
http://radar.oreilly.com/2007/10/operations-is-a-competitive-ad.html
![Page 12: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/12.jpg)
© Copyright 2014 Pivotal. All rights reserved. 12
Agile• Show screen of
tracker
![Page 13: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/13.jpg)
© Copyright 2014 Pivotal. All rights reserved. 13
Design for Failure
• “Sufficiently Paranoid”
•Circuit breaker • http://martinfowler.com/bliki/CircuitBreaker.html
• https://github.com/Netflix/Hystrix
• SpringCloud: http://projects.spring.io/spring-cloud
![Page 14: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/14.jpg)
© Copyright 2014 Pivotal. All rights reserved. 14
http://projects.spring.io/spring-cloud
![Page 15: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/15.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Agile Methodologies Meet Agile (3rd) Platform
15
Agile Development
DEVELOPMENT
App1 John’s dev ver
Other dev apps
DEVELOPMENT
App1 Mary’s dev ver
Other dev apps
QA
QA
App1 QA version
Other QA apps
Production
PRODUCTION
App1 Production version
Other Prod apps
No code or configuration changes!
![Page 16: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/16.jpg)
© Copyright 2014 Pivotal. All rights reserved. 16
DEMO
![Page 17: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/17.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Build Pipelines
17
![Page 18: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/18.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Pipelines - Example
18
CF Runtime
CF Services
Runtime: Dev a MastServices: Dev a Mast
Svc: Mast a CR; RT: Mast a CR
Dijon Tabasco
A1
Prod
Services: Dev; Runtime: CR Services: CR; Runtime: Dev
Services: Mast; Runtime: Mast
Services: Rel; Runtime: Rel
Svc: CR a Rel; RT: CR a Rel
![Page 19: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/19.jpg)
© Copyright 2014 Pivotal. All rights reserved. 19
Distributed, Loosely-coupled Systems…… are very hard to reason about
Foo Bar
Baz
![Page 20: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/20.jpg)
© Copyright 2014 Pivotal. All rights reserved. 20
Your software will be used by humans…And Now You Go Into Prod
• Smoke tests in prod
•App architecture
•Monitoring
•Blue/Green Deployments
![Page 21: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/21.jpg)
© Copyright 2014 Pivotal. All rights reserved. 21
Twelve Factors•One Codebase/Many Deploys
• Explicit Isolated Dependencies
•Config via Environment
•Attached Backing Services
• Separate Build/Release/Run
• Stateless Processes
• Export Services via Port Bindings
• Scale Out via Processes
•Disposable Instances
•Dev/Prod Parity
• Logs == Event Streams
•Admin Tasks == Processes
http://12factor.net
![Page 22: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/22.jpg)
© Copyright 2014 Pivotal. All rights reserved. 22
DEMO
![Page 23: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/23.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Blue/Green Deployments
23
![Page 24: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/24.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Blue/Green Deployments
24
![Page 25: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/25.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Blue/Green Deployments
25
![Page 26: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/26.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Blue/Green Deployments
26
![Page 27: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/27.jpg)
© Copyright 2014 Pivotal. All rights reserved.
Blue/Green Deployments
27
![Page 28: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/28.jpg)
© Copyright 2014 Pivotal. All rights reserved. 28
DEMO
![Page 29: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/29.jpg)
© Copyright 2014 Pivotal. All rights reserved. 29
Summary•Developers are responsible for quality
• Tests
• Techniques for all stages of the application lifecycle
•Design for failure
• Self-healing
• Zero-downtime upgrades
![Page 30: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/30.jpg)
© Copyright 2014 Pivotal. All rights reserved. 30
Thank You!
@cdavisafc
![Page 31: Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)](https://reader030.vdocuments.net/reader030/viewer/2022020207/557d60cdd8b42ae1438b502f/html5/thumbnails/31.jpg)
A NEW PLATFORM FOR A NEW ERA