why automated testing matters to devops
TRANSCRIPT
What is your worst experience with automated testing?
What is automated testing?
Automated Testing:
A process that stimulates a system&
reports whether expectations are met
Types of Automated Testing
Unit Tests
• Written by developers• Intimate with code under test• ONLY interact with unit under test • Mock everything else• May influence design• May expose design concerns• Run FAST (hundreds in secs on dev machine)
Unit Tests
Unit
Test
Test
Mock
Mock
Integration Tests
• Intimate with code under test• May or may not interact with other systems• May use mocks, simulators, scaffolding• May influence design• May expose separation of concerns• May or may not run on Dev machines• May be fast or slow• May or may not encapsulate a user function
Integration TestsSystem Under Test (SUT)
Component A
Component C Component D
Component B
Integration Tests - Example 1System Under Test (SUT)
Reporting Service
Authentication Broker
Parsing Service
Integration Tests - Example 2System Under Test (SUT)
Restful Server
User Interface
Database
Request
End to End (E2E) Tests
• Interacts with System as an “Actor” would• Assumes similar environment to production• May or may not run on Dev machines• Hopefully fast, usually slow (minutes & hours)
End to End (E2E) Tests
What does bad automated testing look like?
Bad Automated Testing
Causes unexpected work, like…– “You filled up the disk”– “The network was flooded”– Data leaks– Tests fail because of poor design– Cascading failures– Numerous false positives– Inconsistent results
What does good automated testing accomplish, how is it different?
Good Automated Tests…Expedite work, like…• “The tests showed me where to look”• Expose error states
– java.lang.StackOverflowError (bad recursion)• High signal-to-noise ratio
– No cascading, except in serious cases– Few false positives
• Act as or establish a “Source of Truth”– Tests fail here, but not there– Tests fail with this data, but not that data
• Isolate problem scope
Good Automated Tests…
Smooth edges via friction
Friction:• Failing tests• Failing systems• Identifying system limits• Forcing consideration of constraints, scenarios, cases
Smoothed:• Pronounce or expose implied contracts
How does this affect you?
Follow the Value
• Your role is, was, or is projected to be valuable to someone somewhere
• Someone has decided:– Consumers won’t buy without your role OR– To speculate that value is increased by your role
Passing the Value
The person who understands your value may not be the person who makes decisions
Or
The person who understands your value may not communicate it in a way that benefits you
If your value is obscured…
You go home!
…So Increase & Promote Value
• By using what you have:– Tests– Code– Systems– Communication
• By digging deeper:– Peers– Customer– Investor– Manager
Checks and Balances
• The code serves as a test case for the test• The code serves as a test case for the deployment
• The tests serve as tests for the code• The tests serve as tests for the deployment
• The deployment serves as a test for the code• The deployment serves as a test for the tests
Checks and Balances
• Each touch-point presents opportunity for greater value through:– Efficiencies– Effectiveness– Goodwill– Making life easier for someone • Customer? • Investor?• User?• Peer?
Creating Value via DevOps Role
• Use friction to create value• What is a failing (or passing) test telling you?• How can you create a better system based on
feedback (like that from automated tests)?
“Release”
The act of coding, testing & deploying
A Release Serves as…
• A test of the market• A test of the customer• A test of the business (or business line)• A test of the investment• A collector of revenue (fuels the business)• A test of your role’s value
• Automated Tests• Deployments as Tests• Code as Tests• Releases as Tests
Each giving feedback you are uniquely positioned to leverage to create value for the
business.
Tests…
Any test could be a pain in your ____ OR
It could be an opportunity to create value in your systems and lines of business!
