testing in a continuous delivery world - londoncd meetup - may 2014
DESCRIPTION
Talking about how Songkick manages software testing whilst following a Continuous Deployment approach.TRANSCRIPT
TESTING IN A CONTINUOUS DELIVERY WORLD
Amy Phillips
Songkick
Testing:
4
A search for information
Quality:
5
Value to some person ! - Gerald Weinberg
6
Test that you rolled a one
7
Test that you rolled a one
8
Test that you rolled a one
9
Test that you rolled a one
10
Test that you rolled a one
11
Test that you rolled a one
What were we doing?
! Checking the result !!!
! Testing the result
12
Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing Elisabeth Hendrickson
13
! You can automate 100% of your system testing
!! Automation will guarantee your system
keeps working - after all you are testing literally everything
14
Some Myths about Automation
! You can never test everything to prove it is working
!! It takes just one test to prove that a
system is broken
15
Does it work?
! Not working according to requirements ! Doesn’t meet the user’s expectations ! Difficult to use ! Inconsistent ! Doesn’t scale ! Can’t be maintained ! Difficult to test ! …
16
What is a bug?
17
The (old) Songkick Way
Automated Test and build
Manual Test Feature & ReleaseQueued
for commit
Commit Deploy
18
Don’t Aim for Perfection
Why do we have this process?
Release
ProblemsMake everything
safe
Scheduled Continuous Releases Integration
!
!
!
20
What were our options?
or
What are our values?
!! Ship new features as soon as possible !! Developers responsible for production
quality
22
The (old) Songkick Way
Automated Test and build
Manual Test Feature & ReleaseQueued
for commit
Commit Deploy
1 Hour 2 Hours
Define a single test strategy
Developers Testers
Unit Tests
Integration Tests
Manual Checks
Manual Tests
Regression Tests
Automated Checks
Define a single test strategy
Developers Testers
Unit Tests
Integration Tests Manual ChecksManual Tests
Regression Tests
Automated Checks
25
!
Testing
100 %Team Effort
CheckingTesting
Agree on how to split test effort
26
!
100 %Team Effort
Testing Machine Human
Agree on how to split test effort
27
!!
! Dev teams are responsible for their own testing
A New Value
28
!! Test at the right level ! Everyone understands what is being
tested and why ! Everyone has a chance to influence
what should be tested ! Fast feedback
Shared ownership of Automated tests
29
!!
! What are we hoping to achieve with this change?
! Identify risks ! Agree on how to mitigate risks
Risk Assess Everything
30
The Songkick Way - low risk
Automated Test and build
Commit DeployAutomated Regression
31
The Songkick Way - medium risk
Automated Test and build
Manual Test on Dev env
Commit DeployAutomated Regression
32
The Songkick Way - high risk
Automated Test and build
Manual Test on Dev env
Commit DeployAutomated Regression
Manual Test on Prod env
Releasing doesn’t have to mean Launching
Trust the Test Results
! Never release if the build is red ! Green builds must mean the tests have
passed
!!You can break anything once ! - Songkick Maxim
35
Learn from your mistakes
Executing the right tests at the right time
Automated build and deploy process
Continuous Deployment
37
Tools
38
Use your brain
39
Keep on talking
Releases
0
28
56
84
112
140
Mar 2011 Jul 2011 Nov 2011 Mar 2012 Jul 2012
But this ended up being about more than just releases…
! Everyone cares about testing ! Fast and maintained automated tests ! Bug fixes often take just minutes
In Summary
! You must automate ! But don’t try to automate everything ! The automation you have doesn’t
necessarily make you safe ! Use problems to drive positive change ! Don’t neglect the human aspect
43
Songkick is hiring
44
Questions?! @itjustbroke !! testingthemind.wordpress.com !
! http://devblog.songkick.com/