fail and win: why a failed test isn’t a bad thing
TRANSCRIPT
![Page 1: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/1.jpg)
Fail and Win: Why a Failed Test Isn’
Caleb Whitmore Founder & CEO Analy<cs Pros
@CalebWhitmore
![Page 2: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/2.jpg)
Hi, I’m @CalebWhitmore
![Page 3: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/3.jpg)
I climbed a mountain… once.
![Page 4: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/4.jpg)
I work @Analy=csPros
![Page 5: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/5.jpg)
And get to work with some awesome clients!
![Page 6: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/6.jpg)
Why Test?
![Page 7: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/7.jpg)
Why Test?
TESTING… • Should focus on improving an outcome • Is the shortest distance between you and improvements • Is done best with rapid itera<on and lots of data
![Page 8: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/8.jpg)
Some things may seem obvious to test…
![Page 9: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/9.jpg)
Bad Form
![Page 10: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/10.jpg)
Good Form
![Page 11: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/11.jpg)
But you should s=ll test them!
![Page 12: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/12.jpg)
Tes=ng depends on data
![Page 13: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/13.jpg)
Op=mizely Data
![Page 14: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/14.jpg)
Google Analy=cs Data
![Page 15: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/15.jpg)
Not all tests are created equally
![Page 16: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/16.jpg)
Some tests will win
![Page 17: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/17.jpg)
Some tests will fail
![Page 18: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/18.jpg)
Is winning a success?
![Page 19: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/19.jpg)
Is failing a loss?
![Page 20: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/20.jpg)
Tes=ng is a Success When:
• A process is followed
• An outcome is measured
• A complete picture is seen
![Page 21: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/21.jpg)
When Things Shake, What Moves?
![Page 22: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/22.jpg)
When Things Shake, What Moves?
![Page 23: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/23.jpg)
Crea=ng a 360 Degree View
• A test will impact more than what you intended
• You must measure the impact across your spectrum of user interac<on
• Measure your test with your exis<ng infrastructure
![Page 24: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/24.jpg)
Crea=ng a 360 Degree View
What happens to… • The “key” conversion point • Engagement on the test page • Length of the en<re visit • Total pageviews/interac<ons in the visit • Secondary conversion goals – view product, view content, start checkout, download a file, etc…
![Page 25: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/25.jpg)
Here’s an example
![Page 26: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/26.jpg)
Example Experiment
ASSUMPTIONS • Auto-‐play repels visitors – Audio without warning
• Thus, bounce rates are high and more people leave • Turn off auto play & more people will stay and engage
![Page 27: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/27.jpg)
Op=mizely Shows a Winner
![Page 28: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/28.jpg)
Bounce Rate Improved
![Page 29: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/29.jpg)
Purchase Rate Tanked!
![Page 30: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/30.jpg)
DO THIS
![Page 31: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/31.jpg)
Setup a 360 Degree View of Tests
• Tip: your Digital Analy<cs plaXorm has this view already!
![Page 32: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/32.jpg)
Op=mizely Automated Integra=ons
![Page 33: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/33.jpg)
Pipes data straight into GA
![Page 34: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/34.jpg)
Is this a good outcome?
![Page 35: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/35.jpg)
What about this?
![Page 36: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/36.jpg)
See the full picture
![Page 37: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/37.jpg)
Capture Experiment Data
• Tie experiment data into your Tag Management and Digital Data frameworks
![Page 38: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/38.jpg)
Leverage Built-‐in Repor=ng
• Create lots of Goals in Op<mizely – pages, clicks, revenue, etc…
![Page 39: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/39.jpg)
QUESTIONS?
@CalebWhitmore of @Analy<csPros
![Page 40: Fail and Win: Why a Failed Test Isn’t a Bad Thing](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d4fa97bb61eb764c8b4584/html5/thumbnails/40.jpg)
Fail and Win: Why a Failed Test Isn’t a Bad Thing
Caleb Whitmore Founder & CEO Analy<cs Pros
@CalebWhitmore