11 ways humans kill good analysis (kevin ertell)
DESCRIPTION
We’re drowning in data in the eCommerce world. We can and do measure everything. But how do we get the most out of those numbers? Those mountains of data can be full of gold if we mine them correctly, or they can just be big piles of useless dirt. All too often, we misuse the valuable data we have and end up flailing away. Many of the reasons we aren’t happy with the results of the analyses come down to fundamental disconnects in human relations between all parties involved. Groups of people with disparate backgrounds, training and experiences gather in a room to “review the numbers.” We each bring our own sets of assumptions, biases and expectations, and we generally fail to establish common sets of understanding before digging in.TRANSCRIPT
#monetatesummit
11 Ways Humans Kill
Good AnalysisKevinErtell, Sur LaTable
FOCUSED
FOCUSEDACTIONABLE
FOCUSEDACTIONABLEMANAGEABLE
FOCUSEDACTIONABLEMANAGEABLEENLIGHTENING
FOCUSED
ACTIONABLE
MANAGEABLE
ENLIGHTENING
FOCUSED
ACTIONABLE
MANAGEABLE
ENLIGHTENING
11 Ways Humans Kill Good Analysis
1. We hire reporters not analysts
Logical
Sequential
Rational
Objective
2. We turn analysts into reporters
Why is conversion down on Google
paid search?
Why is conversion down on Google
paid search?What’s'the'op+mal'marke+ng'mix'to'use'to'launch'Brand'X?
Why is conversion down on Google
paid search?What’s'the'op+mal'marke+ng'mix'to'use'to'launch'Brand'X?
Why'are'return'rates'growing?
3. We expect the data to be perfect and the analysis to be flawless
A man with one watch knows what time it is; a man with two watches is never quite sure.
4. We fail to define objectives and state our assumptions
5. We want numbers for number’s sake
KPIs
Suppor+ng'Metrics
Forensic'Metrics
Supporting metrics
KPIsSupporting metrics
6. We insist on simplicity
How likely are you to recommend this business?
• Large margins of error
• Low precision
• Low detection of movement • Interpretation problems
• Not very predictive
How likely are you to recommend this business?
All'we'did'was'quan+fy'this'common'sense'in'a'way'that'made'sense'to'business'leaders—the'target'audience'for'my'book.'These'prac+cal'leaders'have'liJle'interest'in'advanced'sta+s+cal'methods.'
All'we'did'was'quan+fy'this'common'sense'in'a'way'that'made'sense'to'business'leaders—the'target'audience'for'my'book.'These'prac+cal'leaders'have'liJle'interest'in'advanced'sta+s+cal'methods.'
These practical leaders havelittle interest in advancedstatistical methods
correlations
7. We just want the number
“Plans based on average assumptions are wrong on average.” -Sam Savage
8. We aren’t multilingual in the languages of business & statistics
Standard deviations
Standard deviations
Variances
9. We expect answers immediately
You’re approaching a Coast Guard security zone. … If you don’t stop your vessel, you will be fired upon. Stop your vessel immediately.
You’re approaching a Coast Guard security zone. … If you don’t stop your vessel, you will be fired upon. Stop your vessel immediately.
bang,'bang,'bang,'bangBang! Bang! Bang! Bang!
regression to the mean
10. We ignore our guts
Prefrontal cortex
11. We blow the presentation
