mixed methods research in the age of big data: a primer for ux researchers
TRANSCRIPT
Mixed Methods Research in the Age of Big Data
A Primer for UX Professionals
Zachary Sam ZaissUX Data Scientist | Microsoft Cloud
@zszaiss
2006 2012 2016UX Researcher UX DS
Berkeley MIDS
Gartner Hype Cycle for Emerging Technologies: 2014
http://www.gartner.com/newsroom/id/2819918
Gartner Hype Cycle for Emerging Technologies: 2015
http://www.gartner.com/newsroom/id/3114217
The Education Perspective
https://whatsthebigdata.com/2012/08/09/graduate-programs-in-big-data-and-data-science/ http://uxmastery.com/resources/ux-degrees/
84 78Graduate Degree
Programs inData Science
Graduate DegreePrograms in UX
http://radar.oreilly.com/2013/10/design-thinking-and-data-science.html
We need to makecollaboration with
Data Scientists a priority…
… and it starts with aconversation
Tip #1
Stake Your Claim
Qualitative Evaluation Criteria Talking Points
Quantitative basis for n values
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ http://www.measuringu.com/blog/five-history.php http://www.measuringu.com/blog/five-for-five.php
Qualitative Evaluation Criteria Talking Points
Quantitative basis for n values
Existence Proof
https://www.youtube.com/watch?v=3uqZPnxG4_w
Qualitative Evaluation Criteria Talking Points
Quantitative basis for n values
Existence Proof
Grounded TheoryInductive vs. Deductive Reasoning
http://www.slideshare.net/traincroft/hcic-muller-guha-davis-geyer-shami-2015-0629
Theory from Data
Data from Theory
Qualitative Evaluation Criteria Talking Points
Quantitative basis for n values
Existence Proof
Grounded TheoryInductive vs. Deductive Reasoning
Constructivism vs. Determinism
https://us.sagepub.com/en-us/nam/research-design/book237357
Discussing evaluation criteriafor qualitative research
needs to be second nature.
Example
What is yourmic drop moment?
Tip #1
Stake Your Claim
Tip #2
Speak the Language
vs
Models + Key Aspects of Analysis
Descriptive ModelDescr ipt ive Stat i s t i cs
Statistical SignificanceWhat is the probability of obtaining this result given the null hypothesis is true?
Practical SignificanceIs the effect on the outcome large enough to be considered relevant?
http://fivethirtyeight.com/features/statisticians-found-one-thing-they-can-agree-on-its-time-to-stop-misusing-p-values/
The statement process was lengthier and more controversial than anticipated.
6 Principles for p-values from ASA’s Statement
1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value crosses a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108
Models + Key Aspects of Analysis
Descriptive ModelDescr ipt ive Stat i s t i cs
Statistical SignificanceWhat is the probability of obtaining this result given the null hypothesis is true?
Practical SignificanceIs the effect on the outcome large enough to be considered relevant?
Predictive ModelSuperv i sed Machine Learn ing
AccuracyHow well does the model predict the outcome for new data cases?
Models + Key Aspects of Analysis
Descriptive ModelDescr ipt ive Stat i s t i cs
Statistical SignificanceWhat is the probability of obtaining this result given the null hypothesis is true?
Practical SignificanceIs the effect on the outcome large enough to be considered relevant?
Predictive ModelSuperv i sed Machine Learn ing
AccuracyHow well does the model predict the outcome for new data cases?
Representation ModelUnsuperv ised Machine Learn ing
Optimization CriteriaHow will we determine that we’ve built a reasonable and appropriate representation model for our data?
vs
None of these measuresget at the contextual meaning
behind the model.
A Diagram for Product Manager…
Source: Martin Eriksson, Mind the Product. http://www.mindtheproduct.com/2011/10/what-exactly-is-a-product-manager/
… And a Framework for Attributes
UX
Business
TechExperience Attributes
Customer attributes that can explain how
that customer will experience a product.
Technology AttributesCustomer attributes that can explain whether customers will have technical issues with a product.
Business AttributesCustomer attributes that can explain the extent to which the customer will contribute to business outcomes.
Example: Developer Tools
X
B
T
Prog LanguageTarget PlatformProject Complexity
Project AudienceType of App
Educational Background
Keyboard ProclivityProject Complexity
Example: Freemium Games
X
B
T
Platform UsedFacebook Connected
Whale Status
Completionist Tendencies
Game Session Time
Example: Fitness Bands
X
B
T
Connected DevicesType / Version
Frequency of Exercise
Friends with Same Band
Finger Shape (Fat Fingers)
FarsightednessSkin Irritation
We are uniquely qualifiedto articulate the experienceattributes of our products.
Tip #2
Speak the Language
Tip #3
Get Involved
A Metaphor for A/B Experiments
A Better Metaphor for A/B Experiments
XX
X
XX
X
X
How can we providegreater context toA/B test findings?
Heterogeneous Treatment Effects
control treatment
some kpi
0.71
0.72
productexperts
productnovices
control
treatment
converted didn‘t convert
converted didn‘t convert
control
treatment
converted didn‘t convert
converted didn‘t convert
Heterogeneous Treatment Effect
https://datadialogs.ischool.berkeley.edu/2014/schedule/experiments-action
Tip #1: Stake Your Claim
Tip #2: Speak the Language
Tip #3: Get Involved
Mixed Methods Research in the Age of Big Data
A Primer for UX Professionals
http://www.uxpa.org/sessionsurvey?sessionid=113