asking better survey questions - mozcon 2014

62

Upload: stephanie-beadell

Post on 21-Dec-2014

63 views

Category:

Marketing


0 download

DESCRIPTION

Bad Data, Bad Decisions: The Art of Asking Better Questions, Stephanie Beadell's presentation from MozCon 2014

TRANSCRIPT

Page 1: Asking Better Survey Questions - MozCon 2014
Page 2: Asking Better Survey Questions - MozCon 2014
Page 3: Asking Better Survey Questions - MozCon 2014
Page 4: Asking Better Survey Questions - MozCon 2014
Page 5: Asking Better Survey Questions - MozCon 2014

Ask Questions on a 5-point or 7-point Scale

Page 6: Asking Better Survey Questions - MozCon 2014

Are you happy with your job?

Sample Survey Question

Page 7: Asking Better Survey Questions - MozCon 2014

No 21%

Yes 79%

People Happy at Their Jobs

Page 8: Asking Better Survey Questions - MozCon 2014

I am happy with my job.

Sample Survey Question

Page 9: Asking Better Survey Questions - MozCon 2014

People Happy at Their Jobs

Strongly Disagree Disagree Neutral Agree Strongly Agree

Average = 3.05

Page 10: Asking Better Survey Questions - MozCon 2014

Break Down Big Concepts into Tangible Pieces

Page 11: Asking Better Survey Questions - MozCon 2014

Are you satisfied?

Sample Survey Question

Page 12: Asking Better Survey Questions - MozCon 2014
Page 13: Asking Better Survey Questions - MozCon 2014
Page 14: Asking Better Survey Questions - MozCon 2014

Sample Survey Question

Please Rate Your Satisfaction with Each Feature

Very unsatisfied

Somewhat Unsatisfied Neural

Somewhat Satisfied

Very Satisfied

Feature 1

Feature 2

Feature 3

Page 15: Asking Better Survey Questions - MozCon 2014

Satisfaction by Feature

Feature 1 Feature 2 Feature 3

Page 16: Asking Better Survey Questions - MozCon 2014

Compare Answers Across Questions

Page 17: Asking Better Survey Questions - MozCon 2014

Sample Survey Question

Please Rate Your Experience with Our Business

Very unsatisfied

Somewhat Unsatisfied Neural

Somewhat Satisfied

Very Satisfied

Product Quality

Price

Account Manager

Page 18: Asking Better Survey Questions - MozCon 2014

Business Ratings

Product Quality Price Account Manager

Page 19: Asking Better Survey Questions - MozCon 2014

Business Ratings by Company Size

Product Quality Price Acount Manager Product Quality Price Acount Manager

Small Business Enterprise

Page 20: Asking Better Survey Questions - MozCon 2014

Comparing Answers with SurveyMonkey

Analyze + Compare

Page 21: Asking Better Survey Questions - MozCon 2014

Beware of Bias

Page 22: Asking Better Survey Questions - MozCon 2014

Priming

Page 23: Asking Better Survey Questions - MozCon 2014

Bargh, Chen, and Burrows (1996)

Rude aggressively, bold, rude, bother, disturb, intrude, annoyingly, interrupt, audaciously, brazen, impolitely, infringe, obnoxious, aggravating, bluntly

Polite honor, considerate, appreciate, patiently, cordially, yield, polite, cautiously, courteous, graciously, sensitively; discreetly, behaved, unobtrusively

Neutral exercising, flawlessly; occasionally rapidly, gleefully, practiced, optimistically, successfully, normally, send, watches, encourages. gives, clears, prepares

Sentence Scramble Task

Page 24: Asking Better Survey Questions - MozCon 2014

Bargh, Chen, and Burrows (1996)

People Who Interrupted After Sentence Task

Polite Neutral Rude

Page 25: Asking Better Survey Questions - MozCon 2014

Order Bias

Page 26: Asking Better Survey Questions - MozCon 2014

How important is link building?

Question Order Bias

How much link building do you plan to do this year?

Page 27: Asking Better Survey Questions - MozCon 2014

Randomization in SurveyMonkey and Google Forms

Randomize Order

Page 28: Asking Better Survey Questions - MozCon 2014

Stereotype Threat

Page 29: Asking Better Survey Questions - MozCon 2014

“sports intelligence”

Stone, Lynch, Sjomeling, Darley (1999)

“natural athletic ability”

Page 30: Asking Better Survey Questions - MozCon 2014

Inzlicht (2000)

2 Women 2 Men

Female Math Test Performance when Others Present

Page 31: Asking Better Survey Questions - MozCon 2014

Double-Barreled Questions

Page 32: Asking Better Survey Questions - MozCon 2014

Do you play games on your smart phone or tablet?

Double-Barreled Question

Page 33: Asking Better Survey Questions - MozCon 2014

Leading Questions

Page 34: Asking Better Survey Questions - MozCon 2014

Do you support helping the poor?

Leading Questions

Do you support welfare?

Page 35: Asking Better Survey Questions - MozCon 2014

Understand How and Why People Lie

Page 36: Asking Better Survey Questions - MozCon 2014

Sensitive Information

Page 37: Asking Better Survey Questions - MozCon 2014

Use ranges for demographic info.

Sensitive Information

Page 38: Asking Better Survey Questions - MozCon 2014

Boredom

Page 39: Asking Better Survey Questions - MozCon 2014

Set expectations & show progress.

Boredom

Page 40: Asking Better Survey Questions - MozCon 2014

Guessing is Hard

Page 41: Asking Better Survey Questions - MozCon 2014

Be reasonable about estimations.

Guessing is Hard

Page 42: Asking Better Survey Questions - MozCon 2014

Cool Things You Can Do With Data

Page 43: Asking Better Survey Questions - MozCon 2014

How often do you use OSE?

Sample Survey Question

Page 44: Asking Better Survey Questions - MozCon 2014

Sample Survey Question 1

How often do you use Open Site Explorer for the following tasks?

Never Rarely Sometimes Often Always

Find Backlinks

Anchor Text Analysis

Competitor Analysis

Page 45: Asking Better Survey Questions - MozCon 2014

Sample Survey Question 2

How would you rate Open Site Explorer for the following tasks?

Never Rarely Sometimes Often Always

Find Backlinks

Anchor Text Analysis

Competitor Analysis

Page 46: Asking Better Survey Questions - MozCon 2014

Sample Survey Question 3

How important are the following Open Site Explorer features?

Never Rarely Sometimes Often Always

Find Backlinks

Anchor Text Analysis

Competitor Analysis

Page 47: Asking Better Survey Questions - MozCon 2014

Feature Prioritization Chart

Freq

uenc

y

0

7.5

15

22.5

30

Rating0 4 8 12 16

Page 48: Asking Better Survey Questions - MozCon 2014

Categorical

Page 49: Asking Better Survey Questions - MozCon 2014

Do you drink coffee?

Categorical Response

Yes | No

Page 50: Asking Better Survey Questions - MozCon 2014

Coffee Drinkers

Categorical Response

No

Yes

No Yes

Page 51: Asking Better Survey Questions - MozCon 2014

Ordinal

Page 52: Asking Better Survey Questions - MozCon 2014

Ordinal Response

How much coffee do you drink?0 cups per day 1-3 cups per day 4-5 cups per day 6+ cups per day

Page 53: Asking Better Survey Questions - MozCon 2014

Ordinal Response

Coffee Drinkers

6+ Cups

4-5 Cups

1-3 Cups

0 Cups

0 Cups 1-3 Cups 4-5 Cups 6+ Cups

Page 54: Asking Better Survey Questions - MozCon 2014

Interval & Ratio

Page 55: Asking Better Survey Questions - MozCon 2014

How often do you drink coffee?

Interval & Ratio Response

Never Rarely Sometimes Often Always

Page 56: Asking Better Survey Questions - MozCon 2014

How many cups of coffee do you drink each day?

Interval & Ratio Response

______________ (fill in the blank)

Page 57: Asking Better Survey Questions - MozCon 2014

Source: FIVETHIRTYEIGHT

Mean, Median, Mode

Page 58: Asking Better Survey Questions - MozCon 2014

Source: Pew Global

Correlation

Page 59: Asking Better Survey Questions - MozCon 2014

Source: Wired

Cluster Analysis

Page 60: Asking Better Survey Questions - MozCon 2014

Interval & Ratio Response

Regression

Outcome

Variable 1

Variable 2

Variable 3

(unknown)relative influence on outcome

Page 61: Asking Better Survey Questions - MozCon 2014

Interval & Ratio Response

Excel Add-In: Analysis ToolpakFree Add-In

Page 62: Asking Better Survey Questions - MozCon 2014