ISO 20252 Certified Gregg Peterson

Senior Vice President, Research Operations

Solving the

Unintentional Mobile Challenge

CASRO Online Research Conference – March 2013

Growth in unintentional mobile survey taking mirrors

growth of internet access via cell phones

0%

5%

10%

15%

20%

25%

30%

Q4 '11 Q1 '12 Q2 '12 Q3 '12 Q4 '12

17%

26%

A current client tracker at Market Strategies:

Mobile survey starts are up 50% in past year

Mobile survey starts

Completes

Page 2

UMR happens...do we care?

> Increased survey length

> Poor usability

> High breakoffs

> Possible mode effects

PROS CONS

> Honor respondent choice

> Improve coverage

> Maximize participation rates

Can a better design minimize the

risks and maximize the benefits?

Page 3

Page 4

HIGH COMPLETION RATES

Fast

Easy

Enjoyable

SAME RESPONSES

COMPARED TO PC

DESIGN GOALS

HIGH ENGAGEMENT

No speeding

Meaningful responses

Mobile

Friendly

Page 6

Test cell 1 – Mobile/legacy (plain html)

Page 7

Test cell 2 – Mobile/new (“mobile friendly”)

Page 8

Test cell 3 – Mobile/numeric (“mobile friendly”)

Page 9

Test cell 4 – Mobile/slider (“mobile friendly”)

Page 10

Test cell 5 – Mobile/drop down (“mobile friendly”)

Page 11

Test cell 6 – PC/legacy (plain html)

Page 12

Test cell 7 – PC/new (“mobile friendly”)

Page 13

Test cell 8 – Mobile/app (Mobile optimized)

Smartphone PC/tablet

Legacy (HTML) 1 – Mobile/legacy 6 – PC/legacy

“Mobile friendly”

2 – Mobile/new

3 – Mobile/numeric

4 – Mobile/slider

5 – Mobile/drop down

7 – PC/new

Mobile App 8 – Mobile/app

Experimental Design

Random assignment at sampling stage

*

*

* Random assignment upon

initial survey entry ~ N = 250 / test cell

Page 14

Hypothesis

> Mobile presentations will have longer survey lengths compared to PC,

regardless of presentation

> All mobile presentations will have higher breakoffs compared to PC

> Mobile friendly versions on smartphones will have shorter survey lengths

and lower break-offs compared to the Mobile/legacy version

> Mobile friendly versions on smartphones will elicit higher user experience

ratings compared to the Mobile/legacy (html) version on a smartphone

> Visual design changes, whether on mobile or PC, have no impact on

measures of engagement

> Response distributions across platforms (PC versus smartphone) will be

more similar when they use a similar design e.g. Cell 2: Mobile/new will

be most similar to cell 7: PC/new

H1

H2

H3

H4

H5

H6

Page 15

Page 16

6- PC/ legacy 7- PC/new

A consistent mobile friendly design – most

comparable across devices?

2 - Mobile/new

H6

best match?

Possible self-selection bias for Smartphone groups Experimental challenges

Page 17

Smartphone PC/tablet

Some self selection for Smartphone groups

Invitations

50% 6% Wrong device

Final participation

rate 6% 12%

0%

60%

0%

60%

Are these groups different?

Age

35-54

55+

18-34

Page 18

weighted

0%

60%

0%

60%

0%

60%

Gender

Education

Income

College

Grad school

100K +

Male Female

50K–100K

Some college/tech

Sig difference A few small differences after weighting

Mobile PC

1 - legacy 2 - new 3 - numeric 4 - slider 5 - drop down 8 - App 6 - legacy 7 - new

PC groups access the internet from phone less often

Page 19

23 24 22 23 22 22 23 22

10

15

20

25

0%

60%

0%

100%

0%

60%

Time spent

on Internet (hours/week)

Cell

Internet

usage (past mo.)

Daily

Weekly

Current

wireless

connection

3G

4G

LTE 4G

WIFI

Mean

40+

20–39

Sig difference

Mobile PC

1 - legacy 2 - new 3 - numeric 4 - slider 5 - drop down 8 - App 6 - legacy 7 - new

Survey length and breakoffs

0 3 6 9 12 15 18 21 24Survey length (minutes)

11.0

8.9

10.2

9.4

10.3

7.1

7.0

> All mobile designs slower than PC

> Legacy mobile more than 50% slower than PC with significantly

higher breakoffs

> Better design of Mobile/new improves on length

13%

5%

4%

7%

5%

4%

7%

No

difference

Survey length Breakoffs

13%

H1 – H3

Page 20

Mobile

1 - legacy

2 - new

3 - numeric

4 - slider

5 - drop down

PC

6 - legacy

7 - new

Traditionally formatted instruments create a poor

user experience for mobile survey takers; R’s liked the

mobile app experience best

Enjoy

completing

the survey

Fast or

slow

Easy or

difficult

to answer

Easy or

difficult

to read Interesting

Top 2-box Bottom 3-box

Mobile

1 - legacy

2 - new

3 - numeric

4 - slider

5 - drop down

Mobile 8 - App

PC

6 - legacy

7 - new

Page 21

H4

Mobile respondents as engaged as PC survey takers;

few significant differences across mobile test cells

Straight-

Lining

1+ grids

Trap

question

Self-

reported

satisficing

Number of

OE char

Provided a

Considered

response

Mobile

1 - legacy

2 - new

3 - numeric

4 - slider

5 - drop down

8 - App

PC

6 - legacy

7 - new

Sig difference

Page 22

H5

Page 23

Strategy for comparing response distributions in this

experiment: Each PC cell versus each Mobile cell

10 0 5

6 - PC/legacy 7 - PC/new 2 - Mobile/new

10 0 5 10 0 5 0%

10%

20%

30%

Difference of top 2-box

%

Very favorable Very unfavorable

Question: “Please indicate your overall feelings toward your electric utility provider.”

Categorical

(15 questions)

0-10 ratings

(12 questions)

CHI-SQUARE ANOVA

Mean absolute

differences

legacy new legacy new Legacy new

7 – PC/new

3 0 4%

Mobile

1 - legacy 1 2 0 0 4% 3%

2 - new 1 3 0 0 3% 4%

3 - numeric 4 5 0 4 5% 6%

4 - slider 10 9 7 9 6% 5%

5 - drop down 6 5 0 2 7% 7%

8 - App 5 4 0 3 6% 6%

Comparing PC and Mobile distributions on

27 key measures

Page 24

versus

H6

6 - PC/

legacy

7 - PC/

new

7 – PC/new

Mobile

1 - legacy

2 - new

3 - numeric X X

4 - slider X X

5 - drop down X X

8 - App X X

Comparing PC and Mobile response distributions on

27 key measures - Summary

Page 25

H6

Novel input designs can produce anomalous

results for selected question/scale types

Page 26

0%

60%

Strongly agree Somewhat agree Neither Somewhat disagree Strongly disagree

3 - Mobile/numeric

4 - Mobile/slider

6 - PC/legacy

7 - PC/new

3 - Mobile/numeric input

4 - Mobile/slider

Vertical presentation of choices via drop

down exaggerates use of top choice

Page 27

0%

30%

10 =Very favorable

9 8 7 6 5 4 3 2 1 0 =Very unfavorable

5 - Mobile/drop down

6 - PC/legacy

7- PC/new

4- Mobile/ drop down

Mobile/legacy and Mobile/new produce the same

results on key measures

Page 28

≈

Non-optimized mobile survey presentations do not

automatically lead to different response distributions

Page 29

0%

30%

10 =Extremelysatisfied

9 8 7 6 5 4 3 2 1 0 =Extremely

dissatisfied

1- Mobile/legacy

2 - Mobile/new

7 - PC/new

Hypothesis

> Mobile presentations will have a longer survey lengths compared to

PC, regardless of presentation

> All mobile presentations have higher breakoffs compared to PC

> “Mobile friendly” versions on smartphones will have shorter survey

lengths and lower break-offs compared to the Mobile/ legacy version

> Mobile friendly versions on smartphones will elicit higher user

experience ratings compared to the HTML version on a smartphone

> Visual design changes, whether on mobile or PC, have no impact on

measures of engagement

> Response distributions across platforms (PC versus smartphone) will

be more similar when they use a similar design e.g. Cell 2: Mobile/new

will be most similar to cell 7: PC/new

H1

H2

H3

H4

H5

H6

X

X Page 30

X

Fast

Easy

Enjoyable

Improve completion rates

Decrease survey length

Improve user experience

Mobile friendly designs…

Have no impact on user engagement

2

The right mobile friendly design…

Does not compromise response quality

Positive user experience better data

Page 31

ISO 20252 Certified

Special thanks to

ResearchNow for their

support on this project

Contact:

Gregg Peterson

Senior Vice President, Research Operations

gregg.peterson@marketstrategies.com

Connect with us:

marketstrategies.com | freshmr.com

Twitter | Facebook | LinkedIn

ISO 20252 Certified

Market Strategies project team:

Joanne Mechling

John LaFrance

Janice Swinehart

Gina Ham