the measurement effect in pc smartphone and tablet surveys valerija kolbas university of essex iser...
TRANSCRIPT
The measurement effect in PC smartphone and tablet surveys
Valerija Kolbas
University of EssexISER
Ipsos-MORI
The European Survey Research Association Conference 13-17 July, 2015 Reykjavik
Background and motivation Limiting surveys to the PC mode
affects the size and representativeness of the sample
PC, smartphones, tablets differ in: size of the screen input method speed of connection processing power
Different effect on the measurement error
Background and motivation
Smartphone more break-offs longer completion time More straightlining
Mixed results
Other indicators question order effect primacy effects open-ended answers response distribution
… but non-optimized mobile design affects completion rates, satisfaction with the survey
Tablet Less primacy effect Less straightlining Less or comparable
completion time
Background and motivation
PC and mobile response distributions are equally affected by response formats
Drop-boxes –preference for first options Grids – preference for visible options,
straightlining
- no conclusive evidence which format is better for mobiles
Research Question
How mode of administration and response format affect survey responses
Indicators of measurement error:
overall satisfaction rates straightlining response distribution length of open-end questions
Survey and Questionnaire design National Satisfaction Survey 2014 Administered to final year higher education
students in UK 22 core questions using a 5-point Likert scale 2 open-end questions Mixed-mode: self-selected mail, phone, web 5 response formats – randomly allocated
Sample composition
PCN=5529
SmartphoneN=3196
TabletN=551
Radio-button
Drop-box ()
Drop-box (0)
Drop-box (-)
Drop-box (+)
Radio-buttonWeb
Survey SampleN=9276
Screenshots: response design
• Once clicked a list with options appears on a separate screen.• Responses always in the same order
Screenshots: response design
• All questions are visible on the screen.
• Requires horizontal scrolling .• Portrait or landscape viewing.
• Requires vertical scrolling.
10
Measurement Effect between PC smartphone
tablet responses
Comparisons madeacross all three modes, but within one radio-button response format
Measurement Effect between PC smartphone tablet
F p PC smartphone tablet
Straightlining
3.9 <.05 6.9% 10.3% 6.4%
Modal responses
<1 >.05‘Mostly Agree’
‘Definitely Agree’
&‘Mostly Agree’
‘Mostly Agree’
Mean values
2.7 .07 4.1 4.1 4
Positive feedback
1.3 >.05 195 180 191
Negative feedback
1 >.05 257 240 243
MANOVA to test for differences F=2.3, p<.05
ME between PC, smartphone, tablet summary
Smartphone straightlining significantly higher
Tablet straightlining rate the lowest Signs of the visibility effect for smartphone Other quality indicators comparable
between all three modes
13
Measurement Error between different response designs
presented on a smartphone and a tablet
Comparisons made across five survey response formats within smartphone and tablet modes
MANOVA test of differences F=2.7, p<.05
F p
device 2.7 <.05
format 1.7 <.05
interaction 1.3 .18
Both device and format affected data quality
Responses in Drop-box with a positive initial option
Def.A
gree
Mos
tly A
gree
Neith
er
Mos
tly D
isag
ree
Def. D
isag
ree
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
smartphone
tablet
Smartphones
higher selection of initially suggested response.
less moderately positive responses 11.1% straightliners Mean 4.3
Tablets
Similar selection of extreme and moderately positive responses
More negative responses 5.6% straightliners Mean 4.1
Responses in Drop-box with a negative initial option
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
smartphone tablet
Smartphones
More extreme negative responses Less positive responses 8.4% straightliners Mean 4
Tablets
More positive responses Extremely low ‘Definitely Disagree’
frequency 2.5% straightliners Mean 4.1
Def.A
gree
Mos
tly A
gree
Neith
er
Mos
tly D
isag
ree
Def. D
isag
ree
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
smartphonetablet
Responses in Drop-box with a middle initial option
Smartphone and Tablet
Comparable results
Weak evidence of selecting the middle option
Most frequent ‘Mostly Agree/Disagree’ selection across formats
6.5% and 6.1% of straightliners Mean 4 an 3.9
ME indicators summary
Smartphone – answers affected by response formats.
Initially suggested response is selected more often
Tablet – no strong effect of response formats.
No significant differences in the length of open answers between formats.
Potential Limitations
No reverse-coding Similar question wording Not counterbalanced question or response
order Instructions universal for each response
format Self selected device condition Survey sample: highly educated, IT literate,
similar age, highly motivated
