investigating self-reporting behavior in long-term studies
DESCRIPTION
Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects’ forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in or- der to investigate the accuracy of self-reported information, and used logged data as ground truth to compare the sub- jects’ reports against. Subjects never recorded more than 70% and, depending on the requested reporting interval, down to less than 40% of actual app usages. They significantly over- estimated how long they used apps. While subjects forgot self-reports when no automatic reminders were sent, a high reporting frequency was perceived as uncomfortable and bur- densome. Most significantly, self-reporting even changed the actual app usage of users and hence can lead to deceptive measures if a study relies on no other data sources. With this contribution, we provide empirical quantitative long-term data on the reliability of self-reported data col- lected with mobile devices. We aim to make researchers aware of the caveats of self-reporting and give recommenda- tions for maximizing the reliability of results when conduct- ing large-scale, long-term app usage studies.TRANSCRIPT
INVESTIGATINGSELF-REPORTING BEHAVIOR
IN LONG-TERM STUDIESAndreas Möller ✽, Matthias Kranz ❖,
Barbara Schmid ✽, Stefan Diewald ✽, Luis Roalter ✽
✽ Technische Universität München, Germany❖ Universität Passau, Germany
Logging
Self-Reporting
DATA COLLECTION
Forgetting
Annoyance
Sluggishness
Self-Perception
RESEARCH QUESTIONS
Accuracy? Change over time?
Influence ofreporting frequency?
Reliability maximization?
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
BACKGROUND ■ Electronic diaries show higher
compliance(Hufford & Shields, 2002)
■ Mobile phone as survey tool (Consolvo et al., 2007)
Consolvo et al., 2007
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
CONTRIBUTIONS■ Empirical quantitative long-term data
on reliability of informationcollected with mobile devices
■ Recommendations for maximizingresult reliability
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
METHODOLOGYEvaluatereporting behavior
Ground truthCan be gained in automated
wayLimited effort
Smartphoneusage
GOAL
REQUIRE-MENTS
SOLUTION
Frequently used apps
Installed by everyone
Mail Facebook
Frequently used apps
Installed by everyone
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
PROCEEDING
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 6 + 4
Pre-Questionnaire Reminder Emails Post-Questionnaire
Post-Post-Questionnaire
Requested Self-Reports& Logging
TASK■ Answer questionnaire after Facebook or Mail
usage■ Report as accurate as possible
1. How long did you use the app?
2. How many times did you use the appwithout answering a questionnaire?
1 direct report
n indirect reports
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
SELF-REPORTING AND EXPERIENCE SAMPLING ASSISTANT
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
SERENA■ App usage logging■ Server upload■ Questionnaire triggers
□ Event-based□ Time-based□ Manually
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
3 CONDITIONS
Voluntary Interval Event
No trigger Daily trigger Trigger afterapp usage
30 Participants3,631 Mail usages3,181 Facebook usages
SESSIONS
Voluntary Interval Event
Amount of reported Facebook usages
37.6%
63.8%54.3%
Indirect reports
Direct reports
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
DURATIONS
Voluntary
Interval
Event
1:29
1:29
1:22
2:52
3:02
3:35
Facebook sessions
RealSelf-Reported
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
OVER TIME■ Self-report ratio decreases■ Actual usage decreases
„Answering the questionnairechanged my Facebook usage habits.“
Voluntary 2.3Interval 2.2Event 3.5
5 = strongly agreeA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
DISCUSSION
Subjects reported max. 70% of usage
Commitment decreased
Reports may influence actual
behavior
Subjects overestimated session length
Reminder emails pushed
commitment in 2nd phase
Behavior change with increasing
burden
Little control: forgetting
High control: burden
Trigger influence lower than
hypothesized
App usage decreased
Most subjects would report max. 4 weeks
[email protected]://vmi.lmt.ei.tum.de/serena
Please cite this work as follows:
A. Möller, M. Kranz, B. Schmid, L. Roalter, S. DiewaldInvestigating Self-Reporting Behavior In Long-Term StudiesIn: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2013), pp. 2931-2940, Paris, France, April-May 2013.
If you use BibTex, please use the following entry:@inproceedings{chi2013selfreport, author = {Andreas M\"{o}ller and Matthias Kranz and Barbara Schmid and Luis Roalter and Stefan Diewald}, title = {Investigating Self-Reporting Behavior In Long-Term Studies}, booktitle = {Proceedings of the 2013 ACM annual conference on Human Factors in Computing Systems}, pages = {2931--2940}, series = {CHI '13}, year = {2013}, isbn = {978-1-4503-1899-0}, location = {Paris, France}, numpages = {10}, publisher = {ACM}, address = {New York, NY, USA},}