gami cation of on-line surveys for ... - university of guelph
TRANSCRIPT
Gamification of On-line Surveys for Public Health Data Collection
by
Corey P. J. Alexander
A Thesis
Presented to
The University of Guelph
In partial fulfilment of requirements
for the degree of
Masters of Science
in
Computer Science
Guelph, Ontario, Canada
c© Corey P. J. Alexander, January, 2017
ABSTRACT
Gamification of Online Surveys for Public Health Data
Collection
Corey P. J. Alexander Advisor:
University of Guelph, 2016 Dr. Daniel Gillis
Standard survey methods have many issues associated with them, especially when
they contain a large number of questions, or take a long time to complete. These
problems come in the form of user fatigue, a lack of interest in the survey, or inaccurate
and incomplete answers. We hypothesized that the use of badging - a self element
of gamification - would increase completion rates, increase time spent on the survey,
and have no effect on straight-lining behaviour. This research describes the specific
tools that have been developed to investigate the use of Gamification (specifically
badging) and answer these hypotheses. Results indicated that badging had no effect
on completion rates, although this could be an artifact of participant selection bias.
However, the study does suggest that gamification, specifically badging, increases the
time a user spends on the survey.
iii
DEDICATION
For CPJA, because you know.
iv
ACKNOWLEDGEMENTS
I would like to thank my advisor Dr. Daniel Gillis of the School of ComputerScience at the University of Guelph. I view Dr. Gillis as a mentor and a friend.I would like to thank him for the wisdom and guidance that he has provided mewhenever I needed it, not only in my research and academics but also life in general.I am grateful to have had the pleasure to work with and learn from Dr. Gillis. Thankyou.
I would also like to acknowledge Dr. Judi McCuaig of the School of ComputerScience at the University of Guelph as the second reader of this thesis. Dr. McCuaig,much like Dr. Gillis, was always there to support me and guide me in my researchand academics but also in my life. Thank you.
Without the support and guidance of both Dr. Daniel Gillis and Dr. Judi McCuaig,I would not have made it to where I am today. I would like to sincerely thank bothof them for all the support they have given me during my time with them.
I would like to thank Dr. Stacey Scott for being my external examiner and for thevaluable comments she has provided me.
I would like to thank Colin Howes and Dominic Gagne for their help with buildingparts of the experiment. It was a learning experience for all of us and I wouldn’t havemade it through as easily as I did without their help and dedication.
Finally, I would like to thank all of my friends and family who have supported meduring my degree.
v
Table of Contents
Dedication iii
Acknowledgements iv
List of Tables vii
List of Figures viii
1 Introduction 1
2 Literature Review 32.1 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Gamification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Self Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3.2 Social Elements . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3.3 Branching Elements . . . . . . . . . . . . . . . . . . . . . . . 142.3.4 Flow Elements . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3.5 Pros and Cons of Gamification . . . . . . . . . . . . . . . . . 15
2.4 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 Methods 193.1 Recruitment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.2 Standard Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3 Badge Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3.1 Earning a Badge . . . . . . . . . . . . . . . . . . . . . . . . . 263.4 Building the Experiment . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.4.1 Design of the System . . . . . . . . . . . . . . . . . . . . . . . 283.4.2 Design of the Database . . . . . . . . . . . . . . . . . . . . . . 283.4.3 Design of the Front End . . . . . . . . . . . . . . . . . . . . . 31
3.5 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
vi
3.6 Why are we collecting these data? . . . . . . . . . . . . . . . . . . . . 353.7 Delta Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.8 Relative Delta Times . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4 Results and Discussion 414.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.1.1 Results by Question Category . . . . . . . . . . . . . . . . . . 444.1.2 Results by Survey Question Type . . . . . . . . . . . . . . . . 464.1.3 Results by Survey Question Length . . . . . . . . . . . . . . . 494.1.4 Results by Survey Question . . . . . . . . . . . . . . . . . . . 50
4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5 Conclusion 585.1 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
A Ethics Documents 68A.1 Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
A.1.1 Ethics Approval . . . . . . . . . . . . . . . . . . . . . . . . . . 68A.1.2 Ethics Letter Of Information & Consent Form . . . . . . . . . 70
A.2 Survey Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
B Code & Further Tables 81B.1 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81B.2 Category Code to Category Labels . . . . . . . . . . . . . . . . . . . 81B.3 Wilcoxon Rank Sum Results By Question . . . . . . . . . . . . . . . 83
vii
List of Tables
4.1 Badge Survey Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.2 Non-Badge Survey Metrics . . . . . . . . . . . . . . . . . . . . . . . . 444.3 Wilcoxon Rank Sum Results for Category Delta Timers . . . . . . . . 464.4 Wilcoxon Rank Sum Results for Category Relative Delta Timers . . . 474.5 Ordered Category & Median Question Times . . . . . . . . . . . . . . 484.6 Wilcoxon Rank Sum Results for Answer Type Delta Timers . . . . . 494.7 Wilcoxon Rank Sum Results for Answer Type Relative Delta Timers 504.8 Wilcoxon Rank Sum Results for Category Word Category Delta Timers 514.9 Wilcoxon Rank Sum Results for Category Word Category Relative
Delta Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
B.1 Lookup for the Category ID and the Name of the Category. . . . . . 82B.2 Wilcoxon Rank Sum Results for Question Delta Timers . . . . . . . . 85B.3 Wilcoxon Rank Sum Results for Question Relative Delta Timers . . . 88
viii
List of Figures
3.1 The standard survey. . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.2 An image of the badge survey, without badges. . . . . . . . . . . . . . 253.3 An image of the badge survey, with 1 earned badge. . . . . . . . . . . 263.4 Database visualization. . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.1 Plot of time to complete survey. . . . . . . . . . . . . . . . . . . . . . 56
A.1 Ethics Approval Letter . . . . . . . . . . . . . . . . . . . . . . . . . . 69A.2 Letter of Information & Consent Form, Page 1 . . . . . . . . . . . . . 71A.3 Letter of Information & Consent Form, Page 2 . . . . . . . . . . . . . 72A.4 Letter of Information & Consent Form, Page 3 . . . . . . . . . . . . . 73A.5 Survey Questions & Answers, Page 1-2 . . . . . . . . . . . . . . . . . 75A.6 Survey Questions & Answers, Page 3-4 . . . . . . . . . . . . . . . . . 76A.7 Survey Questions & Answers, Page 5-6 . . . . . . . . . . . . . . . . . 77A.8 Survey Questions & Answers, Page 7-8 . . . . . . . . . . . . . . . . . 78A.9 Survey Questions & Answers, Page 9-10 . . . . . . . . . . . . . . . . 79A.10 Survey Questions & Answers, Page 11-12 . . . . . . . . . . . . . . . . 80
1
Chapter 1
Introduction
In 2013, the Public Health Agency of Canada (PHAC) began designing Foodbook
- a survey created by The Outbreak Management Division & Enteric Surveillance
and Population Studies Division in the Centre for Foodborne, Environmental, and
Zoonotic Infectious Disease - to collect data on Canadian food consumption habits
to better understand risks of food borne infections. To collect these data, the PHAC
opted to use a phone survey in place of other survey methods (e.g. pen and pa-
per or on-line data collection tools). However, recognizing the increasing effort and
growing costs of conducting a national survey, and the added challenges related to
data collection using pen and paper, and phone surveys, including for example lack
of user engagement (Harms et al., 2014; Puleston, 2014), premature survey termina-
tion (Downes-Le Guin et al., 2012) as well as the transition from traditional landline
phones to computers, tablets, hand held devices, and smart phones (Alfi, 2011; Dug-
2
gan, 2013), there is a need to develop alternative data collection tools to mitigate
these challenges. This thesis will explore the development and evaluation of one such
tool that combines elements of Gamification within the standard on-line survey frame-
work. Specifically, this research will show that on-line public health surveys will
have improved user engagement by incorporating self-elements of gamification
through badging compared to non-badged surveys.
This document is structured as follows. To understand the challenges associated
with data collection particularly in the domain of Public Health, we begin with a
review of the literature in Chapter 2. This chapter will also explore the use of gam-
ification to improve user engagement. It ends with a review of the thesis statement,
as well as a list of specific hypotheses that will be explored in subsequent chapters.
To evaluate the hypotheses described, Chapter 3 will describe the implementation
of an on-line Public Health survey tool. The chapter will include a list of relevant
data collected, as well as a description of the experimental design and statistical tests
to be used. Data collected are analyzed in Chapter 4, with further discussion and
conclusions provided in Chapter 5. The chapter concludes with a list of potential
future work
3
Chapter 2
Literature Review
This research will show that on-line public health surveys will have improved
user engagement by incorporating self-elements of gamification through badg-
ing compared to non-badged surveys. We begin by reviewing some of the standard
issues with traditional and on-line surveys from the point of view of user experience,
data quality, and data quantity in Section 2.1. This section will explore underlying
issues with traditional surveys as a data collection tool, specifically looking at user
engagement (described in Section 2.2). Following this, in Section 2.3 we are going to
introduce the concept of Gamification as a tool to increase user engagement. Specif-
ically, we will define what Gamification means, describe the major design elements
related to it, the pros and cons of implementing it, and explain how it has been used
in various domains with a specific focus on the Public Health domain.
4
2.1 Surveys
In this section, we will discuss the current state of surveys as a data collection tool,
including the various issues associated with their use, and with a specific look at user
engagement and how it affects the collected data.
Currently, standard survey methods (including traditional methods using paper and
pencil or phone surveys such as those used to collect data for the PHAC’s Foodbook
project) face various issues. These issues include, but are not limited to, surveys being
uninteresting to participants, or time consuming to complete. In both situations,
this can lead to missing, unrepresentative, or incomplete data. Surveys viewed as
taking excessively long to complete or having a large number of questions, can also
affect the completion rates, defined as the number of people who finish the survey
over the number of people who started the survey, and response rates, defined as
the number of people who completed the survey over the number of people who
were contacted to take part in the survey (Cook, 2000; FluidSurveysTM, 2013, 2014;
Kviz, 1977; Mavletova, 2013). This happens as the participant begins to lose their
motivation or interest with the survey, which in turn also increases data missingness
as a result of a lack of engagement with participants, or premature termination of
the survey. That is, a lack of user engagement due to user fatigue and drop-outs
or participants straight-lining the survey, begins to affect the quality of the data
collected. Straight-lining is when a participant picks either the first answer or the
5
first answer they read, without consuming the information being presented to them
(Herzog and Bachman, 1981). Due to these issues, a lot of time and money are
required to gather a representative sample of the population (Greene et al., 2008;
Harms et al., 2014). This issue is especially significant because of the observed drop
in completion rates noted by Downes-Le Guin et al. (2012). Their findings suggest
that as completion rates continue to fall, the issues outlined above (particularly related
to data missingness), are growing.
To provide context, we consider two separate studies which suffered from very
low completion and response rates. Our first example, a study done on e-cigarette
cessation by Siegel et al. (2016) describes a completion rate for their survey of a mere
4.5% of the 222 respondents. While the paper does not explain why the authors saw
such low completion rates, it does point out that the low completion rate means that
the sample is not representative of all smokers who have tried e-cigarettes, limiting
the authors’ ability to generalize their findings. Whatever the cause, users were not
engaged sufficiently to complete the survey. A second example is a study by Downes-
Le Guin et al. (2012) which describes a response rate of 8% that was attributed in
part to the length of time it took to load the survey (or game) before being able
to begin. That is, respondents disengaged before the survey even loaded. Such low
rates for these (and many more) studies introduce many issues for statistical analyses
due to unrepresentative or missing data sets, making it harder to make accurate
observations, generalizations, and predictions about the topics being studied.
6
2.2 Engagement
While useful, traditional pen and paper or phone surveys are often lengthy, requir-
ing a lot of time to complete, or are simply uninteresting to the intended respondents
(Downes-Le Guin et al., 2012; Harms et al., 2014; Puleston, 2014). When implement-
ing a survey, designers are often faced with several key questions, including:
1. How does one get users interested in the survey? (Downes-Le Guin et al., 2012)
2. How does one keep the attention of survey participants? (Downes-Le Guin
et al., 2012; Galesic and Bosnjak, 2009; Kruat et al., 1975)
3. How can one design a survey to increase engagement and completion rates?
(Wright, 2005)
Ignoring these key questions, or not designing with them in mind can lead to
problems with the data that are collected, including increased costs, longer times to
implement the survey, incomplete data sets, issues with representation, and biases
(Greene et al., 2008; Harms et al., 2014; Siegel et al., 2016).
User engagement is defined as “a desirable – even essential – human response to
computer mediated activities” (O’Brien and Toms, 2008) and with regards to surveys,
it is how interested the participant is with the content of the survey (Puleston, 2014).
As O’Brien and Toms (2008) explain, “engaging experiences have the attention of
7
the user.” and point out that “[engagement] manifests itself in the form of attention,
intrinsic interest, curiosity, and motivation.”
On-line surveys are being used to help reduce some of the issues present in tra-
ditional surveys. However, on-line surveys still suffer from some of the same issues
present in traditional surveys, such as keeping participants motivated and interested
in the content of the survey until completion (Cook, 2000; FluidSurveysTM, 2013;
Kviz, 1977; Mavletova, 2013; Puleston, 2014).
As survey completion and response rates continue to drop, there needs to be explo-
ration of alternative data collection methods to increase user engagement, improve
completion rates, and thereby increasing the quality of data being collected. For the
purposes of this thesis, we define user engagement as how interested the participants
are with the content of the survey (Puleston, 2014).
To make on-line surveys more interesting and enjoyable, researchers have incor-
porated additional visual and interactive features within their on-line survey design,
including visually appealing surveys, animated survey components, and gamified sur-
veys (Downes-Le Guin et al., 2012; Leaman, 2014).
One study points out that making on-line surveys more enjoyable is an important
goal, since it helps to avoid negative respondent behaviours, such as premature ter-
mination (Cechanowicz et al., 2013; Harms et al., 2014). It also keeps users engaged
8
and increases completion rates (Harms et al., 2014). Recently, studies investigating
different methods to make on-line surveys more appealing have shown that Gami-
fication tools show promise in dealing with both user engagement and completion
rates of on-line surveys (Cechanowicz et al., 2013; Goehle, 2013; Hamari et al., 2014;
Leaman, 2014).
Researchers have used Gamification of on-line surveys/activities to increase user
engagement [and completion rates] in several domains. It has been successful in
the educational domain, where on-line homework was Gamified to increase user en-
gagement (Goehle, 2013). Specifically, on-line homework was gamified by adding
achievements that students could unlock throughout the course.
Gamification has also been used “in the real world of corporate learning” (Leaman,
2014), by adding small 15 minute learning games to be played at work, users become
more engaged, and there is an increase in user performance. Another study, used on-
line surveys for market research. Participants were asked to complete various tasks
related to market research, such as image identification, slogan matching, and five-
second memory quizzes (Cechanowicz et al., 2013). On-line surveys have also been
actively used in the public health domain, for investigations of infectious diseases,
like the H1N1 virus (Bults et al., 2011), e-cigarette satisfaction (Siegel et al., 2016),
and substance abuse among young adults (Bachman et al., 2006).
9
This section has shown how researchers have explored user engagement to improve
data collection activities. In many cases, researchers have used Gamification as a
means to improve engagement. The following section will explore Gamification in
further detail.
2.3 Gamification
Gamification is a tool that has been used in numerous disciplines and across mul-
tiple applications to improve user engagement. As such, this section will introduce
the concept of Gamification; explaining the major design elements of it, as well as
the pros and cons of implementing it, and showing how it has been used in various
domains including Public Health.
One definition of Gamification includes the act of taking a task that is to be
completed and making it into a game (Goehle, 2013; Leaman, 2014). As proposed
by Deterding et al. (2011), “Gamification is the use of game design elements in non-
game contexts.” However, Goehle (2013) is more specific, stating that “Gamification
refers to the use of video game mechanics and techniques to increase engagement
and interest in an activity which is, usually, unrelated to video games.” The idea
behind adding various video game mechanics to on-line surveys is that it will make
the surveys more enjoyable or exciting for the respondents (Seaborn and Fels, 2015).
This idea has come about due to the popularity and success of video games, and their
ability to hold the attention of their users (Seaborn and Fels, 2015). For the purposes
10
of this review, the more specific definition of Gamification by Goehle (2013) will be
used.
Gamification has been used in various domains, including Education, Health and
Wellness, on-line Communities and Social Networks, as well as Crowdsourcing (Seaborn
and Fels, 2015). The literature survey done by Seaborn and Fels (2015) shows a broad
range of studies that have implemented various elements of Gamification, ranging
from points and badges, to leaderboards and in-game avatar progression. Looking
specifically at the studies related to Public Health and Wellness, 19 papers were iden-
tified by Seaborn and Fels (2015), with 59% of the papers indicating positive effects
for improving health and well-being while the other 41% were mixed. The papers fo-
cussed on 1) active video games, 2) behavioural modification, 3) rehabilitation games,
and 4) educational games. Although it has been noted that Gamification has been
used in the Public Health domain (such as the study by Cafazzo et al. (2012) which
looked at creating a gamified mobile application to help monitor blood glucose levels
in diabetics) and that surveys are being used to collect public health data or to assess
the effectiveness of gamified tools for public health, there is little evidence to suggest
that the combination of gaming elements within survey design has been investigated
(Johnson et al., 2016; Seaborn and Fels, 2015).
Given the challenges identified with a lack of user engagement within on-line sur-
veys, leading to low completion and response rates, and the effect these have on data
11
quality, and considering that Gamification has been used to improve engagement, it’s
worth exploring this as a potential tool to improve survey design.
To successfully employ Gamification within survey design, it’s important to consider
the four elements of game design. These elements are: self elements, social elements,
branching elements, and flow elements (Csikzentmihalyi, 1975; Hemley, 2012; Hsin-
Yuan Huang and Soman, 2013; Shin, 2006). Each are described in sections 2.3.1
- 2.3.4. For the purposes of this thesis, the focus will be on the self elements of
Gamification.
2.3.1 Self Elements
Self elements are design elements that get a user to compete with themselves (Hsin-
Yuan Huang and Soman, 2013). These elements include but are not limited to points,
badges, or time trials. An example of self elements for surveys would be earning points
or badges when completing a question or a category of questions (Hsin-Yuan Huang
and Soman, 2013).
The use of self elements through addition of points and badges has been successfully
implemented for both on-line based homework as well as market research surveys
(Cechanowicz et al., 2013; Goehle, 2013). The on-line homework study found that
most of the participants in the study actually kept track of the badges that they had
earned during the course of the experiment and actively tried to earn badges. Goehle
12
(2013) concludes that adding elements of Gamification to the on-line homework was
a success and increased the engagement of the users.
In comparison, Cechanowicz et al. (2013) used a bunch of different characteristics
of video games, including a point based system. They also looked at implementing
different levels of Gamification by using one or more gaming elements in their exper-
iments. A standard baseline survey was presented as well as a fully gamified version,
including self elements of Gamification. However, their experiments did not consider
the individual effect of the gamified elements, instead they were all presented at one
time. This poses an issue when trying to determine the effect that the self elements
used had on the participant. Though it should be pointed out that Cechanowicz
et al. (2013) concluded, “Our evaluation demonstrates that the motivational bene-
fits of games increase with the level of gamification, and that these benefits apply
regardless of age, gender, length of a panelist’s tenure, and game experience.” This
provides the impetus to explore the utility of self elements as a method to improve
the engagement of participants.
Various social media sites also use some type of a “scoring” system such as likes,
shares, and follows in order to get users more engaged on the sites (Birman, 2014).
This type of system can be seen as both a self element and social element of Gamifica-
tion because users compete with themselves and other users to earn more of each type
of score (e.g. likes, shares, follows). Currently, there is little evidence to suggest that
13
self elements have been used specifically in on-line Public Health surveys (Johnson
et al., 2016; Seaborn and Fels, 2015).
2.3.2 Social Elements
Social elements include methods of creating competition/interaction between users
(Hsin-Yuan Huang and Soman, 2013). Examples of social elements include leader-
boards, virtual goods, or cooperative goals. Social elements can be included in surveys
simply by giving the participant feedback on how they compare to others who have
completed the survey (Hsin-Yuan Huang and Soman, 2013). Leaderboards and other
on-line social elements have been included in on-line homework to show students how
they compare to others in a course (Goehle, 2013). Social elements have also been
used in a study that attempted to use crowd sourcing as a method to get participants
involved in pro-environmental activities. Virtual and Financial leaderboards were
used to create competition between the various participants and as Massung et al.
(2013) noted, “Quantitative results suggested that the participants near the top of
the Virtual and Financial leaderboards were actively competing for the top position,
and that this competitive aspect can provide extrinsic motivation to maintain en-
gagement with the app.” This shows that social elements of Gamification do have
the potential to increase user engagement. Social elements have been used in various
on-line surveys but not specifically in the domain of Public Health.
14
2.3.3 Branching Elements
Branching elements provide users with the ability to reach a goal along multiple
different pathways; they feel unique to the user, but all lead to the same result (Hsin-
Yuan Huang and Soman, 2013). To include branching elements in on-line surveys,
the user could be given the choice of different sections to complete, allowing them
to take different ‘paths’ to complete the survey, similar to interactive story telling
such as a choose your own adventure type of activity. To our knowledge, branching
elements have not been used in Public Health surveys.
2.3.4 Flow Elements
Flow elements immerse the users within the game (Hemley, 2012). In this way,
users lose track of time and are highly focused on their task (Hemley, 2012; Hsin-
Yuan Huang and Soman, 2013). Flow elements give the user clear goals, involving
a high degree of concentration for a task, or giving the user a sense of personal
control over the situation/activity (Hemley, 2012). Flow elements also give users new
motivations as the task progresses to help avoid them becoming bored or frustrated
by doing the same task at a specific level for a long time (Sy, 2010). The study
completed by Sussane et al. (2011) shows the use of Gamification in on-line idea
competitions. The elements added included the use of “game points, social points,
and leaderboards.” Although these elements fall into the other categories such as
self elements or social elements, the use of them can increase the user’s immersion
15
in the task at hand. Points were earned by the participants for interacting in the
system in various ways, like contributing ideas, writing comments, evaluating ideas,
or leaving other participants a message. After the competition, a survey was used
to collect information about the participants (demographics, general experience with
these types of competitions, and an evaluation of this competition.), this survey was
not gamified but was used to gather information about the participants feeling of
immersion in the activity. The data collected showed that the implementation of
these elements did in fact increase the participants enjoyment of the competition,
and they reported the feeling of time passing faster (Sussane et al., 2011). Flow
elements have not been used in Public Health surveys.
After having defined Gamification, and looking at some of the places that Gami-
fication has been successfully implemented, we will take a look at some of the pros
and cons for implementing elements of Gamification in a system, with a specific look
at on-line Public Health surveys.
2.3.5 Pros and Cons of Gamification
While researchers have begun to explore the use of Gamification in various domains
and across numerous applications, it is important to review some of the challenges and
opportunities that come with implementing Gamfication. In this section we explore
the pros and cons associated with Gamification, with a focus on survey design.
16
From the literature reviewed, it can be seen that the pros of applying Gamification
to on-line surveys are that it may increase the engagement of users, by giving users
different ways to measure their progress throughout the survey, as well as short term
goals to achieve while participating in the survey, leading to users having a feeling of
accomplishment throughout the survey (Downes-Le Guin et al., 2012; Goehle, 2013).
As well, Gamified surveys minimize any negative emotions or behaviours, such as
straight-lining or inaccurate data sets that may be encountered when completing
the survey (Bhaskaran, 2013; Hsin-Yuan Huang and Soman, 2013). Another pro
of Gamification of surveys, as Cechanowicz et al. (2013) points out is that, “The
motivational benefits of games increase with the level of gamification.” This suggests
that the more game-like a survey is, the more likely it is to have the participant
engaged and immersed in the survey.
Of course, Gamification also has a unique set of challenges and issues that must
be considered when trying to implement it within a system. Some of the cons to
be considered when gamifiying a survey are: structuring the survey in such a way
that it is a game can be difficult as some types of surveys lend themselves more
freely to be gamified than others (Goehle, 2013). Another con includes the increased
time and cost of setting up the survey. For example, Downes-Le Guin et al. (2012)
indicated that “The Gamified presentation consumed more than twice as many hours
(as well as significant subcontracted resources to create original artwork) to conceive
and design the game structure, narrative and artwork, and to program and iteratively
17
test” which greatly increases the cost and time of the survey.
2.4 Thesis Statement
Sections 2.1 and 2.2 explained how standard surveys are suffering, specifically
through a lack of user engagement which affects the quality of the data being col-
lected. From there, Section 2.3 introduced the concept of Gamification and showed
how it has been used in various domains to increase user engagement, leading to
better quality data being collected. With these sections in mind, this section will
propose that Gamification can be used to increase the user engagement of on-line
Public Health surveys and explain the steps needed to achieve this goal. It will also
outline a set of hypotheses that we seek to address in this thesis.
Public health surveys are not engaging which has led to reduced completion rates
and response rates, causing a decrease in data quality. Gamification at a broad level
has been applied to many other domains, such as marketing research, education,
and public health (in general) (Batterham, 2014; Cechanowicz et al., 2013; Hsin-
Yuan Huang and Soman, 2013). And with the show of promise for Gamification
tools in other domains for increasing user engagement, it becomes clear that there is
motivation to explore Gamification of on-line Public Health surveys to address some
or all of the issues identified in the previous sections. This research will show that
on-line public health surveys will have improved user engagement by incorporating
18
self-elements of gamification through badging compared to non-badged surveys.
To demonstrate this, this thesis will investigate the following hypotheses:
1. Gamification can be used to increase participant completion rates. More specif-
ically, the use of self elements (such as badges) will increase participants com-
pleting the survey.
2. The participant will spend more time on the gamified survey as opposed to the
non-gamified survey. More specifically, the use of self elements will engage the
participant and they will take more time to answer the survey.
3. There will be no difference between a gamified survey and a non-gamified survey
in terms of participants straight-lining the survey. More specifically, the partici-
pant is just as likely to straight-line a gamified survey as they are a non-gamified
survey.
By answering these hypotheses, we will show that the use of badges have an impact
on the participants’ engagement.
19
Chapter 3
Methods
The goal of this research is to investigate the use of Gamification’s self elements
(e.g. badges or points) by applying them to on-line Public Health surveys, thereby
leading to increased participant completion rates and increased time on survey, while
attempting to reduce participant straight-lining throughout the survey. It is hoped
that increasing engagement in this way will lead to the goal of increasing the quality
of data being collected. To achieve the goal of this research, it is necessary to satisfy
the following objectives:
• develop a standard survey, without any of the elements of Gamification included,
to act as the baseline for the experiment.
• extend the standard survey to incorporate self elements. This will be an exten-
sion to add badges to the survey.
20
• collect data using the various surveys.
• analyze the data, comparing the two different surveys.
The reason for building the platform was to allow access to gather the data needed
to measure engagement, such as the timers per question and the badges. It was
important to build the platform so that we could ensure that the surveys were the
exact same with the only difference being the badges. This allowed for more control
over the experiment and the data that was being gathered.
Each survey (badged, and non-badged, as described in detail in sections 3.2-3.3)
will be on the topic of food and food consumption habits, asking similar questions to
the food consumption habits survey that the PHAC conducted. This allows for the
data collected to potentially be compared in the future to those data collected by the
PHAC to determine if the quality and quantity of data collected through gamified
surveys differ from those collected through non-gamified surveys. Both the badged
and non-badged surveys will contain the same number of categories and the same
number of questions per category. The only difference between the surveys will be
that the badge survey will give out badges to the participant at random points during
the survey. This will be described in detail in section 3.3.1. For a complete list of
the survey categories, questions and answers presented to the respondents, refer to
Appendix B.1 and A.2.
21
The survey was designed to support three different question formats: a radio but-
ton, checkbox, or textfield. Radio button questions will typically be structured in the
following manner:
Q: Did you consume <food item> in the past 7 days?
a) Yes.
b) No.
c) Prefer not to answer.
d) I don’t know.
Checkbox questions will typically be structured in the following manner:
Q: Did you consume any of the following fresh or frozen <food item>?
� Strawberries
� Blueberries
� Raspberries
� Blackberries
� Other berries
� I didn’t eat any of these foods
� I don’t know
� I prefer not to answer
22
Textfield questions will typically be used in a situation where a radio button or
checkbox does not make sense, such as asking for the postal code. Due to the structure
of most questions, textfields have only been used in the Demographics category of
the surveys.
3.1 Recruitment
To recruit participants to the survey, we used a social media and e-mail campaign.
For social media we used Twitter, Facebook, and personal blogs to reach out to
potential participants to inform them that a survey was being conducted. E-mails
were also sent to potential participants, and shared on various list-serves.
The experiment was presented to potential participants as the “Alternative Data
Collection to Understand Food Behaviours” survey. Before beginning the survey,
participants were presented with a Letter of Information and Consent Form (see
Appendix A.1, Figures A.2-A.4). As described in the Letter of Information, users
were informed that there were two types of surveys (standard or badged) but they
would not know which survey type they would get until they began. That is, users
were aware that they could earn badges while completing the survey. It should also
be noted that there was a chance that a participant with the badged survey might
never earn a badge.
23
Several comments need to be outlined regarding the social media campaign and
selection bias. First, there could be the potential for bias in the selection of partic-
ipants since many of the respondents targeted through Facebook were largely made
up of other university students. Further, due to the nature of the e-mails being sent
to various university faculty members (from different universities as well) and list-
serves, there was a potential bias present as some of these participants might also
be researchers and thus more likely to complete the full survey. This could present
an issue when it comes to representativity within the experiment, and our ability to
generalize the findings.
3.2 Standard Survey
The Standard Survey (SS) will be designed to have 1 category per page, with 9
categories in total for the survey. The survey pages will make use of colour (i.e.
they will not be simply black and white), and will include navigation controls (e.g.
next or previous section), and the ability to leave at any time. Each section will
include a short description for the participant. There will be no badges included.
This survey is expected to have the lowest amount of questions or sections completed
and the lowest completion rate compared to the surveys incorporating self elements
of Gamification. It is expected that this type of survey will have a higher amount of
participants straight-lining the survey, due to a lower motivation to complete it since
there are no elements of Gamification present. Figure 3.1 shows an example of the
24
standard survey.
Figure 3.1: An image of the standard survey.
3.3 Badge Survey
The Badge Survey (BS) will be an extension of the SS that will incorporate a
badge system. The respondent will be rewarded with badges at random, this will be
discussed in section 3.3.1. By including badges into the survey, it is hypothesized that
participants will be more likely to complete the entire survey, since the participant
25
will want to continue earning badges (Birman, 2014). This version of the survey is
expected to have a higher number of questions answered, and a higher completion
rate compared to the SS. It is also expected that this type of survey will have a lower
amount of participants straight-lining the survey, due to being more motivated to
continue earning badges (Birman, 2014; Hamari, 2015). Figure 3.2 shows an example
of the badged survey with no badges added while Figure 3.3 shows an example of the
badged survey, with one badge earned.
Figure 3.2: An image of the badge survey, without badges.
26
Figure 3.3: An image of the badge survey, with 1 earned badge.
3.3.1 Earning a Badge
To determine if a participant should earn a badge while they are completing the
survey, a test similar to the FizzBuzz test (Ghory, 2014) is used. When a participant
completes a question, including changing their answer, a time stamp is taken. If
the time stamp is divisible by 20, which was chosen arbitrarily, the participant will
earn a badge. This means that a participant will now have to ability to earn at
least one badge per question if every time stamp taken happens to be divisible by
20. This also means that if the participant chooses to change an answer, they will
27
also have the chance of earning another badge, allowing the badges to be seen as
random. The decision to randomize the badge allocation was to try and discourage
participants from only completing the requirements to earn the next badge instead
of being focussed on the survey.
When a badge is earned, a modal window pops up that the participant can close,
or wait for a maximum of 3 seconds for the window to close itself. This becomes
important when looking at Hypothesis 2.
Due to the nature of every question having the ability to have the participant earn
a badge, badges will look the same for each individual category.
3.4 Building the Experiment
This section will describe how the system was designed and implemented, along
with the design of the database and the technologies that went into the project.
For this experiment, a survey system was designed and implemented. This system
was designed to be easily extensible and scalable, allowing for multiple surveys to be
hosted, requiring an extensive amount of time designing and planning the system.
The database has been designed to allow for multiple different questions, categories,
and surveys to be stored. Figure 3.4 shows the relationships between the various
tables that make up the database and will be discussed more in section 3.4.2.
28
3.4.1 Design of the System
The front end was designed using HTML5, AngularJS (JavaScript), and Bootstrap,
(Cascading Style Sheet). This allowed for the design to be easily extensible with
limited modifications needed to implement new things. The back end of the system
was designed using PHP and MySQLi (Changed to MySQLi because MySQL was
depreciated) and the PHPSlim Framework in order to create a simple yet scalable
API and database, allowing for the back end to be easily extended if there is new
data to be stored or new features are added to the system.
3.4.2 Design of the Database
As for storing the different pieces of information, everything has been split into
separate tables according to how the data is related, with foreign keys being used to
link the data together. For example, there is a table to store the information about a
question. This table does not store the information about the answer set or the link
set, instead storing only foreign keys to the respective tables.
Here, we describe how it all fits together, and how the surveys, categories, questions,
and answers are related.
A survey knows what categories are in it and what type of survey it is (Standard
or Badged). This allows for easy extension in the front end to include different types
29
Figure 3.4: An image of the database with relationships between the various tables.
of surveys if more elements of Gamification are added in future work.
A category is made up of questions, each with their own set of answers, that are
going to be displayed in the survey. The survey will display one category per page,
with all of the questions in the category being displayed at once.
A question is made up of multiple parts, including the question’s text, the answer
type (radio button, checkbox, textfield), the set of answers (answers the participant
30
can choose from as long as answer type is not a textfield), and a set of links (links
to other questions to help control the flow of the survey). The set of links has been
included to try to mimic the PHAC survey; when a participant answers a question,
depending on the answer other questions may or may not appear. For example, if a
participant answers that they have not eaten apples or apple products, the following
questions should not be related to apples or apple products. Currently, links are not
implemented but have been included in such a way that the surveys can easily be
extended to accommodate them.
There are also various pieces of information about the participant that are being
stored, such as their answers to the questions (only if they want this data to be
recorded, by completing the survey) and their measures (outlined in section 3.5).
Since there are no participant accounts, the participants’ measures can be stored
without their consent, due to the data being completely anonymous, with no way to
trace a set of measures back to a specific participant. For example, if information
about a participant’s measures were to be leaked, knowing that it took a participant,
with a numeric ID of x, t amount of time to complete question y, does not give
information about who that participant was.
The various measures that are being collected in relation to a participant are also
stored in their own respective tables, such as a table for timers and badges. By having
these all separated into their own tables, it is very easy to extend the measures that
31
are being stored, by having a table that stores foreign keys to each of the other
measures tables and a foreign key to link to a participant.
With the consent of the participant (by completing the survey and pressing ‘Submit
Survey’), their answers are being stored in the database as well. The format for the
participants’ answers depends on the given answer type, since there are currently
three different types of answers (radio buttons, checkboxes, and textfields). If the
answer type is a radio button, a string that consists of multiple pairs of values, the
question id and the index in the answer set array for the selected answer will be
stored. For a checkbox, an array of true/false, that indicate which answers were
selected, will be stored in place of the index value listed above. And for a textfield,
the text submitted will be stored. Since the questions know which set of answers are
linked to them, it will be easy to look up the participants’ answer.
3.4.3 Design of the Front End
A participant is brought to the landing page, on which the information about con-
sent for the survey is displayed and is presented with the option to accept these terms
and begin the survey or to deny them/leave the survey. Once the participant begins
the survey, multiple timers will be started to record the time it takes to complete a
question, category, and the survey. When a question is answered, a time is recorded.
If the participant goes back to change an answer, the timer will be updated with a
32
new time, with the same holding true for the categories. After the participant sub-
mits a survey, the time is recorded and there will be no way for them to go back and
change their answers.
While going through the survey, the participant is not allowed to move onto another
category without answering all questions in the section. However, the questions can
be answered in any order within a category. The order can then be determined simply
by ordering the times observed. The surveys will also include a button allowing the
participant to leave at any time throughout the survey. This allows the participant
to stop doing the survey at any time if they decide to no longer participate. If the
participant closes their browser, the measures will be stored. That is, their timing
data, or number of badges if applicable, will be stored. However, their specific answers
will not be stored. If the participant loses their internet connection during the survey,
the times for the completed categories will be stored and times for the current category
will be lost.
At any point when the participant attempts to leave, the participant’s measures
will be saved. The only time the participant’s answers will be saved is if they complete
the survey and press the “Submit Survey” button. Upon completion, the participant
will be shown a thank you page for each survey type.
Other pages of the website include:
33
• A page thanking the participant for participating, and
• A page informing participants that they cannot participate without agreeing to
the consent form.
3.5 Data Collection
This section will outline the data that are necessary to investigate the hypotheses
outlined in this thesis, as well as other data that are being collected that could be
useful in answering potential questions of interest. The data that will be collected
to help answer the questions outlined in Section 2.4 are the following, and will be
discussed in full detail in Section 3.6.
• Time spent on a question (in milliseconds),
• Time spent on a category (in milliseconds),
• Time spent on the survey (in milliseconds),
• The deltas of the timers collected,
• The relative delta of the timers collected,
• Number of questions completed,
• Number of questions per category completed,
• Number of categories completed,
34
• The number of people that started the survey,
• The number of people that came to the site,
• The number of people that did not take part in the survey, and
• The number of people that completed the survey.
In addition to these data, the following information will be collected to help answer
other potential questions of interest:
• Basic demographics (e.g. age, gender, education, and income levels), and
• The answers provided by the respondent.
The respondents were required to answer the questions in each of the categories
in the following order: Demographics, Food Habits, Food Insecurity, Fruits, Vegeta-
bles, Meat and Alternatives, Dairy and Alternatives, Grains and Alternatives, Other
Foods. Participants were asked demographic questions at the beginning of the survey
to try to eliminate user fatigue that may have occurred if demographics were located
at the end of the survey. While this does not have an impact on the research question
describe by this thesis, it becomes important for future work where it is necessary to
compare the survey results (i.e. the participant responses to the food survey) with
the survey results collected by the PHAC.
35
3.6 Why are we collecting these data?
It should be noted that although basic demographics are being collected, and they
could be used to separate the data collected by different user groups, like age or
gender, this thesis is looking specifically at the use of the self elements at increasing
overall user engagement. As such, breaking the data up by user groups (although
interesting) is outside the scope of this thesis. This is because we are looking at how
the self elements affect all participants and not individual sub-groups. However, the
data is being collected to allow for further investigation to be done in the future.
Various timers will be collected from the participants and used in multiple ways.
The timers will be used to determine if a participant has straight-lined one or more
categories, or the entire survey, and the order that a participant has answered the
questions. The timers being recorded include: the time spent on a question (tquestion),
the time spent on a category (tcategory), and the time spent on a survey (tsurvey). These
timers are easily scalable for multiple surveys. From here, each of the types of timers
(question, category, survey) had the delta and relative deltas calculated. For the
deltas, the equation used was tx+1 - tx where x was either question, category, or
survey, and for the relative deltas the equation used was (tx+1 - tx)/tx+1, again where
x was either a question, category, or survey. A Wilcoxon Rank Sum test can be used
to determine if there is a difference between the surveys when it comes to the delta
timers and relative deltas. To determine if straight-lining has occurred, it would be
36
expected that the timers would be relatively small and that there may be multiple
timer groups within a survey; a group of very small timers, showing participants
completing the survey quickly, and a group of big timers, showing participants are
taking more time to complete the survey. It is currently assumed that participants
completing the survey rather quickly may not be as engaged in the content of the
survey as those taking more time are. It is assumed that if participants are taking
longer to complete questions or categories this could be an indicator of the participant
being more engaged during the survey.
The number of people that came to the website will be a non-negative value, that
does not represent the actual amount of people that have come to the website but
instead represents the maximum people that could have come to the website. This
value could be inflated due to people refreshing the website, or accidentally closing
the page and having to come back to the site. The first problem is more difficult to
deal with since it is hard to figure out if a participant is refreshing the page but the
website has been equipped with the ability to stop a participant from accidentally
navigating away.
The number of people that have clicked the deny button, denying/ refusing to
accept the terms of consent, will also be a non-negative value. it does not represent
the actual amount of people that have denied/ refused to accept the terms of consent
and left the survey but will be a lower bound on how many participants have not
37
participated. This is because people do not have to click on the deny button for them
to leave the survey. From here an approximation of the amount of people who did not
participate in the survey can be calculated by removing the people who started the
survey from the amount of people who came to the site. This will give the equation
pnotParticipate = pcame - pstarted, where pnotParticipate is the number of people who did
not participate in the survey, pcame is the number of people who came to the website,
and pstarted is the number of people who started the survey. This will only be an
approximation because the number of people who came to the site is not a perfect
representation of the actual number of people who came to the site; the same holds
true for the people who started the survey.
The number of participants that have completed they survey will be the number of
answer sets that are stored, since the only way to store an answer set is to complete
the survey. This piece of data will be used to help determine if there is statistical
evidence to prove that the completion rates of the surveys that have been implemented
are different from one another and if the difference is due to the use of Gamification.
The timers will help to determine how far into the surveys the participants are
getting before quitting or before a noticeable amount of straight-lining or termination
has occurred. It is expected that if straight-lining has occurred, the timing data that
was collected will begin to speed up, while if termination has occurred, the timing
data will stop at a specific question.
38
The participant has multiple methods of leaving the survey; clicking on the exit the
survey button, or by completing the survey. If a participant selects the first option,
the only data that is stored will be the measures of the participant. From here it
is possible to get information on how many questions the participant completed, the
order of the questions and how long it took to complete them. If the participant uses
the second option (completing the survey) to leave, they will also be given the option
to save their answers or not.
Each of the questions will be split into one of three arbitrary categories, based on
the length of both the question and answer text, or the number of words contained
in both. Here, answer text refers to the length of prescribed answers such as those
provided in radio button or checkbox style questions. This excludes answers where the
participant would input using a textfield which which is only used in one question in
the demographics section (i.e. Enter your postal code). The categories are as follows,
short word length; 0 words inclusive to 25 words exclusive, medium word length; 25
words inclusive to 35 words exclusive, and long word length; 35 words inclusive to 50
words inclusive (since the longest question and answer text was 49 words).
The word length categories can be used to determine if participants are more en-
gaged during a specific word length category, since both surveys have the same ques-
tions and therefore the same word length categories, it would be a good indicator of
participants being more engaged for the badged survey or the non-badged survey.
39
3.7 Delta Times
Delta times are calculated using the formula: tx+1 - tx where x represents a question.
Delta times are used to determine the total length of time it takes a participant to
answer a particular question. For example, if the participant answered the first 4
questions of the survey in 10s, 22s, 35s, 50s, respectively, the delta times would be
10s, 12s, 13s, 15s. The delta times were calculated for every question and these were
used to determine delta times for answer type, question word length, and category.
For example, to determine the delta time for the radio button answer type, we simply
summed up the delta times for each question that used this answer type. That is, for
each participant we calculated the delta time per question, their delta time for each
of the 3 answer types, the delta time for each of the 3 question word lengths, and the
delta time for each of the 9 categories. We did this for each participant who answered
the badged survey and the non-badged survey providing us with a distribution of
results that could be compared between the two survey types.
3.8 Relative Delta Times
The relative delta times show the relative difference in time taken to complete two
questions. The formula for the relative delta times is (tx+1 - tx)/tx+1, again where x
represents a question. The relative delta times were calculated for each question and
summed in the same way as described in the previous section for each of the answer
40
types, question word lengths, and categories. The relative delta times are useful
because they allow a comparison between participants to see if there is a difference,
relatively, between the participants being compared. That is to say that if there is
a difference between two participants, the difference will be apparent when viewing
the relative delta times because there will be a huge difference in one or more of the
relative delta time values.
41
Chapter 4
Results and Discussion
Data collection occurred over the course of 4 weeks in March, 2016. Participants
were recruited through an email campaign, as well as through a social media cam-
paign that included Twitter, Facebook, and blog posts (e.g. https://danielgillis.
wordpress.com/2016/03/03/understanding-food-behaviours-survey/). In Sec-
tion 4.1 we provide summary and analyses of data collected during this period, fol-
lowed by a discussion of the findings in Section 4.2.
4.1 Results
During the course of the data collection period, the total number of visits to the
website was 457. This may represent 457 unique individuals but there was no way to
track this due to the use of social media as a method for recruitment. Of these, 111
people (roughly 24%) decided to take part in the survey, while 346 did not. Of the
42
Survey Started Completed Unfinished
Badge 54 45 9
NonBadge 57 46 11
Total 111 91 20
Table 4.1: Completion data collected for each type of survey given.
people who started a survey, 91 (82%) completed the survey, and 20 left the website
before finishing their survey. Table 4.1 breaks down further this information based on
the type of survey a user was randomly assigned; badged or non-badged. Specifically,
54 of the 111 people who started a survey were given a badge survey to complete,
while 57 were given a non-badged survey. Of those who started a badge survey, 45 of
them were able to finish the given survey leaving only 9 unfinished. Similarly, 46 of
57 people were able to complete a non-badged survey. That is, approximately 83%
and 81% of respondents completed the respective surveys.
Respondents who answered the non-badged survey spent an average of 553.53 sec-
onds on the survey, while badged respondents spent an average of 635.34 seconds.
That is, badged survey respondents spent on average 81.81 seconds longer answering
the survey. Median time spent on the surveys was 519.81 seconds and 553.53 seconds
for non-badged and badged surveys, respectively. The difference in medians is 33.72
seconds.
43
By design, respondents who were randomly assigned a badged survey received a
random number of badges. Table 4.2 outlines some basic statistics describing the dis-
tribution of badges received. The maximum number of badges given to a participant
during the experiment was 13, the minimum number of badges given was 0, and the
median number of badges that participants earned was 5. Table 4.2 also shows the
maximum estimated time that earning a badge could add to a participant’s survey
completion time if they were to allow the badge pop up to stay on screen for the full 3
seconds. Recall that the badge pop ups were designed to appear for at most 3 seconds.
Based on this, the longest time added to the total survey completion time (assuming
the extra time is only related to the badge pop ups) was an extra 39 seconds. That is,
we’d expect that the badged survey would be on average 14.7 seconds longer, and no
more than 39 seconds longer than the non-badged survey, if the badges had no effect
on the time a user spends on the survey. Further, we’d expect the median difference
between survey completion times to be approximately 15 seconds.
A Wilcoxon Rank Sum test was preformed on each of the survey types to determine
if there was a difference between the time to complete each of the surveys. The results
of the test showed a p-value of 0.057. Although this p-value was not statistically
significant, further investigation was explored to determine if there were differences
based on different factors. Specifically, we analyzed the time to complete the survey
based on the survey categories, the different answer types, the length of text in each
44
Statistic Number of badges collected Estimated extra time (s)
Mean 4.9 14.7
Median 5.0 15.0
Maximum 13.0 39.0
Minimum 0.0 0.0
Table 4.2: Summary of badges collected by participants in the badged survey, andthe expected extra time these would add to the total survey if the user allowed thebadge pop up to remain on screen for 3 seconds per badge as designed.
question, and by question, to see if there were any differences between the time to
complete the two surveys for any of these factors. A Wilcoxon Rank Sum test was
performed for each of the factors. Wilcoxon Rank Sum tests were applied to both the
delta times, and the relative delta times for each of the factors. Results are provided
in Sections 4.1.1-4.1.4.
4.1.1 Results by Question Category
The first factor examined was the question categories. Wilcoxon Rank Sum tests
were performed to see if there was a significant difference between the time required
to complete each question category. The null hypothesis of each test was that there
was no significant difference between the non-badged and badged category times. A
full list of categories (including the number of questions per category) can be found
in Appendix B.1.
45
Tables 4.3 and 4.4 both show the results from the Wilcoxon Rank Sum tests that
were performed on the aggregated delta times and the relative delta times per cate-
gory, as well as the medians of both sets of times for both surveys. The tables also
include the p-values associated with each of the test statistics, where p-values that
are less than 0.05 represent statistically significant results and are indicated with bold
face font. It should be noted that in the delta times result set, the demographics and
food habits categories have p-values less than 0.05, where in the relative delta timers,
the demographics and food insecurity categories are both close to 0.05, with their
values being 0.065 and 0.078 respectively. Further, it should be noted that the me-
dian times for the badged survey are greater for every category than the non badged
survey, except for food insecurity category.
In the case of the significant results for the tests performed on the delta timers, the
badged survey took longer than the non-badged survey. In particular, badged survey
respondents spent approximately 20 seconds longer on the demographics category,
and 7 seconds longer on the food habits category questions than non-badged survey
respondents. That is, the first two categories presented to the respondents had median
timers that were significantly different between badged and non-badged surveys. In
both cases, the badged survey had a higher median timer.
Table 4.5 provides another view of the data. Here we divide the median time per
category for both badged and non-badged surveys by the number of questions in each
46
category. For both badged and non-badged surveys, the proportion of median time
spent per question seems to increase towards the end of the survey.
Category p-value Median B Median NB
Demographics 0.007 104.165 84.708
Food Habits 0.020 33.784 26.344
Food Insecurity 0.633 45.328 45.442
Fruits 0.145 58.378 56.043
Vegetables 0.086 62.625 59.543
Meat and Alternatives 0.336 78.868 77.833
Dairy and Alternatives 0.377 48.050 45.221
Grains and Alternatives 0.586 36.057 33.530
Other Foods 0.158 48.106 41.795
Table 4.3: Wilcoxon Rank Sum results for both surveys, of the delta times percategory. Where Median Badged and Median Non-Badged are the medians of eachcategory for the badge survey and non-badge survey respectively.
4.1.2 Results by Survey Question Type
The questions were then categorized by answer type: radio button, checkbox, or
textfield. In both surveys, the only textfield was for the postal code. All other
questions were either a radio button, or a checkbox, with 26 answers being radio
47
Category p-value Median B Median NB
Demographics 0.065 2.684 2.558
Food Habits 0.921 0.256 0.256
Food Insecurity 0.078 0.263 0.289
Fruits 0.853 0.265 0.268
Vegetables 0.805 0.214 0.221
Meat and Alternatives 0.412 0.218 0.230
Dairy and Alternatives 0.312 0.106 0.112
Grains and Alternatives 0.249 0.072 0.081
Other Foods 0.780 0.087 0.091
Table 4.4: Wilcoxon Rank Sum results for both surveys, of the relative delta timesper category. Median Badged and Median Non-Badged are the medians of eachcategory for the badge survey and non-badge survey respectively.
48
Category A = Number Questions MedianBA
MedianNBA
Demographics 15 6.94 5.65
Food Habits 6 5.63 4.39
Food Insecurity 8 5.67 5.68
Fruits 5 11.68 11.21
Vegetables 7 8.95 8.51
Meat and Alternatives 9 8.76 8.65
Dairy and Alternatives 6 8.01 7.54
Grains and Alternatives 4 9.01 8.38
Other Foods 5 9.62 8.36
Table 4.5: Total number of questions per category (A), with median badge time (Me-dian B) and median non-badged time (Median NB) divided by number of questionsper category.
49
buttons and 38 answers being checkboxes. Again, we explored whether or not there
was a difference in the timers for each question type between the different surveys
using the Wilcoxon Rank Sum test.
Tables 4.6 and 4.7 both summarize the results from the Wilcoxon Rank Sum tests
that were performed on the various answer types that are used in the two surveys.
In the delta times, radio buttons (answer type 0) have a p-value less than 0.05, and
checkboxes (answer type 1) have a p-value that close the 0.05, at 0.075, while in the
relative deltas set, the radio buttons p-value is close to 0.05 (at 0.062).
AnswerType p-value Median B Median NB
0 0.034 142.722 132.446
1 0.075 395.225 347.945
2 0.335 8.627 8.488
Table 4.6: Wilcoxon Rank Sum results for both surveys, of the delta times per answertype (0 = Radio Button, 1 = Checkbox, 2 = Textfield). Where Median Badged andMedian Non-Badged are the medians of each answer type for the badge survey andnon-badge survey respectively.
4.1.3 Results by Survey Question Length
It should be noted that each of the questions were broken into 3 categories for word
length. The categories were short, [0 - 25), medium, [25 - 35), and long, [35, 50). The
question with the shortest text contained 11 words in both the question text and the
50
AnswerType p-value Median B Median NB
0 0.062 2.746 2.543
1 0.696 1.220 1.282
2 0.789 0.264 0.265
Table 4.7: Wilcoxon Rank Sum results for both surveys, of the relative delta timesper answer type (0 = Radio Button, 1 = Checkbox, 2 = Textfield). Where MedianBadged and Median Non-Badged are the medians of each answer type for the badgesurvey and non-badge survey respectively.
answer text, while the question with the longest text contained 49 words in both the
question text and the answer text. The median word length for both question and
answer of the both surveys was 33 words.
Tables 4.8 and 4.9 sum up the results of the Wilcoxon Rank Sum tests of each
survey for the word length category of each question. For the delta times of the word
length categories, there is one result that is of interest, the questions that fell into the
short category ([0 - 25) total word length) have a p-value of 0.013, while the p-values
for the other categories are greater than 0.05. The relative deltas do not have any
p-values that are less than 0.05.
4.1.4 Results by Survey Question
We further investigated to determine if there was a significant difference in times
spent per question between the two surveys. Wilcoxon Rank Sum tests were per-
51
Word Category p-value Median B Median NB
1 0.013 75.146 59.389
2 0.121 247.014 235.212
3 0.103 225.088 212.318
Table 4.8: Wilcoxon Rank Sum results for both surveys, of the delta times per wordlength category (1 = [0 - 25) total words, 2 = [25 - 35) total words, 23= [35 - 50)total words). Where Median Badged and Median Non-Badged are the medians ofeach word length category for the badge survey and non-badge survey respectively.
Word Category p-value Median B Median NB
1 0.100 1.660 1.514
2 0.216 1.486 1.367
3 0.532 1.205 1.175
Table 4.9: Wilcoxon Rank Sum results for both surveys, of the relative delta timesper word length category (1 = [0 - 25) total words, 2 = [25 - 35) total words, 23= [35- 50) total words). Where Median Badged and Median Non-Badged are the mediansof each word length category for the badge survey and non-badge survey respectively.
52
formed for each of the questions. A full summary of results can be found in Appendix
B.3.
Tables B.2 and B.3 show the results for both surveys, for the delta times and the
relative deltas per question. These results show that there are questions with p-values
less than 0.05 for the delta times per question, specifically questions 2, 11, 22, 24,
29, 54, 57, 60. And that the relative delta times also have a couple of questions with
p-values less than 0.05, specifically questions 9, 16, 22, 54, 63. It should be noted
that questions 22 and 54 show up in with p-values less than 0.05 for both the delta
times and relative deltas.
53
4.2 Discussion
4.1 outlines the various results that have been gathered throughout the experiment.
This section will discuss what the results mean with respect to the experiment [and
hypotheses] outlined in Section 2.4. As Section 4.1 points out, the overall results
from the experiment show that of the 457 people who came to the website, only 111
actually took part in the experiment, which is approximately 24% of the people who
came to the site actually participating in the experiment, with approximately 48.6%
taking part in a badged survey and 51.3% taking part in a non-badged survey. From
here, it can also be seen that 91 people who took part in the experiment were able
to complete it, leaving only 20 unfinished surveys between the two types of surveys.
That is, approximately 81% of the people who took part in the experiment were able
to finish.
Breaking this down by survey type, approximately 83.3% of people who started a
badged survey completed it, and 80.7% of people who started a non-badged survey
completed it. A simple test of proportions indicates that there is no significant dif-
ference (p-value=0.91) in the completion rates for those who took the badged versus
the non-badged survey. Interestingly, this contradicts Hypothesis 1 in Section 2.4.
That is, badging does not appear to have influenced the completion rate.
While the Wilcoxon Rank Sum tests indicate that there was no statistically signif-
icant difference between the time it took to complete either types of surveys, some
54
important observations need to be noted. Upon taking a deeper look at the survey
results, we can see that for the badge survey, the median number of badges that were
given during the experiment was 5, the average number 4.7, and the maximum given
out being 13. Since each badge would produce a pop up window that would appear
for a maximum of 3 seconds (unless the participant closed it), this suggests that the
badged survey should take around 15 seconds longer (using either the median or av-
erage number of badges received) for a user to complete, while at most it should take
a participant around 39 seconds longer. In reality, we observed the following:
• The average time for a non-badged survey was 553.53 seconds.
• The average time for a badged survey was 635.34 seconds.
• The difference in the averages is 76.39 seconds.
• The median time for a non-badged survey was 519.81 seconds.
• The median time for a badged survey was 553.53 seconds.
• The difference in medians was 33.72 seconds.
Referring to Figure 4.1, the difference in the time it takes to finish a badged survey,
on average, is much higher than the time taken to finish a non-badged survey. That
is, the average difference of 76.39 seconds is 5.19 times as much as what would be
expected if we only considered the additional time required to open and close the
badge pop up windows. This hints that participants may be more engaged during a
55
badged survey than with their non-badged counterparts. Further, the median times
between the two survey types also shows that participants appear to be taking long to
complete the badged survey than what would be expected. Specifically, the difference
in median times to complete the survey is 2.25 times longer than what would be
expected if the difference were only related to the badge pop ups. While referring to
Figure 4.1, the orange bar represents the non-badged survey results with the addition
of the time required to receive the average number of badges that were assigned. This
shows that even by adding this extra time to the non-badged survey, there is still a
difference between the two surveys that is larger than might be expected. Again,
this trend suggests that the badged survey respondents are spending more time on
the surveys, and are possibly more engaged. This provides support for Hypothesis 2
outlined in Section 2.4, although it is not statistically significant.
As described in Section 4.1.1, the demographics and food habits categories showed
a significant difference in the delta time taken to complete them, while only the
demographics category showed a significant difference in the relative time taken to
complete it. This suggests that the distribution of times for the demographics cat-
egory is shifted more to the right than the distribution of times for the non-badged
survey. When the categories are ordered, these two categories become the first that
are presented to the participant. The differences observed might be associated with
the participant becoming familiarized with the survey itself.
56
Figure 4.1: A plot of the average and median times to complete the survey for thebadged (in light blue) and non-badged (in dark blue) surveys and the non-badgedsurvey with the addition of the time to earn the average number of badges given (inorange).
57
Other significant results between the surveys were limited to delta timers for radio
button type questions, and questions with fewer than 25 words. No other significant
differences were identified. That is, if straight-lining were occurring, it’s occurring in
the same way for both surveys. However, it should be noted that the median results
for delta timers were always larger for the badged survey than the non badged survey.
That is, despite a lack of statistical significance, it appears that the respondents were
taking longer to complete badged survey questions.
However, when considering the proportion of median time spent per question for
each category, it appears as if the respondents for both badged and non-badged
surveys spent more time answering questions closer to the end of the survey. That is,
it does not appear as if straight-lining occurred, or if it did, it occurred in the same
way. Since none of the categories near the end of the survey were found to have times
that were significantly different between the survey types, this neither supports nor
contradicts Hypothesis 3 in Section 2.4.
58
Chapter 5
Conclusion
As a consequence of issues with pen and paper (or phone) surveys, such as a lack of
user engagement, premature survey termination, and transitions from from traditional
landline phones to computers and hand held devices such as tablets and phones,
investigation into new methods of data collection becomes necessary. Alternative data
collection methods, e.g. on-line surveys, provide a method to collect data without the
use of standard pen and paper, or telephone surveys. Self elements of Gamification
were used to increase user engagement and completion rates during the online survey,
by adding badges that participants can earn during the survey.
5.1 Findings
We now review the hypotheses that this thesis was attempting to answer.
59
1. Gamification can be used to increase participant completion rates.
More specifically, the use of self elements (such as badges) will in-
crease participants completing the survey.
In regards to hypothesis 1, it was found that both types of surveys had very
similar completion rates, with the badge survey having 83.3% of participants
finish it, while 80.7% of non-badge survey participants were able to finish. As
Table 4.1 shows, the number of user’s that started for each survey is close,
with the badged survey having 54 participants start and the non-badged survey
having 57. Table 4.1 also shows that the number of participants that com-
pleted were close as well, with the badge survey having 45 participants finishing
and the non-badged survey having 46, with 9 and 11 participants not finishing
the badged or non-badged survey respectively. That is, we can not conclude
that Gamification was able to increase completion rates, as both the badged
and non-badged surveys have similar completion rates. Evidence suggests that
hypothesis 1 is FALSE.
2. The participant will spend more time on the gamified survey as op-
posed to the non-gamified survey. More specifically, the use of self
elements will engage the participant and they will take more time to
answer the survey.
The average and median times to complete a survey was greater for the badged
60
survey than the non badged survey. The difference, although not statistically
significant, was even larger than what would have been expected when taking
into account the time a badge stays on screen when it is earned. This trend
seems to suggest that hypothesis 2 is TRUE, and perhaps with further investi-
gation and a larger sample size, the true results will be known. Despite the fact
that statistical significance was not observed, the trend suggests that Gamifi-
cation may increase the amount of time that participants were spending on the
questions.
3. There will be no difference between a gamified survey and a non-
gamified survey in terms of participants straight-lining the survey.
More specifically, the participant is just as likely to straight-line a
gamified survey as they are a non-gamified survey.
The results indicate that straight-lining likely did not occur due to the amount
of time that participants of the badged survey spent on each question. However,
since there were mostly non significant results between the two surveys when
compared by delta timers and relative delta timers for category, question type,
question length, question, and overall survey, there is insufficient evidence to
suggest the time spent on each survey differed, or that participants were behav-
ing differently for each survey. That is, the evidence suggests that hypothesis 3
is INCONCLUSIVE.
61
To summarize, hypothesis 1 was found to be true, hypothesis 2 was found to be
false, and hypothesis 3 was found to be inconclusive. Regardless, it appears as if
badging has at least in some way contributed to improving participant engagement.
As such, it is recommended that badging be considered for future online public health
surveys, and that further study be done.
5.2 Future Work
In the future, further investigation of other elements of Gamification, and other
types of Self Elements should be looked at to see if they have an effect on user
engagement. More specifically, Social Elements of Gamification should be investigated
such as leaderboards or being able to share results from the survey if the participant
chooses. While other types of Self Elements that should be further looked at are
Pointsification, the use of numeric scores or points, or immediate feedback as a user
finishes a question, section/category, or survey, to see if these have an impact on the
engagement of users.
Another element of Gamification to look into would be Branching Elements, by
sending the user to different questions or sections of the survey depending on the
answer that was given for a specific question. This would help avoid premature
termination or fatigue for the user by having users skip specific questions that may
be unnecessary for the user to answer based on a previous answer. An example of
62
this would be asking the user if they have eaten a category of food such as fruits; if
the user answers ‘no’ then there is no reason to ask more questions about the type
of fruits a user has eaten just to force them to select an answer such as “I have not
eaten this.”
In regards to the data that was collected, further future work would include taking
a look at the subgroups, such as gender, age, location, within the demographics of
the data and seeing if the same conclusions can be drawn. It should be noted that
there will be biases present in this data due to the biases outlined in section 3.1
As well, since the contents of the survey were modelled to be similar to that of the
survey that the PHAC was running, other future work would include comparing the
data they collected to the data that was collected by this experiment. This would
help to further determine the impact of Gamification by comparing the number of
user’s that completed the survey, timing data if available, or if data collected are of
the same quality.
63
References
C. Alfi. The world in 2011: Ict facts and figures, 2011. URL http://www.slideshare.net/
cokyfauzialfi/ict-facts-figures2011.
Jerald G. Bachman, Lloyd D. Johnston, Patrick M. O’Malley, and John E. Schulenberg. Monitoring
the future. Technical report, Ann Arbor, MI: Institute for Social Research, 2006.
Philip Batterham. Recruitment of mental health survey participants using internet advertising:
content, characteristics and cost effectiveness. International Journal of Methods in Psychiatric
Research, doi: 10.1002/mpr.1421, 2014.
V. Bhaskaran. Increasing confidence in responses to electronic surveys, January 3 2013. URL
https://www.google.com/patents/US20130004933. US Patent App. 13/539,180.
Dan Birman. Ux flows: How to drive deep user engagement, 2014.
Marloes Bults, Desiree JMA Beaujean, Onno Zwart, Gerjo Kok, Pepijn Empelen, Jim E. Steenber-
gen, Jan Hendrik Richardus, and Helene ACM Voeten. Perceived risk, anxiety, and behavioural
responses of the general public during the early phase of the influenza a (h1n1) pandemic in
the netherlands: results of three consecutive online surveys. BMC Public Health, 11(1):1–13,
2011. ISSN 1471-2458. doi: 10.1186/1471-2458-11-2. URL http://dx.doi.org/10.1186/
1471-2458-11-2.
64
A. Joseph Cafazzo, Mark Casselman, Nathaniel Hamming, K. Debra Katzman, and R. Mark
Palmert. Design of an mhealth app for the self-management of adolescent type 1 diabetes:
A pilot study. J Med Internet Res, 14(3):e70, May 2012. doi: 10.2196/jmir.2058. URL
http://www.ncbi.nlm.nih.gov/pubmed/22564332.
Jared Cechanowicz, Carl Gutwin, Briana Brownell, and Larry Goodfellow. Effects of gamification on
participation and data quality in a real-world market research domain. In Proceedings of the First
International Conference on Gameful Design, Research, and Applications, Gamification ’13, pages
58–65, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2815-9. doi: 10.1145/2583008.2583016.
URL http://doi.acm.org/10.1145/2583008.2583016.
Colleen Cook. A meta-analysis of response rates in web- or internet-based surveys. Educational and
Psychological Measurement, 60(6):821–836, 2000.
Mihaly Csikzentmihalyi. Beyond Boredom and Anxiety. Jossey-Bass, 1975.
Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. From game design elements
to gamefulness: Defining ”gamification”. In Proceedings of the 15th International Academic
MindTrek Conference: Envisioning Future Media Environments, MindTrek ’11, pages 9–15, New
York, NY, USA, 2011. ACM. ISBN 978-1-4503-0816-8. doi: 10.1145/2181037.2181040. URL
http://doi.acm.org/10.1145/2181037.2181040.
Theo Downes-Le Guin, Reg Baker, Joanne Mechling, and Erica Ruyle. Myths and realities of
respondent engagement in online surveys. International Journal of Market Research, 54(5):1–21,
2012.
Maeve Duggan. Cell phone activities 2013. Technical report, Pew Research Center, 2013.
FluidSurveysTM. Finding the correct survey length, August 2013. URL http://fluidsurveys.
com/university/finding-the-correct-survey-length/.
65
FluidSurveysTM. What is the difference between a response rate and a completion rate?, 2014. URL
http://fluidsurveys.com/university/difference-response-rate-completion-rate/.
Mirta Galesic and Michael Bosnjak. Effects of questionnaire length on participation and indicators
of response quality in a web survey. Public Opinion Quarterly, 73(2):349–360, 2009.
Imran Ghory. Using fizzbuzz to find developers who grok coding, 2014. URL http://imranontech.
com/2007/01/24/using-fizzbuzz-to-find-developers-who-grok-coding/.
Geoff Goehle. Gamification and web-based homework. PRIMUS, 23(3):234–246, 2013. doi: 10.1080/
10511970.2012.736451. URL http://dx.doi.org/10.1080/10511970.2012.736451.
Jessica Greene, Howard Speizer, and Wyndy Wiitala. Telephone and web: mixed-mode challenge.
Health Serv Res, 43(1 Pt 1):230–48, Feb 2008. doi: 10.1111/j.1475-6773.2007.00747.x.
J. Hamari, J. Koivisto, and H. Sarsa. Does gamification work? – a literature review of empirical
studies on gamification. In System Sciences (HICSS), 2014 47th Hawaii International Conference
on, pages 3025–3034, 2014. doi: 10.1109/HICSS.2014.377.
Juho Hamari. Do badges increase user activity? a field experiment on the effects of gamifica-
tion. Computers in Human Behavior, pages –, 2015. ISSN 0747-5632. doi: http://dx.doi.
org/10.1016/j.chb.2015.03.036. URL http://www.sciencedirect.com/science/article/pii/
S0747563215002265.
Johannes Harms, Christoph Wimmer, Karin Kappel, and Thomas Grechenig. Gamification of online
surveys: conceptual foundations and a design process based on the mda framework. In Proceedings
of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, pages
565–568. ACM, 2014.
Debbie Hemley. 26 elements of a gamification marketing strategy., 2012. URL http://www.
socialmediaexaminer.com/26-elements-of-a-gamification-marketing-strategy/.
66
A. Regula Herzog and Jerald G. Bachman. Effects of questionnaire length on response quality, 1981.
W Hsin-Yuan Huang and D Soman. A practitioner’s guide to gamification of education, 2013.
Daniel Johnson, Sebastian Deterding, Kerri-Ann Kuhn, Aleksandra Staneva, Stoyan Stoyanov, and
Leanne Hides. Gamification for health and wellbing: A systematic review of the literature.
Internet Interventions, doi: 10.1016/j.invent.2016.10.002, July 2016.
Allen I. Kruat, Alan D. Wolfson, and Alan Rothenberg. Some effects of position on opinion survey
items. Journal of Applied Psychology, 60(6):774–776, 1975.
Frederick J. Kviz. Toward a standard definition of response rate. The Public Opinion Quarterly, 41
(2):265–267, 1977. ISSN 0033362X, 15375331. URL http://www.jstor.org/stable/2748344.
Carol Leaman. Report card: Gamification in learning (what works?), 2014.
Elaine Massung, David Coyle, Kirsten F. Cater, Marc Jay, and Chris Preist. Using crowdsourcing
to support pro-environmental community activism. In Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems, CHI ’13, pages 371–380, New York, NY, USA, 2013.
ACM. ISBN 978-1-4503-1899-0. doi: 10.1145/2470654.2470708. URL http://doi.acm.org/10.
1145/2470654.2470708.
Aigul Mavletova. Data quality in pc and mobile web surveys. Social Science Computer Review, 31
(6):725–743, December 2013.
Heather L. O’Brien and Elaine G. Toms. What is user engagement? a conceptual framework for
defining user engagement with technology. Journal of the American Society for Information
Science and Technology, 59(6):938–955, 2008. ISSN 1532-2890. doi: 10.1002/asi.20801. URL
http://dx.doi.org/10.1002/asi.20801.
Jon Puleston. Trimming mobile surveys. Research World, pages 40–42, October 2014.
67
K. Seaborn and D. I. Fels. Gamification in theory and action: A survey. International Journal of
Human-Computer Studies, 74:14–31, 2015.
Namin Shin. Online learner’s ‘flow’ experience: an empirical study. British Journal of Educational
Technology, 37(5):705–720, 2006. ISSN 1467-8535. doi: 10.1111/j.1467-8535.2006.00641.x. URL
http://dx.doi.org/10.1111/j.1467-8535.2006.00641.x.
Michael B. Siegel, Kerry L. Tanwar, and Kathleen S. Wood. Electronic cigarettes as a smoking-
cessation tool. American Journal of Preventive Medicine, 40(4):472–475, 2016/02/01 2016. ISSN
0749-3797. doi: 10.1016/j.amepre.2010.12.006. URL http://dx.doi.org/10.1016/j.amepre.
2010.12.006.
Robra-Bissantz Sussane, Scheiner Christian, and Witt Maximilian. Gamification of online idea
competitions: Insights from an explorative case. Informatik schafft Communities, 2011.
Sharleen Sy. Engagement flow in gamification, 2010. URL https://stratsynergy.wordpress.
com/2010/11/02/engagement-flow-in-gamification/.
Kevin B. Wright. Researching internet-based populations: Advantages and disadvantages of online
survey research, online questionnaire authoring software packages, and web survey services. Jour-
nal of Computer-Mediated Communication, 10(3):00–00, 2005. ISSN 1083-6101. doi: 10.1111/j.
1083-6101.2005.tb00259.x. URL http://dx.doi.org/10.1111/j.1083-6101.2005.tb00259.x.
68
Appendix A
Ethics Documents
A.1 Ethics
A.1.1 Ethics Approval
Since this thesis required experiments involving human participants, it was necessary to review
the experimental design to ensure that risks and harm were to any participant was minimized.
As indicated in Figure A.1, the Research Ethics Boards of the University of Guelph reviewed the
proposed study, and provided the Certification of Ethical Acceptability of Research Involving Human
Participants, and REB Number 15AU004.
69
Page 1 of 1
APPROVAL PERIOD: February 10, 2016 EXPIRY DATE: February 10, 2017 REB: NPES REB NUMBER: 15AU004 TYPE OF REVIEW: Delegated Type 1 PRINCIPAL INVESTIGATOR: Gillis, Daniel ([email protected]) DEPARTMENT: School of Computer Science SPONSOR(S): None TITLE OF PROJECT: Alternative Data Collection to Understand Food
Behaviours
The members of the University of Guelph Research Ethics Board have examined the protocol which describes the participation of the human participants in the above-named research project and considers the procedures, as described by the applicant, to conform to the University's ethical standards and the Tri-Council Policy Statement, 2nd Edition. The REB requires that researchers:
Adhere to the protocol as last reviewed and approved by the REB.
Receive approval from the REB for any modifications before they can be implemented.
Report any change in the source of funding.
Report unexpected events or incidental findings to the REB as soon as possible with an indication of how these events affect, in the view of the Principal Investigator, the safety of the participants, and the continuation of the protocol.
Are responsible for ascertaining and complying with all applicable legal and regulatory requirements with respect to consent and the protection of privacy of participants in the jurisdiction of the research project.
The Principal Investigator must:
Ensure that the ethical guidelines and approvals of facilities or institutions involved in the research are obtained and filed with the REB prior to the initiation of any research protocols.
Submit a Status Report to the REB upon completion of the project. If the research is a multi-year project, a status report must be submitted annually prior to the expiry date. Failure to submit an annual status report will lead to your study being suspended and potentially terminated.
The approval for this protocol terminates on the EXPIRY DATE, or the term of your appointment or employment at the University of Guelph whichever comes first. Signature: Date: February 10, 2016
RESEARCH ETHICS BOARDS Certification of Ethical Acceptability of Research Involving Human Participants
A. Papdopoulos Chair, Research Ethics Board-NPES
Figure A.1: Ethics approval from the University of Guelph Research Ethics Board,February 10, 2016
70
A.1.2 Ethics Letter Of Information & Consent Form
The following Letter of Information and Consent Form (Figures A.2-A.4) was presented to the
participants of the survey prior to beginning the survey.
71
Alternative Data Collection to Understand Food Behaviours
Letter of Information If you have access to the internet and are 18 years of age or older, we’re inviting you to take part in a study being conducted by Dr. Daniel Gillis (an Assistant Professor and Statistician in the School of Computer Science at the University of Guelph). The project has two major goals:
1. We want to know how people across the country consume food (including how often they typically eat, and the kinds of food they eat), and what challenges they might face that could get in the way of them eating the types of foods that they want to eat.
2. We want to figure out how to make surveys more enjoyable for the person answering questions, and in such a way as to provide researchers with better, more complete information. To do this we’re asking some people to answer regular surveys, and others to answer surveys that give them feedback on the information they’ve provided in the form of badges that they can share on Facebook or Twitter (if they choose to).
As a participant, you’re invited to answer a series of questions about yourself, about your food habits, your understanding and level of food insecurity (that is, not being able to eat the way you’d like to eat because of a lack of money), and food consumption. The questionnaire consists of 9 sections: demographics, food habits, food insecurity, fruits, vegetables, meats and alternatives, dairy and alternatives, grains and alternatives, and other foods. To proceed from one section to the next, you’ll need to provide an answer to every question within a section. However, you are not obliged to answer every question. Instead, you have the option of answering “I don’t know”, or “I’d prefer not to answer” on any question. Every question is multiple choice: some requiring a single answer, others that allow for multiple answers. The research program also investigates the use of new technologies (mobile phones, tablets, gesture control devices, etc.) for collecting data for research. While the results are not directly commercializable, they could be used to in the development of tools for marketing or business data collection, and thus might be commercializable in the future. The first question of the survey will ask for your consent for our team to analyze the data provided, as well as share and publish results based on the aggregate answers collected. All of your data are completely anonymous. We won’t (and don’t) ask for your name, address (other than the first three digits of your postal code), or email. To ensure anonymity, please do not enter your name anywhere in the survey.
Figure A.2: Page 1 of the Letter of Information and Consent Form provided to theparticipants prior to participating in the experiment.
72
Alternative Data Collection to Understand Food Behaviours
Letter of Information If you have access to the internet and are 18 years of age or older, we’re inviting you to take part in a study being conducted by Dr. Daniel Gillis (an Assistant Professor and Statistician in the School of Computer Science at the University of Guelph). The project has two major goals:
1. We want to know how people across the country consume food (including how often they typically eat, and the kinds of food they eat), and what challenges they might face that could get in the way of them eating the types of foods that they want to eat.
2. We want to figure out how to make surveys more enjoyable for the person answering questions, and in such a way as to provide researchers with better, more complete information. To do this we’re asking some people to answer regular surveys, and others to answer surveys that give them feedback on the information they’ve provided in the form of badges that they can share on Facebook or Twitter (if they choose to).
As a participant, you’re invited to answer a series of questions about yourself, about your food habits, your understanding and level of food insecurity (that is, not being able to eat the way you’d like to eat because of a lack of money), and food consumption. The questionnaire consists of 9 sections: demographics, food habits, food insecurity, fruits, vegetables, meats and alternatives, dairy and alternatives, grains and alternatives, and other foods. To proceed from one section to the next, you’ll need to provide an answer to every question within a section. However, you are not obliged to answer every question. Instead, you have the option of answering “I don’t know”, or “I’d prefer not to answer” on any question. Every question is multiple choice: some requiring a single answer, others that allow for multiple answers. The research program also investigates the use of new technologies (mobile phones, tablets, gesture control devices, etc.) for collecting data for research. While the results are not directly commercializable, they could be used to in the development of tools for marketing or business data collection, and thus might be commercializable in the future. The first question of the survey will ask for your consent for our team to analyze the data provided, as well as share and publish results based on the aggregate answers collected. All of your data are completely anonymous. We won’t (and don’t) ask for your name, address (other than the first three digits of your postal code), or email. To ensure anonymity, please do not enter your name anywhere in the survey.
Figure A.3: Page 2 of the Letter of Information and Consent Form provided to theparticipants prior to participating in the experiment.
73
2. Understand that your participation is voluntary and you are free to withdraw at any time 3. Understand the provisions for confidentiality
§ I agree to participate.
§ I do not agree to participate.
Figure A.4: Page 3 of the Letter of Information and Consent Form provided to theparticipants prior to participating in the experiment.
74
A.2 Survey Questions
Survey questions and answers presented to the participants of the survey are outlined in Figures
A.5-A.10.
75
Questionnaire Category Category
Description/Instructions Question Answer Set
Demographics The following questions allow us to get to know you a little better. The information you provide here will be used to identify patterns in food consumption, food habits, and food insecurity. If you are unsure of your answer, please select "don't know" or follow the specific instructions of the question. If you prefer not to answer, please select that option, or follow the specific instructions of the question. You can leave the survey at any time by simply closing your browser.
Please enter your gender:
{"0":"Male", "1":"Female", "2":"Other", "3":"prefer not to answer"}
Please enter your age:
{"0":"<18", "1":"18 to 29","2":"30 to 39", "3":"40 to 59", "4":"60+", "5":"prefer not to answer"}
In the past 12 months, were you a part time or full time university or college student?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Which of the following best describes your current
{"0":"Single", "1":"Married/Partnered", "2":"Separated/Divorced/Widowed", "3":"Other", "4":"prefer not to answer"}
relationship status?
Do you have any dependents that live with you?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Please enter the first 3 digits of your postal/zip code. Enter XXX if you prefer not to answer. Enter YYY if you do not have a postal code. Enter ZZZ if you do not know your postal code.
{"0":"Postal Code"}
What is your annual income?
{"0":"<$30000","1":"$30000-$59999","2":"$60000-$79999","3":"$80000-$99999","4":"$100000+","5":"don't know","6":"prefer not to answer"}
What is your highest level of education?
{"0":"Primary","1":"Secondary","2":"University/College Degree/Diploma","3":"Graduate Degree/Diploma","4":"prefer not to answer"}
Which of the following best describes your current university student status?
{"0":"I am not currently a university student","1":"Undergraduate","2":"Master's","3":"PhD","4":"Post Doc","5":"don't know","6":"prefer not to answer"}
If you are a student in university or college, are you an international student?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Do you identify as any of the following?
{"0":"First Nations Status","1":"First Nations Non-Status","2":"Metis","3":"Inuit","4":"Senior Citizen (65+)","5":"Newcomer to Canada (<5yrs in Canada)","6":"I do not identify as any of the above","7":"don't know","8":"prefer not to answer"}
Figure A.5: Page 1 and 2 of the survey questions and answers provided to the par-ticipants
76
In general, how would you describe your physical health?
{"0":"Excellent","1":"Very good","2":"Good","3":"Fair","4":"Poor","5":"Very poor","6":"don't know","7":"prefer not to answer"}
In general, how would you describe your mental health?
{"0":"Excellent","1":"Very good","2":"Good","3":"Fair","4":"Poor","5":"Very poor","6":"don't know","7":"prefer not to answer"}
Thinking about the amount of stress in your life, would you say that most days are:
{"0":"Not at all stressful","1":"Not very stressful","2":"A bit stressful","3":"Quite a bit stressful","4":"Extremely stressful","5":"don't know","6":"prefer not to answer"}
Identify all sources of income for the previous year.
{"0":"Government student loan","1":"Scholarship/Bursary","2":"Bank loan","3":"Research/Teaching Assistantship","4":"Personal savings","5":"Family","6":"Part time employment","7":"Full time employment","8":"Child tax benefits","9":"Social assistance","10":"don't know","11":"prefer not to answer"}
Fruits The following questions are related to different types of fruit you've eaten in the last 7 days. Do not include fruit that was consumed as part of a prepared dessert (such as a pie or a tart), unless it was, for example, a bowl of cherries with whipped cream. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can
Did you eat any of the following fresh or frozen tropical fruits?
{"0":"Bananas","1":"Mangos","2":"Papayas","3":"Kiwis","4":"Pomegranates","5":"Avocado","6":"Other tropical fruits","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
leave the survey at any time by simply closing your browser.
Did you eat any of the following fresh or frozen tree fruits?
{"0":"Apples","1":"Pears","2":"Plums","3":"Peaches","4":"Nectarines","5":"Apricots","6":"Other tree fruits","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
Did you eat any of the following citrus fruits?
{"0":"Lemons","1":"Limes","2":"Grapefruits","3":"Oranges","4":"Tangerines","5":"Clementines","6":"Other citrus fruits","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
Did you eat any of the following fresh or frozen melons?
{"0":"Cantaloupe","1":"Watermelon","2":"Honey Dew","3":"Other melons","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Did you eat any of the following fresh or frozen berries?
{"0":"Strawberries","1":"Blueberries","2":"Raspberries","3":"Blackberries","4":"Cherries","5":"Other berries","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Vegetables The following questions are related to different types of vegetables you've eaten in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.
Did you eat any of the following fresh or frozen leafy vegetables?
{"0":"Lettuce","1":"Cabbage","2":"Spinach","3":"Swiss Chard","4":"Kale","5":"Other leafy vegetables","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Did you eat any of the following fresh or frozen
{"0":"Carrots","1":"Beets","2":"Potatoes","3":"Turnips","4":"Parsnips","5":"Radish", "6":"Other root vegetables/tubers","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
Figure A.6: Page 3 and 4 of the survey questions and answers provided to the par-ticipants
77
root vegetables/tubers?
Did you eat any of the following fresh or frozen strong vegetables?
{"0":"Onions","1":"Leeks","2":"Garlic","3":"Shallots","4":"Scallions","5":"Other strong vegetables","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Did you eat any of the following fresh or frozen stalk vegetables?
{"0":"Celery","1":"Broccoli","2":"Cauliflower","3":"Bok choy","4":"Other stalk vegetables","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you eat any of the following fresh or frozen pod vegetables?
{"0":"Beans","1":"Peas","3":"Other pod vegetables","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Did you eat any of the following fresh or frozen squash vegetables?
{"0":"Acorn","1":"Butternut","2":"Spaghetti","3":"Pumpkin","4":"Other squash","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you eat any of the following fresh or frozen vegetables?
{"0":"Cucumbers","1":"Tomatoes","2":"Mushrooms","3":"Zucchini","4":"Other vegetables","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Meat and Alternatives
The following questions are related to different types of meat (or meat alternatives) that you've eaten in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can
Did you eat any of the following fresh or frozen unprocessed red meats (e.g. steaks, roasts, chops, etc.)? Do not include ground meats or deli meats here.
{"0":"Beef","1":"Deer","2":"Elk","3":"Bison","4":"Lamb","5":"Goat","6":"Other red meat","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
leave the survey at any time by simply closing your browser.
Did you eat any of the following fresh or frozen ground meats?
{"0":"Ground beef","1":"Ground pork","2":"Ground chicken","3":"Ground lamb","4":"Ground turkey","5":"Other ground meat","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Did you eat any of the following fresh or frozen white meats (e.g. breasts, legs, chops, etc.)?
{"0":"Chicken","1":"Pork","2":"Turkey","3":"Other white meat","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Did you eat any of the following fresh or frozen unprocessed fish or seafood?
{"0":"Fish","1":"Mussels","2":"Shrimp","3":"Scallops","4":"Clams","5":"Lobster","6":"Crab","7":"Other seafood or fish","8":"I didn't eat any of these foods","9":"don't know","10":"prefer not to answer"}
Did you eat any of the following nuts or seeds?
{"0":"Pecans","1":"Walnuts","2":"Almonds","3":"Cashews","4":"Hazelnuts","5":"Sunflower seeds","6":"Other nuts or seeds","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
Did you eat any of the following raw or cooked bean/protein replacements?
{"0":"Lentils","1":"Chickpeas","2":"Kidney beans","3":"Black eyed peas","4":"Soy or soy products (e.g. tofu)","5":"Other meat replacement proteins","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Did you eat any of the following processed/deli meats?
{"0":"Sliced ham","1":"Sliced beef","2":"Sliced turkey","3":"Sliced chicken","4":"Salami","5":"Bacon","6":"Other processed meats","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
Did you eat any of the following veggie/vegan protein replacements?
{"0":"Veggie/vegan burger","1":"Veggie/vegan hot dog","2":"Ground veggie meat substitute","3":"Other veggie/vegan meat replacements","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Figure A.7: Page 5 and 6 of the survey questions and answers provided to the par-ticipants
78
Did you eat any of the following nut butters?
{"0":"Peanut butter","1":"Cashew butter","2":"Almond butter","3":"Hazelnut butter","4":"Other nut butters","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Dairy and Alternatives
The following questions are related to different types of dairy (or dairy alternatives) that you've consumed in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.
Did you drink any of the following milks?
{"0":"Milk (cow, goat, etc.)","1":"Almond milk","2":"Soy milk","3":"Cashew milk","4":"Rice milk","5":"Coconut milk","6":"Other milks","7":"I didn't eat any of these foods","8":"don't know","9":"prefer not to answer"}
Did you eat any of the following medium to hard cheeses?
{"0":"Cheddar","1":"Parmesan","2":"Romano","3":"Colby","4":"Monterey Jack","5":"Emmental","6":"Gouda","7":"Edam","8":"Other medium to hard cheeses","9":"I didn't eat any of these foods","10":"don't know","11":"prefer not to answer"}
Did you eat any of the following soft or semi-soft cheeses?
{"0":"Cream cheese","1":"Brie","2":"Havarti","3":"Munster","4":"Other soft or semi-soft cheeses","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you eat any of the following frozen dairy products? Include frozen ice cream bars here.
{"0":"Ice cream (dairy based)","1":"Rice milk ice cream","2":"Soy milk ice cream","3":"Coconut milk ice cream","4":"Other milk ice cream","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you eat any of the following yogurts?
{"0":"Yogurt (dairy based)","1":"Rice milk yogurt","2":"Soy yogurt","3":"Cashew yogurt","4":"Coconut yogurt","5":"Other milk yogurts","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Did you eat any of the following processed dairy based products?
{"0":"Cheese slices","1":"Cheese strings","2":"Pre-made milk based beverage","3":"Other processed dairy","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Grains and Alternatives
The following questions are related to different types of grains (or grain alternatives) that you've eaten in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.
Did you consume any of the following processed grain items?
{"0":"White bread","1":"Hot dog/hamburger buns","2":"White pasta","3":"White rice","4":"Other similar processed carbohydrates","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you consume any of the following whole grain products?
{"0":"Rye bread","1":"Whole wheat bread","2":"Whole wheat pasta","3":"Brown or wild rice","4":"Other complex carbohydrates","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you eat any of the following cold cereals?
{"0":"Bran flakes or other high fibre cereal","1":"Cheerios or other grain based cereals","2":"Granola based cereals","3":"Rice puff cereal","4":"Sugary cereal","5":"Other processed cereals","6":"I didn't eat any of these foods","7":"don't know","8":"prefer not to answer"}
Did you eat any of the following hot cereals?
{"0":"Oatmeal", "1":"Cream of wheat","2":"Multigrain cereal","3":"Other hot high fibre low sugar unprocessed cereals","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Food Insecurity
The following questions allow us to understand your current food security situation. Your answers will help us understand any issues you might have faced
In the past 12 months, have you NOT HAD enough food to eat because of a lack of money?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Figure A.8: Page 7 and 8 of the survey questions and answers provided to the par-ticipants
79
trying to obtain and eat the types of food you want to eat. If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.
In the past 12 months, have you NOT eaten the quality or variety of foods that you wanted to eat because of a lack of money?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 12 months, have you worried that there might not be enough to eat because of a lack of money?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 12 months, have you sought assistance from a food bank or other food provider?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
If you needed to, do you know where you could go to find food in an emergency?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 12 months, have you ever not eaten balanced meals because you couldn't afford to?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 12 months, have you ever reduced your food consumption because there wasn't enough to eat because you couldn't afford it?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 12 months, did you experience weight loss because you couldn't afford to buy food?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Other Foods The following questions are related to other types of food or drinks that you've consumed in the last 7 days. Note that each answer includes an option for "I didn't eat any of these foods". If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.
Did you eat any of the following items?
{"0":"Cake/Cupcakes/Muffins/Dessert Bread","1":"Pie/Tarts","2":"Cookies","3":"Pastries/Croissants","4":"Other desserts or pastries","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Figure A.9: Page 9 and 10 of the survey questions and answers provided to theparticipants
80
Did you consume any of the following beverages?
{"0":"Latte","1":"Capuccino","2":"Milkshake","3":"Other tea/coffee milk based beverage","4":"I didn't eat any of these foods","5":"don't know","6":"prefer not to answer"}
Did you consume any of the following beverages?
{"0":"Soda/Pop","1":"Energy drink","2":"Vitamin water","3":"Sports drink","4":"Other energy drinks","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you consume any of the following alcoholic beverages?
{"0":"Beer/cider","1":"Wine","2":"Clear liquor (e.g. vodka, gin, etc.)","3":"Other liquor (e.g. rye, scotch, etc.)","4":"Other alcoholic beverages","5":"I didn't eat any of these foods","6":"don't know","7":"prefer not to answer"}
Did you consume any of the following items?
{"0":"Chocolate bar","1":"Popcorn","2":"Granola bars","3":"Energy bars","4":"Potato chips","5":"Corn chips","6":"Crackers","7":"Other snack foods","8":"I didn't eat any of these foods","9":"don't know","10":"prefer not to answer"}
Food Habits The following questions give us a better idea of your current eating habits. These will be used to better understand your food consumption habits, which you'll be asked about in following sections. If you are unsure of your answer, please select "don't know". If you prefer not to answer, please select "prefer not to answer". You can leave the survey at any time by simply closing your browser.
In the past week, have you eaten breakfast every day?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past week, have you eaten lunch every day?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past week, have you eaten dinner every day?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Are you a vegetarian or vegan?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 7 days, were you fasting for religious, medical, or other purposes?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
In the past 7 days, have your eating habits changed for any reason?
{"0":"Yes","1":"No","2":"don't know","3":"prefer not to answer"}
Figure A.10: Page 11 and 12 of the survey questions and answers provided to theparticipants
81
Appendix B
Code & Further Tables
B.1 Code
The code to create and implement the surveys used in this thesis can be found in the following
repository: https://bitbucket.org/altdatacollection/alt-data-collection.
B.2 Category Code to Category Labels
Table B.1 outlines the categories used in the survey, and the number of questions in each category.
The categories were presented to the respondent in the following order: 0, 8, 6, 1, 2, 3, 4, 5, 7.
82
Category ID Category Name Number of Questions in Category
0 Demographics 15
1 Fruits 5
2 Vegetables 7
3 Meat and Alternatives 9
4 Dairy and Alternatives 6
5 Grains and Alternatives 4
6 Food Insecurity 8
7 Other Foods 5
8 Food Habits 6
Table B.1: The category IDs for the surveys and the corresponding name for thecategory.
83
B.3 Wilcoxon Rank Sum Results By Question
Question Test Statistic p.value Median B Median NB
1 1150.500 0.151 22.046 18.558
2 1022.000 0.034 13.828 10.321
3 1510.500 0.313 0.000 0.000
4 1463.000 0.656 2.683 2.760
5 1142.000 0.234 7.472 6.840
6 1063.000 0.063 8.640 7.731
7 1334.000 0.956 6.860 7.172
8 1103.000 0.109 6.638 6.036
9 1455.500 0.495 6.398 7.294
10 1275.000 0.524 10.650 9.368
11 1038.000 0.031 9.649 7.669
12 1241.000 0.481 6.947 6.877
13 1146.000 0.143 6.234 5.968
14 1460.000 0.588 7.890 8.814
15 1114.000 0.095 5.597 4.831
16 1486.000 0.479 4.231 4.376
17 1145.000 0.141 13.102 9.607
18 1134.000 0.123 3.508 2.678
19 1180.000 0.213 3.090 2.455
20 1070.000 0.051 3.809 3.273
21 1200.000 0.263 5.766 4.969
22 1136.500 0.018 5.606 4.491
84
23 1245.500 0.084 5.039 4.607
24 1200.500 0.047 3.507 3.089
25 1375.000 0.335 8.627 8.488
26 1341.500 0.314 5.434 5.373
27 1383.000 0.448 5.151 4.877
28 1263.000 0.475 2.587 2.430
29 1038.000 0.031 4.587 3.651
30 1246.500 0.412 3.899 3.719
31 1269.000 0.601 7.539 6.563
32 1097.000 0.231 14.294 11.982
33 1186.000 0.553 7.561 6.453
34 1258.000 0.917 5.944 6.832
35 1116.000 0.448 7.635 6.384
36 1273.000 0.998 8.348 9.202
37 1198.000 0.608 9.215 8.655
38 1292.000 0.773 7.558 7.264
39 1251.000 0.987 6.035 6.202
40 1217.000 0.702 3.740 3.934
41 1247.000 0.742 9.138 9.493
42 1096.000 0.463 8.926 7.101
43 999.000 0.155 6.966 6.667
44 1107.000 0.511 6.126 5.427
45 1024.000 0.213 4.600 4.211
46 1150.000 0.726 6.451 6.479
85
47 1048.000 0.804 10.901 9.764
48 885.000 0.099 7.504 7.299
49 1083.000 0.742 6.445 6.195
50 911.000 0.146 7.400 5.919
51 923.000 0.129 9.671 8.537
52 1155.500 0.886 12.012 11.696
53 1156.000 0.889 7.243 6.705
54 762.500 0.005 8.669 7.119
55 1293.000 0.305 4.449 4.540
56 1301.000 0.274 6.660 6.640
57 1136.000 0.026 6.709 5.282
58 1434.000 0.650 7.228 7.846
59 1333.000 0.290 5.672 5.018
60 1049.000 0.006 4.557 3.686
61 1350.000 0.339 6.852 6.188
62 1360.000 0.455 11.990 10.691
63 1440.000 0.679 4.890 5.091
64 1114.000 0.095 5.100 4.887
65 1309.000 0.675 3.783 3.566
Table B.2: Wilcoxon Rank Sum results for both surveys, of the
delta times per question. Where Median Badged and Median Non-
Badged are the medians of each question for the badge survey and
non-badge survey respectively.
86
Question Test Statistic p.value Median B Median NB
1 1213.000 0.301 0.111 0.097
2 1158.000 0.213 0.049 0.044
3 1510.500 0.313 0.000 0.000
4 1377.000 0.341 0.389 0.344
5 1328.000 0.977 0.030 0.029
6 1189.000 0.297 0.029 0.028
7 1610.000 0.061 0.021 0.025
8 1314.000 0.818 0.023 0.021
9 1652.000 0.050 0.021 0.024
10 1420.000 0.776 0.052 0.052
11 1196.000 0.253 0.042 0.039
12 1371.000 0.894 0.029 0.028
13 1275.000 0.524 0.028 0.027
14 1632.000 0.100 0.054 0.065
15 1373.000 0.993 0.032 0.035
16 1691.000 0.043 0.026 0.031
17 1278.000 0.536 0.096 0.093
18 1311.000 0.684 0.026 0.024
19 1401.000 0.871 0.024 0.021
20 1279.000 0.541 0.023 0.022
21 1365.000 0.952 0.032 0.033
22 1163.000 0.027 0.417 0.350
23 1324.000 0.206 0.283 0.247
87
24 1457.000 0.631 0.153 0.150
25 1585.000 0.789 0.264 0.265
26 1504.000 0.972 0.143 0.140
27 1463.000 0.779 0.117 0.107
28 1539.000 0.295 0.021 0.022
29 1331.000 0.781 0.030 0.031
30 1421.000 0.771 0.030 0.030
31 1510.000 0.300 0.024 0.024
32 1233.000 0.784 0.038 0.039
33 1444.000 0.250 0.018 0.023
34 1403.000 0.383 0.017 0.020
35 1329.000 0.469 0.018 0.019
36 1417.000 0.333 0.020 0.024
37 1417.000 0.333 0.024 0.026
38 1434.000 0.205 0.018 0.021
39 1409.000 0.269 0.016 0.017
40 1405.000 0.376 0.010 0.011
41 1395.000 0.167 0.022 0.023
42 1253.000 0.710 0.019 0.019
43 1203.000 0.986 0.017 0.018
44 1232.000 0.823 0.014 0.014
45 1204.000 0.981 0.012 0.011
46 1411.000 0.135 0.014 0.016
47 1213.000 0.314 0.021 0.023
88
48 1097.000 0.961 0.016 0.018
49 1292.000 0.225 0.012 0.014
50 1050.000 0.685 0.013 0.013
51 1090.000 0.782 0.018 0.018
52 1284.000 0.440 0.025 0.027
53 1282.000 0.449 0.015 0.016
54 862.000 0.034 0.019 0.016
55 1414.000 0.056 0.010 0.011
56 1386.000 0.563 0.142 0.126
57 1331.000 0.285 0.107 0.095
58 1701.000 0.256 0.099 0.111
59 1489.000 0.901 0.072 0.068
60 1349.000 0.336 0.055 0.053
61 1644.000 0.427 0.075 0.076
62 1560.000 0.648 0.116 0.112
63 1724.000 0.026 0.027 0.033
64 1398.000 0.886 0.027 0.030
65 1528.000 0.328 0.019 0.021
Table B.3: Wilcoxon Rank Sum results for both surveys, of the rel-
ative delta times per question. Where Median Badged and Median
Non-Badged are the medians of each question for the badge survey
and non-badge survey respectively.