toward a model of emotions and mood in the online...

Toward a Model of Emotions and Mood in the OnlineInformation Search Process

Irene LopatovskaSchool of Library and Information Science, Pratt Institute, 144 W. 14th Street, New York, NY 10011-7301.E-mail: [email protected]

This article reports the results of a study that examinedrelationships between primary emotions, secondaryemotions, and mood in the online information searchcontext. During the experiment, participants wereasked to search Google to obtain information on thetwo given search tasks. Participants’ primary emotionswere inferred from analysis of their facial expressions,data on secondary emotions were obtained throughparticipant interviews, and mood was measured usingthe Positive Affect Negative Affect Scale (PANAS;Watson, Clark, & Tellegen, 1988) prior, during, and afterthe search. The search process was represented by thecollection of search actions, search performance, andsearch outcome quality variables. The findings suggestexistence of direct relationships between primary emo-tions and search actions, which in turn imply the pos-sibility of inferring emotions from search actions andvice versa. The link between secondary emotions andsearchers’ evaluative judgments, and lack of evidenceof any relationships between secondary emotions andother search process variables, point to the strengthsand weaknesses of self-reported emotion measures inunderstanding searchers’ affective experiences. Ourstudy did not find strong relationships between moodand search process and outcomes, indicating thatwhile mood can have a limited effect on search activi-ties, it is a relatively stable and long-lasting state thatcannot be easily altered by the search experience and,in turn, cannot significantly affect the search. Thearticle proposes a model of relationships betweenemotions, mood, and several facets of the searchprocess. Directions for future work are also discussed.

Introduction

Emotions play an important role in our lives: they focusour attention on the events, objects, and persons we

need to avoid and those we need to seek out; organize ourperception and thinking; color our judgments, decisions,and memories; and provide information to others aboutour current position in the environment (Payne & Cooper,2001). Emotions are an integral part of an informationsearch process because they affect a searcher’s attention,memory, performance, and judgments (Brave, Hutchinson,& Nass, 2002; Dervin & Reinhard, 2007). It is not surpris-ing that emotions have become a popular topic in human–computer interaction (HCI) and information scienceresearch (Lopatovska & Arapakis, 2011). Since earlyresearch on affective aspects of information-seeking behav-ior (Kuhlthau, 1991, 1993), a number of studies haveinvestigated emotive variables in the context of onlineinformation search. Emotions have been shown to affectthe willingness to search (Fulton, 2009), search strategies(Nahl & Tenopir, 1996), search performance (Gwizdka& Lopatovska, 2009; Nahl, 1998; Nahl & Meer, 1997;Wang, Hawk, & Tenopir, 2000), and be affected by thesearch process (Wang et al., 2000), system performance(Bilal, 2000; Bilal & Bachir, 2007; Tenopir, Wang,Zhang, Simmons, & Pollard 2008), interest in theprocess and documents (Kracker, 2002; Kracker &Wang, 2002; Lopatovska & Mokros, 2007), and othervariables.

We conducted a study that focused on the role of affectivevariables in online searching and expanded the line of tra-ditional information science studies of emotions in severalways. We introduced a clear differentiation between thethree types of affective variables: primary emotions, second-ary emotions, and mood, and explored their unique roles inthe online search process. The study uncovered the potentialof each affective variable for providing information about asearcher or the search process, and identified measurementissues associated with each variable. For example, analysisof primary emotions, or instant nonevaluative affective reac-tions to search stimuli, offered rich data about the searchers’immediate affective states that could be used to predict

Received October 9, 2012; revised June 11, 2013; accepted July 12, 2013

© 2014 ASIS&T • Published online 26 Febuary 2014 in Wiley OnlineLibrary (wileyonlinelibrary.com). DOI: 10.1002/asi.23078

JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY, 65(9):1775–1793, 2014

mailto:[email protected]

http://wileyonlinelibrary.com

search actions. However, the data could not explain thenature of affective reactions or pinpoint their specific con-nection to the search stimuli. Analysis of secondary emo-tions, or states that involve cognitive evaluation of anemotion stimulus, helped to understand searchers’ evalua-tive judgments and perceptions involved in informationseeking. However, the measure was heavily dependent onparticipants’ ability to recall and verbalize their experiences.Analysis of moods, longer lasting affective states that are nottied to a particular stimulus, offered insight into searchers’engagement and activity levels during the search. However,the findings related to the mood were limited by the experi-mental study design that removed participants from theirdaily routines necessary for understanding their long-termaffective states.

This article reviews the background literature on emotionresearch, describes the study method and results, and offers aframework of relationships between three affective variablesand several components of an online search process.

Literature Review

Due to the high level of interest in emotions in psychol-ogy, economics, computer science, and other disciplines, thebody of literature on emotion is vast. We organized ourreview of the literature on emotion theories and methodsaround the three affective constructs we chose for the study:primary emotions, secondary emotions, and mood. We alsoreviewed library and information science (LIS) publicationsthat have discussed affective variables in the onlineinformation-seeking context and classified these publica-tions based on their focus on primary or secondary emotionsand mood (for a more detailed analysis of emotion theoriesand emotion studies within a broader HCI context, seeLopatovska and Arapakis, 2011).

Primary and Secondary Emotions

Similar to the lack of agreement on a definition of a basicconstruct of “information,” there is no consensus on a defi-nition of emotion. However, most researchers have agreedthat emotions are “short-lived psychological-physiologicalphenomena” (Levenson, 1994, p. 123) that focus our atten-tion on the relevant information about our environment andguide our behaviors, judgments, and decisions (Payne &Cooper, 2001).

Classical theories of emotion focus on relationshipsbetween emotion and stimuli, structure and types of emo-tions, and emotional expressions. The two dominant viewson the relationships of emotion to stimuli emphasize eithersomatic or cognitive factors. The somatic view (Ekman,1984; Plutchik, 1980; Tomkins, 1984) considers the affec-tive system as the primary motivation system that canamplify any other state (e.g., interference with breathingcauses terror that leads to the struggle for air), and treatsemotions as specific physiological expressions that haveevolved to deal with prototypical life events. The cognitive

view emphasizes the importance of cognitive evaluations inestablishing the meaning of stimuli and ways of coping withthem (Lazarus, 1984). According to the cognitive view, emo-tions have intentionality and include cognitive appraisal,physiological change, and action (e.g., as one senses thepresence of a tiger, the body produces adrenalin, and onestarts to run).

An emotion theory that bridges the gap between cogni-tive and somatic views was proposed by Damasio (2005).Based on the neurophysiological evidence of emotionalstimuli processing, Damasio advocated the existence ofprimary and secondary emotions. Primary, or innate, preor-ganized emotions arise as an immediate response to emo-tional stimuli that also might be caused by internal bodilyprocesses. Secondary emotions “occur once we begin expe-riencing feelings and forming systematic connectionsbetween categories of objects and situations . . . and primaryemotions” (Damasio, 2005, p. 134), and are based on evalu-ative processes. Examples of the primary emotions includehappiness and sadness while relief and hope could be clas-sified as secondary emotions.

For the purposes of our study, we used Damasio’s (2005)classification of emotions to define primary and secondaryemotions. We defined primary emotions as immediate, non-evaluative affective states triggered by subconscious orsemiconscious reactions to emotional stimuli. Secondaryemotions were defined as affective states that “occur oncewe begin experiencing feelings and forming systematic con-nections between categories of objects and situations”(Damasio, 2005, p. 134) and involve cognitive evaluationsof the emotional stimuli.

To identify appropriate methods for studying primary andsecondary emotions, we reviewed the literature on emotionstructure and manifestations. The two major views on thestructure of emotion include continuous and discreteapproaches. The continuous approach assumes the existenceof dimensions such as valence (negative/positive) andarousal (calm/excited) that describe and distinguish betweendifferent emotions (Barrett & Russell, 1999). The discreteapproach claims the existence of basic universal emotionsthat manifest themselves in cross-cultural universals forfacial expressions and antecedent events, and are present inother primates (Darwin, 1872/2005; Ekman & Friesen,2003). Basic emotions usually include fear, anger, disgust,happiness, sadness, and surprise (Ekman, 1992; Plutchik,1980) while other emotions are seen as combinations ofthese basic emotions or as socially learned variants of them(e.g., grief, guilt, and loneliness are variants of basicsadness, [Bower, 1992]).

Both continuous and discrete approaches are associatedwith specific methods for studying emotion. For example, ithas been common for studies that treat emotion as a con-tinuous phenomenon to apply neurophysiological methodsthat link changes in brain activity, skin conductance, andrespiratory, cardiovascular, muscle, and other body func-tions to the investigated stimuli (for an example of a studythat used physiological metrics to examine effects of

1776 JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—September 2014DOI: 10.1002/asi

physical or emotional stress, see Wilhelm, Pfaltz, andGrossman, 2006). Other methods that rely on a dimensionalapproach for studying emotions include participant reportsof their feelings (Barrett & Russell, 1999) or biologicaldetection of arousal/valence that is later mapped onto thetwo- (or more) dimensional space (Peter & Herbon, 2006).

Proponents of the discrete approach often rely on auto-matic facial expression analysis systems that directly inter-pret changes in the facial muscles and classify them intobasic emotion categories (Jaimes & Sebe, 2007). Additionalmethods for studying discrete emotions include, forexample, speech (Murray & Arnott, 1993), gesture analysis(Gunes & Piccardi, 2007), and participant self-report(Tenopir et al., 2008).

Use of both discrete and continuous approaches forstudying emotion is evident in the HCI research. Forexample, Klein, Moon, and Picard (2002) investigatedmeans of minimizing frustration, a discrete emotion, byproviding computer-generated supportive messages. Peterand Herbon (2006) advocated the continuous nature of emo-tions and, within the HIC research, suggested classifyingdifferent emotional states within arousal-valence spacewithout labeling them. While we did not find LIS literaturethat clearly refers to somatic or cognitive theories ofemotion, or discrete and continuous methods for studyingemotion, authors’ methodological positions can often beinferred from the study variables and methods. Several LISstudies that focused on affect are reviewed later.

For our study, we chose to use methods associated withthe discrete view on emotion for two main reasons:

• The notion that each basic emotion is characterized by aunique motivational and behavioral tendency helps to connectdiscrete information search stimuli to discrete emotionalresponses that in turn lead to behavior modifications. Forexample, review of search results can be linked to sadnesswhich in turn leads to query re-formulation.

• Use of the discrete approach offers a number of methodologi-cal advantages, including ease of interpreting emotion read-ings, unobtrusiveness of data collection instruments, andconsistency with traditional methods for investigatingemotion in the information retrieval context.

The Methods section provides a more detailed descrip-tion of the study methods.

Mood

Mood is an affective state that is closely related toemotion, but has its own unique characteristics. Similar toemotion, mood serves both informational and motivationalpurposes (Morris, 1999); it “tells us” when to act and whento preserve energy; it affects our judgments, cognitive, andmemory processes (Thayer, 1996); and guides our behavior.However, unlike emotion, mood is a mild- or moderate-intensity feeling that usually lasts longer than does emotion.Unlike emotion, mood is not felt “about” anything (Morris,1999) but rather represents a summary of affective states.

Mood can be influenced by internal and external stimuli(e.g., weather, life events; [Clark & Watson, 1988]), anddoes not have a clear beginning or ending.

The decision to include mood in our study was informedby previous research that has emphasized the importance ofmood in HCI. Bilal and Bachir (2007) found that moods andattitudes prior to the search affected the search process.Lazar, Jones, Hackley, and Shneiderman (2006) found thatfrustration levels during the search were negatively corre-lated with the mood after the session. In a study of a singlesubject’s reactions on identical stimuli over a period of time,Picard, Vyzas, and Healey (2001) found that features ofdifferent emotions on the same day clustered more tightlythan did features of the same emotions on different days,suggesting that a long-lasting affective state influenced thepattern of emotional responses.

Affect in LIS Research

The following definition of emotion by Dervin andReinhard (2007) accurately captures the major lines ofemotion research in LIS:

. . . emotion conceptualized as: being caused by or arising out ofsituations, tasks, or contexts or their subparts; being attributesof persons—their personalities, demography, genetics, physiol-ogy, or past experiences; being causes of inhibiting or activatingmotivations; causing or leading to specific actor goals or activi-ties; being encoding traces left in information, message, or textpackages; and serving as states of being that have informationalvalue. (p. 55)

The LIS literature on emotions, affect, and feelings rarelyhas defined these constructs or has clearly articulated themethodological framework for studying them. We attemptedto group previous LIS studies of affect into studies that havefocused on primary emotions, secondary emotions, andmood.

A growing area of emotion research includes studies thathave focused on primary emotions or the real-time emo-tional responses to information retrieval stimuli. The mostcommon method for studying immediate emotionalresponses to search stimuli involves the collection of physi-ological signals. Mooney, Scully, Jones, and Smeaton(2006) used physiological measures of galvanic skinresponse and skin temperature to examine the role of search-ers’ emotional states during a data-indexing task. Theauthors were able to link specific search events to patterns ofphysiological responses. Arapakis, Jose, and Gray (2008)used facial expressions of basic emotions to determinetopical relevance of documents and videos and to inform thedesign of a video recommender system. Facial expressionanalysis was used in a study of digital libraries’ interfaces(Lopatovska & Cool, 2008). One of the interesting studyfindings includes the fact that during an experiment whenparticipants were searching digital collections alone, expres-sions of positive emotions spiked mostly during the timeswhen a research assistant entered the room.

JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—September 2014 1777DOI: 10.1002/asi

A majority of the reviewed LIS studies have focused onsecondary emotions, or “experienced” affective states thatparticipants are consciously connecting to the searchstimuli. The preferred methods for studying secondary emo-tions include think-aloud protocols, questionnaires, andinterviews. Examples of such studies include the work ofBilal (2000), Bilal and Kirby (2002), Kalbach (2006),Tenopir et al. (2008), Wang et al. (2000), and others. In astudy of children’s use of a search engine, Bilal (2000)found an association between positive feelings and specificweb browser features such as keyword search option andgraphics (Bilal, 2000). Another study of children’s searchingbehavior has linked positive feelings to the educational andaesthetic features of a digital library. In a study of the Inter-net search behavior of children and adults, Bilal and Kirby(2002) were able to connect feelings of satisfaction andcomfort to the successful completion of the task, and frus-tration to the difficulties in finding the answer. Wang et al.(2000) examined cognitive and affective aspects of searchbehavior on the web, and linked successful search perfor-mance to reduced anxiety and other negative feelings. In astudy of affective and cognitive behaviors during an onlinesearch, Tenopir et al. (2008) found that negative feelingswere associated with system performance, search strategy,and task. Kalbach (2006) analyzed website designs andidentified particular features that help to reduce user uncer-tainty and build confidence during the search. In a recentstudy of relationships between emotions and perceivedsearch success and search effort, researchers based theirquestionnaire on emotions most frequently mentioned in thecontext of online search (Flavián-Blanco, Gurrea-Sarasa, &Orús-Sanclemente, 2011); the authors found positive corre-lations between the following variables: (a) perceived effortand a feeling of joy after the search, (b) cheerful attitudebefore the search and increased regret and frustration afterthe search, and (c) surprise during the search and feelings ofdisgust and anger after the search.

Within the LIS studies of emotion, the think-aloud proto-col is a popular method for studying immediate emotionalresponses to information search stimuli. While investigatingcriteria for online document selection, including the emo-tional value of documents, Wang and Soergel (1998) asked 25participants to verbalize their thought process. The authorsnoted that while emotional values were frequently observed,they were infrequently verbalized, suggesting that self-reportmight not be an effective method for studying emotionalaspects of searchers’ judgments. Tenopir et al. (2008) exam-ined participants’ interactions with the Science Direct systemby collecting real-time participant accounts of their affectiveand cognitive states; they found that positive emotions wereusually associated with satisfactory search results.

A number of LIS studies can be viewed as studies of moodwithin the search context due to their focus on relativelylong-lasting, summative affective states that are not linked toa specific search stimulus. An example of such a study can befound in Nahl (2004), who examined the searching behaviorof senior college students and found that the states of

self-efficacy and optimism were positively correlated with amotivation to complete the search task and satisfaction withsearch results. Another study of college students showed thatlibrary anxiety negatively affected research paper grades(Onwuegbuzie & Jiao, 2004). The majority of reviewedstudies used questionnaires to measure background affectivestates. In the aforementioned study of web searching (Wanget al., 2000), student participants were asked to fill out theState Trait Anxiety Inventory (STAI; Forms Y1 and Y2,Spielberger, Gorssuch, Lushene, Vagg, & Jacobs, 1983) tomeasure their affective states prior to searching. The STAIconsists of two forms: S-anxiety, which measures an indi-vidual’s general tendency of feelings, and T-anxiety, whichmeasures an individual’s current feelings. Kracker (2002),who examined affective and cognitive states involved in acompletion of a research assignment, also applied STAI FormY-1 to measure student anxiety. In an aforementioned studyof student search performance, anxiety levels, and researchachievement (Onwuegbuzie & Jiao, 2004), participants’affective states were measured using several questionnaires:the Library Anxiety Scale, Hope Scale, ProcrastinationAssessment Scale, Multidimensional Perfectionist Scale, andother instruments. Gwizdka and Lopatovska (2009) incorpo-rated questions about participants’ affective states into thepre- and postsearch task questionnaires. They found thatparticipant mood scores were not affected by the searchprocess, and that participants whose mood was generallybetter were less satisfied with their search results and that thequality of their search outcomes was lower. Flavián-Blancoet al. (2011) used the Positive Affect Negative Affect Scale(PANAS; Watson, Clark, & Tellegen, 1988) to capture par-ticipants’ affective states prior to the search and identified thetrend similar to the finding of Gwizdka and Lopatovska:better mood prior to the search resulted in increased regretand frustration after the search.

In summary, we found a large body of literature onemotion theories and studies of affective variables ininformation-seeking and retrieval situations. However, a lackof clear definitions of affective constructs and a scarcity ofstudies that have examined multiple affective variables in thesame context complicates the development of theories ofemotion in online searching. The current study attempts toadvance methodological and theoretical developments in theLIS research on emotion by (a) using three clearly definedaffective variables, (b) identifying the potential of each affec-tive variable to provide useful information about a searcher ora search, (c) outlining the strengths and weaknesses of severalmeasurement instruments, and (d) proposing a framework ofrelationships between primary and secondary emotions andmood and the online search variables.

Methods

The reviewed literature did not provide sufficient supportto develop hypotheses about the specific nature of relation-ships between the primary emotions, secondary emotions,mood and the search process. We therefore designed an


exploratory experimental study to investigate these relation-ships between emotion, mood, search process, searcherdemographics, and task difficulty variables. Some of thestudy results did not produce interesting findings orwere previously published, and are not included here(Lopatovska, 2009, 2011). This article presents additionalfindings, and largely expands on the ideas outlined in theprevious publications by focusing on the following researchquestions:

RQ1: What are the relationships between primary emotions andthe search process?

RQ2: What are the relationships between secondary emotions andthe search process?

RQ3: What are the relationships between mood and search perfor-mance and outcomes?

Affective Variables

Our study examined three distinct affective constructsthat are well-established in psychology and have been pre-viously used in the information-seeking research. Theseconstructs include (a) primary emotions, (b) secondary emo-tions, and (c) mood.

We applied Ekman’s (2003) framework of basic universalemotions to detect primary emotions (neutral, fear, surprise,sadness, happiness, anger, and disgust) that manifestedthemselves in searchers’ facial expressions. We relied onparticipants’ self-reports to collect information about theirsecondary emotions and mood.

Primary emotion data were collected by creating videorecordings of participants’ faces during the search. Thevideo streams were then analyzed for the presence of emo-tional expressions using the eMotion software based on thefacial recognition (FACS) framework. The software ana-lyzes appearance changes in facial features by constructinga three-dimensional, wire-frame mesh over the recordedface, noting the positions of certain facial features (eye brow,corner of a mouth, and eyes, etc.) and feeding the readingsinto the emotion classifier developed from a subset of theCohn–Kanade database of approximately 500 emotionalexpression images from 100 subjects (Cohn & Kanade,2007). To eliminate noise in the data, we analyzed onlyexpressions that were classified as a particular emotion witha .9 probability or higher.

Since secondary emotions include cognitive evaluationsof the emotional stimuli (Damasio, 2005), we chose a self-report method to gauge participants’ self-assessed emotionalstates. Data about participants’ secondary emotions werecollected by asking searchers to describe the feelings thatthey experienced during the search in the post-search inter-views. During the interviews, search screen recordings wereplayed out for participants to refresh their memories andstimulate the discussion about their search experiences.Most of the interview questions were open-ended and gaveparticipants freedom to choose their own words to describetheir emotional states.

For assessing mood, a “background feeling” (Damasio,2005) that is not directly connected to a specific stimulus,participants were asked to fill out a PANAS questionnaireseveral times during the search. The PANAS is comprised oftwo 10-item scales that measure positive affect (i.e., theextent to which the person feels enthusiastic, active, andalert) and negative affect (i.e., the extent to which the personexperiences subjective distress, including anger, contempt,disgust, guilt, fear, and nervousness) (for more informationabout the scale, see Watson et al., 1988).

Search Process Variables

The search process was represented by the three types ofvariables: search actions, search performance, and quality ofsearch outcomes.

Search actions were defined as moves directed at changingthe search screen output and included the following actions:left button down, left button double, right button down, middlebutton down, wheel down/−, wheel up/+, Google results pagechange, and non-Google target page change. We recordedmouse clicks and URL changes using the Tech Smith MoraeRecorder (Version 2.0.1). The included search actions werefurther grouped into the three types of tactical decisions:selection, text manipulation, and (re)examination (Figure 1).

Search performance was defined as a combination ofsearch activities that are indicative of participants’ searchingskills (Bilal & Kirby, 2002; Stenmark, 2008). Following pre-vious studies that have measured search performance, wecollected data on the following search-performance variables:

• Time spent on a search task• Total number of URLs visited• Number of viewed hits, defined as a clicked-on URL from a

search results’ page (Jansen & Spink, 2003)• Number of result pages requested per session; that is, the

number of times a user submitted a query to a search engine,possibly indicating a user’s interactivity levels (Stenmark,2008)

• Number of unique queries per session, which may indicatelevels of a user’s interactivity (Chen & Cooper, 2001)

• Query length, defined as the number of words in a query,where word is “any unbroken string of characters” (Jansen,Spink, & Saracevic, 2000, p. 211) entered into the webbrowser search box; the variable possibly indicating search-ers’ sophistication levels (Stenmark, 2008)

• Time spent on examining each search result page in a searchsession

• Time spent on examining each selected document, defined astime between clicking on the URL from a search engineresults’ page to returning back to the search engine (Jansen &Spink, 2003). The length of time spent viewing the documentscan be linked to user knowledge and thoroughness levels(Stenmark, 2008).

Data on the search-performance variables were collectedfrom the Morae software log files.

Search outcome was defined as the end result of thesearch. As part of the study, each participant produced


written answers to the two search tasks that were posedduring the experiment; these written responses constitutedthe search outcomes. Search tasks were presented to partici-pants as search scenarios requiring them to find informationfor a friend and write their answers in an e-mail format. Thetext of participants’ answers was subsequently given to threeindependent judges who evaluated the general quality ofanswers based on their completeness, correctness, and pre-sentation. The averages of the judges’ scores were used torepresent the quality of the search outcomes (the methodwas previously used by Gwizdka, 2008, and others.)

Figure 1 lists emotions, mood, and search-process vari-ables and the relationships between these variables investi-gated in the study. We examined relationships betweenprimary emotions and secondary emotions and all three com-ponents of the search process, including search actions,search performance, and search outcomes. We investigatedrelationships between pre- and postsearch mood on searchperformance and search outcomes. However, we chose not toexamine relationships between mood and search actions dueto the differences in temporal dimensions of these variables,with mood being a relatively long-lasting state, while most ofthe search actions take only a few seconds. In our analysis ofthe relationships between mood and search-activity levels,we chose the search-performance construct as a better repre-sentation of search process.

Sample and Procedure

Thirty-six undergraduate students enrolled in a psychol-ogy course were recruited for the experiment. Data on 6participants were incomplete and were not included in the

analysis. The average age of participants was 19 years;13participants were males, and 17 were females. The ethnicdistribution of participants was diverse, with Whites andAsians accounting for the majority of participants. Most ofthe students participating in the study were natural sciencemajors, followed by social sciences majors, and a relativelylarge number of undecided majors. All but 1 participantreported online searching to be at least a daily activity. As areward for their participation in the study, students weregiven a course credit.

The study was designed as a laboratory experiment. Themajor advantage of an experimental design over a natural-istic study was the ability to control some of the variablesthat affect emotions and to record every aspect of the onlinesearch, including system performance, users’ search actions,and manifestations of the experienced emotions. The majorlimitation of the selected method included the influence ofthe laboratory setting on participants’ interest, motivation,and general engagement in the search tasks. This limitationwas partially mitigated by the experimental procedure thatincluded search scenarios that were similar to participants’routine search tasks and required participants to use familiarinformation retrieval system: the Google search engine.

We used two different search tasks in the experiment. Thesearch tasks were informed by the task classification of R.W.White (Bell & Ruthven, 2004; White, 2004; White, Ruthven,& Jose, 2005), and included one low-complexity and onehigh-complexity task. A low-complexity task was defined asa task that provided participants with more information;“Subjects generally found tasks in this category the more‘clear’and ‘simple’than those from other categories” (White,2004, p. 185). A high-complexity task was defined as a

FIG. 1. Model of the relationships between emotions, mood, and online information searching variables. [Color figure can be viewed in the online issue,which is available at wileyonlinelibrary.com.]


vaguely formulated task requiring information from multiplesources; “Subjects found these tasks difficult and classifiedtasks in this category as least ‘clear’ and ‘simple’ ” (White,2004, p. 185).

We pretested White’s (2004) tasks in a pilot experimentand selected the two tasks that received the most consistentdifficulty evaluations from participants: a search scenarioabout university enrollment and a music-piracy scenario.

Low-complexity task: “A friend has recently been applying tovarious universities and courses but has been complaining thatthey are finding it difficult to attain a place due to the risingnumbers of students. You were unsure if their assessment wascorrect so you have decided to find out how the size of thestudent population changed over the last 5 years and how it isexpected to change in the coming 5 years.”High-complexity task: “Your friend has just finished reading acopy of a national newspaper in which there is mention ofInternet music piracy. The article stresses how this is a globalproblem and affects compact disc sales worldwide. Unaware ofthe major effects, you decide to find out how and why musicpiracy influences the global music market.”

In the pilot study and the experiment, the music-piracytask was judged as more complex than the enrollment taskand, on average, took participants longer to complete.

The final text of White’s (2004) tasks was slightly modi-fied for the purposes of our study. Search tasks were rotatedusing a Latin square design so that half of the participantsreceived the high-complexity task first while the other halfreceived the low-complexity task first.

The experiment took place in a computer laboratory of alarge university. The room had a computer desk with twomonitors and a keyboard. One monitor displayed theGoogle search engine in Windows Internet Explorer(Version 7) and was used by participants for searching. Thesecond computer monitor was used to display the text ofinstructions, search scenarios, and pre- and postsearch taskquestionnaires. Two web cameras were positioned aboveand below the primary search monitor to video recordsearchers’ faces. The video stream captured by one of thecameras was used as the primary source for the eMotionfacial-expression classification.

Each participant was scheduled for an individual sessionlasting no more than 2 hours, with most of the searchescompleted in under 1 hr. A researcher was not present at thelab during sessions to minimize interference with the searchexperience.

Upon completion of the two search tasks, the researcherreturned to the lab, showed participants a screen recordingof their search sessions, and asked participants to commenton their search activities and emotions. The postsearch inter-view also collected information about the extent to whichparticipants were disturbed by the lab setting, the extent towhich they were clear about the experimental instructionsand tasks, their primary motivation to participate in thestudy, and their general mood that day. Handwritten noteswere created during the postsearch interviews and used as a

primary source of data about participants’ self-reportedstates. Audio recordings of the interviews also were createdand used to verify and expand the handwritten notes whennecessary.

Results

Primary Emotions and Search Actions

Video recordings of participants’ facial expressions wereclassified by the eMotion software into seven emotion cat-egories: neutral, happiness, surprise, anger, disgust, fear, andsadness. Expressions that matched emotion categories with.9 probability or higher were included in the analysis.

We analyzed distribution of emotional expressions acrossall participants and normalized each participant’s emotiondata to account for the differences in search duration andfrequencies of emotional expressions (see Appendix).Figure 2 illustrates the distribution of total emotionalexpressions of all 30 participants during 30 searches. Thex-axis separates the total averages of the seven emotions inno particular order (e.g., although the neutral emotion islisted first, it did not occur before the happiness emotion).The y-axis represents the average percentages of each emo-tion’s occurrences. The figure illustrates that the most fre-quently expressed emotion during the search was surprise,followed by neutral, sadness, fear, happiness, disgust, andanger.

Data on the search activities were extracted from theMorae log file and integrated with the eMotion readingsusing the time stamps associated with the search moves(Morae) and video frames (eMotion).

FIG. 2. Distribution of emotional expressions for all the participantsacross all searches. [Color figure can be viewed in the online issue, whichis available at wileyonlinelibrary.com.]


To examine relationships between expressions of primaryemotions and the search actions, we analyzed the changes ofthe seven emotional expressions before and after each searchaction (e.g., click and the URL change). We selected five3-second intervals before and five 3-second intervals after theaction to examine the pattern of emotional expressions aroundthe search event. The 3-second interval was chosen as a resultof an analysis that showed that the average duration of anemotional expression was 3 seconds and that there were nosignificant changes in emotional expressions within a 3-secondinterval; therefore, the 3-second interval could be representedby the dominant emotion. We calculated the probabilities ofeach emotional expression occurrence within 3-second inter-val and performed a one-way analysis of variance (ANOVA) tocompare each emotion means for the five 3-second intervalsbefore and five 3-second intervals after the search action.

Table 1 lists statistically significant changes in emotionalexpressions around the search actions. For example, theinclusion of the happy expression under the mouse “wheeldown” scrolls indicates that across all 30 participants, happyexpressions occurred statistically more frequently 3 to 9seconds before the wheel down scroll, and dropped signifi-cantly in the 0- to12-second interval after the click.

The analysis of the relationships between expressions ofprimary emotions and the search actions can be summarizedas follows:

• The most frequent emotional expression during the searchwas surprise, followed by the neutral expression.

• The most frequent search behaviors were mouse “wheeldown” scroll (representing the decision to [re]examine infor-mation) and “left down” mouse click (representing contentselection).

• Almost every analyzed search action was characterized by aunique pattern of emotional expressions 15 seconds beforeand 15 seconds after the event (see Table 1).

• In some cases, search actions resulted in an improvement ofsearcher’s emotional state. For example, scrolling up the webpages led to the decrease of fear expressions. However, in mostof the cases, a search move resulted in an immediate increaseof negative second emotions. For example, “left mouse doubleclick” led to increase in disgust and sad expressions.

Primary Emotions, Search Performance, and Outcomes

We examined relationships between seven emotionalexpressions, search performance, and search-outcomes vari-ables by running canonical correlation analysis (CCA;Sherry & Henson, 2005). The CCA method was chosenbecause it allowed us to examine linear relationshipsbetween two multidimensional dependent (emotion) andindependent (performance) constructs. When interpretingthe CCA results, we examined the p value to see if the modelwas statistically significant; we also examined Wilks’s Λ toderive the effect size (effect size or Rc2 = 1 − Λ). Whenexamining the contribution of measurable variables intothe creation of the synthetic dependent and independ-ent variables, we interpreted function coefficients and

structure coefficients. Function coefficients are standardizedcoefficients that are used in the linear equations to combinethe observed variable into the synthetic variable, and areanalogous to beta weights in regression. Structure coeffi-cients are the bivariate correlation between an observed vari-able and a synthetic variable, and are analogous to structurecoefficients in multiple regressions and factor analysis. Thesigns of structure coefficients inform interpretation of therelationships between the variables, for example, a negativesign indicates lower scores or fewer actions while a positivesign indicates higher scores or more actions. A squaredstructure coefficient is analogous to any r 2-type effect sizeand represents a synthetic variable’s variance explained byan observed variable (Sherry & Henson, 2005).

CCA is a multivariate technique that requires a largesample size. Due to the medium sample size (30 participantsperforming two search tasks each resulted in the total of 60search episodes), our analysis did not always produce sta-tistically significant results. However, most of the producedmodels had large effect sizes that indicated the existingrelationships between variables. We made the decision tointerpret CCA models that had large effect sizes but were notstatistically significant because for the exploratory study, themodel effect size serves as an important indicator of theexisting relationships between the variables and points tothe direction of future research even when the model is notstatistically significant (Henson, 2006).

We used CCA to correlate occurrences of the seven emo-tional expressions with the following performance variables:search duration (task time), query length (query length),time examining search results (tview results), time examin-ing target pages (tread hits), number of queries (uniquequeries), number of viewed hits (reviewed hits), number ofresult pages requested per session (Google pages), andratings of the search outcomes’ quality (quality).

The resulting model across all functions was not statisti-cally significant using Wilks’s Λ = .29 criterion, F(56,247.64) = 1.131, p = .261. However, the full model r 2 typeeffect size was .71, which indicated that the canonical func-tions explained a large 71% of the variance shared betweenthe variable sets. The analysis yielded seven functions, noneof which were statistically significant, with squared canonicalcorrelation (Rc

2) of .355 for the first function, squared canoni-cal correlation (Rc

2) of .263 for the second function, squaredcanonical correlation (Rc

2) of .193 for the third function,squared canonical correlation (Rc

2) of.127 for the fourth func-tion, and squared canonical correlation below .10 for theremaining functions. We chose to interpret the first two func-tions that explained the largest amount of variance. Table 2presents the standardized canonical function coefficients(Coef ), structure coefficients (rs), and squared structure coef-ficients, or communalities (rs

2) for the interpreted function.In interpreting Function 1, the strongest contributor to the

dependent synthetic variable was happiness. This was sup-ported by a relatively strong structure coefficient. Judgingby the function coefficients, the strongest contributors to theindependent synthetic variable were task time, unique


TABLE 1. Summary of statistically significant changes in emotional expressions around each analyzed search actions.

Search behaviorEmotional expressions that significantly varied

around the event 15–0 s before the event 0–15 s after the event

(Re)Examination Decision to review the content of a search results’ page or a target pageWheel down (total) (n = 6,495) Neutral Low High

F(9, 64938) = 4.7558, p < .05 (15–0 s before click) (12–15 s after click)Happiness High LowF(9, 64938) = 4.455, p < .05 (3–9 s before click) (0–12 s after click)SurpriseF(9, 64938) = 3.648, p < .05

Fluctuated in the intervals before and after click, no clear pattern

Anger low HighF(9, 64938) = 30.907, p < .05 (15–0 s before click) (0–15 s after click)Disgust Peaked around/during the clickF(9, 64938) = 6.764, p < .05 3–0 s before and 0–3 s after clickFear Low HighF(9, 64938) = 4.289, p < .05 (15–6 s before click) (3–15 s after click)Sadness High LowF(9, 64938) = 24.733, p < .05 (15–6 s before click) (0–15 s after click)

Wheel down non-Google targetpages (n = 5,768)

Neutral Low HighF(9, 57669) = 6.278, p < .05 (15–13 s before click) (9–15 s after click)HappinessF(9, 57669) = 5.365, p < .05

Fluctuated in the intervals before and after click, no clear pattern

SurpriseF(9, 57669) = 3.483, p < .05

Peaked 0–3 s after the click

Anger Low HighF(9, 57669) = 31.257, p < .05 (15–0 s before click (0–15 s after click)DisgustF(9, 57669) = 8.273, p < .05

Peaked around the click (3 s before and 3 s after the click)

Fear Low HighF(9, 57669) = 5.232, p < .05 (15–6 s before click) (3–15 s after click)Sadness High LowF(9, 57669) = 34.473, p < .05 (15–0 s before click) (0–15 s after click)

Wheel up (total) (n = 1,940) HappinessF(9, 19390) = 2.927, p < .05

Peaked around the click (3 s before and 3 s after the click)

AngryF(9, 19390) = 3.307, p < .05

No clear pattern

DisgustF(9, 19390) = 6.555, p < .05

Peaked 12–9 s before the click

Fear High LowF(9, 19390) = 6.555, p < .05 (15–3 s before click) (0–15 s after click)Sadness Low HighF(9, 19390) = 6.175, p < .05 (15–6 s before click) (6–15 s before click)

Wheel up on non-Google targetpages (n = 1,517)

HappinessF(9, 15165) = 3.982, p < .05

Least frequent 9–12 s after the click

DisgustF(9, 15165) = 9.026, p < .05

Peaked 12–9 s before the click

Fear High LowF(9, 15165) = 3.926, p < .05 (15–0 s before click) (9–15 s after click)

Wheel down on Google targetpages (n = 727)

Sadness Low HighF(9, 7259) = 5.143, p < .05 (15–0 s before click) (3–15 s after click)

Wheel up on Google resultpages (n = 423)

AngerF(9, 4215) = 4.680, p < .05


DisgustF(9, 4215) = 4.233, p < .05


Fear High LowF(9, 4215) = 6.226, p < .05 (15–3 s before click) (0–15 s after click)Sadness Low HighF(9, 4215) = 7.860, p < .05 (15–0 s before click) (3–15 s after click)

Selection Decision to change the current view by clicking on a Search button, URL or another application, changing the page focus (scrollingup/down the page, rearranging pages on desktop, etc.).

Left mouse down click (n = 2,839) Neutral High LowF(9, 28379) = 3.138, p < .05 (15–12 s before click) (0–3 s after click)Surprise Low HighF(9, 28379) = 2.413, p < .05 (15–12 s before click) (3–6 s after click)Sadness Low HighF(9, 28379) = 3.447, p < .05 (9–6 s before click) (6–12 s after click)

Left mouse double click (n = 86) Disgust Low HighF(9, 854) = 1.912, p < .05 (15–0 s before click) (6–12 s after click)Sadness Low HighF(9, 854) = 2.239, p < .05 (15–0 s before click) (0–15 s after click)

Middle button down click (n = 13) Neutral High LowF(9, 120) = 2.956, p < .05 (15–12 s before click) (6–15 s after click)HappinessF(9, 120) = 2.713, p < .05


Surprise Low HighF(9, 120) = 3.602, p < .05 (15–12 s before click) (0–15 s after click)DisgustF(9, 120) = 2.750, p < .05

No clear pattern

Non-Google page change (n = 658) Happiness High LowF(9, 6570) = 2.851, p < .05 (9–6 s before click) (6–9 s after click)

Query submission None of the emotional expressions varied significantlyText manipulation Decision to manipulate found text (copy, paste, save, etc.)

Right button down click (n = 90) None of the emotional expressions varied significantly


queries, time viewing results and time reading hits. Thisconclusion was supported by the structure coefficients of thetask time and time reading hits variables, and not supportedby the structure coefficients of the unique queries and timeviewing results. Our interpretation of the signs of the struc-ture coefficients of the dependent and independent variablessuggests that more frequent expressions of happiness coin-cided with longer time spent on search task and time readinghits, and fewer reviewed results and unique queries entered.

In Function 2, the strongest contributors to the dependentsynthetic variable were happiness, fear, and sadness. Thesevalues were also supported by the structure coefficients.Judging by the function coefficients, the strongest contribu-tors to the independent synthetic variable were Googlepages, unique queries, and time reading hits. This conclu-sion was partially supported by the structure coefficients.Our interpretation of the function coefficients and the signsof the structure coefficients of the dependent and indepen-dent variables indicated that less frequent expressions of fearand happiness coincided with more Google pages reviewed,less unique queries, and less time spent reading hits.

Secondary Emotions

Examination of participants’ accounts about their emo-tional states indicated that most of them reported not feelingvery intense emotions during the search (on a scale of 1[Very slightly or not at all] to 5 [Extremely], the mean of theresponse to the question “How intense were the feelings youexperienced during the search?” was 2.) Participants alsoreported not controlling their feelings or their expression (ona scale of 1 [Very slightly or not at all] to 5 [Extremely], themean of the response to the questions “To what extent didyou try to control your feelings by reducing their intensityand shortening their duration?” and “To what extent did youtry to control the expressions of your feelings?” was 1.) Suchreports suggested that for participants, searching experi-ences in a lab were not emotionally different from searchingexperiences in a natural setting.

During the post-search interviews, videos of participants’search sessions were shown to them to facilitate recollectionof search events and experiences. During the interview, par-ticipants were asked to describe their feelings and emotionsassociated with specific search process and reviewed web-sites’ content. None of the participants reported experienc-ing extremely positive or extremely negative emotions. Themost representative participant response was: “<I amfeeling> Nothing. Just searching.”

Several participants reported having feelings associatedwith specific search events. For example, 3 participantsreported feeling frustrated (n = 3) due to the difficulties infinding the answer; 1 participant felt annoyed for the samereason; and 1 participant felt excited and another felt a senseof accomplishment due to the successful task completion.Additional emotions identified in relation to the searchprocess were enjoyment of specific websites and content,and interest in the search process or search task.

The themes that emerged from participants’ accounts ofemotional experiences not directly related to the searchprocess were: feeling generally good and positive (n = 8),feeling content/relaxed/calm (n = 3), feeling worried(n = 3), and feeling gloomy (n = 1).

While describing their emotional experiences, more thanhalf of the participants reported their physiological states suchas feeling tired (n = 17), hungry (n = 4), and sleepy (n = 2).While these states cannot be viewed as emotional states, theycould have affected participants’ mood and activity levels.

Another interesting finding indirectly related to emotionswas participants’ inability to accurately estimate the timethat they spent searching. When asked to estimate how muchtime they spent searching, 11 participants felt that they spentless time on the task than they actually did, 8 participantssaid that they did not pay any attention to the time, and 4participants felt like they spent more time searching thanthey actually did.

Mood and Search Performance and Outcomes Quality

On average, participants experienced more positive affectthan negative affect. Before starting the first search task,

TABLE 2. CCA of performance and emotion variables.

Functions/variables

Functioncoefficient (coef)a

Structurecoefficient (rs)a

Communalities(rs

2) (%)

Function 1: Independent synthetic variableNeutral −.462 −.698 .49Happiness .533 .480 .23Surprise −.223 −.165 .03Anger −.340 −.555 .31Disgust −.314 −.049 .00Fear −.135 −.087 .01Sadness .271 .626 .39Function 1: Dependent synthetic variableTask time 1.079 .737 .54Reviewed hits −.005 .088 .01Google pages −.002 .115 .01Unique queries −.591 .000 .00Query length .025 .221 .05Tview results −.640 .066 .00Tread hits .567 .434 .19Quality −.072 .052 .00Function 2: Independent synthetic variableNeutral −.037 .016 .00Happiness −.688 −.679 .46Surprise −.069 −.382 .15Anger .256 .014 .00Disgust −.048 −.443 .20Fear −.504 −.708 .50Sadness .622 .201 .04Function 2: Dependent synthetic variableTask time .104 .023 .00Reviewed hits −.422 −.402 .16Google pages 1.022 .020 .00Unique queries −1.073 −.434 .19Query length .336 .406 .16Tview results .331 .328 .11Tread hits −.601 −.131 .02Quality .138 .129 .02

Note. ainterpreted coefficients appear in bold.


participants on average reported higher positive affect thanafter completing Search Tasks 1 and 2. Positive (PA) andnegative affect (NA) scores did not vary significantlybetween the two search tasks (Enrollment/Easy task orMusic Piracy/Difficult task).

Pre-search Mood and Search Performance andOutcomes Quality

We investigated the relationships between mood andsearch-performance variables using the CCA test. The testallowed us to examine the linear relationship between per-formance variables (task time, all urls, reviewed hits, Googlepages, unique queries, query length, tview results, tread hits,and quality) and mood variables represented by positive[pre-PA] and negative [pre-NA] affect scores reported priorto the search.

The analysis yielded two not statistically significant func-tions with the full model r 2 type effect size of .37. Thisindicated that the two canonical functions explained a mod-erate 37% of the variance shared between the variable sets.While we are not providing details about specific functionand structure coefficients of the statistical model, their inter-pretation indicated that high PA scores corresponded to thefewer all urls, reviewed hits, Google pages, and uniquequeries. In other words, participants who reported bettermoods visited fewer websites and entered fewer uniquequeries during the course of the search. The function andstructure coefficients of the second function suggested thatthe high pre-task negative affect correlated with more timespent on task, more Google pages reviewed, and more timespent reading hits.

We have noticed that the quality variable was not a majoror even a moderate contributor to the performance syntheticvariable in the two CCA functions. The statistical modelsuggested that pre-PA and pre-NA scores were not related tothe quality of the search results, only the activity levels of asearcher. This observation was further confirmed by runningindividual linear regressions between pre-PA, pre-NA, andquality of search results variables that resulted in small R2

values.

Effect of Search Performance on Mood

We conducted a linear regression test to examine the effectof performance variables on PAand NAscores collected afterthe search-task completion. The statistical analysis of the PAscores yielded a not statistically significant model, with avery modest 11% of post-PA variance explained by task time,all urls, reviewed hits, Google pages, unique queries, querylength, tview results, treadhits, and quality variables,R2 = .108, F(9, 59) = .670, p = .632 at 95% confidence inter-val. None of the independent variables were statisticallysignificant at 95, 90, and 80% confidence intervals.

We ran a linear regression to explain post-task NA scoreswith performance variables (task time, all urls, reviewedhits, Google pages, unique queries, query length, tview

results, treadhits, and quality). The statistical analysisyielded a not statistically significant model, with a modest20% of post-NA variance explained, R2 = .198, F(9,59) = 1.372, p = .226 at 95% confidence interval. None ofthe independent variables were statistically significant at 95,90, and 80% confidence intervals. The findings suggest thatsearch performance did not affect participants’ moods.

Discussion

Relationships Between Primary Emotions andthe Search Process

Analysis of emotion expressions during the search indi-cated that the most frequently expressed emotion was sur-prise, followed by neutral. Surprise expressions also werethe most frequent expressions around the analyzed searchbehaviors. The high frequency of surprise might representthe nature of search experiences that are characterized byhigh uncertainty levels, a high number of surprise reactionsto unexpected results, and reviewed content. It also might belinked to serendipitous discoveries that are “embedded in aprimary information seeking and retrieval episode” (Toms& McCay-Peet, 2009, p. 195). A high number of surpriseexpressions also can indicate the inexactness of the mea-surement instrument—in our case, the eMotion software.Because the software does not adequately interpret thecontext in which the facial expressions occur, it is possiblethat the software misclassified other expressions as surprise.For example, during a highly focused activity such asreading, the mouth might be open or the eyes can be openedwide, which can be misinterpreted as an expression of sur-prise. The literature also has reported similarities betweenthe facial expressions of surprise and fear (Du & Martinez,2011), which might have lead to occasional misclassificationof fear expressions as surprise. Another argument for theinconsistency of surprise measures can be found inReisenzein, Bördgen, Holtbernd, and Matz (2006), whoshowed that emotion readings from facial expressions wereless accurate than were emotion readings from self-reportand behavioral measures. Fine-tuning of the surprise mea-surement and further inquiry into the nature of surpriseduring the search process are necessary.

Our analysis of the patterns of emotional expressionsaround search events resulted in a few general findings.First, analysis of the participants’ search activities indicatedseveral patterns in the use of a search engine. Frequent“wheel down” mouse scrolls and the “left button down”clicks suggested that searchers spent most of the time scroll-ing down the search page to get more information andclicked on a search button or a link. The distribution of theGoogle (n = 508) and non-Google sites (n = 658) points tothe ratio of roughly 1.22 non-Google target pages opened forevery retrieved Google search result page. While on averageparticipants reviewed more non-Google pages than Googleresults pages, the low ratio suggests that searchers madequick judgments about the quality of the retrieved results


from Google result page and either reformulated theirqueries or clicked on the target link that appeared relevant.This conclusion was supported by manual examination ofthe search screen video recordings. The recordings showthat participants rarely went as far as the bottom of the firstGoogle results page. If participants did not see and click ona promising target link after examining a few top-searchresults, they usually retyped their query. This finding isconsistent with the results of previous studies that haveshown that the probability of clicking on a search resultdecreases dramatically with the rank order of a result(Joachims, Granka, Gay, Hembrooke, & Pan, 2005).

Most of the analyzed search actions were characterizedby a unique pattern of emotional expressions around theseactions (Table 1). This finding suggests that search actionscan be characterized, recognized, and classified based on theunique pattern of emotional expressions around them, andthat emotion-recognizing machines potentially can be pro-grammed to recognize, anticipate, and properly react to thesearch moves.

The majority of the search moves were characterized bythe immediate (15 seconds after the click) worsening of emo-tional states. For example, actions directed at changing thescreen view (left button single clicks and left button doubleclicks) resulted in increased sadness; scrolling up and downGoogle result pages was associated with an increased numberof expressions of sadness and disgust. These results suggestthat searchers did not see what they expected as a result oftheir action or had wrong expectations prior to the searchmove (e.g., an expectation to immediately solve the problemand find the answer). Increased sadness during the viewing ofthe Google result pages also helps to explain why mostsearchers did not spend a lot of time examining results andchose to reformulate their query right after examining a fewtop results (Dumais, Buscher, & Cutrell, 2010). Despite nega-tive emotions that immediately followed the clicks, in mostcases searchers did not “give up” and continued clicking (andexpressing negative emotions as a result of their moves). Thelatter finding suggests that negative emotions are either verymoderate (i.e., not extreme enough to terminate the search) oran ordinary (and expected) element of searching.

The only search move that was associated with the consis-tent improvement of emotions was the “wheel up” scroll onnon-Google pages. The decrease in expression of disgust andfear the increase in happy expressions suggest that reexami-nation of the target pages’ content led to improvements insearchers’ feelings. We can hypothesize that these emotionalimprovements are caused by accessing familiar content, con-firmation of relevance judgments made during the page selec-tion, prospects of the near search completion, or other factorsthat warrant further investigation.

We discovered a number of “uneventful” moves: searchactions that were not associated with specific emotionalpatterns. Such moves included “query submission” and“right button down” clicks that represented decisions tomanipulate screen text (copy, paste, etc.). This finding sug-gests that these types of clicks are different from the others.

In the case of the “right button” click, the difference can berelated to the mechanical nature of this move, lack of expec-tations before or evaluative judgments after selection,copying, or pasting text. Lack of a clear emotion patternaround “query submission” action can be related to the timeintervals that we chose to observe. It is possible that ourwindow of observation was skewed by the time it takes forthe page to load, so in the future, we need to determine notthe time when the page registers in the URL address bar butwhen the page actually loads to a browser window.

In summary, our analysis of emotional expressions aroundsearch actions points to the existence of unique emotionalpatterns around most of the search clicks. This finding opensa possibility of programming retrieval systems to “read” andreact to searchers’ moves. The study findings need to beverified by the use of alternative emotion measures and studydesigns. Use of qualitative methods such as immediate self-reported emotion measures or the think-aloud technique isneeded for future studies to verify the relationships betweenemotional reactions and search-related stimuli.

Relationships Between Secondary Emotions and theSearch Process

Our study did not produce as much data on searchers’secondary emotions as it did on primary emotions. Partici-pant reports of their affective search experiences were ratherscarce and limited to statements about the routine, emotion-ally uneventful nature of searching. A few participantsexpressed feelings of frustration, excitement, sense ofaccomplishment, or annoyance regarding specific content ortheir abilities to find information. Several participants com-mented on their general emotional states during the search,which included positive, relaxed, worried, and gloomystates. More than half of the participants reported feelingtired. While this state cannot be strictly classified as emo-tional, it might affect emotional and behavioral search vari-ables. Further investigation into the role of factors that affectsearcher energy levels, distractive factors such as hunger orsleepiness, is warranted. Another finding that needs furtherattention is searchers’ inability to accurately estimate thetime they spent searching. Underestimation or a completedisregard of time during searching might point to highengagement with the search-activity levels (O’Brian &Toms, 2008), general human inability to accurately estimatetime spent/required for a task, or other aspects of searchbehavior that make it (not) engaging, addictive, or irrational.

In light of previous studies that have relied on self-reportdata to investigate searchers’ feelings, our findings suggestthat self-reported data create only a partial picture of theemotional experiences related to searching. Participants hada tendency to recall and report their most dramatic emotionalreactions to the sites’ content and search process; manyimmediate responses to the search stimuli and effects ofemotions on the search strategies were not recalled orreported. Two methodological implications of this findinginclude:


• Self-reported feelings might represent the most memorableand, possibly, important emotional experiences and associ-ated stimuli. In other words, if an emotional reaction wasstrong enough to register in a searcher’s memory, it wasassociated with an important event that affected searcher emo-tions, actions, and memories of the search interaction. Forexample, a self-reported frustration can represent a range ofunderreported negative emotions, and when reported, carrymore weight than does the experienced, but not cognitivelyprocessed and/or remembered, emotion.

• Future studies that aim at a comprehensive investigation ofaffect in an information-seeking context should considerincorporating immediate self-report and physiological mea-sures in addition to relying on retrospective self-reportsbecause the latter measure might prove to be incomplete andlimited to participants’ cognitive and verbal abilities.

Relationships Between Mood and the Search Performanceand Outcomes

Our analysis of the relationship between mood andsearch-performance variables indicated that positive andnegative mood during the search had no effect on the qualityof the search results and a moderate effect on search activi-ties such as number of sites visited and time spent reviewingresults. Relationships between mood and search-activitylevels can be explained by a theory that treats mood as afactor responsible for monitoring resources needed to meetcurrent demands. In light of this theory, positive mood cor-relates with assessment of resource adequacy while negativemood corresponds with assessment of resource inadequacy(Morris, 1999). In the context of our study, this theory helpsto explain why fewer visited sites and reformulated queriescan signal searchers’ assessments of resources’ adequacyand correlate with the positive mood while more reviewedsites and reformulated queries might signal the assessmentof resources’ inadequacy and correspond with the negativemood. Searchers who experience positive mood are reluc-tant to review many websites and reformulate many queriesto not spoil their mood (or other, generally acceptable“status quo”) while searchers whose mood is lower searchmore “actively” to improve the situation. A preference forrisky options among individuals who experience negativemood and risk aversion among individuals who are contentwith the current state and are afraid of loss is well-known inbehavioral research (Isen, 1993; Mano, 1994). This principleneeds further verification in the context of online and offlineinformation-seeking behavior.

The relationships between mood and the quality of resultswarrant further investigation. While we did not identify aconnection between the quality of search results and positiveand negative mood experienced during the search, otherstudies have attained different results. For example, Gwizdkaand Lopatovska (2009), who used similar measures of searchresults quality but different measures of mood, found a nega-tive correlation between mood scores and search resultsquality. In other words, the authors found that the better asearcher felt, the lower the quality of his or her search results

and vice versa. In a study of college students, Onwuegbuzieand Jiao (2004) found a negative effect of library anxiety onresearch-paper quality, as measured in assignment grades.

Additional research is needed to clarify effects of mood onthe search process and the end results of this process (e.g.,search results quality). If good mood makes the searchprocess more efficient, positively affects the quality (or per-ception of quality) of search results or influences other impor-tant aspects of the search, system designers should seek waysto improve searchers’ mood. Another promising line forfuture research includes investigation of the effects of search-ers’ mood on memories of search interactions. Future workmight focus on the role of emotions in constructing memoriesand perceptions of information systems and interactions aswell as the role of emotions, memories, and perceptions insubsequent selection and use of information systems.

As with most of the experiments in the social sciences,our findings might have been influenced by the study design.One can argue that participants did not have a personal stakein the quality of the search outcomes and did not experienceextreme positive and negative reactions to the search stimuli.From the beginning of their search, participants knew thatthey would be awarded research credit for participationregardless of the outcome quality. It is possible that in asituation where participants had to find information tosatisfy their personal information needs, their behavior andaffective profile would be different. The topics of the twotasks might have contributed to the results. It would beinteresting to observe if the relationships between searchtask variables and mood change when people are searchingfor information that is important to them or their loved ones(e.g., researching a medical condition). Our results also canbe attributed to the measurement instrument error (e.g., par-ticipants’ fatigue while filling out the PANAS question-naire). To increase participants’ engagement with the studytasks, further studies should consider naturalistic designs,use of participant-generated search tasks, different measure-ment instruments (e.g., a think-aloud protocol or observa-tion instead of a questionnaire), and other techniques.

Modified Model of Emotions and Mood in the OnlineInformation Search Process

As a result of our investigation, we were able to refine theoriginally proposed model of relationships between affec-tive variables and online information search (Figure 3). Ourexamination of relationships between primary emotions andthe search process showed consistent associations betweencertain emotions and search actions (represented by the two-directional arrow in Figure 3). While the findings needfurther verification, the fact that most of the search actionscorrespond to the unique pattern of emotional expressionssuggests that knowledge of searchers’ emotions can be usedto understand search moves and vice versa.

Examination of the relationships between secondaryemotions and the search process indicated that most of therecalled emotions were associated with assessment of the


search process and search results. Since secondary emotionswere not directly associated with the search stimuli butparticipants’ assessment of these stimuli, we introduced theassessment component to the model. The concept of assess-ment, or a feedback loop, is not new to information retrievalresearch and is usually discussed in the context of behavioralmodifications based on system performance (Spink, 1997).What is interesting in our results is the fact that participantsdid not remember many details about how they achieved theresults but rather had a summative assessment of the searchprocess (e.g., difficult/easy) or results (e.g., satisfied/dissatisfied). Effects of specific search moves and systemperformance on the general emotional assessments of thesearch experience need further investigation. Our partici-pants reported several types of affective (e.g., feeling gen-erally happy) and physiological (e.g., feeling tired orhungry) states that might have affected their searching.Future work will focus on the effects of broader emotionaland physiological states on a search process.

For the RQ3, we investigated relationships betweenmood and the search performance and outcomes. We foundthat the mood reported prior to the search remained rela-tively unchanged throughout the search. This findingsupports the notion that mood is a relatively long-lastingexperience that is not felt about any specific stimuli and is noteasy to change. While we did not find a correlation betweenmood and the quality of the search outcomes, this finding is

not consistent with previous studies and needs further verifi-cation. We found a negative correlation between the moodscores and the frequency of certain search moves; forexample, searchers with higher positive mood scores gener-ally viewed fewer sites and spent less time reviewing results.Some of the study’s open questions related to the role ofmood in searching include: (a) If the mood affects the inten-sity of search activities, to what extent do these activitiesaffect the quality of search results? (e.g., does a searcher in abad mood who reviews more sites, subsequently find betteranswers?); (b) Are there any variables in the search processthat can positively or negatively affect searchers’ mood?(e.g., extremely frustrating or positive experiences that canaffect searcher’s long-term affective state?); and (c) What isthe role of mood experienced during the search in the con-struction of memories and perceptions of search experience?We think that addressing these questions would offer aninteresting line of research for future studies.

The refined model represented by Figure 3 includesoriginal constructs of primary and secondary emotions,mood, and the search process as well as additional variablesthat we think should be included in future work. The solidarrows indicate relationships identified in our study, includ-ing relationships between primary emotions and searchactions and between secondary emotions and searchers’assessments of their experience. The dotted lines representthe existence of relationships between specific variables

FIG. 3. Model of the relationships between primary and secondary emotions, mood, and the online information search process, and additional variablesidentified in the study. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]


within the construct. In the case of primary emotions andsearch performance, we identified that frequent expressionsof happiness coincided with longer time spent on searchtask, reading hits, and reviewing results, and entering moreunique queries. Less frequent happiness and fear and morefrequent sad expressions coincided with more Google pagesreviewed, less unique queries, and less time spent readinghits. We did not find a strong relationship between othervariables within primary emotions and search-performanceconstructs. Similarly, we found relatively weak relationshipsbetween mood and certain search-performance variables(e.g., We found a weak correlation between positive moodand decreased numbers of visited websites and enteredunique queries during the course of the search). We usedcrossed lines to illustrate the cases when we did not find anyrelationships between constructs, including relationshipsbetween search outcomes and primary emotions and mood.The lines representing relationships that were not investi-gated in the reported study but would be interesting toexamine in the future are marked with a question mark.

At the time when we continue to build understanding ofthe importance of emotions in HCI, the framework identifiesproductive areas for future research into the relationshipsbetween emotions and information search variables, broaderpsychological and physiological states, and searchers’ per-ceptions of information systems and interactions. Under-standing affective variables that are part of the searchprocess is the first step in designing retrieval systems forpleasurable, productive, and memorable interactions.

Conclusion

The aim of our study was to understand the relationshipsbetween the three types of affective states—primary andsecondary emotions, and mood—and the online searchprocess. One of the most significant results of the study wasthe fact that most of the search actions, which in turn rep-resented a decision to modify the search, were associatedwith a unique pattern of primary emotional expressions. Ifsearch actions are characterized by a unique set of emotionsand vice versa, it can lead to the development of classifica-tion of search actions and corresponding emotional expres-sions. Such classification can inform the development ofsystems that can recognize emotions, anticipate—and, ifnecessary, influence—searchers’ actions. For example, if asystem can read an emotional pattern that usually precedes acertain search move, it can anticipate, and perhaps preemp-tively execute, a user’s move; a system might provide anencouraging message or suggest a search strategy that mightbe useful in a particular search situation. Our analysis ofemotional patterns indicated that most of the search movesled to an immediate increase of negative emotions. Thecauses of such reactions need further investigation. Perhapssearchers overestimate the positive effects of their movesand are disappointed with the results (consistent withFlavián-Blanco et al., 2011), or perhaps all search-relatedchanges, regardless of intention and execution, result in

immediate negative reactions. The immediate deteriorationof emotions after the click also might indicate increasedlevels of uncertainty. This hypothesis is somewhat supportedby the fact that the only search action that led to theimprovement of emotional states was the “wheel up” scrollassociated with reexamination of previously seen informa-tion. While our study examined emotions expressed within30 seconds of the click, future studies should include longerintervals around the search moves to check when initialnegative reactions change to more positive states and toexplore larger emotional patterns that might be associatedwith the different search phases (e.g., initial query typing,query reformulation, stopping, and other behaviors).

Our investigation of the secondary emotions revealed thatthe affective experiences that participants were able todescribe and recall were associated with the assessments ofquality of search performance, the reviewed content, orsearch outcomes. We did not find any overlap betweenprimary emotions and secondary emotions; for example,while the most frequently expressed primary emotion wassurprise, none of the participants reported being surprisedduring the search. For most participants, searching was aroutine and an emotionally uneventful experience. While theself-reported measures of emotions are the most commonlyused methods in our field, they might be limited to partici-pants’ abilities to recall and verbalize their experiences andshould be used in conjunction with other methods. Inanother interesting finding, when participants were askedabout emotions, they mentioned physiological states that areusually not associated with an affect (e.g., hunger and tired-ness). We did not find any studies of online information-seeking behavior that have mentioned the effects of thesevariables on search performance. It might be interesting andimportant from a methodological viewpoint to examineeffects of searchers’ physiological and psychological stateson their information-seeking behavior.

Our analysis of the effects of mood on the search processand effects of search variables on mood produced severalresults: (a) Neither the search tasks nor other search perfor-mance variables seemed to influence searchers’ mood; thisfinding indicates that it would be difficult to manipulatesearchers’ mood if that was one of the objectives of infor-mation retrieval systems. (b) Positive mood was found to beassociated with fewer search activities (e.g., fewer visitedwebsites and reformulated queries) while negative moodwas associated with increased search activities. (c) Neithermood nor search performance seemed to influence thequality of search results. If future studies confirm thatneither mood nor search-activity levels affects the quality ofthe outcomes, initiating the search in a positive mood mightbenefit the searchers who will invest less effort into a search,get adequate results, and maintain their positive mood.Perhaps, information retrieval systems can improve search-ers’ moods before they start interacting with the system.Some websites already attempt to affectively engage search-ers by using creative graphics (e.g., the changing designs ofthe Google logo on the main search engine’s page known as


“Google Doodles”), designing visually pleasing and notoverly complex sites, or introducing virtual assistants(MacDonald, Park, & Chae, 2012).

The study makes a methodological contribution to the fieldby applying methods and instruments not previously reportedin the LIS literature. We operationalized definitions of twotypes of emotions and mood, and developed a method formeasuring these variables during the search experience. Ourstudy introduced and relied on the universal facial-expressiontheory of emotions and the automated facial-recognition soft-ware. While the current software recognition accuracy rate isnot very high at approximately 60%, conceptually, the auto-mated recognition process is still the most efficient way torecognize emotions from facial expressions. Since most of thehardware used for information retrieval is already equippedwith front-facing cameras, (e.g. smart phones, laptops), rec-ognizing user emotions from facial expressions may prove tobe the most feasible way of gauging immediate emotionalresponses compared to the physiological emotive measureslike heart rate or electromyography (EMG). In the future, theemotion-reading software will most likely improve, and it willbe possible to incorporate it into the mainstream informationretrieval systems.

The study had a number of limitations related to design,sample size, data-analysis methods, and investigated vari-ables. The study was designed as an experiment because weneeded to control for the variables that would affect searchers’emotions and mood and because we needed to use recordingmethods that would be difficult to implement in a naturalisticsetting. The main negative side effect of the experimentalsetting was the fact that participants performed predeterminedsearch tasks and did not have a personal stake in the searchoutcomes; as a result, they could have been indifferent to thequality of the search outcomes. We tried mitigating this effectby designing tasks that would appear interesting and relevantto the studied sample (e.g., music piracy and college enroll-ment), but future research should consider studying searchersin their natural environments, where they look for the infor-mation that solves their real-life problems. While most par-ticipants indicated that they were not aware of the labequipment and that searching in the lab felt similar to search-ing in a public space (e.g., library), the lab setting might haveaffected searchers’ performance and emotional states.

Future studies also should consider integrating additionalmeasures of the search outcomes’ quality. We relied onindependent raters to evaluate completeness, correctness,and presentation as the quality measures of participants’written answers to the search-task questions. An additionalquality measure derived from searchers’ judgments ofsearch results will help to clarify the effects of affectivevariables on perceived and observed search outcomes.

The study used a convenience sample of undergraduatestudents, most of whom very frequently used search engines.The use of such a sample restricts generalizability of the studyfindings. However, since emotion research has supported thenotion of universality of emotional expressions, this suggeststhat other population samples with similar searching skills

would have exhibited comparable emotion patterns under thesame search conditions. Future work is needed to examineemotions and moods of searchers who represent variousdemographics and searching skills.

Another limitation is the relatively small sample size thatcreated issues with some of the statistical analyses used inthe study. While the sample size was adequate for examiningemotion patterns, it was relatively small for some of themultivariate statistical methods used to examine the role ofmood in the search process. Use of a larger sample sizewould be beneficial in future work.

Due to the novelty of emotion research in an informationretrieval context, the study was to a large extent exploratory.In cases when we did not have theories or prior findings forgrounding our decisions, we made judgments based on thepreliminary data analysis. For example, a priori, we did notknow what intervals before and after search actions weneeded to analyze for the presence of emotions. We had todetermine the intervals a posteriori by examining the dataand detecting variations within them. We also did notexamine all possible search actions, and instead chose toexamine simple actions directed toward the system (e.g.,clicks). In the future, we plan to include other search actionssuch as a query typing event and examine emotional patternsduring and around these episodes.

Our investigation of the primary emotions was based onthe assumption that facial expressions represent emotions.While this assumption is based on a well-known andaccepted theory, there have been numerous disagreements inthe psychology literature about the nature of emotions andtheir expressions. Lack of agreement about the nature andexpression of emotion makes it harder to generalize ourfindings. To create a more comprehensive representation ofemotions experienced during the search, future studiesshould consider adding physiological measures of emotionand affect (e.g., monitoring blood pressure) and real-time,self-reporting measures such as think-aloud protocols.

The study explored several affective variables with the aimof developing a comprehensive model of emotions and moodin the context of online searching. We feel that our inquiryinto emotions provided some valuable insights, but alsoinspired many additional questions. Now that we havelearned more the primary, secondary emotions and mood inonline searching, it is important to understand how to use thisknowledge for designing better search experiences. Forexample, the ability to recognize and appropriately respondto primary emotions is critical for building the next genera-tion of retrieval systems (the challenges of this work areevident in design of humanoid robots; see Wright, 2012.)Secondary emotions might be critical for developing percep-tions and memories about search interactions. More work isneeded to determine particular emotions and search episodesthat influence searchers’ perceptions, and examine how theseperceptions and emotional memories affect searchers’ sub-sequent choices of search systems and strategies. Forexample, Lindgaard, Fernandes, Dudek, and Brown (2006)suggested that impressions about a website are formed in the


first 50 seconds of the interaction; perhaps emotions experi-enced at the beginning and end of the search are the mostinfluential in developing perceptions of the search experi-ence. Mood might affect the level of search activity and effort,but might not be correlated with the quality of search results,which leads to the following question: If we can manipulatesearchers’ affective states, should we do it with the aim ofimproving search experiences or improving the quality ofsearch outcomes? Those two objectives might not alwaysalign since pleasant interactions might lead to overconfi-dence, lack of thoroughness, and, possibly, diminishedquality of outcomes. Lastly, the proposed framework ofsearch emotions incorporated with the real-time emotionalfeedback collected on increasingly ubiquitous front-facingcameras may open a new dimension in designing more satis-fying information retrieval systems. Such an approach wouldrepresent both an opportunity for information retrievaladvancements, as well as new ethical challenges.

Acknowledgment

I would like to thank professors Nicholas J. Belkin, TefkoSaracevic, Daniel O. O’Connor, and Gretchen B. Chapmanfor guiding me through this research.

References

Arapakis, I., Jose, J.M., & Gray, P.D. (2008). Affective feedback: Aninvestigation into the role of emotions in the information seekingprocess. Proceedings of the 31st annual International ACM SIGIR(SIGIR ’08) Conference on Research and Development in InformationRetrieval (pp. 395–402), New York, NY.

Barrett, L.F., & Russell, J.A. (1999). Structure of current affect. CurrentDirections in Psychological Science, 8, 10–14.

Bell, D.J., & Ruthven, I. (2004). Searchers’ assessments of task complexityfor web searching. In S. McDonald & J. Tait (Eds.), Lecture Notes inComputer Science, Vol. 2997. Advances in Information Retrieval (pp.57–71). Sunderland, United Kingdom: Springer.

Bilal, D. (2000). Children’s use of the Yahooligans! web search engine: I.Cognitive, physical, and affective behavior on fact-based search tasks.Journal of theAmerican Society for Information Science, 51(7), 646–665.

Bilal, D., & Bachir, I. (2007). Children’s interaction with cross-cultural andmultilingual digital libraries. II. Information seeking, success, and affec-tive experience. Information Processing & Management, 43(1), 65–80.

Bilal, D., & Kirby, J. (2002). Differences and similarities in informationseeking: Children and adults as web users. Information Processing &Management, 38(5), 649–670.

Bower, G.H. (1992). How might emotions affect learning? In S.A.Christianson (Ed.), The handbook of emotion and memory research andtheory (pp. 3–31). Hillsdale, NJ: Erlbaum.

Brave, S., Hutchinson, K., & Nass, C. (2005). Computers that care: Inves-tigating the effects of orientation of emotion exhibited by an embodiedcomputer agent. International Journal of Human–Computer Studies,62(2), 161–178.

Chen, H.-M., & Cooper, M.D. (2001). Using clustering techniques todetect usage patterns in a web-based information system. Journal of theAmerican Society for Information Science and Technology, 52(11), 888–904.

Clark, L.A., & Watson, D. (1988). Mood and the mundane: Relationsbetween daily life events and self-reported mood. Journal of Personalityand Social Psychology, 54(2), 296–308.

Cohn, J.F., & Kanade, T. (2007). Automated facial image analysis formeasurement of emotion expression. In J.A. Coan & J.B. Allen (Eds.),

The handbook of emotion elicitation and assessment. Oxford UniversityPress Series in Affective Science (pp. 222–238). New York, NY: OxfordUniversity Press.

Damasio, A. (2005). Descartes’ error: Emotion, reason, and the humanbrain. New York, NY: Penguin Books.

Darwin, C. (2005). The expression of the emotions in man and animals.Whitefish, MT: Kessinger. (Original work published 1872)

Dervin, B., & Reinhard, C.D. (2007). How emotional dimensions of situ-ated information seeking relate to user evaluations of help from sources:An exemplar study informed by sense-making methodology. In D. Nahl& D. Bilal (Eds.), Information and emotion: The emergent paradigm ininformation behavior research and theory (pp. 51–84). Medford, NJ:Information Today.

Du, S., & Martinez, A.M. (2011). The resolution of facial expressions ofemotion. Journal of Vision, 11(13), 1–24.

Dumais, S.T., Buscher, G., & Cutrell, E. (2010). Individual differences ingaze patterns for web search. Proceedings of the Third Symposium onInformation Interaction in Context (IIiX ’10) (pp. 185–194), New Bruns-wick, NJ. New York, NY: ACM,

Ekman, P. (1984). Expression and the nature of emotion. In K.R. Scherer &P. Ekman, (Eds.), Approaches to emotion (pp. 319–345). Hillsdale, NJ:Erlbaum.

Ekman, P. (1992). An argument for basic emotions. Cognition andEmotion, 6(3/4), 169–200.

Ekman, P. (2003). Emotions revealed: Recognizing faces and feelings toimprove communication and emotional life. NewYork, NY: Times Books.

Ekman, P., & Friesen, W.V. (2003). Unmasking the face. Cambridge, MA:Malor Books.

Flavián-Blanco, C., Gurrea-Sarasa, R., & Orús-Sanclemente, C. (2011).Analyzing the emotional outcomes of the online search behavior withsearch engines. Computers in Human Behavior, 27(1), 540–551.

Fulton, C. (2009). The pleasure principle: The power of positive affect ininformation seeking. Aslib Proceedings: New Information Perspectives,61(3), 245–261.

Gunes, H., & Piccardi, M. (2007). Bi-model emotion recognition fromexpressive face and body gestures. Journal of Network and ComputerApplications, 30(4), 1334–1345.

Gwizdka, J. (2008). Cognitive load on web search tasks. Proceedings of theWorkshop on Cognition and the Web (pp. 83–86), Granada, Spain.

Gwizdka, J. & Lopatovska, I. (2009). The role of subjective factors in theinformation search process. Journal of the American Society for Infor-mation Science and Technology, 60(12), 2452–2464.

Henson, R.K. (2006). Effect size measures and meta-analytic thinking incounseling psychology research. The Counseling Psychologist, 34(5),601–629.

Isen, A.M. (1993). Positive affect and decision making. In M. Lewis & J.Haviland (Eds.), Handbook of emotions (pp. 261–277). New York, NY:Guilford Press.

Jaimes, A., & Sebe, N. (2007). Multimodal human–computer interaction: Asurvey. Computer Vision and Image Understanding, 108(1–2), 116–134.

Jansen, B.J., & Spink, A. (2003). An analysis of web information seekingand use: Documents retrieved versus documents viewed. Proceedings ofthe Fourth International Conference on Internet Computing (pp. 65–69),Las Vegas, NV.

Jansen, B.J., Spink, A., & Saracevic, T. (2000). Real life, real users and realneeds: A study and analysis of users’ queries on the web. InformationProcessing & Management, 36(2), 207–227.

Joachims, T., Granka, L., Gay, G., Hembrooke, H., & Pan, B. (2005).Accurately interpreting click-through data as implicit feedback. Proceed-ings of the Conference on Research and Development in InformationRetrieval (SIGIR’05) (pp. 154–161), Salvador, Brazil.

Kalbach, J. (2006). “I’m feeling lucky”: Tthe role of emotions in seekinginformation on the web. Journal of the American Society for InformationScience, 57(6), 813–818.

Klein, J., Moon, Y., & Picard, R.W. (2002). This computer responds to userfrustration: Theory, design, and results. Interacting with Computers,14(2), 119–140.


Kracker, J. (2002). Research anxiety and students’ perceptions of research:An experiment. Part I. Effect of teaching Kuhlthau’s ISP model. Journalof the American Society for Information Science, 53(4), 282–294.

Kracker, J., & Wang, P. (2002). Research anxiety and students’ perceptionsof research: An experiment. Part II. Content analysis of their writings ontwo experiences. Journal of the American Society for InformationScience, 53(4), 295–307.

Kuhlthau, C.C. (1991). Inside the search process: Information seeking fromthe user’s perspective. Journal of the American Society for InformationScience, 42(5), 361–371.

Kuhlthau, C.C. (1993). A principle of uncertainty for information seeking.Journal of Documentation, 49(4), 339–355.

Lazar, J., Jones, A., Hackley, M., & Shneiderman, B. (2006). Severity andimpact of computer user frustration: A comparison of student and work-place users. Interacting with Computers, 18(2), 187–207.

Lazarus, R.S. (1984). Thoughts on the relations between emotion andcognition. In K.R. Scherer & P. Ekman, (Eds.), Approaches to emotion(pp. 247–259). Hillsdale, NJ: Erlbaum.

Levenson, R.W. (1994). Human emotion: A functional view. In P. Ekman &R.J. Davidson (Eds.), The nature of emotion: Fundamental questions (pp.123–126). New York, NY: Oxford University Press.

Lindgaard, G., Fernandes, G., Dudek, C., & Brown, J. (2006). Attentionweb designers:You have 50 milliseconds to make a good first impression.Behaviour & Information Technology, 25(2), 115–126.

Lopatovska, I. (2009). Searching for good mood: examining relationshipsbetween search task and mood. Proceedings of the 72th Annual Meetingof the American Society for Information Science and Technology.November 6–11, Vancouver, British Columbia, Canada.

Lopatovska, I. (2011). Emotional correlates of information retrieval behav-iors. Proceedings of the SSCI 2011 WACI – 2011 Workshop on AffectiveComputational Intelligence, Paris, France, Apr. 11–15, 2011.

Lopatovska, I., & Arapakis, I. (2011). Theories, methods, and currentresearch on emotions in library information science, informationretrieval and human–computer interaction. Information Processing &Management, 47(4), 575–592. doi:10.1016/j.ipm.2010.09.001

Lopatovska, I., & Cool, C. (2008). Online search: Uncovering affectivecharacteristics of information retrieval experience. Poster presented atthe Association for Library and Information Science Education (ALISE)Annual Conference. Philadelphia, PA, January 8–11.

Lopatovska, I., & Mokros, H. (2007). Willingness to pay and experiencedutility as measures of affective value in the information retrievalobjects: Users’ accounts. Information Processing & Management, 44(1),92–104.

MacDonald, C.M., Park, S.J., & Chae, G. (2012). AMARA: An affectiveagent to enhance users’ enjoyment and engagement with online art col-lections. Proceedings of the American Society for Information Scienceand Technology, 49, 1–4. doi:10.1002/meet.14504901303

Mano, H. (1994). Risk-taking, framing effects, and affect. OrganizationalBehavior and Human Decision Processes, 57(1), 38–58.

Mooney, C., Scully, M., Jones, G.J., & Smeaton, A.F. (2006). Investigatingbiometric response for information retrieval applications. In Proceedingsof the 28th European Conference on Information Retrieval (ECIR 2006)(pp. 570–574). London, U.K., April 2006.

Morris, W.N. (1999). Mood system. In D. Kahneman, E. Diener, & N.Schwarz (Eds.), Well-being: The foundations of hedonic psychology (pp.169–189). New York, NY: Russell Sage Foundation.

Murray, I., & Arnott, J. (1993). Toward the simulation of emotion insynthetic speech: Review of the literature on human vocal emotion.Acoustical Society of America, 93(2), 1097–1108.

Nahl, D. (1998). Learning the internet and the structure of informationbehavior. Journal of the American Society for Information Science,49(11), 1017–1023.

Nahl, D. (2004). Measuring the affective information environmentof web searchers. Proceedings of the 67th annual meeting of theAmerican Society for Information Science & Technology, (41)1, 191–197.

Nahl, D., & Meer, M.P. (1997). User-centered assessment of two webbrowsers: Errors, perceived self-efficacy, and success. Proceedings of the

annual meeting of the American Society for Information Science (pp.89–97).

Nahl, D., & Tenopir, C. (1996). Affective and cognitive searching behaviorof novice end-users of a full-text database. Journal of the AmericanSociety for Information Science, 47(4), 276–286.

O’Brian, H., & Toms, E. (2008). What is user engagement? A conceptualframework for defining user engagement with technology. Journal for theAmerican Society for Information Science and Technology, 59(6), 938–955.

Onwuegbuzie, A.J., & Jiao, Q.G. (2004). Information search performanceand research achievement: An empirical test of the anxiety-expectationmediation model of library anxiety. Journal of the American Society forInformation Science, 55(1), 41–54.

Payne, R., & Cooper, C. (Eds.) (2001). Emotions at work: Theory, researchand applications for management. New York, NY: Wiley.

Peter, C., & Herbon, A. (2006). Emotion representation and physiologyassignments in digital systems. Interacting with Computers, 18(2), 139–170.

Picard, R.W., Vyzas, E., & Healey, J. (2001). Toward machine emotionalintelligence: Analysis of affective physiological state. IEEE Transac-tions on Pattern Analysis snd Machine Intelligence, 23(10), 1175–1191.

Plutchik, R. (1980). A general psychoevolutionary theory of emotion. In R.Plutchik & H. Kellerman (Eds.), Emotion: Theory, research, and expe-rience: Vol. 1. Theories of emotion (pp. 3–33). NewYork, NY: AcademicPress.

Reisenzein, R., Bördgen, S., Holtbernd, T., & Matz, D. (2006). Evidencefor strong dissociation between emotion and facial displays: The case ofsurprise. Journal of Personality and Social Psychology, 91(2), 295–315.Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/16881766

Sherry, A., & Henson, R.K. (2005). Conducting and interpreting canonicalcorrelation analysis in personality research: A user-friendly primer.Journal of Personality Assessment, 84(1), 37–48.

Spielberger, C.D., Gorssuch, R.L., Lushene, P.R., Vagg, P.R., & Jacobs,G.A. (1983). Manual for the State-Trait Anxiety Inventory. ConsultingPsychologists Press, Inc.

Spink, A. (1997). Study of interactive feedback during mediated informa-tion retrieval. Journal of the American Society for Information Science,48(5), 382–394.

Stenmark, D. (2008). Identifying clusters of user behaviour in intranetsearch engine log files. Journal of the American Society for InformationScience and Technology, 59(14), 2232–2243.

Tenopir, C., Wang, P., Zhang, Y., Simmons, B., & Pollard, R. (2008).Academic users’ interactions with Science Direct in search tasks: Affec-tive and cognitive behaviors. Journal of the American Society for Infor-mation Science, 44(1), 105–121.

Thayer, R.E. (1996). The origin of everyday moods: Managing energytension, and stress. Oxford, England: Oxford University Press.

Tomkins, S.S. (1984). Affect theory. In K.R. Scherer & P. Ekman (Eds.),Approaches to emotion (pp. 163–197). Hillsdale, NJ: Erlbaum.

Toms, E., & McCay-Peet, L. (2009). Chance encounters in the digitallibrary. In M. Agosti, J. Borbinha, S. Kapidakis, C. Papatheodorou, & G.Tsakonas (Eds.), Proceedings of the 13th European Conference onResearch and Advanced Technology for Digital Libraries (ECDL’09)(pp. 192–202). Berlin, Germany: Springer-Verlag.

Wang, P., Hawk, W.B., & Tenopir, C. (2000). Users’ interaction with WorldWide Web resources: An exploratory study using a holistic approach.Information Processing & Management, 36, 229–251.

Wang, P., & Soergel, D. (1998). A cognitive model of document use duringa research project: Study I. Document selection. Journal of the AmericanSociety for Information Science, 49(2), 115–133.

Watson, D., Clark, L.A., & Tellegen, A. (1988). Development and valida-tion of brief measures of positive and negative affect: The PANAS scale.Journal of Personality and Social Psychology, 54(6), 1063–1070.

White, R.W. (2004). Implicit feedback for interactive information retrieval.Unpublished doctoral dissertation, University of Glasgow, Glasgow,United Kingdom.

White, R.W., Ruthven, I., & Jose, J.M. (2005). A study of factors affectingthe utility of implicit relevance feedback. Proceedings of the 28th annual


http://www.ncbi.nlm.nih.gov/pubmed/16881766

International ACM SIGIR Conference on Research and Development inInformation Retrieval (pp. 35–41), Salvador, Brazil.

Wilhelm, F.H., Pfaltz, M.C., & Grossman, P. (2006). Continuous electronicdata capture of physiology, behavior and experience in real life: Towards

ecological momentary assessment of emotion. Interacting with Com-puters, 18(2), 171–186.

Wright, A. (2012). Robots like us. Communications of the ACM, 55(5),12–13. doi:10.1145/2160718.2160724

Appendix

Examples of the eMotion data preparation for the analysis

Step 1. A file classifying facial expressions on each video frame was created by eMotion software.

FrameElapsed time

(in ms)Probabilityof neutral

Probabilityof happiness

Probabilityof surprised

Probabilityof anger

Probabilityof disgust

Probabilityof fear

Probabilityof sadness

0 0 0.999 0 0 0 0 0 01 235 0.999 0 0 0 0 0 02 1,000 0.991 0 0 0.007 0 0 0.0023 1,203 0.998 0 0 0.001 0 0 04 1,406 0.998 0 0 0.001 0 0 05 1,719 0.996 0 0 0.002 0 0 0.0026 1,922 0.972 0 0 0.011 0 0 0.0177 2,125 0.97 0 0 0.013 0 0 0.0168 2,328 0.012 0 0.007 0.028 0 0.081 0.8729 2,547 0.145 0.001 0.002 0.087 0.001 0.012 0.751

10 2,766 0 0.002 0.007 0.048 0 0.15 0.79311 2,969 0 0.01 0.011 0.067 0 0.156 0.75512 3,172 0 0.099 0.013 0.023 0 0.723 0.14213 3,391 0 0.01 0.016 0.038 0 0.406 0.5314 3,610 0 0.028 0.001 0 0 0.971 015 3,813 0 0.067 0.001 0.001 0 0.931 016 4,031 0 0.001 0 0 0 0.999 017 4,235 0 0.001 0.006 0 0 0.991 0.00118 4,438 0 0 0.001 0 0 0.998 019 4,703 0 0.001 0 0.001 0 0.996 0.00220 4,906 0 0.001 0.001 0.008 0 0.982 0.009

Step 2. Presence of the dominant emotion (p > .9) was identified for each video frame.

FrameElapsed time

(in ms)Presenceof neutral

Presenceof happiness

Presence ofsurprise

Presenceof anger

Presenceof disgust

Presenceof fear

Presenceof sadness

0 0 1 0 0 0 0 0 01 235 1 0 0 0 0 0 02 1,000 1 0 0 0 0 0 03 1,203 1 0 0 0 0 0 04 1,406 1 0 0 0 0 0 05 1,719 1 0 0 0 0 0 06 1,922 1 0 0 0 0 0 07 2,125 1 0 0 0 0 0 08 2,328 1 0 0 0 0 0 09 2,547 0 0 0 0 0 0 1

10 2,766 0 0 0 0 0 0 011 2,969 0 0 0 0 0 0 012 3,172 0 0 0 0 0 0 013 3,391 0 0 0 0 0 0 014 3,610 0 0 0 0 0 0 015 3,813 0 0 0 0 0 1 016 4,031 0 0 0 0 0 1 017 4,235 0 0 0 0 0 1 018 4,438 0 0 0 0 0 1 019 4,703 0 0 0 0 0 1 020 4,906 0 0 0 0 0 1 0

Step 3. The times associated with emotion expression allowed us to connect eMotion data to the log file data to determine occurrence of primaryemotion in relation to the search process. In the analysis of the frequencies of all emotional states per individual participant or a group, theoccurrences of emotions were summed and divided by the number of frames. In the reviewed example, neutral expression was noted nine times orwas expressed 45% (9 of 20) of the analyzed time.
