hci 2014 (9 of 10): affective factors

Master on Software Engineering :: Human-Computer Interaction

Dr. Sabin-Corneliu Buraga – www.purl.org/net/busaco

affective factors

Human-Computer Interaction

http

://lts5ww

w.ep

fl.ch/



“Designers think, so people can feel.”

Juan-Carlos Fernandez



Emotion

Affective aspects of interaction (Rogers, 2007)

expressivity

how the interface look & feel could affect the user?



Emotion


frustration

how we can detect and reduce it?



Emotion


persuasive technologies

how we can change the user attitude & behaviorby using the actual technologies?



Emotion

Emotion is a reaction to events deemed relevantto the needs, goals, or concerns of an individual

Scott Brave & Clifford Nass, 2008



Emotion

Emotion represents the experience in UX



Emotion

Emotion encompasses physiological, affective, behavioral, and cognitive components




Emotion

emotion in the cognitive context – Moore (2005)



Emotion: classification

Emotions as discrete categories

basic emotions (Paul Ekman et al., 1972):anger

disgustfear

happinesssadnesssurprise




Emotions as discrete categories

additional emotions (Paul Ekman, 1990):amusement contempt contentment

embarrassment excitement guiltpride in achievement relief satisfaction

sensory pleasure shame




Emotions as multi-dimensional models

circumplex model (James Russell, 1980)

2 axes: arousal (calm↔excitement) + valence (pleasantness)





circumplex model (James Russell, 1980)

used to test stimuli of emotion words, emotional facial expressions, and affective states





Plutchik’s wheel of emotions (Robert Plutchik, 1980)

8 primary bipolar emotions: joy vs. sadness; anger vs. fear;

trust vs. disgust; surprise vs. anticipation+

more advanced emotions




Negative & forcefulAnger, Annoyance, Contempt,Disgust, Irritation

Negative & not in controlAnxiety, Embarrassment,Fear, Helplessness,Powerlessness, Worry

Negative thoughtsDoubt, Envy, Frustration,Guilt, Shame

Negative & passiveBoredom, Despair,Disappointment, Hurt, Sadness

AgitationShock, Stress, Tension

Positive & livelyAmusement, Delight, Elation,Excitement, Happiness,Joy, Pleasure

CaringAffection, Empathy,Friendliness, Love

Positive thoughtsCourage, Hope, Pride,Satisfaction, Trust

Quiet positiveCalm, Content, Relaxed,Relieved, Serene

ReactiveInterest, Politeness, Surprise

Douglas-Cowie et al. (2015)





positive activation – negative activation (PANA)Watson and Tellegan, 1985

PAD (Pleasure, Arousal & Dominance) emotional state model – Mehrabian, 1997

used to study nonverbal communication




Lövheim cube of emotion (Hugo Lövheim, 2012)a theoretical model considering the relationships

between monoamine neurotransmitters (serotonin, dopamine, noradrenaline) and emotions





Parrotts’ classification (Gerrod Parrott, 2001)primary emotions: love, joy, surprise, anger, sadness, fear






secondary emotions – e.g., love: affection, lust, longing






secondary emotions – e.g., love: affection, lust, longing

tertiary emotions – e.g., affection: adoration, fondness, liking, attraction, caring, tenderness, compassion,…




discussion

http://uniquelang.peiyinglin.net/visualization/Parrott_Model_b.png



Emotion

Modeling languages & vocabularies

Emotion Annotation and Representation LanguageHUMAINE Project, 2006

http://emotion-research.net/projects/humaine/earl/



Emotion

Modeling languages & vocabularies

Emotion Markup Language (EmotionML) 1.0 W3C Proposed Recommendation, 2013

http://www.w3.org/TR/emotionml/

Vocabularies for EmotionMLW3C Working Group Note, 2014http://www.w3.org/TR/emotion-voc/



Emotion

neurological structure of emotion (LeDoux, 1996)



Emotion

limbic system – the “seat of emotion” – evaluatesthe need/goal relevance of its inputs



Emotion

thalamus + limbic system mainly deal withprimitive (basic) emotions

e.g., startle-based fear, anger, sadness, joy, disgust



Emotion

Emotions that require more extensive cognitive(i.e., knowledge-based) processing are called secondary



Emotion

Emotions that require more extensive cognitive(i.e., knowledge-based) processing are called secondary

examples:frustration

pridesatisfaction

in the cortex



Emotion

An emotion can result from a combination of boththalamic-limbic and cortex-limbic mechanisms



Emotion

An emotion can result from a combination of boththalamic-limbic and cortex-limbic mechanisms

an event causing an initial fear reactioncan be later recognized as harmless

by more extensive, rational evaluation

example: screensaver initialization



Emotion

Emotions could be learned from the social environment



Emotion

emotion versus mood

emotions are intentional



Emotion

emotion versus mood

emotions are intentional

“they imply and involve relationshipswith a particular object”

Frijda, 1994



Emotion

emotion versus mood

moods are not directed at any object in particular andare experienced as more diffuse, global, and general



Emotion

emotion versus mood

moods are not directed at any object in particular andare experienced as more diffuse, global, and general

a person can be sad about something (an emotion)or generally depressed (a mood)



Emotion

emotion versus mood

from a functional point of view,emotions could determine the execution of task(s)



Emotion

emotion versus mood

emotions cause or contribute to moods



Emotion

emotion versus sentiment

sentiments are not states of an individual,but assigned properties of an entity (person, object)



Emotion


sentiments can persist indefinitely, by contrast withemotions (seconds) or moods (hours/days)



Emotion


sentiments are often generalizations about a class ofobjects with a given recognizable property



Emotion


sentiments are often generalizations about a class ofobjects with a given recognizable property

these generalizations must not necessarily be logical“I hate spreadsheet applications!”



How about the role of emotion in HCI?



Emotion

Emotion has influences on

attentionmemory

performanceopinion (assessment)




Emotion

Attention

an interface capable of detecting – or at least predicting –a user’s emotional or mood state could similarly assumean affect-regulation role, helping to guide attention away

from negative and toward more positive stimuli



Emotion

Memory

negative events, which tend to be highly arousing,are typically remembered better than positive events



Emotion

Memory

negative events, which tend to be highly arousing,are typically remembered better than positive events

avoid error messages



Emotion

Memory

emotionality “improves memory for central details,while undermining memory for background details”

Heuer & Reisberg, 1992; Parrott & Spackman, 2000



Emotion

User performance

keeping a user happy may, therefore,not only affect satisfaction,

but may also lead to efficiency and creativity



Emotion

the circumplex model in the context of user experience

P. Desmet & P. Hekkert, “Framework of Product Experience”, International Journal of Design, Vol. 1, No. 1 (2007)

http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/66/15



Emotion

User performance

negative example: objects that appear or move unexpectedly

– e.g., pop-ups, animations – and loud or sharp noises are likely to trigger startle-based fear



Emotion

Assessment

mood influences judgment and decision making

users in a good mood will likely judgeboth the interface and their work more positively,

regardless of any direct emotional effects



Emotion

Causes:needs & goals

Maslow’s hierarchy of basic needs (1968)

physiological, safety/security, social, esteem,cognitive, aesthetic, self-actualization, transcendence



Emotion


interfaces can also directly address a user’s basic needs



Emotion



example (Fogg, 1998):establishing a trusting and safe relationship with users



Emotion



example (Brave & Nass, 2006):educational software should address users’ emotional

needs, not only teaching the relevant content,but also ensuring users believe that they are learning



Emotion



Emotion

Causes:obstacles & control

Ellsworth (1994)

if user has the control over the software,the negative affective consequences are least important



Emotion

Causes:social context – “agency”

(Ellsworth, 1994) & (Friedman & Kahn, 1997)

“This system failed to understand your command.” – blaming itself“The command was not understood.” – blaming no one“You did not speak clearly enough for your command

to be understood.” – blaming the user



Emotion

blaming itself



Emotion

Causes:previous emotional state

repeated pleasurable affective states, therefore,become expected and thus gradually lose intensity



Emotion

How emotion could be detected?

neurological response – via electroencephalogram (EEG) or magneto-resonance imaging (MRI)



Emotion


autonomous activity of the human organism

breathfacial expression

voice



Emotion

categorization of facial responses to determine which emotions they express

Facial Action Coding System – FACS(Paul Ekman et al., 1978, 2002)



Emotion

interpreting micro-expressions (e.g., pushed up cheeks, dropping upper eyelids, eyebrows raised, nose wrinkling,…) – via FACS –

and body language to detect emotions

see also http://tinyurl.com/k47dhen

http://temasys.com.sg/vidyoedm22102013/



Emotion


behavioral changes



6 + 1 emotions with universal facial gestures(N. Lazzaro, Why We Play: Affect and the Fun of Games, 2008)

Emotion Example

Frustration Figuring out how to get character off a roof in Tom Clancy’s Splinter Cell (usability issues that detract from the player experience)

Fear Falling into boiling lava, fast-moving projectiles aimed at the player in Doom

Surprise Using Myst’s linking books for the first time to transport toa new world

Sadness When the young magician Aerith, in Final Fantasy VII is murdered

Amusement When two Sims get married in The Sims, or rollingover and picking up sumo wrestlers in Katamari Damacy

Disgust Becoming a social outcast (social disgust)after losing the dancing challenge in Sid Meier’s Pirates

Curiosity Wanting to know what happens by driving the race trackthe wrong way in Project Gotham Racing 3



Emotion

Expressivity

induces an emotional state via UI elements:color, icons, sound, graphics,…



Emotion

Expressivity

lack of expressivity in communication at a textual levelemoticons

:) :| >:-(



Emotion

Expressivity

abbreviations could have emotional connotations

see leet speak

I 12 CU 2NITE



Emotion

To stimulate emotions, products must haveattributed a personality based on appearance

and interaction means

Reeves & Nass, 1989



How to create emotions in a person?



Emotion

1. Auto appraisal2. Reflective appraisal3. Memory of an emotion4. Imagination5. Talking about6. Empathy or witnessing another’s emotion7. Instruction by others on how to feel8. Violation of social norm9. Making facial expression of an emotion

Ekman, 2004



Emotion


a dialog window appears



Emotion


thinking about previous message



Emotion


rememberinglast game



Emotion


thinking about what happensnow on Web



Emotion


discussing UI features with others



Emotion


another person’s facial expression

(video-chat)



Emotion


another person’s review of a task



Emotion


deleting others’ files



Emotion


laughing after defeatin front of friends makes

it feel more positive



Emotion: design solutions

metaphors+

idioms




metaphors + idioms

“Metaphors create insight. But they also distort.They have strengths. But they also have limitations.”

Gareth Morgan




metaphors + idioms

use metaphors to convey and/or createmental associations

Dan Saffer, 2005




metaphors + idioms

proper metaphors used to denote UI objects/devices contribute to the personification of the software,

increasing the usability




icons

denote a relation between a visual representationand a certain concept




icons

denote a relation between a visual representationand a certain concept

based on similarity – e.g., an empty page = new fileanalogue – example: scissors = cut operation

arbitrary (based on an idiom) – e.g., X = close window




icons

most effective are the icons designed with respect to similarity between visual representation and the concept

Rogers, 2007




arbitrary applications

system programs

remark: the orientation is differentsee also GNOME Human Interface Guidelines




icons

beware of the cultural context




earcons

earcon = the aural equivalent of an icon




earcons

earcon = the aural equivalent of an icon

using sounds to denote a specific concept, event, or object

M. Blattner et al, “Earcons and Icons”, Human-Computer Interaction, Vol. 4 (1989)

http://www.daimi.au.dk/~dsound/DigitalAudio.dir/Papers/Earcons_and_Icons.pdf




earcons

design guidelines (Karen Kaushansky, 2012):choose the right type of audio

embed meaning in audio earconsdesign in context

consider the “non-use cases”decide between recorded prompts and text-to-speech

www.smashingmagazine.com/2012/09/14/guidelines-for-designing-with-audio/




earcons

examples:mobile phones, operating system sound schemes,

instant messaging, electronic games, 3D environments,…

commonly, not used in the Web context




tactons

communicate non-visual informationby using the tactile sense

“structured tactile messages” (Brewster & Brown, 2004)http://eprints.gla.ac.uk/3443/1/tactons_aussi.pdf




tactons

communicate non-visual informationby using the tactile sense

a common example: coded tactile representations (Braille code)

text informationnon-visual form




tactons

context:wearable computing




tactons

examples:haptic I/O devices – e.g., sensors

case study:L. Brown, Tactons: Structured Vibrotactile Messages for Non-Visual

Information Display, Microsoft Research (2006)http://research.microsoft.com/apps/video/default.aspx?id=135935



Emotion: design solutionsabstract

representationcognitive

dimensionusage

icon visual conveying information

earcon audio attention

tacton haptic appeal to a known touch

emoticon emotion conveying an emotional state

gesticon dynamic indicating a state change

kineticon dynamic state change, progress

some are more suitable to (visually) impaired users



persuasive computing




Positive psychology

hedonomics

focusing on the pleasant or enjoyable aspects of HCI

Hancock, Pepe & Murphy, 2005




Persuasive (seductive) technologies

interactive computational systems deliberately designed to change human attitudes and behaviors

B. J. Fogg (2003)





interactive computational systems deliberately designed to change human attitudes and behaviors

captology(computers as persuasive technologies)





persuasion in the context (Maria Miceli et al., 2011):accidental vs. intentional

communicative vs. non-communicativecoercive vs. non-coercive





typical examples:advertisement techniques,

warning boxes, reminders, prompts,personalized messages, recommendations,…





case studies:games with social impact – for people with special needs

applications helping the cure of certain phobiasinfant simulators

virtual pets – to diminish “bad habits” of children…



Emotion

Herman the bug (Lester et al., 1997)





interesting uses in the HCI context:get understanding of user behaviour

get understanding of cognitive factors





multimodal factors (Guerini, 2011):embodied conversational agents – ECAs

kinetic typographymusic

use of images





multimodal factors (Guerini, 2011):embodied conversational agents – ECAs

kinetic typographymusic

use of images

virtual characters




Virtual characters

avatars – artificial (visual) representations of people




Virtual characters

user-facing agents

showing simulated human behavior(knowledge, inferring, choices,…artificial intelligence)




Virtual characters

autonomy no human interventionreactivity takes stimuli from environmentproactivity takes initiative to actcollaboration able to collaborate with other agents




Virtual characters

autonomy no human interventionreactivity takes stimuli from environmentproactivity takes initiative to actcollaboration able to collaborate with other agents

multi-agent systems



Emotion: design solutionsintelligent virtual agents having human appearance and the capability to teach

foreign language vocabulary

M. Macedonia et al., “Intelligent virtual agents as language trainers facilitate multilingualism”, Frontiers in Psychology (2014)

http://journal.frontiersin.org/Journal/10.3389/fpsyg.2014.00295/full

case study




The use of anthropomorphism

adding human qualities to virtual things

reducing the user’s anxiety regarding computers(computer and/or Internet phobia)





example:“Hi, Daniel, welcome back! What we learnt last time?

Oh, yes, that experiment on Web navigation…Let’s repeat it!”

versus“User 33, start the experiment 7!”





warnings:user might feel inferior, anxious, or stupid




Affective interaction

goal: a program/device able to register emotions and other stimuli and to generate a correct response

to these stimuli





input:

sensors measuring (or learning) the user affective state

facial expression, gestures,heartbeat rate, blood pressure, transpiration, speech





output:

expressing empathy using computational approaches

“intelligently” / “emotionally” reacting on a situation



iCAT – a robot hardware platform using a setof logical rules to convey emotional states (Philips, 2005)

context: human-robot interaction (HRI)

www.research.philips.com/technologies/projects/robotics/

case study




Aspect of interest: computational humour

humour is a powerful generator of emotions

theories of humour applied in computer science

details in P. Petta et al. (Eds.), Emotion-Oriented Systems, Springer, 2011





humour generation

building computational models to generate humourous content (text)





humour recognition

identifying humourin natural language and/or human behaviour



Emotion: abuse on software

Humans could have an offensive behaviorregarding the digital entities

www.agentabuse.org





example (Tatsuya Nomura, 2005):

impolite behavior of the patients,when they discovered that the therapist is a virtual agent





example (Antonella De Angeli & Rollo Carpenter, 2005):

“injuries” on conversational – e.g., IRC – robots:“liar”, “stupid”, “dumb” software





experiment (Christoph Bartneck et al., 2005):

humans tend to hurt more easily robots/devicesthan people




User: What is your address?

Jabberwacky: I don't think I'll tell you.

You'd kill me if I told you. Wouldn't you!

User: Yes I would.

Jabberwacky: What is your favourite food?

User: I WILL KILL YOU! I WILL HUNT YOU DOWN!!!

Jabberwacky: Please don't... I really have to leave.

User: I WILL KILL YOU!

Jabberwacky: Good.



Emotion: open question

With which emotionshould HCI designers be most concerned?




When and how should interfaces attemptto directly address users’ emotions and

basic needs (versus application-specific goals)?




When and how should users be informedthat their affective states are being monitored

and adapted to?




How does emotion play outin computer-mediated communication?



affective factorsemotions, moods, sentiments

persuasive technologies

“Conclusion”



next lecture: ubiquitous computing

hci 2014 (9 of 10): affective factors

Design

emotion emotion

software engineering

sabincorneliu buraga

classification emotions

stimuli of emotion words

advanced emotions

primary bipolar emotions

affective states