beyond eye tracking: using user temperature, rating dials, and facial analysis to understand the...
DESCRIPTION
Dan Berlin, Jon Strohl, David Hawkins and I presented this at UXPA 2013. Eye tracking is well known and accepted in the UX community. Here we present preliminary evidence for the usefulness of adding electrodermal activity (EDA), continuous dial ratings, etc. to user experience research.TRANSCRIPT
![Page 1: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/1.jpg)
Beyond Eye Tracking Using user temperature, rating dials, and facial analysis to understand the user experience Jen Romano Bergstrom, Jon Strohl, David Hawkins Dan Berlin UXPA2013 | Washington, DC @romanocog @forsmarshgroup @banderlin
![Page 2: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/2.jpg)
2
Client’s needs • Traditionally…
– What works well – What needs help
![Page 3: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/3.jpg)
3
Client’s needs • Traditionally…
– What works well – What needs help
• Measure the UX
Observations
Selection/click behavior
Contextual observations
Time to complete task Reaction time
Accuracy Ability to complete tasks
![Page 4: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/4.jpg)
4
Task efficiency and accuracy Accuracy
Steps to Complete Task*
Time to Complete Task*
Users 10% 8 170 seconds
Admins 21% 8.3 32 seconds
All Participants
15% 8.2 101 seconds
![Page 5: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/5.jpg)
Session observations
5
• Observational click behavior • Facial expressions of frustration • Fidgeting and other observations of emotion
Areas of the website that participants explored first.
![Page 6: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/6.jpg)
6
Explicit Post-task satisfaction questionnaires
Moderator follow up
In-session difficulty ratings
Verbal responses
Real-time +/- dial
Measure the UX by asking questions
![Page 7: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/7.jpg)
Think aloud protocol
7
• Rooted in cognitive psychology and the study of thinking • Makes explicit what is implicitly present to participants • Concurrent vs. retrospective
“This is really confusing!”
![Page 8: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/8.jpg)
Satisfaction questionnaires & difficulty ratings
8
• Assess users subjective satisfaction • Consistent questionnaire used across interfaces or
customized for its features and capabilities • Structured vs. unstructured
Satisfaction Questionnaire Please circle the numbers that most appropriately reflect your impressions about using this Web-based instrument.
terrible wonderful
1. Overall reaction to the Web site: 1 2 3 4 5 6 7 8 9 not applicable
confusing clear 2. Screen layouts: 1 2 3 4 5 6 7 8 9 not applicable
inconsistent consistent 3. Use of terminology throughout the Web site: 1 2 3 4 5 6 7 8 9 not applicable
inadequate adequate 4. Information displayed on the screens: 1 2 3 4 5 6 7 8 9 not applicable
illogical logical 5. Arrangement of information on the screen: 1 2 3 4 5 6 7 8 9 not applicable
never always 6. Tasks can be performed in a straight-forward manner: 1 2 3 4 5 6 7 8 9 not applicable
confusing clear 7. Organization of information on the site: 1 2 3 4 5 6 7 8 9 not applicable
impossible easy 8. Forward navigation: 1 2 3 4 5 6 7 8 9 not applicable
impossible easy 9. Backward navigation: 1 2 3 4 5 6 7 8 9 not applicable
difficult easy 10. Overall experience of finding information: 1 2 3 4 5 6 7 8 9 not applicable
too frequent appropriate 11. Census Bureau-specific terminology: 1 2 3 4 5 6 7 8 9 not applicable
12. Overall reaction to the Web site:
Terrible Wonderful 1 2 3 4 5 6 7
Frustrating Satisfying
1 2 3 4 5 6 7 Difficult Easy
1 2 3 4 5 6 7 13. Additional Comments (use the back of this paper if necessary):
![Page 9: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/9.jpg)
9
Client’s needs • For this project…
– What grabs attention? – What is engaging? – What is a turn off? – What about the videos? – Good parts? Bad? – Is green better than…?
![Page 10: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/10.jpg)
A volunteer please
10
![Page 11: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/11.jpg)
11
Client’s needs • For this project…
– What grabs attention? – What is engaging? – What is a turn off? – What about the videos? – Good parts? Bad? – Is green better than…? Explicit
Post-task satisfaction questionnaires
Moderator follow up
In-session difficulty ratings
Verbal responses
Real-time +/- dial
Observations
Selection/click behavior
Contextual observations
Time to complete task Reaction time
Accuracy Ability to complete tasks
![Page 12: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/12.jpg)
Implicit measures
12
• Physiological responses are difficult to control • Implicit responses are unfiltered • Responses occur before explicit measures
Definition: Underlying reactions (e.g., eye tracking, arousal) that people are unaware of, cannot control, or cannot express at a granular level
Stimulus Implicit Responses
Thought Processes
Explicit Responses
![Page 13: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/13.jpg)
Why don’t we measure the implicit?
13
• Very difficult, if even possible, to communicate the subconscious.
• Responses occur in a very short time interval.
• A lot of noise in the signal
• Unfamiliar lexicon used in the literature.
• The technology is just beginning to become usable by a wider audience.
• Analyses appear overwhelmingly time consuming and complicated.
• It’s difficult to justify the ROI.
![Page 14: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/14.jpg)
Why should we measure the implicit?
14
• Evaluates thought processes and emotions (not what the participant tells you)
• Quantifiable data that goes beyond task performance • Moment by moment interaction • Cause and effect triggers • Deeper insights
![Page 15: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/15.jpg)
Why should we measure the implicit?
15
• Evaluates thought processes and emotions (not what the participant tells you)
• Quantifiable data that goes beyond task performance • Moment by moment interaction • Cause and effect triggers • Deeper insights
Traditional research is good at explaining what people say and do, not what they think and feel.
![Page 16: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/16.jpg)
16
Observations Selection/click behavior
Ethnography
Time to complete task Reaction time
Accuracy
Ability to complete tasks
The Complete UX
Explicit Post-task satisfaction questionnaires
Moderator follow up
In-session difficulty ratings
Verbal responses
Real-time +/- dial
Implicit
Eye tracking Electrodermal activity (EDA)
Behavioral analysis
Pupil dilation
Facial expression coding
Implicit associations
Linguistic analysis of verbalizations
Heart rate variability
![Page 17: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/17.jpg)
Two categories of implicit measures
17
Biometrics Neuroimaging
![Page 18: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/18.jpg)
Neuroimaging metrics
18
• Indirectly or directly measures activity in the brain.
• Typically measures the hemodynamic response or brain electrical activity.
• Examine what “people are thinking”
![Page 19: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/19.jpg)
Why don’t we collect neuroimaging measures?
19
• Lots of resources • Expensive equipment • Complex analyses • Strict protocols • Unnatural environment
![Page 20: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/20.jpg)
Two categories of implicit measures
20
Biometrics Neuroimaging
![Page 21: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/21.jpg)
Biometrics
21
• Established in UX research – Eye Tracking
• New to UX – Electrodermal Activity
• Skin conductance response • Body temperature
– Facial expression analysis – Pupil dilation – Heart rate variability – Respiration – Blood pressure
![Page 22: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/22.jpg)
Eye Tracking
22
![Page 23: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/23.jpg)
What is eye tracking
23
• Observing and recording eye movements as a participant interacts with a product – Allows us to gain deeper insight into how users
perform tasks
• Allows UX researchers to collect objective behavioral data
• Doesn’t include observing pupil dilation, blink rate, or facial recognition
Yesterday
![Page 24: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/24.jpg)
Eye tracking today
24
![Page 25: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/25.jpg)
Qualitative heat maps
25
• Aggregate of fixation count or duration across participants
Example: • Participants have similar fixation counts across links • Displays uncertainty of where to click to get started
![Page 26: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/26.jpg)
Qualitative gaze plots
26
• Plot of fixations for a single participant Example:
• Participant fixates back and forth between two different sections
• Displays uncertainty on how to use the sections
• The instructional paragraph did not facilitate web reading
![Page 27: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/27.jpg)
27
Example: • Participant has
repeated fixations in the upper right hand corner
• Participant said that he/she was looking for a search tool on the page
• The search tool was contained within a disappearing banner on the page
Qualitative gaze plots
![Page 28: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/28.jpg)
Quantitative eye-tracking data
28
• Quantitative data – Attention
• Time to first fixation – Are users finding the important content quickly?
• Total number of fixations in an area of interest • Percentages of fixations in an AOI compared to the total page
– Are users spending an inordinate amount of time looking at a single area?
– Processing • Fixation duration
– Are users spending a long period of time in this area? – Efficiency
• Repeat fixations – Is information clear and presented efficiently?
![Page 29: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/29.jpg)
Quantitative eye tracking
29
• Break the page up into separate “areas of interest” or AOIs
• Compare the fixation data between important areas and less important ones – Or compare data between
designs
Areas of Interest
![Page 30: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/30.jpg)
Combining quantitative and qualitative data
30
• Using multiple sources of data makes the evidence more compelling
• Example: “LAUNCH” was expected to be the most clicked • Heat map supports the quantitative eye-tracking data
![Page 31: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/31.jpg)
Beyond eye tracking
31
• Eye tracking is just one type of biometric measure • It tells us where participants are looking • It does not tell us
– Emotional state – Level of arousal – Level of mental workload
![Page 32: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/32.jpg)
Facial expression analysis
32
![Page 33: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/33.jpg)
33
Emotion Recognition Software • Real-time and continuous tracking of facial expressions
(Terzis, Moridis, Economides, 2010) • Distinguishes between happy, angry, sad, surprised, scared,
disgusted, and neutral – Overall accuracy of 89%
![Page 34: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/34.jpg)
34
Emotion Recognition Software
![Page 35: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/35.jpg)
35
Emotion Recognition Software
![Page 36: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/36.jpg)
Bringing biometrics to UX research
36
![Page 37: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/37.jpg)
Electrodermal Activity
37
![Page 38: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/38.jpg)
What is it?
38
• Electrodermal activity (EDA) encompasses skin conductance responses and body temperature.
• Nerve fibers release sweat in response to a stimulus.
• Sweat facilitates the travel of an electrical signal.
• After a stimulus onset, glands return to a baseline status.
• Sweat secretion is related to sympathetic nervous system activity.
![Page 39: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/39.jpg)
Who cares?
39
• Skin conductance is an established measure of arousal • Arousal can indicate engagement, fear, frustration, or other
emotional changes • Continuously measure changes in arousal throughout a test • Establish bench marks and use them to compare previous
iterations • Determine if the design facilitated typical levels of arousal
or if there were specific triggers
![Page 40: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/40.jpg)
EDA in UX research
40
• EDA can indicate usability problems • Assess “good” and “bad” interfaces and compare biometrics (Ward
& Marsden, 2002) • “Bad” interface causes higher skin conductivity, lower blood
volume, and increased pulse rate • Assess frustration while playing a game (Lin and Hu, 2005)
![Page 41: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/41.jpg)
41
How do I do it?
• The electrodes on an EDA sensor measure the resistance electricity faces when traveling across the skin.
• Electrodes can be placed on three locations – Best option - Palm – Good option - Finger – Acceptable option – Wrist
• Wired and wireless available
EDA recording device & analysis software
![Page 42: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/42.jpg)
The device that required the least amount of training
42
![Page 43: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/43.jpg)
A less commonly used explicit measure: Dial rating
43
![Page 44: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/44.jpg)
Dial Rating
44
FMG Rating Dial
• Continuous real-time feedback on videos and commercials
• Researcher can choose anchors for the ratings • Tear dropped knob allows participant to remain
focused on the video • Time sensitive
Position of dial
Max position of dial
Min position of dial
Dial Recorder Software
![Page 45: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/45.jpg)
Visa Video Ad
45
![Page 46: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/46.jpg)
46
EDA data System Time Movement Data Temperature Raw EDA Signal Event Marker
![Page 47: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/47.jpg)
47
• Tonic and phasic activity – Tonic activity is slow, state-based level of arousal – Phasic activity is a rapid, stimulus based change in arousal
• EDA activity is long periods of gradual change with a series of peaks in activity.
2.6
2.8
3.0
0 4 8 11 15 19 23 26 30
µS
Seconds
Processing the EDA signal
![Page 48: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/48.jpg)
48
• The phasic response begins 1-4 seconds after onset of stimulus • The signal is analyzed in discrete time intervals • The area under the curve is analyzed to determine changes
2.6
2.8
3.0
0 4 8 11 15 19 23 26 30
µS
Seconds
Response onset Returning to baseline Response onset Peak is delayed
Analyzing EDA data
![Page 49: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/49.jpg)
49
Traditional Measures of Attention and Emotion
![Page 50: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/50.jpg)
50
P
I found my mind wandering while the
advertisement was on
While the advertisement was on, I found myself
thinking about other things
I had a hard time keeping my mind
on the advertisement
Average
P1 1 1 1 1.0
P2 1 2 1 1.3
P3 1 1 1 1.0
P4 3 3 3 3.0
P5 2 2 2 2.0
P6 2 2 2 2.0
Explicit rating of attention: Please indicate how much you agree with the following statements
Response options: 1 (Not at all) | 2 | 3 | 4 | 5 | 6 | 7 (Extremely)
![Page 51: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/51.jpg)
51
Explicit rating of emotion: Please indicate how much you experienced each of the following while viewing the advertisement
P
Amused, fun-loving, silly
angry, irritated, or annoyed
disgust, distaste, or revulsion
guilty, repentant, or blameworthy
inspired, uplifted, or elevated
interested, alert, or curious
joyful, glad, or happy
sad, downhearted, or unhappy
scared, fearful, or afraid
sympathy, concern, or compassion
surprised, amazed, or astonished
P1 2 1 1 1 1 3 2 1 1 1 1
P2 2 3 1 1 1 1 1 1 1 1 1
P3 4 1 1 1 2 3 3 1 1 1 2
P4 1 2 1 1 1 1 1 1 1 1 1
P5 4 1 1 1 3 4 4 1 1 1 1
P6 5 1 1 1 3 4 4 1 1 1 2
Response options: 1 (Not at all) | 2 | 3 | 4 | 5 | 6 | 7 (Extremely)
![Page 52: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/52.jpg)
52
• When? – When did minds start to wander? – When were people engaged?
• What? – What did people focus on? – What did people miss? – What caused the negative/positive emotions?
• Was it something specific or overall?
Unanswered Questions
![Page 53: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/53.jpg)
53
New Measures of Attention and Emotion
![Page 54: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/54.jpg)
54
Traditional Likert-Scale Overall Rating
New Continuous Dial Rating
Visa Video Ad Example Question: Please indicate how much you experienced each of the following while viewing the advertisement. Response options: Not At All | A little bit| Moderately | Quite a bit | Extremely
P amused, fun-loving, or silly
angry, irritated, or annoyed
disgust, distaste, or revulsion
guilty, repentant, or blameworthy
inspired, uplifted, or elevated
interested, alert, or curious
joyful, glad, or happy
sad, downhearted, or unhappy
scared, fearful, or afraid
sympathy, concern, or compassion
surprised, amazed, or astonished
P1 2 1 1 1 1 3 2 1 1 1 1
P2 2 3 1 1 1 1 1 1 1 1 1
P3 4 1 1 1 2 3 3 1 1 1 2
P4 1 2 1 1 1 1 1 1 1 1 1
P5 4 1 1 1 3 4 4 1 1 1 1
P6 5 1 1 1 3 4 4 1 1 1 2
-1.1
0.0
1.1 P1
P2
P3
P4
P5
P6
Mean
![Page 55: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/55.jpg)
55
1.6 1.65
1.7 1.75
1.8 1.85
1.9 1.95
2 2.05
2.1
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Electrodermal Activity: Visa Video Ad
You can put notes here, but if you don’t it won’t appear when you present
[music only, screen change from bright to dark]
[drama<c screen change to black with white words, "without the worry of currency exchange"; music consistent]
[almost falls in water] [tail end of previous screen which appeared for several seconds and then change to first men<on of brand]
[middle of second screen change—MUSIC changes]
+ + + +
+
[music change]
[scene bright and beachy]
+
![Page 56: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/56.jpg)
56
Traditional Likert-Scale Overall Rating
New Physiological Measure of Arousal
Visa Video Ad Example Question: Please indicate how much you experienced each of the following while viewing the advertisement. Response options: Not At All | A little bit| Moderately | Quite a bit | Extremely
P amused, fun-loving, or silly
angry, irritated, or annoyed
disgust, distaste, or revulsion
guilty, repentant, or blameworthy
inspired, uplifted, or elevated
interested, alert, or curious
joyful, glad, or happy
sad, downhearted, or unhappy
scared, fearful, or afraid
sympathy, concern, or compassion
surprised, amazed, or astonished
P1 2 1 1 1 1 3 2 1 1 1 1
P2 2 3 1 1 1 1 1 1 1 1 1
P3 4 1 1 1 2 3 3 1 1 1 2
P4 1 2 1 1 1 1 1 1 1 1 1
P5 4 1 1 1 3 4 4 1 1 1 1
P6 5 1 1 1 3 4 4 1 1 1 2
1.6
1.65
1.7
1.75
1.8
1.85
1.9
1.95
2
2.05
2.1
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
![Page 57: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/57.jpg)
Artery Video Ad
57
![Page 58: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/58.jpg)
Artery Video Ad Example: Traditional Measures
58
Traditional Likert-Scale Overall Rating
Question: Please indicate how much you experienced each of the following while viewing the advertisement. Response options: Not At All | A little bit| Moderately | Quite a bit | Extremely
P amused, fun-loving, or silly
angry, irritated, or annoyed
disgust, distaste, or revulsion
guilty, repentant, or blameworthy
inspired, uplifted, or elevated
interested, alert, or curious
joyful, glad, or happy
sad, downhearted, or unhappy
scared, fearful, or afraid
sympathy, concern, or compassion
surprised, amazed, or astonished
P1 1 1 2 1 1 1 1 1 1 1 1
P2 1 1 5 1 1 1 1 2 1 1 4
P3 3 1 3 1 1 2 1 1 1 3 3
P4 1 3 5 1 1 3 1 3 1 1 5
P5 1 1 3 1 1 3 1 2 1 1 1
P6 1 1 5 1 1 1 1 1 1 1 3
![Page 59: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/59.jpg)
Artery video example
59
Traditional Likert-Scale Overall Rating
New Continuous Dial Rating
Question: Please indicate how much you experienced each of the following while viewing the advertisement. Response options: Not At All | A little bit| Moderately | Quite a bit | Extremely
P amused, fun-loving, or silly
angry, irritated, or annoyed
disgust, distaste, or revulsion
guilty, repentant, or blameworthy
inspired, uplifted, or elevated
interested, alert, or curious
joyful, glad, or happy
sad, downhearted, or unhappy
scared, fearful, or afraid
sympathy, concern, or compassion
surprised, amazed, or astonished
P1 1 1 2 1 1 1 1 1 1 1 1
P2 1 1 5 1 1 1 1 2 1 1 4
P3 3 1 3 1 1 2 1 1 1 3 3
P4 1 3 5 1 1 3 1 3 1 1 5
P5 1 1 3 1 1 3 1 2 1 1 1
P6 1 1 5 1 1 1 1 1 1 1 3
-‐1.2
-‐1
-‐0.8
-‐0.6
-‐0.4
-‐0.2
0
0.2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
P2, video 1
P3, video 1
P4, video 1
P5, video 1
P6, video 1
Mean
![Page 60: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/60.jpg)
-‐1.2
-‐1
-‐0.8
-‐0.6
-‐0.4
-‐0.2
0
0.2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
P2, video 1
P3, video 1
P4, video 1
P5, video 1
P6, video 1
Mean
Continuous dial rating: Artery video
60
[sound of rushing air] "this much was found stuck to the aorta..."
"every cigareWe is doing you damage"
![Page 61: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/61.jpg)
Electrodermal activity: Artery video
61
Traditional Likert-Scale Overall Rating
New Physiological Measure of Arousal
Question: Please indicate how much you experienced each of the following while viewing the advertisement. Response options: Not At All | A little bit| Moderately | Quite a bit | Extremely
P amused, fun-loving, or silly
angry, irritated, or annoyed
disgust, distaste, or revulsion
guilty, repentant, or blameworthy
inspired, uplifted, or elevated
interested, alert, or curious
joyful, glad, or happy
sad, downhearted, or unhappy
scared, fearful, or afraid
sympathy, concern, or compassion
surprised, amazed, or astonished
P1 1 1 2 1 1 1 1 1 1 1 1
P2 1 1 5 1 1 1 1 2 1 1 4
P3 3 1 3 1 1 2 1 1 1 3 3
P4 1 3 5 1 1 3 1 3 1 1 5
P5 1 1 3 1 1 3 1 2 1 1 1
P6 1 1 5 1 1 1 1 1 1 1 3
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
P1
P2
P3
P4
P5
P6
Mean
![Page 62: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/62.jpg)
Electrodermal activity: Artery video
62
"...the main artery from the heart"
"every cigareWe is doing you damage"
[voice, pace change] "authorized by the Australian government"
"this much was found stuck to the aorta..."
[sound of rushing air] [first faWy deposits emerge]
+ + + + + +
“every cigareWe is doing you damage "
[sound effect; no text] “age 32“ [heartbeats] [sound of crackling embers]
+ + + +
![Page 63: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/63.jpg)
1.6 1.65
1.7 1.75
1.8 1.85
1.9 1.95
2 2.05
2.1
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
EDA does not capture valence
63
You can put notes here, but if you don’t it won’t appear when you present
P1: Artery ad (Negative emotion)
P1: Visa ad (Positive emotion)
![Page 64: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/64.jpg)
Continuous Dial Rating: Artery vs. Visa
64
-1.1
0.0
1.1 P1
P2
P3
P4
P5
P6
Mean
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
P2, video 1
P3, video 1
P4, video 1
P5, video 1
P6, video 1
Mean
![Page 65: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/65.jpg)
EDA advantages and disadvantages
65
• Advantages – Continuous measure of
automatic physiological response
– Sensitive to minor changes in arousal
– Informs order of magnitude • Disadvantages
– Does not inform valence – Peak of physiological response
is slow – Sometimes difficult to collect
0
0.5
1
1.5
2
2.5
Dial Eye Tracker EDA
Mea
n In
trusi
vene
ss R
atin
g
Debriefing question: On a scale of 1 to 5, how intrusive was ____ while you were trying to complete the tasks and watch videos?
Dial: Two participants rated the dial as very intrusive (4): “I was having to concentrate on what my reaction was, not just have it.”
“It’s not something I normally do, or something I do consciously.” EDA: Three participants rated the wrist band as moderately intrusive (3): “It was itchy.” “I had to remember not to move it.” “I didn’t know where to put it.”
![Page 66: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/66.jpg)
The future of implicit measures
66
![Page 67: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/67.jpg)
We need to be taking a collaborative approach
67
• Disparate measures of physiological response can tell a cohesive story! • By analyzing different streams of data we can uncover a very rich level
of analysis.
![Page 68: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/68.jpg)
We need to be taking a collaborative approach
68
![Page 69: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/69.jpg)
Combining implicit measures for meaningful insights
69
-1.100
0.000
1.100 • Simulated pupil diameter data
• Simulated heart rate variability data
• Simulated EDA data
![Page 70: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/70.jpg)
EDA: promising future
70
• Promising results – When data is good, EDA provides continuous, “objective” arousal
measure – There is consistency between:
• The Likert scale and the continuous dial data • Self-reported emotion overall and EDA data
– EDA provides additional data above and beyond self-report measures
– Most complete story can be told with a combination of measures.
![Page 71: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/71.jpg)
71
• Data Analyses – Compare to baseline – different baseline per person and per stimulus – How does pupil dilation data compare with EDA? – Reduce the intrusiveness ratings for all metrics
Lessons learned
• Dial – If ET is not used, allow participants to look at the dial when making
responses – Include simple practice task to increase familiarity
• Eye Tracker – Instruct participants to visually search as if they were at home on their own
computer
• EDA – Improve quality of EDA data; explore equipment – Provide a cushion/pad to rest arm – Over-recruit
![Page 72: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/72.jpg)
Select your measure carefully
72
• Where are participants dwelling on instructions and tasks? – Eye tracking
• Which specific elements on a page are particularly stressful? – Eye tracking, EDA
• Which content is very engaging for the user? – Eye tracking, EDA, satisfaction questions, debriefing interview
• Which design causes more stress on the user? – EDA, debriefing interview
![Page 73: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/73.jpg)
Not just about usability but also interaction
73
![Page 74: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/74.jpg)
Interfaces that adjust based on affective state and workload
74
![Page 75: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/75.jpg)
Video games that adapt to a user’s experience
75
![Page 76: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/76.jpg)
Cognitive training programs that adjust to a person’s ability
76
![Page 77: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/77.jpg)
But for UX…
77
![Page 78: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/78.jpg)
Pushing our research further
78
• There are lessons to be learned from neuromarketing – Neuromarketing researchers have used EDA, heart rate
variability and even fMRI and EEG in an attempt to determine how users experience an advertisement.
• UX has a different set of requirements – To become more usable for practitioners, we need:
• Portable technology that can be taken when traveling • Software that has a short learning curve • Customizations that allow for sensors to be wrist mounted and
more literature to substantiate the use of this sensor location • Analysis protocols that can be completed in a short period of
time.
![Page 79: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/79.jpg)
Issues to keep in mind
79
• We want to mimic real-world experiences during a usability study • Complex setup will confound our experimental design • Participant comfort is paramount
• Concurrent think-aloud vs. Retrospective think-aloud • A talking participant is a distracted participant
• We always need to provide support for a ROI
![Page 80: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/80.jpg)
Where do we go from here?
80
• We need to: – Collaborate to move our
field forward – Share methods and
analysis protocols – Empirically test our
hypotheses – Continually provide proof
for ROI
![Page 81: Beyond Eye Tracking: Using User Temperature, Rating Dials, and Facial Analysis to Understand the User Experience](https://reader033.vdocuments.net/reader033/viewer/2022052822/554f0730b4c90547648b52f8/html5/thumbnails/81.jpg)
Thank you!
81
Jennifer Romano Bergstrom [email protected] | @romanocog
Dan Berlin
[email protected] | @banderlin
Jon Strohl [email protected] | @jonstrohl
David Hawkins
[email protected] | @dHawk87 UXPA2013 | Washington, DC