cross-modal perception of motion- based visual-haptic stimuli ian oakley & sile omodhrain...
TRANSCRIPT
Cross-modal perception of motion-based visual-haptic stimuli
Ian Oakley & Sile O’Modhrain
Palpable Machines Research Group
http://www.mle.ie/palpable
[email protected]@media.mit.edu
Media Lab EuropeSugar House Lane
BellevueDublinIreland
Overview
• Background – Touch in Broadcast Media
• The Touch TV Project• Role of cross modal perception in
TTV• Experimental Study - Pilot• Conclusions and Future Work
Touch in Broadcast Media
• Adding haptic information to a broadcast may enhance viewing
• Motivation Consumer demand for richer media
• Led to advances in audio and video technologies• Touch may provide greater richness• Latest advances not enhanced viewing experience
e.g. digital and interactive TV
Evidence supporting this assertion• Touch in virtual environments linked to increased
ratings of presence and co-presence• Adoption of touch in gaming and simulator systems
The Touch TV Project
• Ongoing work investigating integration of touch into broadcast content Currently, focused on a practical and
theoretical exploration of the design space available for this work
Involves a number of research threads ranging from authoring of media to psychological study
Cross-Modal Perception in TTV• Touch TV inherently cross-modal
Involves construction of immersive and involving media with graphics, audio and touch components
Techniques for combining graphics and audio established in this context
• E.g. visual capture
How to add touch to this format is an open question
• How do we add haptic information to an audio-visual stream ensuring that it will be interpreted as we intend?
Experimental Study
• Interested in combination of visual and haptic stimuli representing movement in complex graphical scenes Relatively complex stimuli
• Initial questions: Can haptic cues be associated with visual
movement? What is the effect of visual distractors? What is the effect of visual target position? What kind of haptic cues should we use?
• Attempted to address these questions in a pilot study
Experimental Study
• In each trial, the visual display consisted of a number of bouncing balls controlled by a simple mathematical model
• Each ball had a different spring constant – different “bounciness” and was subject to gravity
• Force displayed on a PHANToM haptic interface either represented the velocity of one of the displayed balls, or of an unseen ball
• Participant’s task was to determine whether or not what they could feel related to one of the displayed balls
Experimental Study
• Varied: Number of distractors (5 possibilities) Target spring constant (6 possibilities) Target position (5 possibilities) Haptic-Visual match (2 possibilities)
• Total of 300 trials presented in random order• Practice consisted of a random selection of 50 trials• 10 Participants
All employees at Media Lab Europe 6 male, 4 female, all right-handed, none reported tactile or
visual impairment, range of familiarity with haptic devices (2.5 on a scale from 0-5)
• Participants wore noise cancelling headphones• Were able to rest between each trial
Experimental Study
• Gathered trial time, error rate• Trial time for visual presence/absence of
haptic stimuli:
• T-test revealed no significant differences
4.128 3.993
00.5
11.5
22.5
33.5
44.5
Mea
n
Tri
al T
ime
(sec
onds)
Presence Absent
Visual presence/ absence of haptic stimuli
Experimental Study
• Further analysis restricted to trials in which haptic ball was visually present
• Used ANOVA, post-hoc t-tests with Bonferroni CI adjustments
• Did not include trials in which subject made errors in temporal analysis However, relatively large error rate Collapsed position factor for temporal analysis
• Used a 5 (distractors) x 6 (spring values)• No interactions found (F=0.399, p=0.991)
• Error data 5 (distractors) x 6 (spring values) x 5 (positions) One interaction: Distractors by Spring Value (F=1.974, p<0.01)
Results
• Trial time for differing numbers of visual distractors
• ANOVA: F=8.235, p<0.001
00.5
11.5
22.5
33.5
44.5
Tri
al T
ime
(sec
onds)
0 1 2 3 4
Number of Distractors
No of Distracto
rs
0 1 2 3 4
0 X Not Sig Not Sig P<0.01 P<0.01
1 X x Not Sig P<0.05 P<0.01
2 X X X Not Sig P<0.05
3 X X X X Not Sig
4 X X x X X
Results of post-hoc t-tests
Results
• Trial time for differing spring values• ANOVA: F=83.881, p<0.001
0
1
2
3
4
5
6
Tri
al T
ime
(sec
onds)
0.15 0.2 0.25 0.3 0.35 0.4
Spring valuesLarger figures correspond to
"bouncier" balls
Spring Values
0.15 0.2 0.25 0.3 0.35 0.4
0.15 X Not Sig P<=0.01
P<=0.01
P<=0.01
P<=0.01
0.2 X X P<=0.01
P<=0.01
P<=0.01
P<=0.01
0.25 X X X P<=0.01
P<=0.01
P<=0.01
0.3 X X X X P<=0.01
P<=0.01
0.35 X X X X X Not Sig
0.4 X X X X X X
Results of post-hoc t-tests
Results
• Error rate for differing numbers of distractors
• ANOVA: F=20.614, p<0.001
0
5
10
15
20
25
30
Per
centa
ge
of Tri
als
Answ
ered
Inco
rrec
tly
0 1 2 3 4
No of DistractorsResults of post-hoc t-tests
No of Distracto
rs
0 1 2 3 4
0 X P<0.05 P<0.01 P<0.01 P<0.01
1 X x Not Sig P<0.05 P<0.01
2 X X X Not Sig P<0.01
3 X X X X Not Sig
4 X X x X X
Results
• Error rate for differing spring values
• ANOVA: F=4.861, p<0.001• Post-hoc t-tests revealed 0.15 spring value led
to significantly higher error rate than 0.4 spring value
0
5
10
15
20
25
Per
centa
ge
of
Tri
als
Answ
ered
In
corr
ectl
y
0.15 0.2 0.25 0.3 0.35 0.4
Spring valuesLarger figures correspond to
"bouncier" balls
Results
• Error rate for target positions• ANOVA: F=0.399, p<0.05• Post-hoc t-tests did not attain significance
0
5
10
15
20
25
Per
centa
ge
of Tri
als
Answ
ered
Inco
rrec
tly
-45 -25 0 25 45
Screen position (centre at 0)
Interpretation
• Subjects merged visual-haptic motion stimuli Error rate always less than chance
• Increase in trial time and error rate with increased numbers of distractors indicates serial search Haptic target did not “pop out” as in parallel search
• Equality of present and absent results suggests subjects performed an exhaustive search
• Possible speed/accuracy trade off Errors decreased, time increased with larger spring values
• Eccentricity effect in position error data suggests subjects focused on the centre of the screen
Conclusions and Future Work• Study successfully informed basic
questions relevant for our TTV domain Relevant to our authoring process
• Many questions remain Contrast same scenario with visual/audio and
visual/audio/haptic stimuli Role of visually distinct objects Is mapping force to motion the best solution?
• Basic scaling research on force presentation in this domain
…
Cross-modal perception of motion based visual-haptic stimuli
Ian Oakley & Sile O’Modhrain
Palpable Machines Research Group
http://www.mle.ie/palpable
[email protected]@mle.ie
Media Lab EuropeSugar House Lane
BellevueDublinIreland
EuroHaptics 2003 6th-9th July www.mle.ie/palpable/
eurohaptics2003