![Page 1: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/1.jpg)
THE UNIVERSITY THE UNIVERSITY OF BRITISH COLUMBIAOF BRITISH COLUMBIA
Multimodal Design & Technologies
Sidney Fels
![Page 2: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/2.jpg)
Fels: Multimodal Design and Technologies 2
Overview
• Introduction
• Human I/O: Interface Examples
• Bringing Modalities Together
• Summary
![Page 3: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/3.jpg)
Fels: Multimodal Design and Technologies 3
Introduction
• Two messages1. knowledge of modalities motivates
design
2. modalities can complement each other– challenges and pitfalls
• Applications in virtual environments– communication of experience– intimacy and embodiment
![Page 4: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/4.jpg)
Fels: Multimodal Design and Technologies 4
Human Information Processing• Input
• 5 usual senses– vision, hearing, touch, taste, smell
• position and motion sensing systems
• Output• intentional
– neuromuscular, movable, verbal
• non-intentional / biopotentials– galvanic skin response (GSR), heart rate, brain,
muscle activation
• Cognitive– memory, decision making, tracking, learning
![Page 5: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/5.jpg)
Fels: Multimodal Design and Technologies 5
Visual Display Technologies
• examples of designs exploiting human capabilities– Virtual Retinal Display (VRD)– Cubby– CAVE Automatic Virtual Environment
![Page 6: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/6.jpg)
Fels: Multimodal Design and Technologies 6
Virtual Retinal Delay (VRD)
ArgonLaser
Delivery Optics
C&D Electronics
Scanners
AO Modulators
Red LaserDiode
From HIT lab,U. Of Washington(Furness et al., 1991)Microvision continuing work
![Page 7: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/7.jpg)
Fels: Multimodal Design and Technologies 7
Cubby (Djajadiningrat and Gribnau, 2000)
QuickTime™ and aCinepak decompressorare needed to see this picture.
![Page 8: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/8.jpg)
Fels: Multimodal Design and Technologies 8
CAVE Illustration (U. of Illinois, 1992)
![Page 9: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/9.jpg)
Fels: Multimodal Design and Technologies 9
Haptics Display Technologies
• examples of touch and force feedback– Pantograph– Phantom– CyberForce
![Page 10: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/10.jpg)
Fels: Multimodal Design and Technologies 10
Pantograph (Hayward, McGill)
• 2 dof translation - 10cm X 10cm• FF/B mouse• acceleration best for shock and hard
contacts
![Page 11: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/11.jpg)
Fels: Multimodal Design and Technologies 11
Phantom (Massey, SensAble Devices)
• thimble on finger– single point force feedback– 3 degrees of freedom
![Page 12: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/12.jpg)
Fels: Multimodal Design and Technologies 12
CyberForce (Kramer, Immersion Corp.)
• Exoskelton– arm forces
• Tendons– grasp forces
![Page 13: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/13.jpg)
Fels: Multimodal Design and Technologies 13
tricks: visual / aural illusionsexamples of metacognitive gap
• visual: never show the point penetrating the surface, even if it is
• aural: play a crisp contact sound on contact
this makes the surfaceappear stiffer/harder(very robust effect)
actual:
displayed:
BUT: if time offset too great, opposite effect
![Page 14: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/14.jpg)
Fels: Multimodal Design and Technologies 14
Other Haptic Devices
• Tactile feedback
• Temperature
• Sheer forces
![Page 15: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/15.jpg)
Fels: Multimodal Design and Technologies 15
challenges for haptic interaction design
• continuous vs. discrete manual control
• displaying interaction potential
• embedding haptic interfaces
• ensuring tight sensory coupling
• interacting with other modalities
![Page 16: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/16.jpg)
Fels: Multimodal Design and Technologies 16
Olfactory Devices
• How do we smell?– different theories:
• i.e. chemical, infrared absorption
– different perceptual mappings:• smell prism• four odors: fragrant, acrid, burnt and caprylic• domain specific: wine, beer, etc.• many, many others...
– Acuity is great - 10,000 times more sensitive than taste
– negative adaptation occurs• you get used to the smell
![Page 17: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/17.jpg)
Fels: Multimodal Design and Technologies 17
Olfactory Interfaces• Smell camera
– Susnick & Raknow, Nature, 2000, 406, 710-714.
• Smell display– liquids, gels, microencapsulation
• Application Research:– Joe Kaye:
• inStink, Dollars & Scents, Scent Reminder, and more...
– Fels, Gauthier, Smith: Interactive Yoga system
• Smell Display Research and Products:– DigiScents (bankrupt), TriSenx (bankrupt)– DIVEpak (Southwest Research Institute, 1993)– see notes
![Page 18: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/18.jpg)
Fels: Multimodal Design and Technologies 18
Olfaction Display Challenges
• smell synthesis– many smells vs. mixing base smells
• control breathing space– sealed room with air filtration– air control in front of and behind user– sealed pod– tethered mask– tubes into an HMD from pack– built into HMD
![Page 19: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/19.jpg)
Fels: Multimodal Design and Technologies 19
Taste
• taste buds for– sensations of sour, salty, bitter, sweet and
umami– receptors not completely resolved
• umami receptor (Zucker et al., 2002)
– extremely complex and poorly understood
• interacts with olfaction
• display = food?
![Page 20: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/20.jpg)
Fels: Multimodal Design and Technologies 20
Biopotentials: Examples• Galvanic Skin Response (GSR)
– Affective Computing, i.e. Galvactinator, Scheirer and Picard
• Heart Rate (HR)– 2 Hearts Musical System, (McCaig and Fels,
2002)
• Brain activity (EEG)– Brain Computer Interface (BCI)
• Muscle activity (EMG)
![Page 21: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/21.jpg)
Fels: Multimodal Design and Technologies 21
• Multitude of input/output systems– all active at once
• I/O mechanisms usually depend upon – cognitive context– emotional contexts
• All these systems available for applications– complement each other
• Multimodal design looks at:– integration– substitution– complement
Summary of Human I/O
![Page 22: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/22.jpg)
Fels: Multimodal Design and Technologies 22
Multimodal Design: Bringing it Together
• User centred and non-user centred
• Intimacy and Embodiment– automatic behaviour– sources of aesthetics
![Page 23: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/23.jpg)
Fels: Multimodal Design and Technologies 23
Intimacy and Embodiment
• Want interfaces that feel “good” to use
• Humans and machines intimately linked– degree of intimacy supported may
determine success
• Types of relationships:– human to human– human to machine
![Page 24: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/24.jpg)
Fels: Multimodal Design and Technologies 24
Intimacy• Intimacy is a measure of subjective match
between the behaviour of an object and the control of that object.– extension of “control intimacy” from electronic
musical instruments analysis (Moore, 1997)
• High intimacy implies:– object feels like an extension of self– satisfaction derives from interacting with object– emotional expression flows
• requires cognitive effort to prevent
![Page 25: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/25.jpg)
Fels: Multimodal Design and Technologies 25
Intimacy, Embodiment and Aesthetics
Case 1: Object disembodied from Self
Case 2: Self embodies Object
Case 3: Self disembodied from Object
Case 4: Object embodies Self
self object
objectself
objectself
object self
Aesthetic
Control
Result
Reflection
Belonging
![Page 26: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/26.jpg)
Fels: Multimodal Design and Technologies 26
Intimacy and Embodiment Design Examples
• Many excellent examples out there of interesting designs– Many at Siggraph:
• Jam-o-Drum (Blaine et al.), Wooden Mirror (Rozin), etc.
– Tangible Bits work (Ishii et al.)– Ubicomp (Weiser and more)– Wearable computing (Mann and others)– Art, Entertainment– Virtual reality
![Page 27: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/27.jpg)
Fels: Multimodal Design and Technologies 27
Application: Iamascope (Fels and Mase, 1997)
![Page 28: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/28.jpg)
Fels: Multimodal Design and Technologies 28
Iamascope Overview Video
QuickTime™ and aCinepak decompressorare needed to see this picture.
![Page 29: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/29.jpg)
Fels: Multimodal Design and Technologies 29
FlowField: Semantics of Caress (Chen, Fels and Min, 2003)
• Whole hand interaction in VR– use multi-touch sensitive pad – immersive display
• Idea:– allow users direct manipulation of fluid
• use particle simulation for fluid– aesthetics was important
• hand manipulation on hard surface mapped to obstructions in the flow field
![Page 30: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/30.jpg)
Fels: Multimodal Design and Technologies 30
FlowField: Semantics of Caress
QuickTime™ and aYUV420 codec decompressorare needed to see this picture.
![Page 31: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/31.jpg)
Fels: Multimodal Design and Technologies 31
• Multimodal interfaces need to consider– human information processing– matching interface to task
• use complementary modes where appropriate
– intimacy and embodiment
• Plenty of research opportunities
Summary
![Page 32: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/32.jpg)
Fels: Multimodal Design and Technologies 32
![Page 33: THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b3a00550346ae7a8b4674/html5/thumbnails/33.jpg)
THE UNIVERSITY THE UNIVERSITY OF BRITISH COLUMBIAOF BRITISH COLUMBIA
Thank-you!
Please fill out your comments forms.
www.icics.ubc.ca/hci/multimodal/www.icics.ubc.ca/hci/multimodal/