Slide 1

A Framework for Haptic Broadcasting Presented by Cong Ly CMPT-820 March 16, 2009 Jongeun Cha, Ian Oakley, Yo-Sung Ho, Yeongmi Kim, and Jeha Ryu

Slide 2

Overview 1.Introduction 2.Types of Haptic 3.Proposed Framework Content Creation Transmission Viewing & Interactions 4.Implementation 5.Demonstration 2

Slide 3

Overview 1.Introduction 2.Types of Haptic 3.Proposed Framework Content Creation Transmission Viewing & Interactions 4.Implementation 5.Demonstration 3

Slide 4

What is Haptic? There is no agreement in the precise definition among researchers. In this paper Haptic is used to define two sub- categories of feedback Tactile Kinesthetic 4

Slide 5

Motivations Broadcast programs are generally linear A begin, middle and end Entertainment is a multi-billion industry Consumers are actively seeking for interactive content. We have the technology MPEG-4 BIFS (Binary Format for Scenes) Reachin API - VRML 5

Slide 6

Overview 1.Introduction 2.Types of Haptic 3.Proposed Framework Content Creation Transmission Viewing & Interactions 4.Implementation 5.Demonstration 6

Slide 7

Passive Haptic No direct interaction 7

Slide 8

Active Haptic Semi-interactions Tactile and kinesthetic 8

Slide 9

Types of Haptic Two types of Haptic Media Linear and Non-Linear Linear Haptic Sequential progression Human touches, impacts, sounds, etc 9

Slide 10

Types of Haptic Non-linear Haptic Interactivity, tactile information Able to feel the surfaces Dynamic content Kinesthetic devices PHANToM 10

Slide 11

Overview 1.Introduction 2.Types of Haptic 3.Proposed Framework Content Creation Transmission Viewing & Interactions 4.Implementation 5.Demonstration 11

Slide 12

Framework 12

Slide 13

Content Creation Audio and Video Standard video camera Microphone for audio Three Approaches for capturing Haptic data 1.Physical sensors 2.Modeling tools 3.Analysis of other associated media 13

Slide 14

Physical Sensors Capturing haptic surfaces Piezoelectric resonance Touch sensors Movement data 3D robotic arm Accelerometer Force-torque sensors 14

Slide 15

Content Creation Audio and Video Standard video camera Microphone for audio Three Approaches for capturing Haptic data 1.Physical sensors 2.Modeling tools 3.Analysis of other associated media 15

Slide 16

Modeling Tools Capturing 3D scenes 3D scanner to capture objects ZCam, depth video camera (2.5D) 3D Modeling tool K-HapticModel HAMLAT Motion capturing 16

Slide 17

Content Creation Audio and Video Standard video camera Microphone for audio Three Approaches for capturing Haptic data 1.Physical sensors 2.Modeling tools 3.Analysis of other associated media 17

Slide 18

Automatic Generation Extract trajectory of object from video Dr. Greg Moris work SFU Vision and Media Lab 18

Slide 19

Transmission 19

Slide 20

MPEG-4 BIFS BIFS (Binary Format for Scenes) Scenes are encoded and transmitted separately Local and remote animations User Objects interaction Enables different points of view (3D) Scenes description Consist of information about the objects Time and place Relations between the objects 20

Slide 21

MPEG-4 BIFS 21 Proposed extended BIFS nodes

Slide 22

MPEG-4 BIFS BIFS Nodes Content Store data gathered during Creation Ie. Piezoelectric sensors, modeling tools DepthMovie Node Identical to DepthImage Added MovieTexture for tactile content 22

Slide 23

Viewing & Interaction 23

Slide 24

Viewing & Interaction Haptic Compositor Route elements to renderers Haptic Renderer Decode objects positions Generate interaction forces Tactile Renderer Decode tactile information Thermal perception, intensities of tactile 24

Slide 25

Overview 1.Introduction 2.Types of Haptic 3.Proposed Framework Content Creation Transmission Viewing & Interactions 4.Implementation 5.Demonstration 25

Slide 26

Implementation Implementation by the authors 26

Slide 27

Implementation Components GPAC Project on Advanced Content multimedia framework BIFS Broadcaster MPEG-4 BIFS Darwin Streaming Server Apple's QuickTime Streaming Server Standard RTP and RTSP protocols Osmo4 Player From GPAC framework 27

Slide 28

Overview 1.Introduction 2.Types of Haptic 3.Proposed Framework Content Creation Transmission Viewing & Interactions 4.Implementation 5.Demonstration 28

Slide 29

Demonstration Home Shopping Scenario 29

Slide 30

Demonstration Movie with Tactile Feeling 30

Slide 31

Industry D-Box http://www.d-box.com Pneumatic actuated chairs Used for movies and simulations 31

Slide 32

PHANToM Sensable Technologies Developed by a student at MIT Industry 32

Slide 33

Conclusions Haptic Media enhances existing multimedia Such as movies Haptic can be used in Surgical Training Military Commercial Proposed Framework is feasible Tools needed are readily available 33

Slide 34

Thats all Thats all folks Questions? 34

Slide 35

References 1.Jongeun Cha, Ian Oakley, Yo-Sung Ho, Yeongmi Kim, and Jeha Ryu A Framework for Haptic Broadcasting, IEEE Multimedia Magazine 2.G. M. Krishna and K. Rajanna, Tactile Sensor Based on Piezoelectric Resonance, IEEE Sensors Journal, vol. 4, no. 5, 2004, pp. 691-697. 3.Y. Kim, S. Kim, T. Ha, I. Oakley, W. Woo, and J. Ryu, Air-Jet Button Effects in AR, Intl Conf. Artificial Reality and Telexistence, LNCS 4282, 2006, pp. 384-391. 4.SFU Visual and Modeling Lab, http://www.cs.sfu.ca/research/groups/VML/index.html 5.MIT Tech, Robotic Gripper with Phantom Sensable Technologies, http://techtv.mit.edu/videos/467-robotic-gripper-with-phantom-sensable-technologies 6.Sensable Technologies, PHANToM, http://www.sensable.com/haptic-phantom-premium-6dof.htm 7.Raunhofer Institute, MPEG-4 BIFS Binary Format for Scenes, http://www.iis.fraunhofer.de/Images/MPEG-4%20BIFS_tcm389-67584.pdf 35