simon julier department of computer science university ... · event-based haptics • detect when a...

64
Introduction to Virtual Environments: Haptics Simon Julier Department of Computer Science University College London http://www.cs.ucl.ac.uk/teaching/VE

Upload: others

Post on 13-Aug-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Introduction to Virtual Environments: Haptics

Simon JulierSimon JulierDepartment of Computer ScienceUniversity College London

http://www.cs.ucl.ac.uk/teaching/VE

Page 2: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Outline

• Introduction• Perception of Haptic Stimuli• Synthesis of Haptic Stimuli

Page 3: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Introduction

• Introduction• Perception of Haptic Stimuli• Synthesis of Haptic Stimuli

Page 4: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

The Importance of Touch

• Touch is vital for normal interaction with the real world• Provides information about:

– Surfaces• Hot, cold, smooth, rough, slippery, rubbery

– 3D Space and 3D Shapes• Size, curvature, where things are

– Feedback• Holding objects, etc.

• Case studies show that no sense of touch is highly debilitating– Must use mental effort at all times to confirm what’s going on

Page 5: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Example: Soft vs. Hard Keyboards

Touch-tone phones:

•Rich haptic cues

•Can be done without looking

•Effortless

Softkeyboard phones:

•Limited haptic cues

•Must be looked at

•Lots of development effort needed to be usable

Page 6: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Why Simulate Touch in Virtual Environments?

• Interaction with objects very important for utility / presence of virtual environments

• Touch provides valuable feedback and information– Weight, strength give feedback on how well user is holding object– Weight, strength give feedback on how well user is holding object– Hardness, texture, vibration, interia or weight convey information about

the properties of objects

• Visual dominance can be used but– It works only when you can see what’s doing the manipulating and what’s

being manipulated– It does not give the fine-grained feeling of manipulating the actual object

Page 7: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Teleoperation

Page 8: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Medical Training

Page 9: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Medical Training

Training for Bone Milling, Stanford BioRobotics

QuickTime™ and a decompressor

are needed to see this picture.

Page 10: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Medical Training

The Virtual Haptic Back Project, University of Ohio

Page 11: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Industrial Training

SPIDAR Industrial putty gun training, Ortega and Coquillart

Page 12: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Design and Simulation

Haptics can be used to provide a sculpting metaphor and more organic designs

Page 13: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Haptics to Preview CAD

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Page 14: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Cultural Heritage

Museum of PureForm / PERCO Exo-Skeleton

Page 15: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Science

UNC Nanomanipulator

Page 16: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Perception of Haptic Stimuli

• Introduction• Perception of Haptic Stimuli• Synthesis of Haptic Stimuli• Evaluation Study

Page 17: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Information

• Tactile (Cutaneous)– Force receptors in the skin

• Proprioceptive (Kinesthetic)– Force receptors in the muscle– Force receptors in the muscle

• Haptic– Combination of tactile and proprioceptive– Active

• We do not discuss temperature and pain, although they are important and temperature-based interfaces have been developed

Page 18: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Tactile

• Measured through mechanoreceptors• These lie in multiple layers of the skin• Types of information include:

– Pressure– Temperature– Temperature– Softness– Wetness– Friction

• Adhesion, slip

• Vibro-tactile extremely important– Texture

Page 19: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Slow and Fast Adaptation in Receptors

• Receptors can be fast or slow• Fast adaptation to onset and offset of skin deformation

– FAI receptors: small, well-defined areas; respond to rate of deformation; particularly important for manipulation

– FAII receptors: large, poorly-defined areas; respond to acceleration (and – FAII receptors: large, poorly-defined areas; respond to acceleration (and higher order derivative) of deformation

• Slowly adapting mechanoreceptors have a continual response to skin deformation– SAI receptors: strong dynamic sensitivity and irregular response to

sustained and small indentations (less than 1 µm!); particularly important for textures

– SA II receptors: less dynamic sensitivity but regular response to sustained indentation; skin stretch; detection of slippage

Page 20: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Processing of Receptors is Not Uniform

Page 21: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Two-Point Threshold

• One way to test tactile “resolution” is to apply the two-point threshold– Touch subject in various parts

of the body with a two-headed of the body with a two-headed probe

– Keep widening the distance until the subject can detect twodistinct points of contact

– Provides a measure of fidelity / spatial actuation

Page 22: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Typical Two-Point Thresholds

Page 23: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Vibro-Tactile and Temporal Response

• Temporal “two point experiment”– Pulses of 1ms duration have to be at least 5.5ms apart to be

perceived as two separate events

• Up to 700Hz can be detected in adult skin

Page 24: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Proprioception

• Measured by mechanoreceptors

• They lie in muscles, tendons, joints and the tendons, joints and the skin of the hand

Page 25: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptically Perceptible Properties

• Two main types of properties:– Geometric

• Specific to a particular object• Specific to a particular object

– Material• Common for all objects of the same type

Page 26: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Geometric Properties

• Micro-geometric:– Small enough to fall in a single skin region, e.g., fingertip– Encodes the shape of the object which contacts the fingers

• Macro-geometric:– Too large to fit in a single region– Proprioception needed to map fingertips over the object

Page 27: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Material Properties

• Texture:– Includes roughness, stickiness and spatial density– Microtexture:

• Micron-level • Micron-level

– Macrotexture:• One to two orders of magnitude greater

• Hardness (compliance):– Tactile (e.g., cotton wool)– Proprioceptive (e.g., piano key)

Page 28: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Perception of Weight

• The perception of weight is dependent upon many factors– Objects feel heavier if wielded than resting on the skin– Smaller objects of the same weight feel heavier– Smaller objects of the same weight feel heavier– Objects with a higher inertia tensor feel heavier– Cognitive factors (the golf-ball illusion)

Page 29: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Role of Manual Exploration

• Haptic systems are not passive• User moves and adopts strategies to enhance feelings from

objects• Different strategies optimise discovery of different properties

Page 30: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Summary

• Touch is a proxal modality and is extremely important for finding out properties of objects and providing feedback for manipulation

• There are two main types: tactile and proprioceptive• There are two main types: tactile and proprioceptive– Tactile caused by deformation of the skin– Proprioceptive caused by forces in the muscles and joints

• Users adopt exploration strategies to optimise finding out about the environment

Page 31: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Synthesis of Haptic Stimuli - Haptic Devices

• Introduction• Perception of Haptic Stimuli• Synthesis of Haptic Stimuli

– Haptic devices– Haptic rendering

• Evaluation Study

Page 32: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Hardware Platforms

• An ideal platform would:– Be fully bi-directional– High spatial and temporal resolution– Produce both tactile and kinesthetic information– Not encumber a user’s motion

• However, no practical display incorporates all of these features and any haptic display is a compromise of one kind or another

• Haptic displays can be broadly classified as– Tactile vs. kinesthetic– World-grounded vs. body-grounded vs. hand-grounded– Passive vs. active

Page 33: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Pin Arrays

Active, tactile, world-grounded

Page 34: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Finger Tip Vibro-Tactile Display

Active, tactile, hand-grounded

Page 35: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Tactors

Active, tactile, body-grounded or world-grounded

Page 36: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Air Jets

• High pressure air forced onto user’s fingers

• Can be used to provide short-duration high short-duration high frequency events– Button clicks on a virtual

mobile phone

Air-Jet Button Effects in AR, Kim et al.

Page 37: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Robotic Arms

SENSABLE 6-DOF Phantom PERCRO GRAB

World-grounded, kinesthetic, active

Page 38: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Exoskeletons

World-grounded, kinesthetic, active

Page 39: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

PureForm Exoskeleton in Action

Page 40: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

String and Wire-Based Haptic Devices

World-grounded, kinesthetic, active

Page 41: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Stringed Workbench in Action

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Page 42: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Forcefeedback Gloves

Active, kinesthetic, hand-grounded

Page 43: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Pneumatic Gloves

Page 44: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Novint Falcon

Active, tactile, world-grounded

Page 45: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Hybrid System

Page 46: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Synthesis of Haptic Stimuli - Haptic Rendering

• Introduction• Perception of Haptic Stimuli• Synthesis of Haptic Stimuli

– Haptic devices– Haptic rendering

Page 47: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Rendering Loop

Interaction DevicePosition Sensing

Collision DetectionCollision Detection

Response ForceComputation

Force Feedback

Virtual ObjectDatabase

1kHzupdate

rate

Page 48: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Rendering Using Virtual Proxies

• The actual location of the haptic device is the haptic interface point (HIP)

• The virtual representation • The virtual representation which does not intersect the objects is a sphere known as the virtual proxy (VP)

• The haptic forces try to bring the HIP and VP together

Page 49: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Penalty-Based Haptic Rendering

• Check if the haptic interface point (HIP) penetrates the object

• If so, find the closest point on the surface

xF

the surface• Compute the force

proportional to the penetration

• The force is only a function of position, not of the trajectory taken

Courtesy M. C. Lin, UNC Chapel Hill

Page 50: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Problem with Penalty Methods

• Force discontinuities when crossing boundaries of internal Voronoi cells.

F1

F2

Unexpected force discontinuities (both in

magnitude and direction) are very disturbing!

Courtesy M. C. Lin, UNC Chapel Hill

Page 51: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Problem with Penalty Methods

• Pop-through thin objects.

After the mid line is crossed, the force helps popping

through.

motion

Courtesy M. C. Lin, UNC Chapel Hill

Page 52: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

HIP(t)

Perform collision detection between the

path of the HIP and path of the HIP and the C-obstacles

HIP(t+∆t)

Courtesy M. C. Lin, UNC Chapel Hill

Page 53: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

Set the subgoal and the constraint plane(s)

HIP(t)

HIP(t+∆t)

Courtesy M. C. Lin, UNC Chapel Hill

Page 54: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

Find a new subgoal using the active planes

HIP(t)

Courtesy M. C. Lin, UNC Chapel Hill

HIP(t+∆t)

Page 55: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

Since the subgoal is in free space, drop the constraints,

set the HIP as the new subgoal and perform

collision detection between

HIP(t)

collision detection between the path and the C-obstacles

Courtesy M. C. Lin, UNC Chapel Hill

HIP(t+∆t)

Page 56: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

HIP(t)

Recompute subgoal with new constraintswith new constraints

Courtesy M. C. Lin, UNC Chapel Hill

HIP(t+∆t)

Page 57: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

The path to the new subgoal intersects

another plane, so this is added to the set of

HIP(t)

is added to the set of constraints

HIP(t+∆t)

Page 58: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Using Constraint Planes

Compute active constraints (in 2D

there are only 2) and find subgoal

HIP(t)

The final position of the virtual proxy

HIP(t+∆t)

Page 59: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Shading

• Because objects are polygons, shading is used to smooth the effects of vertices

• Similar approach to Phong shading: Unshadedshading:– Linearly interpolate between

normals between vertices– User-supplied normals

• However, the distance between the HIP and VP can increase– Puts energy in– Users didn’t notice this

Unshaded

Shaded

Page 60: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Rendering Friction

• Friction is controlled by the movement of the VP

• Static friction:– Force is proportional to the

distance between the VP and the HIPHIP

– VP does not move if the component of the force tangential to the plane lies within the friction cone

• Dynamic friction:– Movement of VP determined by

Coulomb and dynamic friction equations

Page 61: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Rendering Surface Stiffness

• If the surface is not perfectly stiff, it will deform under applied pressure

• The VP is moved closer to • The VP is moved closer to the HIP according to the affine combination

Page 62: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Haptic Textures

• Haptic textures correspond to fine-grained roughness of the surface

• Can be achievd with:• Can be achievd with:– Image-based textures which

specify friction, stiffness and normal properties over the surface

– Add additional constraint planes to reflect sharp discontinuities

Page 63: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Limitations in Frequency and Spatial Response

• Even though haptic loops run at a high update rate, there are problems:– Stability in the control loops means that the

spring stiffness between the VP and HIP is fairly lowfairly low

– Mechanical compliance of the haptics device means that gratings as coarse as 10mm become distorted and the effective bandwidth is only a few tens of hertz

• Therefore, haptic displays are poor at exhibiting high frequency behaviours– Fine-grained textures– Tapping is a common exploration strategy

• Most surfaces feel spongy

Page 64: Simon Julier Department of Computer Science University ... · Event-Based Haptics • Detect when a collision event occurs • Play pre-recorded force profiles in an open-loop manner

Event-Based Haptics

• Detect when a collision event occurs

• Play pre-recorded force profiles in an open-loop profiles in an open-loop manner

• Scale the amplitude of the response with the impact speed