virtual environments: tracking and interaction

1

Virtual Environments: Tracking and Interaction

Simon JulierDepartment of Computer ScienceUniversity College London

http://www.cs.ucl.ac.uk/teaching/VE

Outline

• Problem Statement:– Models of Interaction– Tracking Requirements

• Tracking Systems:– Hardware– Sources of errors

• Interaction:– Basic interaction– Locomotion – Selection & Manipulation

Problem Statement




2

Tracking and Interaction

User

Interface Devices

Computer

User

Synthetic Environment

Real Environment

Tracking and interaction happens

here

Basic Interaction Tasks

• Locomotion or Travel– How to effect movement through the space

• Selection– How to indicate an object of interest

• Manipulation– How to move an object of interest

• Symbolic– How to enter text and other parameters

Models of Interaction

• “Extended Desktop” Model– The user needs tools to do 3D tasks

• “Virtual Reality” Model– The user is using their body as an interface to the world– The system responds to everything they do or say

3

Extended Desktop Model

• Focus on analysing a task and creating devices that fit the task

• Study ergonomics of the device and applicability/suitability for the role

Limits of ED Model

• 3D tasks are quite complicated to perform

• Tasks can become very specialised

• Leads to a proliferation of real (and virtual) devices

Fakespace Cubic Mouse

Types of Device

Ascension Wanda

3DConnexion Spaceball

Polhemus Isotrak 3-Ball

Logitech 3D Mouse

3DConnexion Spacemouse

Inition 3DiStick

4

Virtual Reality Model

• Need to track the user precisely and interpret what they do

• Focus is on users exploring the environment

• Tension between magical and mundane responses of the environment– Mundane are where the world responds as

if it was controlled by laws of physics– Magical are everything else (casting spells,

automatic doors, etc…)

Limits of VR Model

• Can’t track user over very large areas– E.g. Some form of locomotion

metaphor will be required for long distance travel (see later)

• Physical constraints of systems• Limited precision and tracking

points• Lack of physical force feedback

Tracking System




5

Connection Between Interaction and Tracking

• Irrespective of interaction model, user must be instrumented in some way to convey information to the system

• This is carried out using the tracking system

Requirements for Trackers

• Resolution– Be able to detect small changes in the system

• Accuracy– The size of the range of the correct positions reported by the

system

• Sample Rate– The frequency the sensors are checked for new data. Sampling

rate must be greater than the data rate

• Data Rate– The no. of computed position/sec, the higher the rate, the more

desirable the system will be.


• Update rate– The rate new positions are reported to the host computer

• Lag– the delay between the new movement made and the new position

reported

• Range of operations– The area/range/volume in which the tracker can accurately report

the positions. E.g., the distance, the height. This is determined by the wire length, signal strength, etc.

6


• Robustness– The ability the tracker can cope with the amount of uncertainty and

noise. (e.g. water, metal, keys)

• Fitness for tracking multiple objects– Ability to independently determine the positions of multiple objects.

This is determined by the design of the system architecture.– Ability to cope with alteration caused by the one remote object onto

the other. For example, if one sensor is occluded by another sensor.

Types of Tracking Technology

• Many types of tracker are available– From ultrasonic, consumer devices ($10s) through to

very precise mechanical trackers ($100,000s)– Not all trackers are suited to all applications

• E.G. mechanical trackers aren’t that suitable for CAVEs since you see the device

– Cost is still a big problem if you want to track at a fine enough scale for head-tracked virtual reality

The Ideal Tracker

Magical, ideal tracker would have these characteristics:

• Tiny (transistor size)• Self-Contained• Complete (6 DoF)• Accurate (1mm position, 0.1 degree orientation)• Fast (1000Hz, <1ms latency)• Immune to occlusions (no line-of-sight requirement)• Robust (no interference)• No range limitation• Cheap

7

Tracking Technologies• 5 main types: mechanical, inertial, acoustic, optical,

magnetic. • Most can be classed as:

• Outside-In: user emits signal to indicate its location to the system

• Inside-Out: systems emits signal to user which senses location

Outside-in Inside-out

Mechanical Trackers

• First & simplest systems

• Use prior knowledge or rigid mechanical pieces and measurements from sensors.

• Typically boom-type tracked displays with counterweights.

Mechanical Trackers

• Some example systems

8

Mechanical Trackers

• Pros– Accurate– Low latency– Force-feedback– No Line of Sight or Magnetic Interference Problems

• Cons– Large & cumbersome– Limited range

Inertial Trackers

• 3 linear accelerometers measure acceleration vector• Rotated using current rotation matrix (orientation)

determine by gyroscopes

Inertial Trackers

• Pros– Small (chip form), self-contained.– Immune to occlusions– No interference– Low latency (typically <2ms)– High sample rate

• Cons– Drift is the show stopper– Accelerometer bias of 1 milli-g 4.5m drift after 30s– Close, but no silver-bullet

• High potential as part of hybrid systems…

9

Acoustic Trackers

• Uses sound waves for transmission and sensing

• Involves pulses at intervals• SONAR is best known,

determining time of a pulse• Uses ultrasound• Outside-In (microphone

sensors)• (Logitech Acoustic Tracker)• (Samba De Amigo Maracas)

Acoustic Trackers

• Pros– Very small so can be worn– Line of sight less of an issue than with optical systems– Better range than mechanical systems

• Cons– Size proportional to range– Environment considerations (temperature, humidity)– Acoustic issues can cause slow update rate (10Hz) (5-100ms)– Attenuation at desirable high frequencies (reduced interference)– Jingling of keys

Magnetic Trackers

• Measures changes in the magnetic field• Can be done by magnetometers (for

DC)• Or by induced current in an

electromagnetic field (for AC)• 3 sensors orthogonally arranged will

produce a 3D vector• In tracking, a multi-coil source unit with

each coil energised (excited) and when measured results in position and orientation.

• Compass: uses the earth’s naturally occurring DC magnetic field to determine heading, can be used here

• (Ascension spacePad)

10

Magnetic Trackers

• Pros– User-worn component small– No line of sight issues (magnetic fields go through us)– One source unit can excite many sensor units– Very low latency (~5ms)– Ability to track multiple users using a single source unit

• Cons– Field distortions (foreign objects, natural magnetic field)

• Requires some compensation– Jingling of keys (or anything magnetically conductive)– Need to wait for energised excitation of coil to subside before the

next one so update is slow– Jitter increases over distance from emitter/sensor

Optical Trackers

• Measures reflected or emitted light• Involves a source (active or passive) and

sensor• Sensors can be analogue or digital• Photo sensing (light intensity) or Image

forming (CCD)• Triangulation with multiple sensors• Possible to be both outside-in and inside-

out

Optical Trackers

• Pros– Analogue sensors with active light source gives high update and

spatial precision– Passive with image-forming sensors could be used in an

unaffected environment– Image forming sensors provide closed-loop feedback of real

environment and tracker

• Cons– Line of sight is critical– Target’s orientation harder to determine

11

Hybrid Trackers

• No single solution that suits all applications

– Many different approaches, each with advantages and limitations

– Can address the limitations by building hybrid systems which combine the advantages of each approach

• Inertial sensors have provided the basis for several successful hybrid systems due to their advantages

• Example, the VisTracker users an opto-inertial hybrid

Hybrid Tracking Algorithms

• Hybrid tracking is an example of a data fusionalgorithm:– Information from a set of disparate modalities– Fused together to provide consistent estimate

• Most common implementation is to use a Kalman filter

Kalman Filtering

• The Kalman filter is a recursive minimum mean squared error estimator

• It uses a predict-update cycle:

• This makes it possible to combine lots of types of information in an asynchronous manner

Initialize Predict Update

12

Fusing Multiple Measurements

t

Camera

t+50ms

Camera

Inertial

Update UpdatePredict

t+100ms

Prediction (using motion model)

Predict

Hybrid Trackers

• InterSense IS-900– Tracking system for VR-Walkthrough

applications– Inertial (orientation & position) &

Ultrasonic (drift correction) hybrid tracker which has highly accurate 6 degree of freedom tracking in a wide area.

– Features fast updates, low latency, filtering to reduce jitter and advanced prediction algorithms to reduce latency very smooth and precise

– The four sensors, including a head tracker, a hand tracker, a wand (with four buttons and an integrated joystick), and a stylus (with two buttons).

Tracking Errors

• Static Tracked Object– Misalignment– Spatial Distortion (inaccuracy)– Spatial Jitter (noise)– Creep

• Dynamic Tracked Object– Lag (time delay, tracker + subsystems complex relation)– Latency Jitter (variations in latency)– Dynamic Errors (other inaccuracies, e.g. prediction algorithms)

13

Tracking Errors of < 1 Degree Noticable

Misalignment

M

S

W

B

WM

WB

BS

SM• Referentials:– W: world– B: base (referential) of tracker– S: sensor of tracker– M: display (manipulator)

• Transformation (pose) AB: – Transformation that modify the

referential A into B – Pose of B with respect to A– 4x4 Homogeneous

transformation matrix – AB = (BA)-1 and AB = AC.CB

WM = WB.BS.SM

Spatial Distortion

• Repeatable errors at different poses in the tracking volumes

• Many factors including incorrect calibration and persistent environmental disturbances

14

Spatial Jitter

• These are caused by noises in the sensor

• Even with same noise on sensors, the jitter on pose estimates can change with the pose

• Hybrid sensors can improve the performance

“A General Method for Comparing the Expected Performance of Tracking and Motion Capture Systems” - Dannett Allen, Greg Welch

Creep

• Slow but steady changes in tracker output over time

• Caused by temperature drift or other similar “start up”transients in a system

Measurements from stationary gyro“Evaluation of a Solid State

Gyroscope for Robotics Applications”- Barshan and Durrant-Whyte

System Latency

• Mine, M. Characterization of end-to-end delays in head-mounted displays. Tech. Rep. TR93-001, Department of Computer Science, The University of North Carolina, 1993.

• Definition: End to end delay – Total time required for image displayed by HMD to

change in response to the movement of the user’s head.

15

Delays in HMD Pipeline

• Tracking system comprises– Physical sensing– Filtering on tracking device– Transmission delays (RS232, Ethernet, etc.)

• Application delay– Collision detection, interaction events, etc.

• Image generation– At roughly the display refresh rate

• Display system– Time taken to transfer and display an image

from Mine (1993)

Measuring delay

• Mine constructed a system to measure delay in HMD systems– Measurement at several points in pipeline.

Tracking Application Image generation Display

Tstart Treport Tdisplay Tdisplay+17ms

16

Measuring delay

from Mine (1993)

Results

• Tracking delays– Best had delays ~10ms.– Worst, delays of ~60ms.– More tracked units implies longer delay

• Application/Image generation– 55ms on average.– Although application delay was minimal.

• Display system delay– NTSC has delay of 16.67ms.

Tracking Summary

• Quite a complex and challenging problem– No real ideal solution

• Several tracking technologies exist with different levels of suitability based on the application in question. All of the technologies display both pros and cons. – The ultimate tracker will probably not be developed from a single

technology, but as a hybrid of these technologies.

• A VR application should provide the following:– High data rates for accurate mapping without lag– High tolerance to environmentally induced errors– Consistent registration between physical and virtual environments– Good sociability so that multiple users can move freely

17

Interaction









Direct Locomotion

• User walks from one part of the environment to another

• Intuitive, easy to use• Requires a great deal

of space

18

Constrained Walking

• User walks but motion is constrained– VirtuSphere– Treadmills

• However, most forms can be very difficult to use– Mismatch in perceptual cues– Dynamics / inertia of device

make it hard to navigate effectively

CirculaFloor

• Floor consists of a set of movable tiles

• As the user walks forwards, tiles move in front of the user’s feet to allow near infinite movement

CirculaFloor

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

19

Walking-in-Place

• User “walks in place”• Movement detected by

gait analysis• No perceptual

mismatch

Redirected Walking in the CAVE

• Problems with walking in the CAVE:– You eventually hit the walls– You can turn and see the

missing back wall• One means of countering

this is to rotate the environment– The user is directed back to

the front wall

Redirected Walking in the CAVE

• Apply a small rotation to the scene to cause user to turn towards centre– Sufficiently small that not consciously

noticed– Subject responds to maintain balance

• Increase rate when user is navigating or rapidly turning head

• Results:– Variance in number of times user saw back

wall decreased– Rates of simulator sickness were not

increased– Some users did not notice the rotation

20






Locomotion

• User points (somehow) in the direction of motion

• User presses a button

Selection and Manipulation

• User points at object with their hand

• User selects by pressing a button

• User grabs by pressing 2nd button– Object is rigidly

attached to hand coordinate system

21

Selection Only

• Occlusion selection

• Similar to selection with a mouse

• Put hand over object (occlude it) to select it

Locomotion

• Travel in Immersive Virtual Environments: An Evaluation of Viewpoint Motion Control Techniques, Bowman, Koller and Hodges

• One of the first rigorous studies of some of the trade-offs between different travel techniques

Taxonomy of Travel

Bowman, Koller and Hodges

22

Quality Factors

• 1. Speed (appropriate velocity)• 2. Accuracy (proximity to the desired target)• 3. Spatial Awareness (the user’s implicit knowledge of his position and

orientation within the environment during and after travel)• 4. Ease of Learning (the ability of a novice user to use the technique)• 5. Ease of Use (the complexity or cognitive load of the technique from

the user’s point of view)• 6. Information Gathering (the user’s ability to actively obtain

information from the environment during travel)• 7. Presence (the user’s sense of immersion or “being within” the

environment)

Experiment 1

• Absolute motion task– Gaze v. Point AND constrained v. unconstrained

• Note the immediate trade-offs with point and gaze– Bowman claimed expected gaze to be better

• Neck muscles are more stable• More immediate feedback

• Eight subjects, each doing four times 80 trials (five times 4 distances to target, four target sizes)

Experiment 1

• No difference between techniques

• Significant factors were target distance and size


23

Experiment 1

• No difference between techniques

• Significant factors were target distance and size


Experiment 2

• Relative motion task• No prior expectation

– Though there is an obvious one

• Need forward and reverse direction

• Nine subjects, four sets of 20 trials


Experiment 2

• Obvious difference• Can’t point at target

and look departure point simultaneously


24

Summary of 1st Two Experiments


Experiment 3

• Testing spatial awareness based on four travel variations– Constant speed (slow)– Constant speed (fast)– Variable speed (smooth acceleration)– Jump (instant translation)

• Concern is that jumps and other fast transitions confuse users

Experiment 3

• However, there was no main effect

• This is still worth further study


25

Other Locomotion Techniques

• Direct walking• Constrained movement• Redirected walking

Selection and Manipulation

• Moving Objects In Space: Exploiting Proprioception In Virtual-Environment Interaction, Mine, Brooks Jr. and Sequin

• One of the first papers to discuss a range of selection and manipulation tasks

Body-Relative Interaction

• Provides– Physical frame of reference in which to work– More direct and precise sense of control– “Eyes off” interaction

• Enables– Direct object manipulation (for sense of position of object)– Physical Mnemonics (objects fixed relative to body)– Gestural Actions (invoking commands)

26

Working within Arms Reach

• Takes advantage of proprioception• Provides direct mapping between hand motion and object

motion• Provides finer angular precision of motion

Ray-Based Interaction

• Ray-Based– Ray is centred on user’s

hand– All manipulations are relative

to hand motion• Translation in beam

direction is hard• Rotation in local object

coordinates is nearly impossible

Mark Mine, http://www.cs.unc.edu/~mine/isaac.html

Object-Centred Interaction

• Object-Centred– Select with ray as

before• Local movements of

hand are copied to object local coordinates

Mark Mine, http://www.cs.unc.edu/~mine/isaac.html

27

Go-Go Hand Interaction

• Arm stretches to reach object

• Amplifies local movements

Stretch Go-Go Hand Technique, Bowman & Hodges, based on Go-Go Hand from Pouyrev,

Billinghurst, Weghorst, Ichikawa

World in Miniature (WIM) Interaction

• Smaller version of the world created and superimposed on the real world

• User controls WIM using hanheld ball

• Can interact with environment by selecting 1:1 scale or same object on WIM

World in Miniature, Stoakley and Pausch

Scaled-World Grab

• Automatically scale world, so that selected object is within arms reach– Near and far objects easily moved– user doesn’t always notice scaling– dramatic effects with slight head movement

28

Mine, Brooks Jr, Sequin

Scaled-World Grab for Locomotion

• User transports himself by grabbing an object in the desired travel direction and pulling himself towards it

• User can view the point of interest from all sides very simply

• For exploration of nearby objects , virtual walking is more suitable; while going much further, invoking a separate scaling operation or switch to an alternate movement mode is better

Physical Mnemonics

• Storing of virtual objects and controls relative to user’s body1.Pull-down menus2.Hand-held widgets3.Field of View-Relative mode

switching

29

Pull-Down Menus

• Problems with virtual menus– Heads-up are difficult to manage– Fixed in world often get lost

• Could enable menu with ..– Virtual button (too small) – Physical button (low acceptability)

• Instead “hide” menus around the body, e.g. above FOV

Hand-Held Widgets

• Hold controls in hands, rather than on objects

• User relative motion of hands to effect widget changes


FOV-Relative Mode Switching

• Change behaviour depending on whether a limb is visible– Hand visible, use occlusion selection– Hand not visible, use ray selection

30

Gestural Actions

• Head – butt zoom• Look – at Menus• Two – handed flying• Over – the – shoulder

deletion


Experiment 1

• Align docking cube with target cube as quickly as possible

• Comparing three manipulation techniques– Object in hand– Object at fixed distance– Object at variable distance (scaled by arm extension)

Experiment 1

• 18 subjects• In hand was significantly faster


31

Experiment 2

• Virtual widget comparison• Comparing

– Widget in hand– Widget fixed in space

• 18 subjects (as before)• Performance measured by accuracy not time

Experiment 2

• Widget in hand was significantly better


Putting it All Together

QuickTime™ and a decompressor

are needed to see this picture.

32

Summary

• Tracking systems provide a way to model the user (VR model) or provide direct input to control system (EM model)

• A lot of work has been done and is being done in 3D interaction– Covered locomotion and selection & manipulation

• However it is still quite tedious to use most 3D user interfaces– Lack of precision is probably main problem

• However, people are able to interact

virtual environments: tracking and interaction

Documents