Input devices and interaction

Ruth Aylett

Contents

Tracking– What is available

Devices– Gloves, 6 DOF mouse,

WiiMote

Why is it important?

Interaction is basic to VEs– We defined them as ‘interactive in real-time’

No interaction => NOT a VE Ideal interaction:

– Very low latency - i.e fast– Multi-modal– Unencumbered– Intuitive

Technology falls well short of this of course

Tracking the human body Large displays require position and orientation of viewer’s

body to be tracked– tracking information fed to runtime system as input signal.

Most commonly tracked is head butsometimes also hands, arms,legs, eyes etc.– Head tracking used to update virtual

viewpoint orientations.

Body tracking needed for lifelikeinteraction with objects and creatures.– say user wishes to wave at another person

in the VE: their real-world motions can betracked and replicated in the VE.

Interaction types

Navigation– Staying on the ground?– Walking v flying

• Depends on size of model wrt display system• Degree of immersion

Interaction with other users– Gesture

Interaction with objects– Depends on the object and interaction– Select, lift, rotate, throw, steer, hit

Virtual Tennis

MovieVirtual Tennis

Tracking the human head

An essential basic requirement in immersive VRsystems.

Imagine axes mounted on top of your head– pans, tilts and yaws of head

measured around those axes.

HMDs often have rotation sensorsto measure these three angles.

Angles passed to run-time VRsoftware which updates viewingangles.

HMD

Tracking devices

Many tracking devices and systems developed over theyears– some aimed specifically at VR systems– others borrowed from other areas.

Some systems are portable and cheap - some requirepermanent installations in large rooms and are veryexpensive indeed.– Trackers can be magnetic, electro-magnetic, acoustic, inertial,

optical, or mechanical. Electro-magnetic trackers

– transmitter generates electromagnetic signals– received by a receiver (or sensor).– Signal strength used to determine absolute position and

orientation of receiver relative to transmitter.

Example: Polhemus FASTRAK

FASTRAK electro-magnetic sensor from Polhemus– accurately computes the position and orientation of tiny receiver

as it moves through space. Dynamic, real time six degree-of-freedom measurement

of position (X, Y, and Z) and orientation (yaw, pitch, androll)– RS-232 signal updated at 120 records/sec.

Transmitter constantly puts out a weak magnetic field.– passive receiver generates an electric signal as it is moved

through the field.

– Polhemus' processing electronics then amplify and analyse thissignal to determine the real-world position and orientation of thereceiver relative to the transmitter.

Polhemus FASTRAK system

Polhemous trackers well proven and widely used since thevery early 1990’s.

The FASTRAK system shownhere has one receiverand one transmitter.

System expandedby adding up to three morereceivers– can attach receivers to

different parts of body– log data for gait and limb

analysis or computeranimation.

Electromagnetic TrackingPolhemus

Electromagnetic TrackingAscension

Ascension market a number of systems based on DCrather than AC fields including Flock of Birds and a full

gait analysis system called MotionStar.

Electromagnetic TrackingAdvantages Small receivers Reasonably cheap Line-of-sight (LOS) not requiredDisadvantages Accuracy diminishes with distance Not very large working volume High latency due to filtering Transmitter/receiver required

Electro-magnetic interference

Major problem of electro-magnetic trackers– magnetic fields easily affected by the surrounding environment.

Large metal objects produce eddy currents in thepresence of the magnetic fields– These can interfere and distort the original signal causing

inaccurate measurements.– same effect appears near electric currents, such as in cabling– also ferromagnetic materials– Also electromagnetic sources such as computer monitors.

Ferromagnetic and/or metal surfaces cause fielddistortion

Ultrasonic trackers

Two main components– transmitter generating an ultrasound signal– receiver detecting the signal.

Distance is calculated bymeasuring time-of-flightof ultrasonic pulse.– Three transmitters and receivers

needed to calculate full 3Dposition and orientation.

Ultrasonic tracking used by Logitech Head Tracker(shown) and 3D mouse.

Ultrasonic trackers The Power Glove made by toy

company Mattel (who make Barbie)– introduced in 1989 for use with

the Nintendo EntertainmentSystem (NES).

Ultrasonic device for use in place ofstandard Nintendo controllers

Detected finger motion– Plus full set of buttons on the wrist.

In fact not much use for Nintendogamers– But amazingly advanced piece of VR

kit for its time.

Acoustic Tracking

Advantages Well known transducers (mics), lightweight Low cost deviceDisadvantages Line-of-sigh (LOS) required Echoes Low accuracy (speed of sound in air varies) Transmitter/receiver required

Inertial tracking systems

Very popular (because cheap)– based on inertial gyro technology– Detects acceleration and thus can calculate velocity (since mass

in known) giving 3DoF– Newish example is the Intersense IS-300.

Can be coupled with ‘add-on’ ultrasonic system to give 6DoF sensing– example of a hybrid technology tracker.

IS-300 can operate in metallicenvironments,– 6 DoF tracker operates only

in LoS of transmitter. Other examples:

Intersense Intertrax2 and the Ascension 3D-Bird.

Inertial TrackingAdvantages Cheap Small size No transmitter/receiver required LOS not requiredDisadvantages Only 3 DOF on their own Drift Not accurate for slow movements

Optical tracking methods

Many different forms– Often use image processing and pattern recognition and matching– Much work outside of VR: numerous ideas suitable for tracking

object position and pose

For example fiducial mark detection– light sources or reflective

colour markers attached toobject at important locations suchas joints or extremities.

Easier for image processingalgorithm to track in clutteredconditions.

How it is done

Optical tracking methods

Outside-in tracker– tracking apparatus is fixed– object to be tracked (e.g. the user) is viewed from the "outside".

Inside-out systems– take tracking measurements from the object to be tracked– for instance a camera can be mounted on the HMD– images analysed to produce pose and distance estimations based on

the position of fixed patterns within the environment.

Visible images or infra-red used. Many optical systems (but not all!) are one-offs, expensive

and require careful calibration procedures.

Infra-red cameras

Optical Tracking

Advantages Can work over a large area. Inherently wirelessDisadvantages LOS needed Transmitter/receiver required Expensive Requires computer vision technology

Eye trackers

Eye tracking systems are examples of optical trackingdevices.– viewpoint in the virtual world follows

the gaze of user’s eye.

Originally developed as a mousereplacement– simply look at object– interact through eye movement

(such as a slow blink).

Support physically impaired users. Combined eye and head tracking systems

• also exist - use in practice is complicated.

Mechanical trackers

Mechanical linkage system– arm-like structure of several joint, one end fixed, the

other free to move with the user.

Measure position and angular orientation of freeend– by measuring angles at each joint and

factoring in length of each segment.

Fake Space BOOM (right)

Mechanical TrackingAdvantages Simple sensors, no need for transmitter/receiver low-cost device very low latency High positional accuracyDisadvantages The user is tethered Lots of inertia Typically small working volume Mechanical parts wear out

Unencumbered tracking

Depends on identifying hand/hand onvideo– One approach is using blobs

Cybergloves and similar

Inherent in the folklore and hype of VR is the cyberglove- a wearable device that monitors the the position andorientation of hand and fingers.

The name CYBERGLOVE® isregistered by Virtual Technologies Inc(VTi).– uses 18 or 22 patented

angular sensors for tracking theposition of fingers and hand.

GlovesVirtual Technologies

CyberGlove - 18-sensor model - 22-sensor modelVariants are: - CyberTouch - CyberGrasp

GlovesFifth Dimensions Technologies - Data Glove

Data Glove finger flexure hand orientation -roll & pitch

Gloves

Fakespace - Pinch GlovePinch Glove gesture recognition reliable low cost electrical sensors in

each fingertip contact among any 2 or more

digits

Mouse as input device in VR

Normal 2D mouse can be used (as in Cortonafor example).– Need user selectable modes to switch between

DoF’s. More sophisticated mice provide 3 or more

DoF: these include the Spaceball (shownhere) and Spacemouse.

Standard games joysticks orgamepads also usedto give 2 or more DoF’s.

6 DOF “Mice”

3 translation DOF 3 rotation DOF

6 DOF “Mice”Spaceball

byLabtec

Spacemouse by DLR (Logitech - USA)

6 DOF “Mice”

CyberpuckSpaceOrb

The WiiMote 3 accelerometers

– Enough for 6 DOF– But will drift– Bluetooth connection

to 10m Optical (IR) sensor

– To 5m from sensorbar

– Triangulation fromends of bar

– Allows accuratepointing

Speaker

WiiMote interaction

Head-tracking– WiiMote stationary, head-mounted IR source

Finger-tracking - touch-free interaction– IR tape on finger + fixed IR source

Gesture recognition– Using accelerometers– Feature classification– Fast movements work better; beware variable arm

orientation

Software

Free libraries– WiiGLE

• http://mm-werkstatt.informatik.uni-augsburg.de/documents/WiiGLE/doku.php

• Provides a set of classifiers– WiiGee

• http://www.wiigee.org/index.html• Java-based, one classifier

Issues with Bluetooth stacks– Flakey implementations, especially Vista– BlueSoleil seems a good driver

• http://www.bluesoleil.com/