vr inputs and outputs. output devices visual visual auditory auditory haptic haptic
TRANSCRIPT
VR Inputs and OutputsVR Inputs and Outputs
Output DevicesOutput Devices
VisualVisual AuditoryAuditory HapticHaptic
Visual DevicesVisual Devices
DimensionsDimensions• Stereoscopic/monoscopicStereoscopic/monoscopic• Image resolutionImage resolution• Field of viewField of view• Display techDisplay tech• Ergonomic factorsErgonomic factors• CostCost
Human EyeHuman Eye
126 million 126 million photoreceptorsphotoreceptors
Eye-gaze Eye-gaze technology not technology not useful yetuseful yet• CalibrationCalibration• ErrorError• Overlaying controlOverlaying control
IPDIPD• 53-75 mm53-75 mm
Head-tracked DisplaysHead-tracked Displays
Displays in which the user’s head is Displays in which the user’s head is tracked and the image display screen tracked and the image display screen is located at a fixed location in is located at a fixed location in physical space.physical space.
Examples?Examples?• CRTCRT• Virtual Workbench or ImmersaDeskVirtual Workbench or ImmersaDesk• CAVECAVE• Many large screen displaysMany large screen displays
CRT HTD (Fishtank VR)CRT HTD (Fishtank VR)
3D Glasses
Head Tracker
Stationary Display
Stereoscopic DisplayStereoscopic Display
User
C
AB
Screen
C
A
B
Standard Display Stereoscopic Display
Virtual WorkbenchVirtual Workbench3D Glasses
3D Display
3D Object
CAVECAVE
Any Projected Large Stereoscopic ScreenAny Projected Large Stereoscopic Screen
CharacteristicsCharacteristics
LargeLarge Projection-Based (except for Fishtank Projection-Based (except for Fishtank
VR)VR) StereoscopicStereoscopic Head TrackedHead Tracked Stationary Display Screen(s)Stationary Display Screen(s) Let’s try to identify the pros and cons Let’s try to identify the pros and cons
and application domainsand application domains
3D Glasses
3D Display
3D Object
Large Screen ProjectionLarge Screen Projection
InfrastructureInfrastructure• Front Projection Front Projection
(user occlusion)(user occlusion)• Back Projection Back Projection
(takes up space)(takes up space)
Large Screen ProjectionLarge Screen Projection
AdvantagesAdvantages Viewer not isolatedViewer not isolated CollaborationCollaboration Minimal physical gearMinimal physical gear Good resolutionGood resolution Large field of viewLarge field of view
DisadvantageDisadvantagess One person “sweet-One person “sweet-
spot” spot” Real objects may Real objects may
occlude virtual objectsocclude virtual objects SynchronizationSynchronization LightLight
• ReflectionReflection• BleedingBleeding
Equipment footprintEquipment footprint AlignmentAlignment
Head-Mounted DisplayHead-Mounted Display
Visually Coupled Systems Visually Coupled Systems
Operator uses natural Operator uses natural visual and motor skills visual and motor skills for system controlfor system control
Basic ComponentsBasic Components• An immersive visual An immersive visual
display (HMD, large display (HMD, large screen projection screen projection (CAVE), dome (CAVE), dome projection)projection)
• Tracking system for Tracking system for head and/or eye motionhead and/or eye motion
HMD Optical SystemHMD Optical System
Image SourceImage Source• CRT or Flat Panel (LCD)CRT or Flat Panel (LCD)• Video, See–Through or Video-Pass ThroughVideo, See–Through or Video-Pass Through
HMD Optical HMD Optical SystemSystem
Image SourceImage Source• CRT or Flat Panel (LCD)CRT or Flat Panel (LCD)• Video, See–Through or Video, See–Through or
Video-Pass ThroughVideo-Pass Through
Head-Mounted Displays Optical Head-Mounted Displays Optical SystemSystem
Mounting ApparatusMounting Apparatus• What are some factors?What are some factors?
EyeglassesEyeglasses WeightWeight EarphonesEarphones TrackersTrackers
Field of ViewField of ViewMonocular FOV is the angular subtense (usually expressed in degrees) of the displayed image as measured from the pupil of one eye.
Total FOV is the total angular size of the displayed image visible to both eyes.
Binocular(or stereoscopic) FOV refers to the part of the displayed image visible to both eyes.
FOV may be measured horizontally, vertically or diagonally.
Focal Length & DiopterFocal Length & Diopter
Focal LengthFocal Length - The - The distance from the distance from the surface of a lens at surface of a lens at which rays of light which rays of light converge.converge.
Diopter Diopter - The power - The power of a lens. Equal to of a lens. Equal to 1/(focal length of the 1/(focal length of the lens measured in lens measured in meters)meters)
OcularityOcularityOcularityOcularity MonocularMonocular - HMD - HMD
image goes to only image goes to only one eye.one eye.
Biocular Biocular - Same HMD - Same HMD image to both eyes.image to both eyes.
Binocular Binocular (stereoscopic) (stereoscopic) - - Different but matched Different but matched images to each eye.images to each eye.
IPDIPD
Interpupillary Interpupillary Distance (IPD)Distance (IPD)
IPD is the horizontal IPD is the horizontal distance between a distance between a user's eyes.user's eyes.
IPD is the distance IPD is the distance between the two between the two optical axes in a optical axes in a binocular view binocular view system.system.
Vignetting and Eye ReliefVignetting and Eye Relief
VignettingVignetting The blocking or The blocking or
redirecting of light rays redirecting of light rays as they pass through as they pass through the optical system.the optical system.
Eye Relief DistanceEye Relief Distance Distance from the last Distance from the last
optical surface in the optical surface in the HMD optical system to HMD optical system to the front surface of the the front surface of the eye.eye.
Basic Eye Basic Eye
Cornea
CrystallineLens
Fovea
RetinaOpticNerve
Properties of the EyeProperties of the Eye
Approximate Field of ViewApproximate Field of View• 120 degrees vertical120 degrees vertical• 150 degrees horizontal (one eye)150 degrees horizontal (one eye)• 200 degrees horizontal (both eyes)200 degrees horizontal (both eyes)
AcuityAcuity• 30 cycles per degree (20/20 Snellen 30 cycles per degree (20/20 Snellen
acuity).acuity).
Simple FormulasSimple Formulas
Visual Resolution in Cycles per degree Visual Resolution in Cycles per degree (V(Vresres) = Number of pixels /2 (FoV in ) = Number of pixels /2 (FoV in degrees)degrees)
Example:Example: (1024 pixels per line)/(2*40 (1024 pixels per line)/(2*40 degrees) = Horizontal resolution of degrees) = Horizontal resolution of 12.8 cycles per degree12.8 cycles per degree
To convert to Snellen acuity (as in To convert to Snellen acuity (as in 20/xx)20/xx)
VVresres = 600/xx (20/47) = 600/xx (20/47)
Optical SystemOptical System
Move image to a distance that can Move image to a distance that can be easily accommodated by the eye.be easily accommodated by the eye.
Magnify the imageMagnify the image
Simple Magnifier HMD DesignSimple Magnifier HMD Design
p
q
Eyepiece(one or more
lenses)Display
(Image Source)
Eye f Image
1/p + 1/q = 1/f where1/p + 1/q = 1/f wherep = object distance (distance from image source to eyepiece)p = object distance (distance from image source to eyepiece)q = image distance (distance of image from the lens)q = image distance (distance of image from the lens)f = focal length of the lensf = focal length of the lens
Virtual Image Virtual Image
Lens Display
VirtualImage
ResolutionResolution
(low) 160 x 120 color pixels per eye(low) 160 x 120 color pixels per eye (current low) 640x480 – 1024x768(current low) 640x480 – 1024x768 (high) 1000+ x 1000+(high) 1000+ x 1000+
Note that resolution and FOV are Note that resolution and FOV are independentindependent
Another important factor: pixel densityAnother important factor: pixel density• Pixels per degree of FOVPixels per degree of FOV
How can we use the make up of the eye to How can we use the make up of the eye to better leverage resolution?better leverage resolution?
Basic Eye Basic Eye
Cornea
CrystallineLens
Fovea
RetinaOpticNerve
LEEP OpticsLEEP Optics
Large Expanse Large Expanse Extra PerspectiveExtra Perspective
Very wide FOV for Very wide FOV for stereoscopic imagesstereoscopic images• Higher resolution in Higher resolution in
the middle of FOVthe middle of FOV• Lower resolution on Lower resolution on
the peripherythe periphery Pincushion Pincushion
distortiondistortion
Fresnel LensFresnel Lens
A lens consisting of a A lens consisting of a concentric series of concentric series of simple lens sectionssimple lens sections
Result is a thin lens with a Result is a thin lens with a short focal length and short focal length and large diameterlarge diameter
More even resolution More even resolution distributiondistribution
Less distortionLess distortion from lanternroom.comfrom lanternroom.com
Distortion in LEEP OpticsDistortion in LEEP Optics
A rectangle Maps to this
How would you correct this?
To correct for distortionTo correct for distortion Predistort imagePredistort image This is a pixel-based This is a pixel-based
distortiondistortion Graphics rendering uses Graphics rendering uses
linear interpolation!linear interpolation! Too slow on most systemsToo slow on most systems Pixel shaders!Pixel shaders! Render to TextureRender to Texture
Distorted Field of ViewDistorted Field of View
Your computational model (computer Your computational model (computer graphics) assumes some field of graphics) assumes some field of view.view.
Scan converter may over or Scan converter may over or underscan, not all of your graphics underscan, not all of your graphics image may appear on the screen.image may appear on the screen.
Are the display screens aligned Are the display screens aligned perpendicular to your optical axis?perpendicular to your optical axis?
Distance alongz-axis
Distorted FoV Distorted FoV (cont.)(cont.)
Compound Microscope HMD Compound Microscope HMD DesignDesign
Relay lens produces a real image of the display Relay lens produces a real image of the display image source (screen) at some intermediate image source (screen) at some intermediate location in the optical train. The eyepiece is then location in the optical train. The eyepiece is then used to produce an observable virtual image of used to produce an observable virtual image of this intermediate image.this intermediate image.
Relay Lens
ImageIntermediateReal Image
Eyepiece
ExitPupil
Exit PupilExit Pupil The area in back of the optics from which The area in back of the optics from which
the entire image can be seen. Important if the entire image can be seen. Important if IPD not adjustable.IPD not adjustable.
Compound microscope optical systems Compound microscope optical systems have a real exit pupil.have a real exit pupil.
Simple magnifier optical systems do not Simple magnifier optical systems do not have an exit pupil.have an exit pupil.
Relay Lens
ImageIntermediateReal Image
Eyepiece
ExitPupil
Virtual Research V8 HMDVirtual Research V8 HMD DisplayDisplay
• Dual 1.3” diagonal Dual 1.3” diagonal Active Matrix LCDActive Matrix LCD
• Resolution per eye: 640 Resolution per eye: 640 x 480x 480
• focal length = 1mfocal length = 1m OpticalOptical
• Field of view: 60° Field of view: 60° diagonaldiagonal
SolveSolve• What is the cycles per What is the cycles per
degree?degree?• What is its horizontal What is its horizontal
and vertical field of and vertical field of view?view?
Pros/ConsPros/Cons
Characteristics of Characteristics of HMDsHMDs
ImmersiveImmersive• You are inside the computer You are inside the computer
worldworld• Can interact with real world Can interact with real world
(mouse, keyboard, people)(mouse, keyboard, people)• Mask out real world (including Mask out real world (including
body)body) ErgonomicsErgonomics
• Headborn weightHeadborn weight• Length of useLength of use• Cue conflict Cue conflict
accommodation vs. parallaxaccommodation vs. parallax PerspectivePerspective
• Resolution and field of viewResolution and field of view• TetheredTethered• AvatarsAvatars
Multi-tile and see-through HMDsMulti-tile and see-through HMDs
Hand Mounted DisplaysHand Mounted Displays
BinocularsBinoculars• 1280x10241280x1024
$19900$19900
Floor Supported Floor Supported DisplaysDisplays
Articulated mechanical arm Articulated mechanical arm • Offload weight Offload weight • 0.2 ms Latency0.2 ms Latency• High accuracyHigh accuracy• 0.1 degree orientation error0.1 degree orientation error
Pole in the wayPole in the way Limited Space (6’ diam, 3’ Limited Space (6’ diam, 3’
hgt)hgt) Good FOVGood FOV Good resolution Good resolution
(1280x1024)(1280x1024) Fakespace Boom3CFakespace Boom3C WindowsVRWindowsVR
• DifferencesDifferences StereoStereo TrackingTracking
• AdvantagesAdvantages
Desk Supported DisplaysDesk Supported Displays Fishtank VRFishtank VR Autostereoscopic Autostereoscopic
DisplaysDisplays Addresses weight Addresses weight
fatiguefatigue Tracking?Tracking? DTI 2018XL Virtual DTI 2018XL Virtual
WindowWindow Elsa Ecomo4DElsa Ecomo4D
Monitor Large Monitor Large Volume DisplaysVolume Displays
Multiple peopleMultiple people Monitor Based LVDMonitor Based LVD
• Fishtank VRFishtank VR• With limited FOV, what With limited FOV, what
is a possible solution?is a possible solution? Exaggerate trackingExaggerate tracking More monitors (synch, More monitors (synch,
bezel) bezel) Most use active stereo Most use active stereo
(shutterglasses)(shutterglasses)• 29-32% transmittance29-32% transmittance• $100-800$100-800• Can be used for Can be used for
prolonged periodsprolonged periods
Projector LVDProjector LVD WorkbenchWorkbench CAVECAVE
• $300-500k with SGI$300-500k with SGI• $100k with PC clusters$100k with PC clusters• Issues?Issues?
Large wall displaysLarge wall displays
Sound DisplaysSound Displays What are good goals for VR Sound DisplaysWhat are good goals for VR Sound Displays Compare importance to:Compare importance to:
• MoviesMovies• Video GamesVideo Games
What role does sound play?What role does sound play?• InteractivityInteractivity• ImmersionImmersion• Perceived image quality(!)Perceived image quality(!)
DimensionsDimensions• Mono/Stereo/”virtual sound”Mono/Stereo/”virtual sound”
TrackerTracker• OcclusionOcclusion• What does it take to uniquely place a sound source?What does it take to uniquely place a sound source?
3D Sound3D Sound Azimuth CuesAzimuth Cues
• Interaural Time DifferenceInteraural Time Difference head radius/(speed of sound(head radius/(speed of sound(θθ+sin +sin θθ))))
• Interaural Intensity DifferenceInteraural Intensity Difference High frequencyHigh frequency Head shadow effectHead shadow effect
Elevation CuesElevation Cues• How are ears modeled?How are ears modeled?• If simple holes what is the problem?If simple holes what is the problem?• Pinna effects sound propagationPinna effects sound propagation
Frequency attenuation and amplificationFrequency attenuation and amplification
3D Sound3D Sound Range CuesRange Cues
• IntensityIntensity• Motion ParallaxMotion Parallax
As the user moves his/her head, the As the user moves his/her head, the changes in azimuthchanges in azimuth
Head Response Transfer FunctionsHead Response Transfer Functions• Two microphones at the earsTwo microphones at the ears• Record sounds (HR impulse Record sounds (HR impulse
responses)responses)• Corresponding Fourier transforms Corresponding Fourier transforms
(HRTFs)(HRTFs)• Depends on sound frequencyDepends on sound frequency• No two people are exactly the sameNo two people are exactly the same
Given HRTFsGiven HRTFs• Take soundTake sound• Apply filter Apply filter • Get two signals that Get two signals that shouldshould
replicate the sound to the userreplicate the sound to the user However what’s the effect of using However what’s the effect of using
another person’s HRTF?another person’s HRTF?
Sound Display HardwareSound Display Hardware Offload compuational loadOffload compuational load ConvolvotronConvolvotron
• Crystal RiverCrystal River• First virtual 3D sound First virtual 3D sound
GeneratorGenerator• NASA 1988NASA 1988• Real-time DSPReal-time DSP• Calculate new HRTFs givenCalculate new HRTFs given
SoundSound Head position (tracker)Head position (tracker)
AudigyAudigy Soundblaster EAX/ Aureal A3DSoundblaster EAX/ Aureal A3D
• Simple geometrySimple geometry• Occlusion, reflectionOcclusion, reflection
Tracking?Tracking? High-end SDKs - $14kHigh-end SDKs - $14k
Speaker LayoutsSpeaker Layouts
StereoStereo Quad speakersQuad speakers 5.1 surround sound (6.1/7.1)5.1 surround sound (6.1/7.1) Multiple users?Multiple users?
Haptic FeedbackHaptic Feedback Hapthai – touchHapthai – touch Greatly improves realismGreatly improves realism When is it needed?When is it needed?
• Other cues occluded/obstructedOther cues occluded/obstructed• Required for task performanceRequired for task performance• High bandwidth!High bandwidth!
Why are hands and wrist the most Why are hands and wrist the most important?important?• High density of touch receptorsHigh density of touch receptors• Full body are still in research and Full body are still in research and
not very usable (back, body)not very usable (back, body) Two kinds of feedbackTwo kinds of feedback
• Touch Feedback – information on Touch Feedback – information on surface geometry, roughness, surface geometry, roughness, temperature, etc. Does not resist temperature, etc. Does not resist user contactuser contact
• Force Feedback – information on Force Feedback – information on compliance, weight, and inertia. compliance, weight, and inertia. Actively resists contact motionActively resists contact motion
Passive HapticsPassive Haptics
Not controlled by Not controlled by systemsystem
ProsPros• CheapCheap• Large scaleLarge scale• AccurateAccurate
ConsCons• Not dynamicNot dynamic• Limited useLimited use
Active HapticsActive Haptics Actively resists contact Actively resists contact
motionmotion Dimensions?Dimensions?
• Force resistanceForce resistance• Frequency ResponseFrequency Response• Degrees of FreedomDegrees of Freedom• LatencyLatency• IntrusivenessIntrusiveness• SafetySafety• ComfortComfort• PortabilityPortability
Human Haptic System SensorsHuman Haptic System Sensors Meissner and PacinianMeissner and Pacinian
• Quick (high frequency) Quick (high frequency) sensors (50-300 Hz)sensors (50-300 Hz)
• Vibration, accelerationVibration, acceleration• Low resolutionLow resolution
Merkel and Ruffini - Slow Merkel and Ruffini - Slow (low frequency) adapting (low frequency) adapting (0-10 Hz)(0-10 Hz)• Constant, surfaces, edgesConstant, surfaces, edges• High resolutionHigh resolution
Thermoreceptors – Thermoreceptors – temperaturetemperature
Proprioceptors/KinesthesiaProprioceptors/Kinesthesia• Perception of own body Perception of own body
motionmotion Page 96Page 96
Tactile Feedback InterfacesTactile Feedback Interfaces
Goal: Stimulate skin tactile receptorsGoal: Stimulate skin tactile receptors How?How?
• Air bellowsAir bellows• JetsJets• Actuators (commercial)Actuators (commercial)• Micropin arraysMicropin arrays• Electrical (research)Electrical (research)• Neuromuscular stimulations (research)Neuromuscular stimulations (research)
Tactile MouseTactile Mouse
Logitch iFeel Logitch iFeel MouseMouse
Electrical ActuatorElectrical Actuator• Shakes up and Shakes up and
down (do not down (do not disturb XY motion)disturb XY motion)
• Mouse over buttonsMouse over buttons Haptic BumpHaptic Bump Rumble PackRumble Pack
CyberTouch GloveCyberTouch Glove CyberGloveCyberGlove Immersion CorporationImmersion Corporation Six Vibrotactile actuatorsSix Vibrotactile actuators
• Back of fingerBack of finger• PalmPalm
Off-centered actuator Off-centered actuator motormotor
Rotation speed=frequency Rotation speed=frequency of vibration (0-125 Hz)of vibration (0-125 Hz)
When tracked virtual hand When tracked virtual hand intersects with virtual intersects with virtual object, send signal to glove object, send signal to glove to vibrateto vibrate
$15000$15000
Force Feedback DevicesForce Feedback Devices
Attempts to stop the user’s motion if Attempts to stop the user’s motion if necessarynecessary
Larger ActuatorsLarger Actuators GroundedGrounded Mechanical bandwidth Mechanical bandwidth - perceived - perceived
frequency of a haptic interaction (in frequency of a haptic interaction (in Hz)Hz)
Control bandwidthControl bandwidth – system – system control bandwidth to devicecontrol bandwidth to device
Force Feedback JoysticksForce Feedback Joysticks
WingMan Force 3DWingMan Force 3D Inexpensive ($60)Inexpensive ($60) Actuators that can Actuators that can
move the joystick move the joystick given system given system commandscommands
Max 3.3 N of forceMax 3.3 N of force Force feedback Force feedback
driving wheeldriving wheel
SensAble PHANToM ArmSensAble PHANToM Arm
3 and 6 DOF force feedback 3 and 6 DOF force feedback devicedevice
Most popular (1000+) in VR Most popular (1000+) in VR researchresearch
3 DC motors3 DC motors Little interitia/frictionLittle interitia/friction 6.4 N (1.7 continuous)6.4 N (1.7 continuous) Control loop = 1000 HzControl loop = 1000 Hz $1500 – Omni$1500 – Omni $16000 – standard Desktop$16000 – standard Desktop $56000 – 6 DOF high end$56000 – 6 DOF high end
Other Force FeedbackOther Force Feedback
Large ForcesLarge Forces• Haptic Master ArmHaptic Master Arm
250 N250 N
• Sarcos ArmSarcos Arm• http://www.princeton.ed
u/~jmelli/papers/ch4-haptics.pdf
GlovesGloves• CyberGraspCyberGrasp
$39000$39000
• CyberForceCyberForce $56000$56000
Why is it hard?Why is it hard?
Control bandwidthControl bandwidth Mechanical bandwidthMechanical bandwidth ForcesForces HygieneHygiene Gender differencesGender differences PortabilityPortability EncumbranceEncumbrance Solutions?Solutions?
• Specific engineeringSpecific engineering