the light field stereoscope | siggraph 2015

The Light Field Stereoscope:Immersive Computer Graphics via Factored

Near-Eye Light Field Displays with Focus Cues

Fu-Chung Huang1,2 Kevin Chen1 Gordon Wetzstein1

1Stanford University2Now at NVIDIA Research

Top View

Vergence & Accommodation Match!

Left Eye Right Eye

(Rotation) (Focus)R

eal World

Parallax Over Pupil

Top View

HMDScreen

Stereo H

MD

Vergence & Accommodation Mismatch!

Left Eye Right Eye Real W

orld

No Parallax!

Parallax Over Pupil

LensesDisplaysDrivers

………

$20$70$60

On ebay:

DIY Light Field Stereoscope

Instructions: www.computationalimaging.org

Image Courtesy of SONY, OCULUS VR, HTC, Samsung, Razer, Zeiss

Gaming

Image Courtesy of Oculus VR

Image Courtesy of Jaunt VR

Image Courtesy of IIN Engineering

Image Courtesy of RCSB.org

Image Courtesy of INITION

Da vinci surgical system

Image Courtesy of INTUITIVE SURGIVAL

HEALTH and SAFETY WARNING

HEALTH and SAFETY WARNING

focus at the object’s distance

Vergence & accommodation are coupled!

Vergence & accommodation are coupled!

Immersive HMD Magnified Display

Immersive HMD Magnified Display

Decoupled!

Immersive HMD Magnified Display

!@#$$^R$%%&$

discomfort and fatigue, eyestrain, diplopic vision, headaches, nausea and …

0.001

0.01

0.1

1

1m 10m 100m 1000m 10000m

Personal Action Vista

StereopsisVergence

Accommodation

Motion Parallax

Relative Height

ArialPerspective

Dep

th C

ontra

st

Distance [Cutting and Vishton 1995]

Current HMD

Depth Cues

Image Courtesy of Nimble VR

Backlight

Thin Spacer & 2nd panel(6mm)

Magnifying Lenses

LCD Panel

Front Focusing

Rear Focusing

StereoscopeWheatstone, Brewster

1838

VR/ARIvan Sutherland

1968

VR explosionOculus, Sony, Valve, MS, …

Now Future

Future HMDs

ResolutionLatency

FOVForm Factor

…

Accommodation

StereoscopeWheatstone, Brewster

1838

VR/ARIvan Sutherland

1968

VR explosionOculus, Sony, Valve, MS, …

Now Future

Future HMDs

[Lanman and Luebke 2013]

[Hu and Hua 2014]

Accommodation

Object emits natural light rays

Light field enters eyes

Layered3D, Tensor Display[Wetzstein et al. 2011, 12]

Small Angle

Stationary

TV vs. HMD• Reduced brightness problematic• Shared: Need wide viewing angle

– High Rank– Multiple Layers– Temporal Multiplexing

• Viewer adapts to reduced brightness• Personal: Small viewing angle

– Low Rank– 2 Layers– Native Refresh Rate

Image Courtesy of ShutterStock.com

Low-rank, 2-panelNo temporal multiplexing

Both binocular & focus cuesto form 3D perception

Low-rank, 2-panelNo temporal multiplexing

Target Light Field

Input: 4D light field for each eye

Model Courtesy of Bushmills Irish Whiskey

Multiplicative Two-layer Modulation

1

𝑙 (𝑥 ,𝑣 )

𝑥

𝑣

𝑡 1𝑡 2

𝑙 (𝑥 ,𝑣 )=𝑡1 (𝑥 )𝑡 2(𝑥− (𝑣− 𝑥)𝑑𝑒 ) 𝑑𝑒

Model Courtesy of Bushmills Irish Whiskey

Input: 4D light field for each eye

Multiplicative Two-layer Modulation

1𝑡 1𝑡 2

𝑙 (𝑥 ,𝑣 1 )=𝑡 1 (𝑥 )𝑡 2(𝑥− (𝑣1−𝑥)𝑑𝑒 ) 𝑑𝑒

Model Courtesy of Bushmills Irish Whiskey

Input: 4D light field for each eye

Multiplicative Two-layer Modulation

1𝑡 1𝑡 2

𝑙 (𝑥 ,𝑣 4 )=𝑡1 (𝑥 )𝑡 2(𝑥− (𝑣 4−𝑥 )𝑑𝑒 ) 𝑑𝑒

Model Courtesy of Bushmills Irish Whiskey

Input: 4D light field for each eye

Multiplicative Two-layer Modulation

1𝑡 1𝑡 2

𝑙 (𝑥 ,𝑣 7 )=𝑡1 (𝑥 ) 𝑡2(𝑥− (𝑣 7−𝑥)𝑑𝑒 ) 𝑑𝑒

Model Courtesy of Bushmills Irish Whiskey

Input: 4D light field for each eye

Multiplicative Two-layer Modulation

𝑡 1𝑡 2

Model Courtesy of Bushmills Irish Whiskey

Input: 4D light field for each eye

Parallax over the Pupil

Multiplicative Two-layer Modulation

𝑡 1𝑡 2

~l=(ϕ1 𝑡 1)o (ϕ2 𝑡 2)Reconstruction:

Model Courtesy of Bushmills Irish Whiskey

Input: 4D light field for each eye

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒‖𝛽 l− (ϕ 1𝑡 1 )o (ϕ2 𝑡 2 )‖2{𝑡 1, 𝑡 2 } 𝑠 .𝑡 .0≤𝑡 1 , 𝑡 2≤1

𝑡1←𝑡 1oϕ1𝑇(𝛽 l o (ϕ2 𝑡2))ϕ1𝑇 ( ~l o (ϕ2 𝑡2 ) )+𝜖

for layer t1[Wetzstein et al 2012]

Input light field to the eyeMultiplicative Two-layer Modulation

𝑡 1𝑡 2

Left Eye: Front Panel Left Eye: Rear Panel

Front Panel Rear Panel

Output: 2 layers for each eye

Model Courtesy of Bushmills Irish Whiskey

Front Focus Mid Focus Rear Focus

Front Object Mid Object Rear Object

Multiplicative Two-layer Modulation

𝑡 1𝑡 2

Front Panel Rear Panel Model Courtesy of Bushmills Irish Whiskey

MultiplicativeMultilayer

AdditiveMultilayer

+

*

Multilayer Displays

Akeley et al. [2004]

Love et al. [2009]Narain et al.[2015]

Opt

ical

Ove

rlay

Tem

pora

lM

ultip

lexi

ng

Central Light Field

Right Light Field

Left Light Field

With Additive Multilayer

Left Light FieldA

dditi

veZoom-In

Mul

tiplic

ativ

e

Model Courtesy of Paul H. Manning

IMPLEMENTATION&

RESULTS

The Hardware- Adafruit based 3D printed housing learn.adafruit.com/3d-printed-wearable -video-goggles

- Aspheric lens, 2” diameter, 5cm focal length, $10 each

- TOSHIBA LTD056EV7F, 5.6” 1280x800, $60 each- CHIMEI INNOLUX N070ICG-LD1, 7” $35 each (need to replace polarizer!)

- PCB800099 driver board, $30 each

E-Tech DemoSouth Hall G, Booth ET18

The Hardware

Content Generation• 5x5x2 Stereo Light Field

• GPU Factorization

• 3 ~ 5 Iterations per frame• 25(views) x 640x800 x 2(eyes) • 5 ~ 10ms per iterations Left eye Right eye

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Model Courtesy of ixlrlxi

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Model Courtesy of ixlrlxi

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Model Courtesy of Utah 3D Repository

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Model Courtesy of Utah 3D Repository

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Model Courtesy of Paul H. Manning

Photograph Results

Traditional HMDs- No Focus Cues

The Light Field HMDStereoscope

Model Courtesy of Paul H. Manning

Designing Real-world ExperienceLeft eye Right eye

Fron

tFo

cusin

gRe

arFo

cusin

g

Technical Details in the Paper

Good Parameter Space

Limited by Diffraction

100 200 300 400 500

2

3

5

6

4

Front Panel PPI

Num

ber o

f Vie

ws

Minimum Views to Support Accommodation

IPD / 2 * M2

IPD / 2 * M1

Center

Center

Diffraction Analysis Light Field Distortion Asymmetric Image Formation

Discussion• Reduction in brightness

• Latency: 5x5x2 views

• Diffraction

Future Work

Image Courtesy of Jaunt and Google

Light Field Content Human Vision/Perception

Emerging Technologies

South Hall G, Booth ET-18

www.computationalimaging.org/ sites.google.com/site/jonash123/

IPD / 2 * M2

IPD / 2 * M1

Center

Center

Top View Light Field Side View Light Field