the light field stereoscope | siggraph 2015
TRANSCRIPT
The Light Field Stereoscope:Immersive Computer Graphics via Factored
Near-Eye Light Field Displays with Focus Cues
Fu-Chung Huang1,2 Kevin Chen1 Gordon Wetzstein1
1Stanford University2Now at NVIDIA Research
Top View
Vergence & Accommodation Match!
Left Eye Right Eye
(Rotation) (Focus)R
eal World
Parallax Over Pupil
Top View
HMDScreen
Stereo H
MD
Vergence & Accommodation Mismatch!
Left Eye Right Eye Real W
orld
No Parallax!
Parallax Over Pupil
LensesDisplaysDrivers
………
$20$70$60
On ebay:
DIY Light Field Stereoscope
Instructions: www.computationalimaging.org
Image Courtesy of SONY, OCULUS VR, HTC, Samsung, Razer, Zeiss
Gaming
Image Courtesy of Oculus VR
Image Courtesy of Jaunt VR
Image Courtesy of IIN Engineering
Image Courtesy of RCSB.org
Image Courtesy of INITION
Da vinci surgical system
Image Courtesy of INTUITIVE SURGIVAL
HEALTH and SAFETY WARNING
HEALTH and SAFETY WARNING
focus at the object’s distance
Vergence & accommodation are coupled!
Vergence & accommodation are coupled!
Immersive HMD Magnified Display
Immersive HMD Magnified Display
Decoupled!
Immersive HMD Magnified Display
!@#$$^R$%%&$
discomfort and fatigue, eyestrain, diplopic vision, headaches, nausea and …
0.001
0.01
0.1
1
1m 10m 100m 1000m 10000m
Personal Action Vista
StereopsisVergence
Accommodation
Motion Parallax
Relative Height
ArialPerspective
Dep
th C
ontra
st
Distance [Cutting and Vishton 1995]
Current HMD
Depth Cues
Image Courtesy of Nimble VR
Backlight
Thin Spacer & 2nd panel(6mm)
Magnifying Lenses
LCD Panel
Front Focusing
Rear Focusing
StereoscopeWheatstone, Brewster
1838
VR/ARIvan Sutherland
1968
VR explosionOculus, Sony, Valve, MS, …
Now Future
Future HMDs
ResolutionLatency
FOVForm Factor
…
Accommodation
StereoscopeWheatstone, Brewster
1838
VR/ARIvan Sutherland
1968
VR explosionOculus, Sony, Valve, MS, …
Now Future
Future HMDs
[Lanman and Luebke 2013]
[Hu and Hua 2014]
Accommodation
Object emits natural light rays
Light field enters eyes
Layered3D, Tensor Display[Wetzstein et al. 2011, 12]
Small Angle
Stationary
TV vs. HMD• Reduced brightness problematic• Shared: Need wide viewing angle
– High Rank– Multiple Layers– Temporal Multiplexing
• Viewer adapts to reduced brightness• Personal: Small viewing angle
– Low Rank– 2 Layers– Native Refresh Rate
Image Courtesy of ShutterStock.com
Low-rank, 2-panelNo temporal multiplexing
Both binocular & focus cuesto form 3D perception
Low-rank, 2-panelNo temporal multiplexing
Target Light Field
Input: 4D light field for each eye
Model Courtesy of Bushmills Irish Whiskey
Multiplicative Two-layer Modulation
1
𝑙 (𝑥 ,𝑣 )
𝑥
𝑣
𝑡 1𝑡 2
𝑙 (𝑥 ,𝑣 )=𝑡1 (𝑥 )𝑡 2(𝑥− (𝑣− 𝑥)𝑑𝑒 ) 𝑑𝑒
Model Courtesy of Bushmills Irish Whiskey
Input: 4D light field for each eye
Multiplicative Two-layer Modulation
1𝑡 1𝑡 2
𝑙 (𝑥 ,𝑣 1 )=𝑡 1 (𝑥 )𝑡 2(𝑥− (𝑣1−𝑥)𝑑𝑒 ) 𝑑𝑒
Model Courtesy of Bushmills Irish Whiskey
Input: 4D light field for each eye
Multiplicative Two-layer Modulation
1𝑡 1𝑡 2
𝑙 (𝑥 ,𝑣 4 )=𝑡1 (𝑥 )𝑡 2(𝑥− (𝑣 4−𝑥 )𝑑𝑒 ) 𝑑𝑒
Model Courtesy of Bushmills Irish Whiskey
Input: 4D light field for each eye
Multiplicative Two-layer Modulation
1𝑡 1𝑡 2
𝑙 (𝑥 ,𝑣 7 )=𝑡1 (𝑥 ) 𝑡2(𝑥− (𝑣 7−𝑥)𝑑𝑒 ) 𝑑𝑒
Model Courtesy of Bushmills Irish Whiskey
Input: 4D light field for each eye
Multiplicative Two-layer Modulation
𝑡 1𝑡 2
Model Courtesy of Bushmills Irish Whiskey
Input: 4D light field for each eye
Parallax over the Pupil
Multiplicative Two-layer Modulation
𝑡 1𝑡 2
~l=(ϕ1 𝑡 1)o (ϕ2 𝑡 2)Reconstruction:
Model Courtesy of Bushmills Irish Whiskey
Input: 4D light field for each eye
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒‖𝛽 l− (ϕ 1𝑡 1 )o (ϕ2 𝑡 2 )‖2{𝑡 1, 𝑡 2 } 𝑠 .𝑡 .0≤𝑡 1 , 𝑡 2≤1
𝑡1←𝑡 1oϕ1𝑇(𝛽 l o (ϕ2 𝑡2))ϕ1𝑇 ( ~l o (ϕ2 𝑡2 ) )+𝜖
for layer t1[Wetzstein et al 2012]
Input light field to the eyeMultiplicative Two-layer Modulation
𝑡 1𝑡 2
Left Eye: Front Panel Left Eye: Rear Panel
Front Panel Rear Panel
Output: 2 layers for each eye
Model Courtesy of Bushmills Irish Whiskey
Front Focus Mid Focus Rear Focus
Front Object Mid Object Rear Object
Multiplicative Two-layer Modulation
𝑡 1𝑡 2
Front Panel Rear Panel Model Courtesy of Bushmills Irish Whiskey
MultiplicativeMultilayer
AdditiveMultilayer
+
*
Multilayer Displays
Akeley et al. [2004]
Love et al. [2009]Narain et al.[2015]
Opt
ical
Ove
rlay
Tem
pora
lM
ultip
lexi
ng
Central Light Field
Right Light Field
Left Light Field
With Additive Multilayer
Left Light FieldA
dditi
veZoom-In
Mul
tiplic
ativ
e
Model Courtesy of Paul H. Manning
IMPLEMENTATION&
RESULTS
The Hardware- Adafruit based 3D printed housing learn.adafruit.com/3d-printed-wearable -video-goggles
- Aspheric lens, 2” diameter, 5cm focal length, $10 each
- TOSHIBA LTD056EV7F, 5.6” 1280x800, $60 each- CHIMEI INNOLUX N070ICG-LD1, 7” $35 each (need to replace polarizer!)
- PCB800099 driver board, $30 each
E-Tech DemoSouth Hall G, Booth ET18
The Hardware
Content Generation• 5x5x2 Stereo Light Field
• GPU Factorization
• 3 ~ 5 Iterations per frame• 25(views) x 640x800 x 2(eyes) • 5 ~ 10ms per iterations Left eye Right eye
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Model Courtesy of ixlrlxi
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Model Courtesy of ixlrlxi
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Model Courtesy of Utah 3D Repository
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Model Courtesy of Utah 3D Repository
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Model Courtesy of Paul H. Manning
Photograph Results
Traditional HMDs- No Focus Cues
The Light Field HMDStereoscope
Model Courtesy of Paul H. Manning
Designing Real-world ExperienceLeft eye Right eye
Fron
tFo
cusin
gRe
arFo
cusin
g
Technical Details in the Paper
Good Parameter Space
Limited by Diffraction
100 200 300 400 500
2
3
5
6
4
Front Panel PPI
Num
ber o
f Vie
ws
Minimum Views to Support Accommodation
IPD / 2 * M2
IPD / 2 * M1
Center
Center
Diffraction Analysis Light Field Distortion Asymmetric Image Formation
Discussion• Reduction in brightness
• Latency: 5x5x2 views
• Diffraction
Future Work
Image Courtesy of Jaunt and Google
Light Field Content Human Vision/Perception
Emerging Technologies
South Hall G, Booth ET-18
www.computationalimaging.org/ sites.google.com/site/jonash123/
IPD / 2 * M2
IPD / 2 * M1
Center
Center
Top View Light Field Side View Light Field