Download - Marc Levoy - Stanford University
![Page 1: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/1.jpg)
Light field sensing
Marc Levoy
Computer Science Department Stanford University
![Page 2: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/2.jpg)
© 2008 Marc Levoy
The scalar light field (in geometrical optics)
Radiance as a function of position and direction in a static scene with fixed illumination
L is radiance in watts / (m2 steradians) 5-dimensional function
![Page 3: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/3.jpg)
© 2008 Marc Levoy
The vector light field [Gershun 1936]
• amplitude gives irradiance at that point
• direction tells which way to orient a surface for maximum brightness under uniform illumination
adding two light vectors the vector light field produced by a luminous strip
![Page 4: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/4.jpg)
© 2008 Marc Levoy
Visualizing the vector irradiance field
flatland scene with partially opaque blockers
under uniform illumination
scalar irradiance at each point
vector directions,visualized using line integral convolution (LIC) [Cabral 1993]
![Page 5: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/5.jpg)
© 2008 Marc Levoy
Dimensionality of the scalar light field
• for general scenes ⇒ 5D function “plenoptic function”
L ( x, y, z, θ, φ )
• in free space ⇒ 4D function “ the (scalar) light field”
L ( ? )
![Page 6: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/6.jpg)
Some candidate parameterizations for the 4D light field
Point-on-plane + direction (or point-on-surface + direction)
L ( x, y, θ, φ ) or L ( u, v, θ, φ )
• convenient for measuring BRDFs • restriction to line gives looming field
x
y
![Page 7: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/7.jpg)
•
flight path through a flatland scene
corresponding looming light field (see also [Hasinoff 2006])
θ
t
t
θ
![Page 8: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/8.jpg)
The looming field
[Gibson]
![Page 9: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/9.jpg)
More parameterizations
Chords of a sphere L ( θ1, φ1, θ2, φ2 )
• convenient for spherical gantry • facilitates uniform sampling
![Page 10: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/10.jpg)
Two planes (“light slab”)
L ( u, v, s, t )
• uses projective geometry – one plane at infinity ⇒ array of orthographic images – fast incremental display algorithms
![Page 11: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/11.jpg)
© 2008 Marc Levoy
The free-space assumption
• Where can you use free-space light fields? – the 3D space around a compact object – the 3D space inside an uncluttered environment
• stitching together light fields [Chen, Levoy, Hanrahan (unpublished) ]
– partition scene into disjoint cells – links between cells are light fields – hierarchy of cells, links, light fields – # of light fields is linear in # of cells
![Page 12: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/12.jpg)
© 2008 Marc Levoy
Light field rendering
flipbook animation (QuickTime VR)
rebinning the rays to create new views
(movie is available at http://graphics.stanford.edu/papers/light)
![Page 13: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/13.jpg)
Alternative parameterizations for the 5D plenoptic function
• Two-plane ray field
L ( u, v, s, t, z )
• allows multiple colors, in sequence, along one line • alternative to L ( x, y, z, θ, φ ) • inspired by Salesin’s ZZ-buffer [1989]
z
![Page 14: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/14.jpg)
© 2008 Marc Levoy
Devices for recording light fields
small scenes
big scenes • handheld camera [Buehler 2001]
• array of cameras [Wilburn 2005] • plenoptic camera [Ng 2005] • light field microscope [Levoy 2006]
![Page 15: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/15.jpg)
© 2008 Marc Levoy
Stanford Multi-Camera Array [Wilburn SIGGRAPH 2005]
• 640 × 480 pixels × 30 fps × 128 cameras
• synchronized timing • continuous streaming • flexible arrangement
![Page 16: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/16.jpg)
© 2005 Marc Levoy
Ways to use large camera arrays
• widely spaced light field capture
Manex’s bullet time array
(movie is available at http://graphics.stanford.edu/projects/array)
![Page 17: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/17.jpg)
© 2005 Marc Levoy
Ways to use large camera arrays
• widely spaced light field capture • tightly packed high-performance imaging
![Page 18: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/18.jpg)
© 2005 Marc Levoy
Ways to use large camera arrays
• widely spaced light field capture • tightly packed high-performance imaging • intermediate spacing synthetic aperture photography
![Page 19: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/19.jpg)
© 2008 Marc Levoy
Synthetic aperture photography
Σ
![Page 20: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/20.jpg)
© 2008 Marc Levoy
Example using 45 cameras [Vaish CVPR 2004]
![Page 21: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/21.jpg)
one camera’s view synthetic aperture view
(movie is available at http://graphics.stanford.edu/projects/array)
![Page 22: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/22.jpg)
Light field photography using a handheld plenoptic camera
Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan
(Proc. SIGGRAPH 2005 and TR 2005-02)
![Page 23: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/23.jpg)
© 2008 Marc Levoy
Conventional versus light field camera
![Page 24: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/24.jpg)
© 2008 Marc Levoy
Conventional versus light field camera
uv-plane st-plane
![Page 25: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/25.jpg)
Prototype camera
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125µ square-sided microlenses
![Page 26: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/26.jpg)
Typical image captured by camera (show here at low res)
![Page 27: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/27.jpg)
© 2008 Marc Levoy
Digital refocusing
• refocusing = summing windows extracted from several microlenses
Σ
Σ
![Page 28: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/28.jpg)
© 2006 Marc Levoy
A digital refocusing theorem
• an f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera
– or –
• it can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera
![Page 29: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/29.jpg)
© 2007 Marc Levoy
Example of digital refocusing
![Page 30: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/30.jpg)
© 2007 Marc Levoy
Example of digital refocusing
![Page 31: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/31.jpg)
© 2007 Marc Levoy
Example of digital refocusing
![Page 32: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/32.jpg)
© 2007 Marc Levoy
Example of digital refocusing
![Page 33: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/33.jpg)
© 2007 Marc Levoy
Example of digital refocusing
![Page 34: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/34.jpg)
© 2007 Marc Levoy
Refocusing portraits
(movie is available at http://refocusimaging.com)
![Page 35: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/35.jpg)
Extending the depth of field
conventional photograph, main lens at f / 22
conventional photograph, main lens at f / 4
light field, main lens at f / 4, after all-focus algorithm [Agarwala 2004]
![Page 36: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/36.jpg)
© 2008 Marc Levoy
Digitally moving the observer
• moving the observer = moving the window we extract from the microlenses
Σ
Σ
![Page 37: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/37.jpg)
© 2008 Marc Levoy
Example of moving the observer
![Page 38: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/38.jpg)
© 2008 Marc Levoy
Example of moving the observer
![Page 39: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/39.jpg)
© 2008 Marc Levoy
Example of moving the observer
![Page 40: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/40.jpg)
© 2008 Marc Levoy
Moving backward and forward
![Page 41: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/41.jpg)
© 2008 Marc Levoy
Moving backward and forward
![Page 42: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/42.jpg)
© 2008 Marc Levoy
Moving backward and forward
![Page 43: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/43.jpg)
© 2008 Marc Levoy
Moving backward and forward
![Page 44: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/44.jpg)
© 2008 Marc Levoy
Lego gantry for capturing light fields (built by Andrew Adams)
calibration point
plane + parallax [Vaish 2004]
![Page 45: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/45.jpg)
© 2008 Marc Levoy
Flash-based viewer for light fields (written by Andrew Adams)
(light field can be viewed at http://lightfield.stanford.edu/lfs.html)
![Page 46: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/46.jpg)
© 2008 Marc Levoy
Implications / commercialization (see refocusimaging.com)
• cuts the unwanted link between exposure (due to the aperture) and depth of field
• trades off (excess) spatial resolution for ability to refocus and adjust the perspective
• sensor pixels should be made even smaller, subject to the diffraction limit
36mm × 24mm ÷ 2.5µ pixels = 266 Mpix 20K × 13K pixels 4000 × 2666 pixels × 20 × 20 rays per pixel
or 2000 × 1500 pixels × 3 × 3 rays per pixel = 27 Mpix
![Page 47: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/47.jpg)
Light Field Microscopy
Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz
(Proc. SIGGRAPH 2006)
![Page 48: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/48.jpg)
© 2008 Marc Levoy
A traditional microscope
objective
specimen
intermediate image plane
eyepiece
![Page 49: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/49.jpg)
© 2008 Marc Levoy
A light field microscope (LFM)
• 40x / 0.95NA objective ↓ 0.26µ spot on specimen × 40x = 10.4µ on sensor
↓ 2400 spots over 25mm field
• 1252-micron microlenses ↓ 200 × 200 microlenses with 12 × 12 spots per microlens
objective
specimen
intermediate image plane
eyepiece sensor
→ reduced lateral resolution on specimen = 0.26µ × 12 spots = 3.1µ
![Page 50: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/50.jpg)
© 2008 Marc Levoy
A light field microscope (LFM)
objective
specimen
intermediate image plane
eyepiece sensor
![Page 51: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/51.jpg)
Example light field micrograph
• orange fluorescent crayon • mercury-arc source + blue dichroic filter • 16x / 0.5NA (dry) objective • f/20 microlens array • 65mm f/2.8 macro lens at 1:1 • Canon 20D digital camera
ordinary microscope light field microscope
![Page 52: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/52.jpg)
© 2006 Marc Levoy
The geometry of the light field in a microscope
• microscopes make orthographic views
• translating the stage in X or Y provides no parallax on the specimen
• out-of-plane features don’t shift position when they come into focus
• front lens element size = aperture width + field width
• PSF for 3D deconvolution microscopy is shift-invariant (i.e. doesn’t change across the field of view)
f
objective lenses are telecentric
![Page 53: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/53.jpg)
© 2008 Marc Levoy
Example light field micrograph
panning sequence focal stack
(movies are available at http://graphics.stanford.edu/projects/lfmicroscope)
![Page 54: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/54.jpg)
© 2008 Marc Levoy
Real-time viewer
(movie is available at http://graphics.stanford.edu/projects/lfmicroscope/2007.html)
![Page 55: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/55.jpg)
© 2008 Marc Levoy
Other examples
fern spore (60x, autofluorescence)
Golgi-stained neurons (40x, transmitted light)
zebrafish optic tectum (calcium imaging of neural activity)
(movies are available at http://graphics.stanford.edu/projects/lfmicroscope)
![Page 56: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/56.jpg)
© 2008 Marc Levoy
3D reconstruction
• 4D light field → digital refocusing → 3D focal stack → deconvolution microscopy → 3D volume data
• 4D light field → tomographic reconstruction → 3D volume data
(from Kak & Slaney)
(DeltaVision)
![Page 57: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/57.jpg)
© 2008 Marc Levoy
Silkworm mouth (40x / 1.3NA oil immersion)
slice of focal stack slice of volume volume rendering
100µ
![Page 58: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/58.jpg)
© 2008 Marc Levoy
GFP-labeled zebrafish neurons (40x / 0.8NA water immersion)
focal stack
deconvolved
volume rendering
![Page 59: Marc Levoy - Stanford University](https://reader030.vdocuments.net/reader030/viewer/2022012801/61bd091161276e740b0eb15d/html5/thumbnails/59.jpg)
© 2005 Marc Levoy http://graphics.stanford.edu