computational photography light field rendering
DESCRIPTION
Computational Photography Light Field Rendering. Jinxiang Chai. Image-based Modeling: Challenging Scenes. Why will they produce poor results? lack of discernible features occlusions difficult to capture high-level structure illumination changes specular surfaces. Some Solutions. - PowerPoint PPT PresentationTRANSCRIPT
Computational PhotographyLight Field Rendering
Jinxiang Chai
Image-based Modeling: Challenging Scenes
Why will they produce poor results?
- lack of discernible features
- occlusions
- difficult to capture high-level structure
- illumination changes
- specular surfaces
Some Solutions
- Use priors to constrain the modeling space
- Aid modeling process with minimal user interaction
- Combine image-based modeling with other modeling approaches
Videos
Morphable face (click here)
Image-based tree modeling (click here)
Video trace (click here)
3D modeling by ortho-images (Click here)
Spectrum of IBMR
Images user input range
scans
Model
Images
Image based modeling
Image-based renderingGeometry+ Images
Light field
Images + Depth
Geometry+ Materials
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
Outline
Light field rendering [Levoy and Hanranhan SIG96]
3D light field (concentric mosaics) [Shum and He Sig99]
Plenoptic Function
Can reconstruct every possible view, at every moment, from every position, at every wavelength
Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality!
An image is a 2D sample of plenoptic function!
P(x,y,z,θ,φ,λ,t)
Ray
Let’s not worry about time and color:
5D• 3D position
• 2D direction
P(x,y,z,)
Static object Camera
No Change in
Radiance
Static Lighting
How can we use this?
Static object Camera
No Change in
Radiance
Static Lighting
How can we use this?
Ray Reuse
Infinite line• Assume light is constant (vacuum)
4D• 2D direction
• 2D position
• non-dispersive medium
Slide by Rick Szeliski and Michael Cohen
Only need plenoptic surface
Synthesizing novel views
Assume we capture every ray in 3D space!
Synthesizing novel views
Light field / Lumigraph
Outside convex space
4DStuff
Empty
Light Field
How to represent rays?
How to capture rays?
How to use captured rays for rendering
Light Field
How to represent rays?
How to capture rays?
How to use captured rays for rendering
Light field - Organization
2D position
2D direction
s
Light field - Organization
2D position
2D position
2 plane parameterization
su
Light field - Organization
2D position
2D position
2 plane parameterization
us
t s,tu,v
v
s,t
u,v
Light field - Organization
Hold u,v constant
Let s,t vary
What do we get?
s,tu,v
Lumigraph - Organization
Hold s,t constant
Let u,v vary
An image
s,tu,v
Lightfield / Lumigraph
Light field/lumigraph - Capture
Idea 1• Move camera carefully over u,v
plane
• Gantry> see Light field paper
s,tu,v
Stanford multi-camera array
640 × 480 pixels ×30 fps × 128 cameras
synchronized timing
continuous streaming
flexible arrangement
For each output pixel• determine s,t,u,v
• either• use closest discrete RGB• interpolate near values
s u
Light field/lumigraph - rendering
Light field/lumigraph - rendering
Nearest• closest s
• closest u
• draw it
Blend 16 nearest• quadrilinear interpolation
s u
Ray interpolation
s u
Nearest neighbor
Linear interpolation in S-T
Quadrilinear interpolation
Image Plane
Camera Plane
Light FieldLight Field CaptureCapture RenderingRendering
Light Field/Lumigraph Rendering
Light fields
Advantages:• No geometry needed
• Simpler computation vs. traditional CG
• Cost independent of scene complexity
• Cost independent of material properties and other optical effects
Disadvantages:• Static geometry
• Fixed lighting
• High storage cost
3D plenoptic function
Image is 2D
Light field/lumigraph is 4D
What happens to 3D?
- 3D light field subset
- Concentric mosaic [Shum and He]
3D light field
One row of s,t plane• i.e., hold t constant
s,t u,v
3D light field
One row of s,t plane• i.e., hold t constant
• thus s,u,v
• a “row of images”
s
u,v
Concentric mosaics [Shum and He]
Polar coordinate system:
- hold r constant
- thus (θ,u,v)
Concentric mosaics
Why concentric mosaic?
- easy to capture
- relatively small in storage size
Concentric mosaics
From above
How to captured images?
Concentric mosaics
From above
How to render a new image?
Concentric mosaics
From above
How to render a new image?
- for each ray, retrieval the closest captured rays
Concentric mosaics
From above
How to render a new image?
- for each ray, retrieval the closest captured rays
Concentric mosaics
From above
How to render a new image?
- for each ray, retrieval the closest captured rays
Concentric mosaics
From above object
How to retrieval the closest rays?
Concentric mosaics
From above object (s,t) interpolation plane
How to retrieve the closest rays?
Concentric mosaics
From above object (s,t) interpolation plane
How to retrieve the closest rays?
Concentric mosaics
From above object (s,t) interpolation plane
How to retrieve the closest rays?
Concentric mosaics
From above object (s,t) interpolation plane
How to retrieve the closest rays?
Concentric mosaics
From above object (s,t) interpolation plane
How to synthesize the color of rays?
Concentric mosaics
From above object (s,t) interpolation plane
How to synthesize the color of rays? - bilinear interpolation
Concentric mosaics
From above
Concentric mosaics
From above
Concentric mosaics
What are limitations?
Concentric mosaics
What are limitations? - limited rendering region?
- large vertical distortion