03/12/03© 2003 university of wisconsin last time npr assignment projects high-dynamic range capture...

03/12/03 © 2003 University of Wisconsin

Last Time

• NPR Assignment

• Projects

• High-Dynamic Range Capture

• Image Based Rendering Intro


Today

• Several Image-Based-Rendering (IBR) systems


IBR Systems

• Methods differ in many ways:– The range of new viewpoints allowed

– The density of input images

– The representation for samples (known images)

– The amount of user help required

– The amount of additional information required (such as intrinsic camera parameters)

– The method for gathering the input images


Movie-Map Approaches

• Film views from fixed locations, closely spaced, and store– Storage can be an issue

• Allow the user to jump from location to location, and pan

• Appropriate images are retrieved from disk and displayed

• No re-projection – just uses nearest existing sample

• Still used in video games today, but with computer generated movies– Which games (somewhat dated now)?


Quicktime VR (Chen, 1995)

• Movie-maps in software

• Construct panoramic images by stitching together a series of photographs– Semi automatic process, based on correlation

• Scale/shift images so that they look most alike

– Works best with >50% overlap

• Finite set of panoramas – user jumps from one to the other


Results - Warping


Results - Stitching


View Interpolation(Chen and Williams, 1993)

• Input: A set of synthetic images with known depth and camera parameters (location, focal length, etc)

• Computes optical flow maps relating each pair of images– Optical flow map is the set of vectors describing where each point

in the first image moves to in the second image

• Morphs between images by moving points along flow vectors

• Intermediate views are “real” views only in special cases


View Morphing (Seitz and Dyer, 1997)

• Uses interpolation to generate new views such that the intermediate views represent real camera motion

• Observation: Interpolation gives incorrect intermediate views (not resulting from a real projection)


View Morphing

• Observation: Interpolation gives correct intermediate views if the initial and final images are parallel views


View Morphing Process

• Basic algorithm:– User specifies a camera path

that rotates and translates initial camera onto final camera

– Pre-warp input images to get them into parallel views

– Interpolate for intermediate view

– Post-warp to get final result


View Morphing Process

• Requires knowledge of projection matrices for the input images– Found with vision algorithms. User may supply correspondences

• Intermediate motion can be specified by giving trajectories of four points


Plenoptic Modeling (McMillan and Bishop, 1995)

• Input: Set of panoramic images gathered by panning a video camera about its vertical optical axis on a tripod

• The algorithm:– Determines the properties of each camera and registers the images

associated with each pan

– Determines the relative locations of each camera

– Determines the depth of each point seen by multiple cameras

– Generates new views by reconstructing the plenoptic function from the available samples


Fitting Each Camera

• Problem:– Given several images all taken with the same camera rotated about

its optical center

– Determine the camera parameters

– Determine the angle between each image

• Approach:– Multi-stage optimization

– First stage estimates angle between images and focal length

– Second stage refines and gets remaining parameters

• When done, can map a pixel in one image to its correct position in any other images


Locating the Cameras

• Given cylindrical projections (panoramic images) for two cameras

• User identifies 100-500 tie-points (points seen in both images)

• Each tie-point defines two rays – one from the center of each camera through the tie-point– These rays should intersect at the world location for the point

• Minimization step finds the camera locations and some other parameters that minimize the perpendicular distance between all the rays


Determining Disparity

• The minimization algorithm gives us the world location of the tie-points, but what about the rest of the points in the image?

• Use standard computer vision techniques to find the remaining disparities– Disparity is the angular difference between the locations of a point

in two images

– Directly related to the depth of the point

• Makes heavy use of the epipolar constraint


The Epipolar Constraint

• The location of a point in one image constrains it to lie on a ray in space passing through the camera and image point

• This ray projects to a curve in the second image– Line for planar projection

– Sine curve for cylindrical projection

• Since the point must lie on the ray in world space, it must lie on the curve in the second image

• Reduces the search for correspondences to a linear one along the line


Panoramas


Reconstructing a New View

• The disparity from a known cylinder pair can be transferred to a new cylinder, and then re-projected onto a plane (the image)– Can all be done in one step

• Have to resolve occlusion problems– Two points from the reference image could map to the same point in

the output image

– Solution: Define a fill ordering that guarantees correct occlusion (an important contribution of this paper)

• Also have to fill holes


Rendering Order


Working from Photos

• Image-based rendering obviously relies heavily on computer vision techniques– Particularly: Depth from stereo

– The problem is very hard with real images

– These techniques are not perfect!

• Sampling remains a problem– Images tend to appear blurry

– Relatively little work on reconstruction algorithms


Light Field Rendering or Lumigraphs

• Aims:– Sample the plenoptic function, or light field, densely

– Store the samples in a data structure that is easy to access

– Rendering is simply averaging of samples

• The plenoptic function gives the radiance passing through a point in space in a particular direction

• In free space: Gives the radiance along a line– Recall that radiance is constant along a line


Storing Light Fields

• Each sample of the light field represents radiance along a line

• Required operations:– Store the radiance associated with an oriented line

– Look up the radiance of lines that are “close” to a desired line

• Hence, we need some way of describing, or parameterizing, oriented lines– A line is a 4D object

– There are several possible parameterizations


Parameterizing Oriented Lines

• Desirable properties:– Efficient conversion from lines to parameters

– Control over which subset of lines is of interest

– Ease of uniform sampling of lines in space

• Parameterize lines by their intersection with two planes in arbitrary positions– Take (s,t) as intersection of line in one plane, (u,v) as intersection in

other: L(s,t,u,v)

– Light Slab: use two quadrilaterals (squares) and restrict each of s,t,u,v to (0,1)


A Slab


Line Space

• An alternate parameterization is line space– Better for looking at subset of lines and verifying

sampling patterns– In 2D, parameterize lines by their angle with the

x-axis, and their perpendicular distance form the origin

– Extension to 3D is straightforward

• Every line in space maps to a point in line space, and vice versa– The two spaces are dual– Some operations are much easier in one space

than the other


Verifying Sampling Patterns


Capturing Light Fields

• Render synthetic images

• Capture digitized photographs– Use a gantry to carefully control which images are captured

• Makes it easy to control the light field sampling pattern

• Hard to build the gantry

– Use a video camera• Easy to acquire the images

• Hard to control the sampling pattern


Render synthetic images

• Decide which line you wish to sample, and cast a ray, or• Render an array of images from points on the (u,v) plane –

pixels in the images are points on the (s,t) plane• Antialiasing is essential, both in (s,t) and (u,v)

– Standard anitaliasing and aperture filtering


Tightly Controlled Capture

• Use a computer controlled gantry to move a camera to fixed positions and take digital images

• Looks in at an object from outside– Must acquire multiple slabs to get full coverage

– Care must be taken with camera alignment and optics

• Object is rotated in front of gantry to get multiple slabs– Must ensure lighting moves with the object

• Effectively samples light field on a regular grid, so rendering is easier


Capture from Hand Held Video

• Place the object on a calibrated stage– Colored to allow blue-screening

– Markers to allow easy determination of camera pose

• Wave the camera around in front of the object– Map to help guide where more samples are required

• Camera must be calibrated beforehand

• Output: A large number of non-uniform samples

• Problem: Have to re-sample to get regular sampling for rendering


Re-Sampling the Light Field

• Basic problem:– Input: The set of irregular samples from the video capture process

– Output: Estimates of the radiance on a regular grid in parameter space

• Algorithm outline:– Use a multi-resolution algorithm to estimate radiance in under-

sampled regions

– Use a binning algorithm to uniformly resample without bias


Compression

• Light fields samples must be dense for good rendering

• Dense light fields are big: 1.6GB– When rendering, samples could come from any part of the light

field

– All of the light field must be in memory for real-time rendering

– But lots of data redundancy, so compression should do well

• Desirable compression scheme properties:– Random access to compressed data

– Asymmetric – slow compression, fast decompression


Compression Scheme

• Vector Quantization:– Compression:

• Choose a codebook of reproduction vectors

• Replace all the vectors in the data with the index into the “nearest” vector in the codebook

– Storage: The codebook plus the indexes

– Decompression:• Replace each index with the vector from the codebook

• Follow up with Lempel-Ziv entropy encoding (gzip)– Decompress into memory


Alternate Compression Schemes

• Neighboring “images” in (u,v) are likely to be very similar– Picture doesn’t change much as you move the camera a little

– We know what the camera motion is

– BRDF changes smoothly for many cases

– Use MPEG or similar to encode a sequence of images

– This has been discussed but not implemented

• “Textures” should compress well– Use hardware rendering from compressed textures


Rendering

• Ray-tracing: For each pixel in the image:– Determine the ray passing through the eye and the pixel

– Interpolate the radiance along that ray from the nearest rays in the light-field

• Texture Mapping:– Finding the (u,v) and (s,t) coordinates is exactly the texture mapping

operation

– Use graphics hardware to do the job, or write a software texture mapper (maybe faster – only have to texture map two polygons)

• Use various interpolation schemes to control aliasing


Exploiting Geometry

• When using the video capture approach, build a geometric model– Use a volume carving technique

• When determining the “nearest” samples for rendering, use the geometry to choose better samples

• This has been further extended:– Surface point used for improving sampling determines focus

– By default, we want focus at the object, so use the object geometry

– Using other surfaces gives depth of field and variable focus


Surface Light Fields

• Instead of storing the complete light-field, store only lines emanating from the surface– Parameterize the surface mesh (standard technique)

– Choose sample points on the surface

– Sample the space of rays leaving the surface from those points

– When rendering, look up nearby sample points and appropriate sample rays

• Best for rendering complex BRDF models– An example of view dependent texturing


Summary

• Light-fields capture very dense representations of the plenoptic function– Fields can be stitched together to give walkthroughs

– The data requirements are large

– Sampling still not dense enough – filtering introduces blurring

• Next time: Using domain specific knowledge

03/12/03© 2003 university of wisconsin last time npr assignment projects high-dynamic range capture...

Documents

university of wisconsinresults

university of wisconsinmovie

university of wisconsinfitting

final images

real views

correlationscaleshift

panappropriate images

initial camera