interactive rendering of acquired materials on dynamic geometry using bandwidth prediction

Mahdi M. Bagher / Cyril Soler / Kartic Subr / Laurent Belcour / Nicolas HolzschuchMahdi M. Bagher / Cyril Soler / Kartic Subr / Laurent Belcour / Nicolas Holzschuch

Interactive rendering of Interactive rendering of acquired materials on dynamic geometryacquired materials on dynamic geometry

using bandwidth predictionusing bandwidth prediction

• Direct reflection under distant illumination with no visibility and global illumination effects

• Rendering an image is computationally expensive.

Shading

fro

m W

ikip

edi

a

Illumination BRDF

Acquired Materials

• Using measured materials• photo-realistic rendering• large memory footprint

(33MB for a single isotropic BRDF)

• Not suitable for real-time rendering

Phong shading Measured “Color-Changing-Paint-3”BRDF measurement Gantry [Matusik2002]

Monte Carlo Sampling

• Monte Carlo approximation to the shading integral

• Importance sampling• less noisy

• Data driven reflectance• no analytical importance function

• pre-computed Importance samples

Monte Carlo sampling

BRDF importance sampling

Sampling

• Dealing with two types of sampling• Reconstruction• sampling in image space• to render an image

• Integration• sampling in the space of incident directions • to shade each individual pixel

Illumination BRDF Cosine

The Problem

• Shading is slow…

• How to speed-up shading?

Case Study

• What difference various materials make in rendering an image?

Measured reflectance data from [Matusik2002]

Diffuse Materials

• Diffuse materials• shading varies slowly across the image (low frequencies)• many integration samples for each pixel (wide BRDF lobe)

Measured Teflon [Matusik2002] BRDF Lobe [BRDFLab]

Specular Materials

• Specular materials• shading varies quickly across the image (high frequencies) • few integration samples for each pixel (narrow BRDF lobe)

Measured color-Changing-Paint3 [Matusik2002] BRDF Lobe [BRDFLab]

The Problem

• Rendering is slow…• How to speed-up shading?

• Can we exploit the coherency between • Reconstruction samples…• Integration samples…

• …for adaptive sampling?

• How to measure the coherency between samples?

Intuition

• Study the frequencies in the image• As a function of what happens to the light• Traveling from the light source to the camera

The Traveling Light

• Light when traveling is affected by• Travel through free space• Object’s curvature• Reflectance• BRDF• Texture

• Occlusion

Our Contribution

• How to speed-up shading?By sparse sampling the image• How to combine the sparse samples?• Multi-resolution rendering and bi-lateral up-sampling

By adaptive sampling for integration

Our Contribution (Illustrated)

Sparse reconstruction samples Adaptive Integration samples

,

Final shaded image

Multi-resolution shading and bi-lateral up-sampling

∝ the average variance of the shading integrand∝ the local maximum variations in the image

Assumptions

• Direct reflection under distant illumination

• No visibility and global illumination effects

• Allow dynamic editable geometry

• Screen-space rendering in the context of deferred-shading

• Any shading model is possible

• Save more if shading cost is higher e.g. measured materials

• BRDF importance sampling

Normal

Dynamic Geometry Animation

QuickTime™ and a decompressor

are needed to see this picture.

Roadmap

• Related work

• Our technique

• The goal

• Sparse sampling for reconstruction

• Adaptive sampling for integration

• Validation

• The rendering algorithm

• Results and comparisons

• Extension: MSAA

• Conclusion / Future work

Related Work GPU Rendering of Complex Materials

[Heidrich&Seidel1999]

[Ramamoorthi2002]

[Kautz2002]

[Claustres2007]

[Kautz&McCool1999]

[Latta&Kolb2002]

[McCool2001]

[Wang2009]

Related WorkMulti-Resolution Screen-Space Algorithms

[Nichols&Wyman2009] [Shopf2009][Nichols&Wyman2010]

[Soler2010]

[Nichols2010][Segovia2006]

Related WorkPre-Computed Transport

[Sloan2002]

[Ramamoorthi2009]

[Sun2007]

[Liu2004]

[Loos2011]

[Loos2012]

Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


Goal

• Sparse sampling for reconstruction• ∝ maximum local variations (local bandwidth)

• Adaptive sampling for integration• ∝ variance of the shading integrand

• Multi-resolution rendering and bilateral up-sampling.• a single coarse-to-fine rendering pass

Roadmap

• Related work

• Our technique

• The goal


• Local light-field definition

• Spectral Operations

• 2D local bandwidth

• Image-space local bandwidth


• Validation

• …

Local Light-Field

• 4D local light-field illustrated in 2D

Central ray

Sp

ace

Angle

Roadmap

• Related work

• Our technique

• The goal







• Validation

• …

Spectral Operations

•Spectral operations [Durand et al. 2005][D

urand

et al. 2005]

Roadmap

• Related work

• Our technique

• The goal







• Validation

• …

2D Local Bandwidth

• 2D Local Bandwidth• maximum local variations about a light path both in angle and space.

spatial angular

2D local bandwidth

Propagating Local Bandwidth

Propagating Local Bandwidth

• One-bounce 2D bandwidth propagation• Transport (from light to reflector)• Reflection (BRDF & Texture)• Transport (from reflector to pixel)

• Matrix operators on 2D bandwidth

where

Bandwidthat pixel p

Reflectionoperator

Illumination bandwidth

Scale Curvature Mirror BRDF and texture

Curvature ScaleTransport to image plane

Transport to the surface

Input Signals

• Illumination• environment map

• Reflectance• Acquired isotropic BRDFs

• TexturesGold-paint[ Matusik2003]

Grace cathedral [Debevec2001]

Input Signals’ Bandwidth

• Illumination• environment map• Purely angular bandwidth

• Reflectance• Acquired isotropic BRDFs• Purely angular bandwidth

• Textures• Purely spatial bandwidth

Gold-paint[ Matusik2003]

Grace cathedral [Debevec2001]

Computing Local Bandwidth

• Discrete Wavelet Transform to compute bandwidth• environment map and reflectance (BRDF & Texture)

• Discrete Wavelet Transform instead of WFFT • wavelets are well localized both in space and frequency.

Space Frequency

Daubechies 4 tap wavelet

FFT

Local Bandwidth Examples

Gold-paint[ Matusik2003]

Angular bandwidth

Angular bandwidthGrace cathedral [Debevec2001]

Roadmap

• Related work

• Our technique

• The goal







• Validation

• …

Image-Space Bandwidth

• We combine bandwidth from various incident directions to get the final bandwidth at a given pixel.

• Taking a max is too conservative, we take the weighted average instead.

• Bandwidth at each pixel

Bandwidth estimate based on max

Bandwidth estimate Rendered image

Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


Variance of the Shading Integrand

• # shading samples ∝ sum of bandwidths, weighted by the illumination times reflectance squared.

• The sum is a conservative approximation of the actual variance of the shading integrand.

Angular bandwidth arriving at the surface

Angular bandwidth of the reflectance

Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


Validation of the Bandwidth and Variance

Estimated variance Measured varianceEstimated bandwidth Measured bandwidth(WFFT)

Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


The Algorithm

• Step 1• Compute g-buffers• Load Illumination and reflectance bandwidth

• Step 2• Estimate bandwidth and variance

• Step 3• Mip-map bandwidth and variance

• Step 4• Shade and up-sample

Video: algorithm



Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


Result (Final Image)

Result (Bandwidth)

Result (Variance)

Timings

• Fast bandwidth/variance estimation

• (8 ms)

(ms)(ms)

Timings

• Computation time scales linearly with the total number of shading samples

Comparison with Reference

Our algorithm(1015 ms)

Similar time(906 ms)

Similar quality(2639 ms)

Comparison with Spherical Gaussian Approximation (Wang et al. 2009)

Fast but different from ground-truth (notice that color changing effects are missing)

Path-traced reference

Interactive but more accurate

Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


ExtensionAdaptive Multi-Sample Anti-Aliasing

Video: adaptive MSAA



Timings

• Sub-linear performance for MSAA

Roadmap

• Related work

• Our technique

• The goal



• Validation



• Extension: MSAA


Conclusion

• Interactively shading dynamic geometry with acquired materials• Shade a fraction of pixels using bandwidth• Adaptively sample shading integrals• Combine shaded pixels using up-sampling

• Exploiting bandwidth to sub-linearly scale MSAA with deferred shading

Future Work

• Local light sources

• Better up-sampling algorithm

• Depth-of-field

• 6D bandwidth computation for SVBRDFs

• Visibility and indirect illumination

Thank you

Interactive rendering of Interactive rendering of acquired materials on dynamic geometryacquired materials on dynamic geometry

using bandwidth predictionusing bandwidth prediction

interactive rendering of acquired materials on dynamic geometry using bandwidth prediction

Documents