general and robust error estimation and reconstruction for ...€¦ · results –gid („removing...
TRANSCRIPT
![Page 1: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/1.jpg)
General and Robust Error Estimation and Reconstruction for Monte Carlo Rendering
Pablo Bauszat1, Martin Eisemann1,2, Elmar Eisemann2, Marcus Magnor1
1 Computer Graphics Lab, TU Braunschweig, Germany2 Delft University of Technology, Netherlands
![Page 2: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/2.jpg)
Monte Carlo Rendering
• Today‘s industry standard
• General and unbiased
• Covers variety of natural phenomena
• Requires extensive sampling• Pixel (2D integral)• Camera lens (2D integral)• Time (1D integral)• Global illumination (2D integral per bounce)• … and more …
2
![Page 3: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/3.jpg)
Noise
3
![Page 4: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/4.jpg)
Filtering
4
Noisy Reference
Uniform filter(small)
Uniform filter(large)
Adaptive filtering
![Page 5: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/5.jpg)
Adaptive Reconstruction
• Filter bank• Set of filters with different properties
• Select best filter on a per-pixel level
5
Filter bank
![Page 6: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/6.jpg)
Problem statement
How to choose the best filterfrom the set for a pixel?
6
![Page 7: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/7.jpg)
Previous work
7
Li et al. 2012Overbeck et al. 2009 Rousselle et al. 2011/2012/2013
Kalantari et al. 2013 Moon et al. 2014
![Page 8: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/8.jpg)
Limitations of previous work
• Filter selection based on noisy image
• Often tailored for specific filters
• Switching filters may cause seams
8
Local selection
![Page 9: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/9.jpg)
Our method
9
![Page 10: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/10.jpg)
Insights
Our method is based on three key insights:
1. Filter selection is often more crucial than sampling rate
2. Filter error is locally smooth for most image regions
3. Often multiple filters are close-to-optimal choices
10
![Page 11: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/11.jpg)
1. Filter selection is often more crucial than sampling rate
11
Filter bank of 4 Gaussian and4 Joint Bilateral filters
32 sppSURE
12.3 MSE-3
32 sppBest choice
1.6 MSE-3 (x 7.7)
16 sppBest choice
2.3 MSE-3 (x 5.3)
Recently employedby [Li2012] and[Rousselle2013]
![Page 12: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/12.jpg)
1. Filter selection is often more crucial than sampling rate
12
Scene SURE32 spp
Best choice32 spp
Best choice16 spp
Conference 12.327 1.605 (x 7.7) 2.344 (x 5.3)
Sibenik 0.758 0.157 (x 4.8) 0.258 (x 2.9)
Toasters 0.187 0.096 (x 1.9) 0.156 (x 1.2)
San Miguel 16.880 6.419 (x 2.6) 9.831 (x 1.7)
Mean squared error (MSE) * 10-3 – Same filter bank
![Page 13: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/13.jpg)
Insights
Our method is based on three key insights:
1. Filter selection is often more crucial than sampling rate
2. Filter error is locally smooth for most image regions
3. Often multiple filters are close-to-optimal choices
13
![Page 14: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/14.jpg)
2. Error smoothness – Gaussian filters
14
Gaussian σ=7 Gaussian σ=11 Gaussian σ=13
![Page 15: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/15.jpg)
2. Error smoothness – Guided Image Filtering [He2010]
15
Guided radius=4 Guided radius=8 Guided radius=16
![Page 16: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/16.jpg)
Insights
Our method is based on three key insights:
1. Filter selection is often more crucial than sampling rate
2. Filter error is locally smooth for most image regions
3. Often multiple filters are close-to-optimal choices
16
![Page 17: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/17.jpg)
3. Often multiple filters are close-to-optimal choices
17
Reference Filter A Filter B Filter C
![Page 18: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/18.jpg)
3. Often multiple filters are close-to-optimal choices
18
Optimal selection via ground truth
• MSE down to 8.0% from noisy image
Regularized selection
• MSE down to 8.4% from noisy image
• Variations in selection are penalized
Regularized selection
• MSE down to 9.1% from noisy image
• Variations in selection are penalized
![Page 19: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/19.jpg)
What do we learn from the insights?
• Filter selection is crucial
• Filter error is piece-wise smooth
• Non-optimal filter selection does not imply large error
19
![Page 20: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/20.jpg)
Our Method
20
Filter bankgeneration
1
Sparsereference pixels
2
Dense errorinterpolation
4
Filter compositing
5
Sparse errorcomputation
3
![Page 21: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/21.jpg)
1. Filter bank generation
21
16 spp
Sample Budget
Filter bank
Filter 1 Filter 2 Filter n…
… … …
32 spp16 spp
![Page 22: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/22.jpg)
2. Sparse reference pixels
22
16 spp
Sample Budget
Filter bank
Filter 1 Filter 2 Filter n…
… … …
16 spp
128 spp per reference pixel
![Page 23: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/23.jpg)
3. Sparse error computation
23
16 spp
Sample Budget
Filter bank
Filter 1 Filter 2 Filter n…
… … …
128 spp per filter cache
• Serves as reference
•Used to estimate filter error
• Low-variance estimator
![Page 24: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/24.jpg)
4. Dense error interpolation
• Interpolation of sparse error estimate (per filter)
24
Sparse error(zoom-in)
Interpolated error(zoom-in)
Filter error using reference(zoom-in)
![Page 25: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/25.jpg)
4. Dense error interpolation
25
• Best selection from interpolated error leads to seams
Optimal selection(per-pixel)
Seams(closeup)
![Page 26: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/26.jpg)
5. Filter compositing
Globally optimize filter selection (seek labeling 𝐿)
argmin𝐿
𝐸 𝐿
Globally optimize filter selection (seek labeling 𝐿)
argmin𝐿
𝐸 𝐿 = 𝐸𝐷𝑎𝑡𝑎 𝐿
26
Data termLocal errormaps
Regularization termSolution image gradients
Minimize MSE Avoid seams
Globally optimize filter selection (seek labeling 𝐿)
argmin𝐿
𝐸 𝐿 = 𝐸𝐷𝑎𝑡𝑎 𝐿 + 𝜆 ∙ 𝐸𝑟𝑒𝑔𝑢𝑙𝑎𝑟𝑖𝑧𝑒𝑟 𝐿
![Page 27: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/27.jpg)
5. Filter compositing
27
• Solve by graph-cuts„Fast approximate energy minimization via graph cuts”, Boykov et al. 2001
Globally optimized label map
Filter 1 Filter 2 Filter n…
… … …
Cut
![Page 28: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/28.jpg)
5. Filter compositing
28
• Solve by graph-cuts„Fast approximate energy minimization via graph cuts”, Boykov et al. 2001
Local selection Global selection
![Page 29: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/29.jpg)
Our Method
29
Filter bankgeneration
1
Sparsereference pixels
2
Dense errorinterpolation
4
Filter compositing
5
Sparse errorcomputation
3
![Page 30: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/30.jpg)
Bells & Whistles
• Choice of regularization in filter compositing
• Integration of high-quality radiance values (not included the filter bank)
• Select „best“ pixels for sparse error estimate
30
![Page 31: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/31.jpg)
Adaptive placement of sparse estimates
• Required for highly variant error regions
• Reduces residual variance in radiance estimate
31
Filter bankvariance
Monte Carlovariance
Poissonsampling
Importancesampling
![Page 32: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/32.jpg)
Results
32
![Page 33: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/33.jpg)
Results – San Miguel
Global illumination
33
MC 4096 spp15,449 sec
Our result 32 spp146 + 13 sec
MC 32 spp146 sec
![Page 34: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/34.jpg)
Results - Chess
Depth-of-field
34
MC 4096 spp1,492 sec
Our result 8 spp9 + 29 sec
MC 8 spp9 sec
![Page 35: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/35.jpg)
Results - Poolball
Motion blur
35
MC 4096 spp10,989 sec
Our result 8 spp25 + 25 sec
MC 8 spp25 sec
![Page 36: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/36.jpg)
Results - Teapot
Glossy materials
36
MC 4096 spp3,619 sec
Our result 16 spp14 + 8 secMC 16 spp
14 sec
![Page 37: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/37.jpg)
Results - Dragon
Participating media
37
MC 4096 spp12,464 sec
Our result 32 spp95 + 12 sec
MC 32 spp95 sec
![Page 38: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/38.jpg)
Results - Timings
38
Intel Core i7-2600, 3.40 GHz, 16 GB RAM, NVIDIA GeForce 780 GTX, Windows 7 64-bitRendered with PBRT 2 path tracing.
8 filter 4 filter 8 filter 4 filter 8 filter 4 filter 8 filter 4 filter 8 filter 4 filter
Rendering Filtering Error estimation Filter composite
![Page 39: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/39.jpg)
Error analysis
39
Two error sources
Residual variance in radianceInterpolation error
![Page 40: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/40.jpg)
Results – GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari et al. 2013)
40
GID (8 spp) Ours (8 spp) GID (32 spp) Ours (32 spp) Reference
Chess scene MSE=2.6491SSIM=0.9516
MSE=1.38179SSIM=0.9874
MSE=2.4006SSIM=0.9558
MSE=0.8962SSIM=0.9948
![Page 41: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/41.jpg)
Results – RD(„Robust Denoising using Feature and Color Information”, Rousselle et al. 2013)
41
RD (16 spp) Ours (16 spp) RD (32 spp) Ours (32 spp) Reference
Dragon scene MSE=13.6693SSIM=0.9654
MSE=10.1914SSIM=0.9599
MSE=9.3887SSIM=0.9781
MSE=7.8838SSIM=0.9768
![Page 42: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/42.jpg)
Error sparsity
• Sparsity of error maps in transform domain (CDF 9/7 wavelets)
• Redundant information
42
Gaussian σ=7 Gaussian σ=11 Gaussian σ=13
86.46% 88.58% 89.86%
Guided radius=4 Guided radius=8 Guided radius=16
81.34% 87.07% 89.43%
NLM BM3D BLS-GSM
60.06% 67.35% 73.62%
![Page 43: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/43.jpg)
Results – SURE [Stein1981]
43
Noisy SURE Our approach Reference
Sibenik scene MSE=6.0644SSIM=0.9066
MSE=0.7681SSIM=0.9643
MSE=0.3556SSIM=0.9829
![Page 44: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/44.jpg)
Conclusion
• Summary• Redistributing samples can improve filter selection
• Global filter selection removes image seams
• Benefits• Works with arbitrary filters
• No assumptions regarding scene and image content
• Easy integration into existing rendering frameworks
44
![Page 45: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/45.jpg)
Outlook
• Investigate other interpolation schemes
• Adaptive sampling feedback loop
• Temporal coherence
45
![Page 46: General and Robust Error Estimation and Reconstruction for ...€¦ · Results –GID („Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms”, Kalantari](https://reader036.vdocuments.net/reader036/viewer/2022071023/5fd85b14dbfa5c278a690755/html5/thumbnails/46.jpg)
Thank you for your attention!
46
Pablo Bauszat Elmar EisemannMartin Eisemann Marcus Magnor
graphics.tudelft.nl
graphics.tu-bs.de