interactive brdf estimation for mixed-reality applications
DESCRIPTION
Interactive BRDF Estimation for Mixed-Reality Applications. Martin Knecht, Georg Tanzmeister, Christoph Traxler, Michael Wimmer Institute of Computer Graphics and Algorithms Vienna University of Technology. Motivation. Goal of our mixed reality framework - PowerPoint PPT PresentationTRANSCRIPT
Interactive BRDF Estimation for Mixed-Reality Applications
Martin Knecht, Georg Tanzmeister, Christoph Traxler, Michael Wimmer
Institute of Computer Graphics and Algorithms
Vienna University of Technology
Motivation
Goal of our mixed reality frameworkLight interaction between real and virtual objects
Materials of real objects must be known
Martin Knecht 2
Problem Statement
Material estimation should not need any preprocessing
Use Kinect sensor and fish-eye lense camera for data acquisition
Should run at interactive framerates
Use GPU wherever possible
Should estimate Phong parameters Used in mixed reality framework
3Martin Knecht
Similar to pipeline of Zheng et al. 2009
Estimation Pipeline
4Martin Knecht
Input Data for Estimation
5Martin Knecht
Highlight Removal
6Martin Knecht
Highlight Removal
7Martin Knecht
Diffuse Reflectance Estimation
8Martin Knecht
Diffuse Reflectance Estimation
9Martin Knecht
Inverse shading:
Clustering
10Martin Knecht
Clustering 1/5
Assumption: similar color same material
Same material same specular parameters
Clustering executed on the diffuse estimation
Novel hybrid CPU/GPU K-Means1) Initialize cluster centers
2) Assign pixel to nearest cluster center
3) Calculate new cluster centers
4) Repeat steps 2 & 3
11Martin Knecht
Clustering 2/5
1) Initialize cluster centersRandom cluster centers
Exploit temporal coherenceReuse of cluster centers of previous frame
12Martin Knecht
Clustering 3/5
2) Assign pixel to nearest cluster center
13Martin Knecht
Cluster ShaderRGBC1
RGBC2
...
...
Cluster 1 Cluster 2 Cluster 6 Bitmask 1 Bitmask 2...
Clustering 4/5
3) Calculate new cluster centers
1x1 Mipmap is the average over all pixel
New cluster center:
14Martin Knecht
1x1 RGBD
1x1 Bitmask
TRGBD
T*
Clustering 5/5
4) Repeat steps 2 & 3
Repeat until no pixel changes cluster Standard stopping criteria too conservative
Max. 20 iterations
Check variance change of distances
15Martin Knecht
Clustering 5/5
16Martin Knecht
Specular Reflectance Estimation
17Martin Knecht
Specular Reflectance Estimation
Done on a per cluster basis same material
CPU based nonlinear function solver
Variables: Specular parameter
Light positions
Evaluation of objective function done on GPUSimilar mipmap method used as for clustering
18Martin Knecht
Specular Reflectance Estimation
19Martin Knecht
Results - Estimation
20Martin Knecht
Diffuse component
Phong shaded image
Specular component+
Results – Timings
BRDF estimation runs at ~2.8 fps
Two tasks with major impact K-Means clustering
Specular estimation
Martin Knecht 21
Normal estimation 0,57 msHighlight removal 0,94 msDiffuse estimation 0,23 msK-Means 39,08 msSpecular estimation 315,76 ms
< 0.5 %
~ 11 %
~ 88,5 %
Differential Instant Radiosity
22Martin Knecht
Results – Mixed Reality Integration
23Martin Knecht
Limitations
Kinect sensor does not work everywhere
Bright objects are discarded from estimation
Shadows are not considered
No estimation of optimal amount of clusters
No integration of data over time
Simplifications lower quality of estimation
24Martin Knecht
Conclusion & Future work
BRDF estimation without any preprocessing
Hybrid CPU/GPU K-Means implementation
Runs at interactive framerates
Future workImprove speed specular estimation
Improve quality BRDF estimation
Exploit temporal coherence more often
25Martin Knecht
Thank you for your attention!
Supported by grand from the FFG-Austrian Research Promotion Agency under the Program “FIT-IT Visual
Computing” Project Nr.: 820916
26