planematch: patch coplanarity prediction for …planematch: patch coplanarity prediction for robust...
TRANSCRIPT
![Page 1: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/1.jpg)
PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction
Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,
Thomas Funkhouser
Princeton University National University of Defense Technology
Technical University of Munich Google
![Page 2: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/2.jpg)
RGB-D Reconstruction
Microsoft Kinect Structure Sensor Xtion
![Page 3: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/3.jpg)
RGB-D ReconstructionB
un
dle
Fu
sio
n [
Dai
et
al. 1
7]
![Page 4: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/4.jpg)
RGB-D Reconstruction
BundleFusion[Dai et al. 2017]
ElasticFusion[Whelan et al. 2016]
VoxelHashing[Niessner et al. 2013]
KinectFusion[Newcombe/Izadi et al. 2011]
Robust Recon.[Choi et al. 2015]
![Page 5: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/5.jpg)
Vo
xelH
ash
ing
Bu
nd
leFu
sio
n
Loop Closure
![Page 6: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/6.jpg)
Loop Closure -> Feature Descriptor
RGB Features:
- SIFT, SURF, ORB, Freak, …
- LIFT, MatchNet, …
Geometric Features:
- SHOT, FPFH, SpinImages, …
- 3DMatch, …
Are there additional primitives?
Keypoint-based
![Page 7: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/7.jpg)
Our Idea: Planar Feature Descriptors
Coplanar
Surface
Patches
![Page 8: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/8.jpg)
Existing Planar Matching is Local
Online Structure Analysis [Zhang et al. 2015]
Fine-to-Coarse Registration [Halber and Funkhouser 2017]
Point-to-Plane ICP [Chen & Medioni 91]
![Page 9: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/9.jpg)
Long-Range Constraints for SLAMCoplanar
Surface Patches
![Page 10: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/10.jpg)
Task: Co-planarity Matching?
… …
![Page 11: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/11.jpg)
PlaneMatch: Learning Co-planarity Features
➢Color
➢Depth
➢Normals
➢Plane Segmentation (Mask)
➢…
…Learn from 3D data!
![Page 12: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/12.jpg)
Siamese Network Architecture
256D
256D
256D
![Page 13: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/13.jpg)
Siamese Network Architecture
256D
256D
256D
![Page 14: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/14.jpg)
Siamese Network Architecture
256D
256D
256D
![Page 15: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/15.jpg)
Siamese Network Architecture
256D
256D
256D
![Page 16: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/16.jpg)
Step 1: Extract Planar Patches
RGB
Depth
Planar Patches
![Page 17: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/17.jpg)
Step 2: Extract Global Rep. / Patch
RGB
Patch Mask
Depth
Normals
![Page 18: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/18.jpg)
Step 3: Extract Local Rep. / Patch
RGB
Patch Mask
Depth
Normals
![Page 19: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/19.jpg)
Local / Global Representations
Local
Representation
Global
Representation
![Page 20: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/20.jpg)
Siamese Network Architecture
256D
256D
256D
![Page 21: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/21.jpg)
Training: Self-Supervised Learning
ScanNet [Dai et al. 2017]
Anchor Positive Negative
Anchor Positive Negative…
10 million triplets
![Page 22: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/22.jpg)
Triplets for Training
Positive NegativeAnchor
![Page 23: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/23.jpg)
Benchmark for Task of Co-planarity Matching
Positive pair (6k) Negative pair (6k)
By patch size
By pair distance
![Page 24: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/24.jpg)
PlaneMatch Evaluation
![Page 25: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/25.jpg)
PlaneMatch Ablation Study
![Page 26: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/26.jpg)
PlaneMatch Registration
: transformation matrix : indicator variables (∈ [0,1] )
![Page 27: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/27.jpg)
PlaneMatch Registration
: transformation matrix : indicator variables (∈ [0,1] )
: plane pair set : plane-to-plane distance
: confidence weight
Pairspredicted by coplanarity
network
cop
cop
cop cop
π
![Page 28: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/28.jpg)
PlaneMatch Registration
: transformation matrix : indicator variables (∈ [0,1] )
: point pair set : point-to-point distance
: confidence weight
kp kp
kp
kp
π
Pairspredicted by
SIFTkeypoints
![Page 29: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/29.jpg)
PlaneMatch Registration
: transformation matrix : indicator variables (∈ [0,1] )
: threshold for error (0.01 m)
If > , = 0If < , = 1
Robust optimization following [Choi et al. 15] / [Zollhoefer et al. 14] / [Zach et al. 14]
![Page 30: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/30.jpg)
PlaneMatch Registration Results
.
![Page 31: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/31.jpg)
PlaneMatch Registration Results
PlaneMatch (Ours)BundleFusion [Dai et al.17]
![Page 32: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/32.jpg)
PlaneMatch Registration Results
PlaneMatch (Ours)BundleFusion [Dai et al.17]
![Page 33: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/33.jpg)
Evaluation on TUM-RGBD
RMSE in cm (lower is better)
![Page 34: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/34.jpg)
Ablation on TUM-RGBD
RMSE in cm (lower is better)
![Page 35: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/35.jpg)
Effect of Long-range Co-planar Pairs
50% deduction 100% deduction0% deduction
1-5m
1-5m
1-5m
1-5m
![Page 36: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/36.jpg)
Effect of Long-range Co-planar Pairs
50% deduction 100% deduction0% deduction
1-5m
1-5m
1-5m
1-5m
![Page 37: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/37.jpg)
Effect of Long-range Co-planar Pairs
50% deduction 100% deduction0% deduction
1-5m
1-5m
1-5m
1-5m
![Page 38: PlaneMatch: Patch Coplanarity Prediction for …PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz,](https://reader033.vdocuments.net/reader033/viewer/2022060509/5f25dcb752b5663491443b90/html5/thumbnails/38.jpg)
Conclusion
1. New task: co-planarity matching
2. Feature learning using self-supervision
3. Integration with robust optimization into SLAM
Yifei Shi Kai Xu Tom FunkhouserSzymon Rusinkiewicz
Thank You!