Exploiting Segmentation for Robust 3D Object Matching Michael Krainin, Kurt Konolige, and Dieter Fox

2

Shape-based Pose Quality Metrics

Given an object model (triangle mesh) and a depth map,

Should not assume any object texture Should be robust to clutter and occlusion

3

Outline

Progression of quality metrics Iterative Closest Point Beam-based Sensor Models Segmentation Sensor Model

Key Contributions Gradient-based optimization for beam-based models

Improves on ICP by reasoning about free-space Novel sensor model framework

Uses segmentation to relax beam-independence assumptions

Explicitly reasons about surface extents

4

ICP Error Metric

Mean squared distance for pairs of closest points [Besl & McKay‘92]

No notion of amount of model or scene explained by the correspondences

Discards viewpoint/free-space information

5

x xx

Beam-Based Sensor Models

Maximize data likelihood:

Typically assume pixel independence:

Measured Depth – Rendered Depth0l

p(p

ixel |

mod

el, p

ose

)

x

x

x

x

6

Beam Model Sensor Models

Beam-based sensor model properties: Consider viewpoint Prefer to explain more pixels using the model

Use in pose estimation: 2D particle-based localization [Thrun et al. ‘05]

Coarse to fine grid search for body tracking [Ganapathi et al. ‘10]

Annealed particle filters for vehicle detection and tracking [Petrovskaya & Thrun ‘09]

Has not been used with gradient-based optimization

7

Beam Model Optimization

We propose a gradient-based optimization

Levenberg-Marquardt using the g2o graph optimization framework [Kümmerle et al. ‘11]

Error function evaluations by re-rendering OpenGL rendering, CUDA sensor model

evaluation 1 ms per evaluation on a mid-range graphics

card ~500 evaluations per g2o optimization

ICP

Beam Optimization

observed rendered

8

Surface Extents

Can still match to the wrong surface

Idea: Use size of surfaces to rule out matches like these

9

Segmentation

(Over-)Segmentation as an estimate of surface extents

Connected components using depths and normals

10

Segmentation Sensor Model

Usual beam-based sensor model:

Segmentation model:

Allows consistent classification of entire segment as being generated from the model or not

11

Segmentation Sensor Model

Let S be a partition (segmentation) of D. Then given model M and pose T,

Let mi be an indicator for whether Si was generated from M

Segment pixels conditionally independent given classification

12

Error Functions

Standard Beam-Based Model Segmentation Model

x

13

Test Data

Recorded 46 cluttered scenes with ground truth poses

14

Pose Estimation Results

Optimization algorithms:

Random restart evaluation functions (upper bounded by 93% from previous result):

ICP Beam Optimization

Successful convergence [%] (exists a correct end pose among 20 random restarts)

86 ± 3 93 ± 2

ICP Mean Squared Error

Standard Beam Model

Segmentation Beam Model

Successful matches [%](determines correct end pose among 20 restarts)

79 ± 3 67 ± 4 85 ± 3

15

Robustness to Segmentation Granularity

Spam 1 of 5 0 degrees

8 degrees 12 degrees

x

x

x

(Fixed to 8 degrees in other experiments)

16

Conclusions

Contributions Novel formulation for beam-based sensor models Demonstration of gradient-based approach for optimizing

beam-based sensor models All code and test data freely available

Future Work Detecting failed segmentations Extensions such as color and normal direction Directly optimizing the segmentation model