3D Object Modelling 3D Object Modelling and Classificationand Classification

Intelligent Robotics Research Centre (IRRC)

Department of Electrical and Computer Systems Engineering

Monash University, Australia

Visual Perception and Robotic Manipulation

Springer Tracts in Advanced Robotics

Chapter 4Chapter 4

Geoffrey Taylor

Lindsay Kleeman

2Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ContentsContents

• Introduction and motivation.

• Split-and-merge segmentation algorithm

• New method for surface type classification based on Gaussian image and convexity analysis

• Fitting geometric primitives

• Experimental results

• Conclusions

3Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

IntroductionIntroduction

• Motivation: enablea humanoid robotto perform ad hoctasks in a domesticor office environment.

• Flexibility in anunknown environmentrequires data driven segmentation to support object classification.

Metalman: an upper-torsohumanoid robot

4Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

IntroductionIntroduction

• Object modelling in robotic applications:

– CAD models (Kragić, 2001)

– Generalized cylinders (Rao et al, 1989)

– Non-parametric (Müller & Wörn, 2000)

– Geometric primitives (Yang & Kak, 1986)

• Many domestic objects can be adequately modelled with geometric primitives.

• Colour/range data provided by robust stereoscopic light stripe scanner (Taylor et al, 2002).

5Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

SegmentationSegmentation

• Basic techniques:– Region Growing: iteratively grow seed segments.

– Split-and-Merge: find region boundaries.

– Clustering: transform and group points.

• Region growing requires accurate range data for fitting primitives to small seed regions.

• Split-and-Merge maintains large regions that can be robustly fitted to primitives.

6Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

SegmentationSegmentation

• Raw range/colour data from stereoscopic light stripe camera.

• Calculate normal vector and surface type for each range element.

7Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

SegmentationSegmentation

• Remove range discontinuities and creases.

• Fit primitives.

• Compare best model to dominant surface type.

• Split poorly modelled regions by surface type and fit primitives again.

8Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

SegmentationSegmentation

• Iteratively grow regions by adding unlabelled pixels that satisfy model.

• Merge regions using iterative boundary cost minimization to compensate for over-segmentation.

9Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

SegmentationSegmentation

• Extract primitives and add texture using projected colour data.

• Use models for object classification, tracking and task planning

10Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

Surface TypeSurface Type

• Determine local shape of NxN element patch:

11Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

Classification methodsClassification methods

• Conventional method:– Fit surface, calculate mean and Gaussian curvature

– Classify based on curvature sign ( > 0, < 0, = 0) Sensitive to noise (second-order derivatives required) Arbitrary approximating function introduces bias.

• Our novel method:– Based on convexity and principal curvatures.

– Non-parametric (no approximating surface)

– Robust to noise

12Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ClassificationClassification

Number of non-zero principal curvatures

Con

vexi

ty

conv

ex

co

ncav

e

ne

ithe

r

Zero One Two

13Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

Principal CurvaturesPrincipal Curvatures

• Determine number of principal curvatures from Gaussian image of surface patch.

Surface representation Gaussian image

14Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

Principal CurvaturesPrincipal Curvatures

• Spread of normal vectors in Gaussian image of patch indicates non-zero principal curvature.

plane

ridge/valley pit/peak/saddle

15Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

• Align central normal to z-axis.

• Measure spread in direction using MMSE:

• Optimize with respect to • Two solutions:

(, e)max and (, e)min

• Non-zero curvature whenemax > eth or emin > eth

Principal CurvaturesPrincipal Curvatures

2)sincos( ii yxe

min

max

y

x

amin

16Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ConvexityConvexity

Convex Concave

n0

n1

d

n1 x n0

(n1 x n0) x dn1

n0

d

n1 x n0

(n1 x n0) x d

17Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

• For each element in patch, calculate:

• Let S=Ncv/Ncc, ratio of convex to concave elements.

• Global convexity given by dominant local property:

ConvexityConvexity

concave : 0

convex : 0 ])[( 001 ndnn

convex (peak, ridge): S > Sth

concave (pit, valley): S < 1/Sth

neither (plane, saddle): 1/Sth < S < Sth

18Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

Surface Type SummarySurface Type Summary

principal curvatures

convexity

raw 3D scan surface type

19Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

• Planes:– Principal component analysis

• Spheres, cylinders, cones:– Minimize distance to fitted surface:

– Levenberg-Marquardt numerical optimization.

– Initial estimate of parameters required.

• Choose model with minimum error, e < eth.

Fitting PrimitivesFitting Primitives

Ri

i fe 2|)(|)( pmp

20Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

Cylinder EstimationCylinder Estimation

• Estimate cylinder axis from Gaussian image:

min

y

x

a

Cylindrical region and axis

Gaussian image and direction of minimum spread

a

21Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ResultsResults

• Box, ball and cup:

Raw colour/range scan Discontinuities, surface type

22Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ResultsResults

• Box, ball and cup:

Region growing, merging Extracted object models

23Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ResultsResults

• Bowl, funnel and goblet:

Raw colour/range scan Discontinuities, surface type

24Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ResultsResults

• Bowl, funnel and goblet:

Region growing, merging Extracted object models

25Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ResultsResults

• Comparison with curvature-based method:

Besl and Jain, 1988Non-parametric result

26Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics

ConclusionsConclusions

• Split-and-merge segmentation using surface type and geometric primitives is capable of modelling a variety of domestic objects using planes, spheres, cylinders and cones.

• New surface type classifier based on principal curvatures and convexity provides greater robustness than curvature-based methods without additional computational cost.