1

Segmentation of Salient Regions in Outdoor Scenes

Using Imagery and 3-D Data

Segmentation of Salient Regions in Outdoor Scenes

Using Imagery and 3-D Data

Gunhee Kim Daniel Huber Martial Hebert

Carnegie Mellon University

Robotics Institute

< WACV2008 >

2

Perception for Unmanned VehiclesPerception for Unmanned Vehicles

Imagery

3-D scan

Sensors

Input Data

Vrml_file

3

Problem StatementProblem Statement

• Our ultimate goal is…

• Here, we focus on a crucial pre-requisite step !!

4

Object DetectionObject Detection

• Naïve Scanning

5

Object DetectionObject Detection

• Prioritize the searching regions

Salient Regions

6

Problem StatementProblem Statement

• Saliency Detection using Imagery and 3-D Data

– A Mid-level vision task

– No high-level priors, models, or learning

– Only low-level information

– Ex) pixel colors, (x,y,z)-coordinates

7

Problem StatementProblem Statement

< Input >

1. An image

2. 3D scan data

< Output >

Information-theoreticoptimal clustering

Segmentation of top-k most salient regions

8

Saliency Saliency

• Saliency: The quality of standing out relative to neighboring items

• Top-down

– Driven by high-level concepts

– Memories and Experiences

• Bottom-up

– Driven by low-level features

– Intensity, contrast, color, orientation, and motion

9

Two Bottom-Up ModelsTwo Bottom-Up Models

• Itti-Koch-Niebur Model [1]

[1] L.Itti, C.Koch and E.Niebur A Model of Saliency-based Visual Attention for Rapid Scene Analysis, PAMI, 1998.

10

Two Bottom-Up ModelsTwo Bottom-Up Models

• Kadir-Brady Saliency detector [3]

– Find the region which are locally complex, and globally discriminative.

[3] T.Kadir and M. Brady, Scale, Saliency and Image Description. IJCV 45 (2):83-105, 2001.

More flatter, higher complex

Scale, rotation, affine invariance

11

Previous Work for Saliency in 3-DPrevious Work for Saliency in 3-D

[1] S. Frintrop, E. Rome, A. N¨uchter, and H. Surmann. A bimodal laser-based attention system. CVIU, 2005.[2] D. M. Cole, A. R. Harrison, and P.M. Newman. Using naturally salient regions for slam with 3d laser data, ICRA workshop on SLAM, 2005.

• BILAS [3]: Itti et al [1]’s model

• Cole et al. [4]:

Kadir-Brady [2]’s model

12

Saliency of 3-D Data Saliency of 3-D Data

• A point cloud

• Gestalt laws of grouping [5]

Proximity laws Continuity & Simplicity laws

13

Saliency of 3-D Data Saliency of 3-D Data

• How to detect salient regions in 3-D data?

• (Answer) Find the set of clusters that best fit 3-D pdfs– Gaussian

– Uniform

GaussianUniform

14

Robust Information-Theoretic Clustering (RIC) [6]Robust Information-Theoretic Clustering (RIC) [6]

• Input: A feature set + Families of pdfs

• Output: Clusters according to how well they fit the pdfs

• Minimum Description Length (MDL) principle

– Goodness of fit = compression costs

– Huffman-like coding

[6] C.Bohm,C.Faloutsos, J.Y.Pan, and C.Plant, Robust information-theoretic clustering, KDD 2006

15

Proposed ApproachProposed Approach

Clustering of 3-D Data using RIC

Projection of Clusterson an Image

Compute Saliency Valuesof Clusters

Image Segmentation of Top-ranked Saliency Regions

16

Clustering in 3-D DataClustering in 3-D Data

• RIC clustering

Uniform distGaussian dist

InformationTheoretic Optimal Clustering

17

Proposed ApproachProposed Approach

Clustering on 3-D Data using RIC

Projection of Clustersto an Image

Compute Saliency Valuesof Clusters

Image Segmentation of Top-ranked Saliency Regions

Projection of Clustersto an Image

18

Saliency FeaturesSaliency Features

)( ii Cf )( ie Cf

)( ih Cf

)( is Cf

)( is Cf

3D-data Image data (RGB color)

Local Regional Global

19

Saliency FeaturesSaliency Features

)( ii Cf

Local 3D-data

1. Compression Cost- how well the cluster is fit to the reference family of pdfs

20

Saliency FeaturesSaliency Features

Local RGB-data

2. Entropy of RGB histograms

[7] - Follow Kadir-Brady’s definition

)( ie Cf

[7] T. Kadir and M. Brady. Saliency, scale and image description. IJCV, 45(2):83–105, 2001.

21

Saliency FeaturesSaliency Features

Regional RGB-data

3. Center-surround contrast [8]

- Follow Itti et al’s definition

[8] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum.Learning to detect a salient object, CVPR, 2007.

)( ih Cf

22

Saliency FeaturesSaliency Features

Global RGB-data

4. Color spatial distribution [8] - Rare color: More salient

[8] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum.Learning to detect a salient object, CVPR, 2007.

)( is Cf

)( is Cf

23

Proposed ApproachProposed Approach

Clustering on 3-D Data using RIC

Projection of Clustersto an Image

Compute Saliency Valuesof Clusters

Image Segmentation of Top-ranked Saliency Regions

Projection of Clustersto an Image

24

Image SegmentationImage Segmentation

• From Sparse Points to Dense Regions

• Using Conventional Markov-Random Field (MRF) Models

– Labeling Problem

25

Qualitative ResultsQualitative Results

26

Qualitative ResultsQualitative Results

27

Future WorkFuture Work

• Integration with the Recognition

• Over-segmentation & Under-segmentation

– Model-specific pdfs

28

ConclusionConclusion

• Bottom-up saliency detection using Imagery and 3-D data as a mid-level task

– (x,y,z)-coordinates of 3-D data and Colors at pixels.

• Practically useful building block for perception of unmanned vehicles