Semantically Integrating Laser and Vision in Pedestrian Detection

Luciano Oliveira

Advisors:Prof. Urbano NunesProf. Paulo Peixoto

Motivation

SegmentationRecognition

TrackingSearching

Clustering methods

Kalman Filter

Efficient sub-window searching

(image)

Where is the pedestrianin the scene?

Goals

Object detectionusing laser/vision

Proof-of-concept: pedestrian detection, but can be applied to several other objects

Recover objectlocalization

DO NOT entirelyrely on laser, as previous methods do

Perform the fusion in a context-aware mode

Overview of the proposed method

Coarsesegmentation

Finesegmentation

Sensor registration

Parts-based ensemble detector

Semantic/contextualinterpretation

3D sliding windowsearching

Inference and decision outputs

Procrustesanalysis

Referenceshapes

Laser points

For each2D window

(label, confidence) for each fm

Images

For each3D window

(object, confidence)

Templating matching

Laser-image registration

Laser segmentation and labeling

MLN

HLSM-FINT

Ground MRF

{ }Nnnc 1=

{ }Mmmf 1=

for each cn

Experimental setupPointgrey camera

Sick LMS200 laser

Odometry

Sensor-driven detectors

Coarsesegmentation

Finesegmentation

Sensor registration

Parts-based ensemble detector

Semantic/contextualinterpretation

3D sliding windowsearching

Inference and decision outputs

Procrustesanalysis

Referenceshapes

Laser points

For each2D window

(label, confidence) for each fm

Images

For each3D window

(object, confidence)

Templating matching

Laser-image registration

Laser segmentation and labeling

MLN

HLSM-FINT

Ground MRF

{ }Nnnc 1=

{ }Mmmf 1=

for each cn

Ensemble of classifiers HFI

Perimeter rate

Distance / max(w,w´)

Fuzzy System

C1 scaled score

C2 scaled score

FuzzySystem

Join

t co

nfid

ence

Inte

rsec

tion

rate

Fina

l con

fiden

ceFuzzy

System

Hierarchical Fuzzy IntegrationFuzzy inputs

C2

C1

It suffers from exponential growing of rules and low overall performance over challenging situations

Drawbacks

Initially evaluated on Haar-like features / Adaboost and HOG / SVM classificationsystems

Ensemble of classifiers HLSM-FINT

HLSM-FINT – Rationale

• CNN – expert in background (BG) (60% of hit rate in NiSIS competition)• HOG/SVM – expert in objects (OB) (70% of hit rate in NiSIS competition)• Fuzzy integral (Sugeno) – providesa comprehensive framework andgreat synergism• 95.67% of hit rate in NiSIScompetition over 6125 croppedimages (ped + non-ped), usingHeuristic Majority Vote method• 96.4% of hit rate over fullDaimlerChrysler datasets : ~15.000 images

BG

BG BG

BG OB

OB

Parts-based HLSM-FINT

Upper HLSM-FINT (shoulder + header)

Lower HLSM-FINT (waist)

Laser detector

Laser detector

arm armtorso

armtorso

torsoarm armpartial

segment

• Featureless approach

• Coarse-to-fine segmentation

• Relative Neighboorhood Graph (RNG) clustering + clustering index

• Procrustes Analysis (PA) labeling procedure

Laser detector

Laser detector

Occlusion problem:

• z-buffer analysis +

• angle between start and endpoint (proportional to laser angle resolution)

Laser-image registration

Coarsesegmentation

Finesegmentation

Sensor registration

Parts-based ensemble detector

Semantic/contextualinterpretation

3D sliding windowsearching

Inference and decision outputs

Procrustesanalysis

Referenceshapes

Laser points

For each2D window

(label, confidence) for each fm

Images

For each3D window

(object, confidence)

Templating matching

Laser-image registration

Laser segmentation and labeling

MLN

HLSM-FINT

Ground MRF

{ }Nnnc 1=

{ }Mmmf 1=

for each cn

Laser-image registration

Zhang and Pless’ calibration method(with an error of 6 mm in the calibration)

Semantic Fusion

Coarsesegmentation

Finesegmentation

Sensor registration

Parts-based ensemble detector

Semantic/contextualinterpretation

3D sliding windowsearching

Inference and decision outputs

Procrustesanalysis

Referenceshapes

Laser points

For each2D window

(label, confidence) for each fm

Images

For each3D window

(object, confidence)

Templating matching

Laser-image registration

Laser segmentation and labeling

MLN

HLSM-FINT

Ground MRF

{ }Nnnc 1=

{ }Mmmf 1=

for each cn

Semantic fusion

Semantic fusion

MRF

Wi

• MRFs given by FOL formulas• Weights given by the MRF training (gradient ascent method over theconditonal log-likelihood)

Semantic fusion – Examples

ConclusionsHFI has achieved better performance than its components, but failedto get the gist of the fusionHLSM-FINT has succeeded to capture the aimed synergism of thefusion, but has had difficulties on hard situations (e.g. occlusion). Parts-based occlusion has improved this issue.The introduction of the laser sensor has brought significantimprovementThe proposed fusion method offers two main advantages:

Contextual and spatial relationship among the parts of theobject, dropping the false alarm rateIt is able to detect the object in spite of laser failing

The whole system is not able to run on-the-fly, although there is no code optimization. Nevertheless, parallel hardware can provideinteresting plataform to make the system faster. It will be subject offuture research.

Publications and awardsJournals

OLIVEIRA, L.; NUNES, U.; PEIXOTO, P.; SILVA, M. and MOITA, F. SemanticFusion of Laser and Vision in Pedestrian Detection, Journal of PatternRecognition, Elsevier, accepted for publication (ISI impact factor: 3.279).OLIVEIRA, L.; NUNES, U. and PEIXOTO, P. On Exploration of ClassifierEnsemble Synergism in Pedestrian Detection, IEEE Transactions onIntelligent Transportation Systems, pp. 16-21, 2010 (ISI impact factor: 2.844).

Awards3rd place in Intel/GV Entrepreneurship and Venture Capital Competition(2008)1st place in NiSIS Competition - Best accuracy model over Daimler Chrysler image dataset. Scheme of Primate's Visual Cortex Cells for Pedestrian Recognition (2007)

5 international conferences