Using spatio-temporal Using spatio-temporal probabilistic framework for probabilistic framework for

object trackingobject tracking

By: Guy Koren-BlumsteinBy: Guy Koren-Blumstein

Supervisor: Dr. Hayit Supervisor: Dr. Hayit GreenspanGreenspan

Emphasis on Face Detection & Tracking

AgendaAgenda

►Previous research overview (PGMM)Previous research overview (PGMM)►Under-segmentation problemUnder-segmentation problem►Face tracking using PGMMFace tracking using PGMM

Modeling skin color in [L,a,b] color space – Modeling skin color in [L,a,b] color space – over-segmentation problemover-segmentation problem

►Optical flow – overviewOptical flow – overview►Approaches for using optical flowApproaches for using optical flow►ExamplesExamples

Previous researchPrevious research

► Complementary research to an M.Sc. Thesis Complementary research to an M.Sc. Thesis research conducted by A.Mayer under the research conducted by A.Mayer under the supervision of Dr. H.Greenspan and Dr. J. supervision of Dr. H.Greenspan and Dr. J. Goldberger.Goldberger.

► Research Goal: Building a probabilistic Research Goal: Building a probabilistic framework for spatio-temporal video framework for spatio-temporal video representation.representation.

► Useful for:Useful for: Offline – automatic search in video databasesOffline – automatic search in video databases Online – characterization of events and alerting Online – characterization of events and alerting

on those that are defined as ‘suspicious’on those that are defined as ‘suspicious’

Previous researchPrevious researchParsing clip to

BOF

Build feature Space [L a b]

Build GMM modelIn [Lab] space

Label BOF pixels

Connect. Comp. On [L,a,b,x,y,t]

Learn GMM modelOn [L,a,b,x,y,t]

Source Clip BOF 1 Labeled BOF

Blob Extraction

Under segmentation problem…

Face Detection & TrackingFace Detection & Tracking

►Most of the known techniques can be Most of the known techniques can be divided into two categories :divided into two categories : Search for skin color and apply shape Search for skin color and apply shape

analysis to distinguish between facial and analysis to distinguish between facial and non-facial objects.non-facial objects.

Search for facial features regardless of Search for facial features regardless of pixel color pixel color (eyes,nose,mouth,chin,symmetry etc.)(eyes,nose,mouth,chin,symmetry etc.)

Apply framework to track Apply framework to track facesfaces

►The framework can extract and track The framework can extract and track after objects in an image sequence.after objects in an image sequence.

►Applying shape analysis to each skin-Applying shape analysis to each skin-colored-blob can label the blob as colored-blob can label the blob as ‘face’ or ‘non-face’.‘face’ or ‘non-face’.

►The face will be tracked by virtue of The face will be tracked by virtue of the tracking capabilities of the the tracking capabilities of the frameworkframework

Skin color in [L a b]Skin color in [L a b]

► Skin color is modeled in [a b] components onlySkin color is modeled in [a b] components only► Supplies very good discriminability between ‘skin’ Supplies very good discriminability between ‘skin’

pixels and ‘not-skin’ pixels (high rate of True-Negative)pixels and ‘not-skin’ pixels (high rate of True-Negative)► Not optimal in terms of True-Positive (leads to mis-Not optimal in terms of True-Positive (leads to mis-

detection of skin color pixels)detection of skin color pixels)

Over-segmentation of facesOver-segmentation of faces

► Building blobs is done in [L a b] color space.Building blobs is done in [L a b] color space.► More than one blob might have skin color [a b] More than one blob might have skin color [a b]

componentscomponents► Solution : Unite all blobs whose [a b] are close enough Solution : Unite all blobs whose [a b] are close enough

to the skin color model (adaptive TH can be used)to the skin color model (adaptive TH can be used)

Under SegmentationUnder Segmentation

► Faces moving in front of skin-color Faces moving in front of skin-color background are not extracted well.background are not extracted well.

► Applying shape analysis on the middle map Applying shape analysis on the middle map yields mis-detection of faces.yields mis-detection of faces.

Employing motion Employing motion informationinformation

►Motion information helps to distinguish Motion information helps to distinguish between foreground dynamic objects and between foreground dynamic objects and static backgroundstatic background

►2 levels of motion information2 levels of motion information Binary – indicates for each pixel whether it is Binary – indicates for each pixel whether it is

in motion or not. Does not supply motion in motion or not. Does not supply motion vector. Feature space: [L a b x y t m] where vector. Feature space: [L a b x y t m] where m={0,1}m={0,1}

Optical flow - supplies motion vector Optical flow - supplies motion vector according to a given model. according to a given model. Feature space: [L a b x y t V Feature space: [L a b x y t Vxx V Vyy]]

Is binary information good Is binary information good enough?enough?

Optical FlowOptical Flow

►Optical flow is an apparent motion of Optical flow is an apparent motion of image brightnessimage brightness

► If I(x,y,t) is the brightness, two main If I(x,y,t) is the brightness, two main assumptions can be made:assumptions can be made: I(x,y,t) depends on coordinates x,y in I(x,y,t) depends on coordinates x,y in

greater part of the imagegreater part of the image Brightness of every point of moving object Brightness of every point of moving object

does not change in timedoes not change in time

Optical FlowOptical Flow

► If object is moving during time If object is moving during time dtdt and its and its displacement is (displacement is (dx,dydx,dy) then using Taylor series) then using Taylor series

...),,(),,(

dtt

Idyy

Idxx

ItyxIdttdyydxxI

► According to assumption 2:According to assumption 2:

0...

),,(),,(

dtt

Idyy

Idxx

I

tyxIdttdyydxxI

► Dividing by Dividing by dt dt gives the optical flow equation:gives the optical flow equation:

dt

dyv

dt

dxu

t

Ivy

Iux

I

,

Optical Flow – Block MatchingOptical Flow – Block Matching

► Does not use the Does not use the equation directly.equation directly.

► Divides the image to Divides the image to blocks blocks

► For every block in IFor every block in Itt it it search for the best search for the best matching block in Imatching block in It-1t-1..

► Matching criteria: Cross Matching criteria: Cross Correlation, Square Correlation, Square Difference, SAD etc.Difference, SAD etc.

Working with 8-D feature Working with 8-D feature spacespace

► Connected component Connected component analysis:analysis: Does not require Does not require

initialization of the order initialization of the order of the modelof the model

Hard decision proneHard decision prone

► GMM model via EM:GMM model via EM: Initialized by K means. Initialized by K means.

Requires initialization of Requires initialization of K.K.

Impose elliptic shape on Impose elliptic shape on the objectsthe objects

Soft Decision proneSoft Decision prone

Parsing clip to BOF

Build feature Space [L a b]

Build GMM modelIn [Lab] space

Label BOF pixels

Connect. Comp.On [x,y,t,Vx,Vy]

Learn GMM model [x,y,t,Vx,Vy]

Frame By Frame Tracking

Frame by frame trackingFrame by frame tracking

►Widely used in the Widely used in the literatureliterature

► Can handle variations Can handle variations in object’s velocityin object’s velocity

► Tracking can be Tracking can be improved by improved by employing Kalman employing Kalman filter to predict filter to predict object’s location and object’s location and velocityvelocity

Predict params for next frame

merge blobs

Create new blobs

split blobs

Kill old blobs

Label by updatedparameters

Update blob’s params

Label by predicted parameters

ExamplesExamples

►Opposite directions:Opposite directions: Optical FlowOptical Flow, , Connected componentConnected component ( (

Extracted FacesExtracted Faces), ), GMMGMM ► Same direction, different velocitySame direction, different velocity

Optical FlowOptical Flow, , Connected componentConnected component, , GMM GMM ((FacesFaces))►Different directions – complex backgroundDifferent directions – complex background

Optical FlowOptical Flow, , Connected componentConnected component, GMM: , GMM: K=5K=5,,K=3K=3,,FacesFaces

► Variable velocityVariable velocity Optical FlowOptical Flow, , Connected componentConnected component, , GMMGMM, Frame , Frame

By FrameBy Frame

Real World SequencesReal World Sequences

► Face trackingFace tracking Optical FlowOptical Flow No motion infoNo motion info Connected componentConnected component GMMGMM Frame By FrameFrame By Frame

► Car TrackingCar Tracking Optical FlowOptical Flow No Motion infoNo Motion info GMMGMM

► Flower gardenFlower garden Optical FlowOptical Flow No motion infoNo motion info Connected componentConnected component GMMGMM

SummarySummary

► Applying probabilistic framework to track Applying probabilistic framework to track faces in video clipsfaces in video clips

►Working in [L,a,b] color space to detect Working in [L,a,b] color space to detect facesfaces

►Handling over segmentation Handling over segmentation ►Handling under segmentation by employing Handling under segmentation by employing

optical flow information in 3 different ways:optical flow information in 3 different ways: Connected Component AnalysisConnected Component Analysis Learning GMM modelLearning GMM model Frame By Frame trackingFrame By Frame tracking

Further ResearchFurther Research

►Adaptive face color modelAdaptive face color model►Variable length BOF (using MDL)Variable length BOF (using MDL)►Using more complex motion modelUsing more complex motion model

Thank you for Thank you for listeninglistening

Questions ?Questions ?