bmc 2012 - invited talk

Background Modeling and Foreground Detection for Video Surveillance:

Recent Advances and Future Directions

Thierry BOUWMANSAssociate Professor

MIA Lab - University of La Rochelle - France

2

Plan

Introduction Fuzzy Background Subtraction Background Subtraction via a

Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal

Component Analysis (RPCA) Conclusion - Perspectives

3

Goal

Detection of moving objects in video sequence.

Pixels are classified as: Foreground (F)Background(B)

Séquence Pets 2006 : Image298 (720 x 576 pixels)

4

Background Subtraction Process

Background Maintenanc

e

ForegroundDetection

Background Initialization

F(t)Foreground Mask

Video

t ≤ N

t > N

t ≥ N t=t+1

N images

I(t+1)N+1

Incremental Algorithm

Classification task

Batch Algorithm

5

Related Applications

Video surveillance Optical Motion Capture Multimedia Applications

Projet ATON – Université de Californie San Diego

Séquence Danse [Mikic 2002] Séquence Jump [Mikic 2002]Projet Aqu@theque – Université de La Rochelle

6

Background Subtraction

Processing

Acquisition

Convex Hull

Pattern Recognition

Tracking

On the importance of the background subtraction

7

Challenges

Critical situations which generate false detections :

Shadows -

Illumination variations…

Source : Séquence Pets 2006 Image 0298 (720 x 576 pixels)

Multimodal Backgrounds

Rippling Water

Water Surface

Camera Jitter

Waving Trees

8

Source: http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html

9

Background Subtraction Web Site: References (553), datasets (10) and codes (27).

Statistical Background Modeling

Source: http://sites.google.com/site/backgroundsubtraction/Home.html (6256 Visitors, Source Google Analytics).

10

Plan

Introduction Fuzzy Background Subtraction Background Subtraction via a

Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal

Component Analysis (RPCA) Conclusion - Perspectives

11

Fuzzy Background Subtraction

A survey in Handbook on Soft Computing for Video Surveillance, Taylor and Francis Group [HSCVS 2012]

Three approaches developed at the MIA Lab: Background modeling by Type-2 Fuzzy Mixture of

Gaussians Model [ISVC 2008]. Foreground Detection using the Choquet Integral

[WIAMIS 2008][FUZZ’IEEE 2008] Fuzzy Background Maintenance [ICIP 2008]

12

Weakness of the original MOG1. False detections due to the matching test

13

Weakness of the original MOG2. False detections due to the presence of outliers in the training

step

Exact distribution

14

Mixture of Gaussians with uncertainty on : the mean and the variance [Zeng 2006]

(T2 FMOG-UM) (T2 FMOG-UV)

15

Mixture of Gaussians with uncertainty on the mean(T2 FMOG-UM)

: Intensity vector in the RGB color space

16

Mixture of Gaussians with uncertainty on the variance (T2 FMOG-UV)

: Intensity vector in the RGB color space

17

Classification B/F by T2-FMOG

Matching test:

Classification B/F as the MOG ⇒

Results on the “SHAH” dataset(160 x 128 pixels) – Camera Jitter

Original sequence MOG

T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)18

Video at http://sites.google.com/site/t2fmog/

Results on the “SHAH” dataset(160 x 128 pixels) – Camera Jitter

Method Error Type

Image 271

Image 373

Image 410

Image 465

Total Error

Variation in %

MOG FNFP

02093

11204124

48182782

20501589 18576

T2-FMOG-UM FNFP

0203

1414153

6043252

252046 10631 42,77

T2-FMOG-UV FNFP

03069

9571081

22171119

10691158 10670 42.56

19

Results on the “SHAH” dataset(160 x 128 pixels) – Camera Jitter

20

[Stauffer 1999]

[Bowden 2001] – Initialization [Zivkovic 2004] – K is variable

Results on the sequence “CAMPUS” (160 x 128 pixels) – Waving Trees

Original Sequence MOG

T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)21

Video at http://sites.google.com/site/t2fmog/

Resultat on the sequence “Water Surface” (160 x 128 pixels) – Water Surface

Original Sequence MOG

T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)22

Video at http://sites.google.com/site/t2fmog/

23

Fuzzy Foreground Detection :

Features: color, edge, stereo features, motion features, texture.

Multiple features: More robustness in presence of illumination

changes, shadows and multimodal backgrounds

24

Choice of the features

Color (3 components)

Texture (Local Binary Pattern [Heikkila – PAMI

2006])

For each feature, a similarity (S) is computed following its value in the background image and its value in the current image.

25

Aggregation of the Color and Texture features with the Choquet Integral

Color Features Texture Features

Similarity mesure for the

Color

Similarity measure for the

Texture

Fuzzy Integral

Classification B/F

Foreground Mask

BG(t)

I(t+1)

SC,1 SC,2 SC,3 ST

How to compute S for the Color and the Texture?

T<TifT

TT=Tif1

T<TifT

T

=S

BIB

I

IB

IBI

B

T

IT

k I,C

Background Image

Current Image

For the TextureFor the Color

kFkIkF

kI

kIkF

kIkFkI

kF

kC

CCifC

CCCif

CCifC

C

S

,,,

,

,,

,,,

,

, 1

FT

k F,C

0 ≤ S ≤ 1

k=one of the color components

0 ≤ T,C ≤ 255

26

27

Fuzzy operators

« Sugeno Integral» et «Choquet Integral»

Uncertainty and imprecision Great flexibility Fast and simple operations

ordinal cardinal

28

Data Fusion using the Choquet Integral

Mesures floues :

Intégrale de Choquet :

29

Fuzzy Foreground Detection

Classification using the Choquet integral

If then else

where Th is constant threshold. is the value of the Choquet integral for the pixel (x,y)

Aggregation Color, Texture Aqu@thèque (384 x 288 pixels) - Ohta color space

a) Current image b) Ground truth

c) Choquet integral d) Sugeno integral [Zhang 2006]

IntegralColor space

ChoquetOhta

SugenoOhta

S(A,B) 0.40 0.27

Comparison between the Sugeno and Choquet Integrals30

Aggregation Colors, Texture : Ohta, YCrCb, HSV Aqu@thèque (384 x 288 pixels)

Choquet - Ohta Choquet - YCrCb Choquet - HSV

IntegralColor Space Ohta YCrCb HSV

S(A,B) 0.40 0.42 0.30Evaluation of the Choquet integral for different color spaces

0.6

0.5

0.5

0.5

0.53

0.3

0.4

0.3

0.39

0.34

0.1

0.1

0.2

0.11

0.13

0.9

0.9

0.8

0.89

0.87

0.7

0.6

0.7

0.61

0.66

0.4

0.5

0.5

0.5

0.47

1

1

1

1

1

Values of the fuzzy measures

31

Texture

Color

32

Aggregation Color, Texture VS-Pets 2003 (720 x 576)

Current Image Choquet - YCrCb Sugeno – Ohta [Zhang 2006]

Aggregation Colors : Pets 2006 (384 x 288

pixels) Original sequence Ground truth

OR Sugeno Integral Choquet Integral

YCrCb

Ohta

HSV33

34

Fuzzy Background maintenance No-selective rule

Selective rule

Here, the idea is to adapt very quickly a pixel classified asbackground and very slowly a pixel classified as foreground.

35

Fuzzy adaptive rule

Combination of the update rules of the selective scheme

and

Original Image 1850 Ground Truth

No selective rule Selective rule Fuzzy adaptive rule

No selective Selective

Fuzzy adaptive

S(A,B)% 58.40 57.08 58.96

36

Results on the Wallflower datasetSequence Time of Day

Similarity measure

37

Computation Time

Algorithm Frames/Second

T2-FMOG-UM 11

T2-FMOG-UV 12

MOG 20

Choquet integral 31

Sugeno integral 22

OR 40

Resolution 384*288, RGB, Pentium 1,66GHz, RAM 1GB

38

Assessment

Fuzzy Background Modeling by T2-FMOG Multimodal Backgrounds

Fuzzy Foreground Detection using multi-features

Fuzzy Background Maintenance

Perspectives

- Using fuzzy approaches in other statistical models.

- Using more than two features- Fuzzy measures by learning

39

Plan

Introduction Fuzzy Background Subtraction Background Subtraction via a

Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal

Component Analysis (RPCA) Conclusion - Perspectives

40

Background Modeling and Foreground Detection via a Discriminative Subspace Learning (MIA Lab)

Reconstructive subspace learning models (PCA, ICA, IRT) [RPCS 2009]

Assumption: The main information contained in the training sequence is the background meaning that the foreground has a low contribution.

However, this assumption is only verified when the moving objects are either small or far away from the camera.

41

Discriminative Subspace Learning

Advantages More efficient and often give better classification results. Robust supervised initialization of the background Incremental update of the eigenvectors and eigenvalues.

Approach developed at the MIA Lab: Background initialization via MMC [MVA 2012] Background maintenance via Incremental Maximum

Margin Criterion (IMMC) [MVA 2012]

42

Background Subtraction via Incremental Maximum Margin Criterion

Denote the training video sequences S ={I1, ...IN}

where It is the frame at time t

N is the number of training frames.

Let each pixel (x,y) be characterized by its intensity in the grey scale and asssume that we have the ground truth corresponding to this training video sequence, i.e we know for each pixel its class label that can be foreground or background.

43

Background Subtraction via Incremental Maximum Margin Criterion

Thus, we compute respectively the inter-class scatter matrix Sb and the intra-class scatter matrix Sw:

where c = 2

I is the mean of the intensity of the pixel (x,y) over the training video

Ii is the mean of samples belonging to class i

pi is the prior probability for a sample belonging to class i (Background,

Foreground).

44

Background Subtraction via Incremental Maximum Margin Criterion

Batch Maximum Margin Criterion algorithm.

Extract the first leading eigenvectors that correspond to the background. The corresponding eigenvalues are contained in the matrix LM and the leading eigenvectors in the matrix ΦM.

The current image It can be approximated by the mean background and weighted sum of the leading eigenbackgrounds ΦM.

45

Background Subtraction via Incremental Maximum Margin Criterion

The coordinates in leading eigenbackground space of the current image It can be computed :

When wt is back projected onto the image space, the background image is created :

46

Background Subtraction via Incremental Maximum Margin Criterion

Foreground detection

Background maintenance via IMMC

47

Principle - Illustration

Current Image

IBackground

IForeground

Background image

Foreground mask

48

Results on the Wallflower dataset

Original image, ground truth , SG, MOG, KDE, PCA, INMF, IRT, IMMC (30), IMMC (100)

49

Assessment

Advantages Robust supervised initialization of the background. Incremental update of the eigenvectors and

eigenvalues.

Disadvantages Needs ground truth in the training step.

Others Discriminative Subspace Learning methods such as LDA.

Perspectives

50

Plan

Introduction Fuzzy Background Subtraction Background Subtraction via a

Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal

Component Analysis (RPCA) Conclusion - Perspectives

51

Foreground Detection via Robust Principal Component Analysis

PCA (Oliver et al 1999): Not robust to outliers. Robust PCA (Candes et al. 2011):

Decomposition into low-rank and sparse matrices

Approach developed at the MIA Lab: Validation [ICIP 2012][ICIAR 2012][ISVC 2012] RPCA via Iterative Reweighted Least Squares [BMC

2012]

52

Robust Principal Component Analysis

Candes et al. (ACM 2011) proposed a convex optimization to address the robust PCA problem. The observation matrix A is assumed represented as:

where L is a low-rank matrix and S must be sparse matrix with a small fraction of nonzero entries.

http://perception.csl.illinois.edu/matrix-rank/home.html

53

Robust Principal Component Analysis

This research seeks to solve for L with the following optimization problem:

where ||.||* and ||.||1 are the nuclear norm (which is the l1-norm of singular value) and l1-norm, respectively, and λ > 0 is an arbitrary balanced parameter.

Under these minimal assumptions, this approach called Principal Component Pursuit (PCP) solution perfectly recovers the low-rank and the sparse matrices.

54

Algorithms for solving PCP

Algorithms Accuracy Rank ||E||_0 # iterations

time (sec)

IT 5.99e-006 50 101,268 8,550 119,370.3

DUAL 8.65e-006 50 100,024 822 1,855.4

APG 5.85e-006 50 100,347 134 1,468.9

APGP 5.91e-006 50 100,347 134 82.7

ALMP 2.07e-007 50 100,014 34 37.5

ADMP 3.83e-007 50 99,996 23 11.8

10,000timesspeedup!

Time required to solve a 1000x1000=106 RPCA problem:

Source: Z. Lin , Y. Ma “The Pursuit of Low-dimensional Structures in High-dimensional (Visual) Data: Fast and Scalable Algorithms”

Time required is still acceptable for ADM but for background modeling and foreground detection?

55

Application to Background Modeling and Foreground Detection

Source: http://perception.csl.illinois.edu/matrix-rank/home.html

n is the amount of pixels in a frame (106)m is the number of frames considered (200)Computation time is 200* 12s= 40 minutes!!!

56

PCP and its application to Background Modeling and Foreground Detection

Only visual validations are provided!!!

Limitations:

Spatio-temporal aspect: None! Real Time Aspect: PCP takes 40 minutes with the

ADM!!! Incremental Aspect: PCP is a batch algorithm. For

example, (Candes et al. 2011) collected 200 images.

57

PCP and its variants

Source: T. Bouwmans, Foreground Detection using Principal Component Pursuit: A Survey, under preparation.

How to improve PCP?

Algorithms for solving PCP (17 Algorithms) Incremental PCP (5 papers) Real-Time PCP (2 papers)

Validation for background modeling and foreground detection (3 papers) [ICIP 2012][ICIAR 2012][ISVC 2012]

58

PCP and its variants

Source: T. Bouwmans, Foreground Detection using Principal Component Pursuit: A Survey, under preparation.

59

Validation Background Modeling and Foreground Detection: Qualitative Evaluation

RSL

PCP-EALM

PCP-IADM

PCP-LADM

BPCP-IALM

Original image

Ground truth

PCA

PCP-LSADM

Source: ICIP 2012, ICIAR 2012, ISVC 2012

60

Validation Background Modeling and Foreground Detection : Quantitative Evaluation

F-Measure

Source: ICIP 2012, ICIAR 2012, ISVC 2012

Block PCP gives the best performance!

61

PCP and its application to Background Modeling and Foreground Detection

Recent improvements: BPCP (Tang et Nehorai (2012)) : Spatial but not

incremental and not real time! Recursive Robust PCP (Qiu and Vaswani (2012) ):

Incremental but not real time! Real Time Implementation on GPU (Anderson et al.

(2012) ): Real time but not incremental!

What we can do? Research on real time incremental robust PCP!

62

Conclusion Fuzzy Background Subtraction Background Subtraction via a Discriminative

Subspace Learning: IMMC Foreground Detection via Robust Principal

Component Analysis (RPCA)

Fuzzy Learning Rate Other Discriminative Subspace Learning methods

such as LDA Incremental and real time RPCA

Perspectives

Publications Chapter

T. Bouwmans, “Background Subtraction For Visual Surveillance: A Fuzzy Approach”, Handbook on Soft Computing for Video Surveillance, Taylor and Francis Group, Chapter 5, March 2012.

International Conferences :

F. El Baf, T. Bouwmans, B. Vachon, “Fuzzy Statistical Modeling of Dynamic Backgrounds for Moving Object Detection in Infrared Videos”, CVPR 2009 Workshop, pages 1-6, Miami, USA, 22 June 2009.

F. El Baf, T. Bouwmans, B. Vachon, “Type-2 Fuzzy Mixture of Gaussians Model: Application to Background Modeling”, ISVC 2008, pages 772-781, Las Vegas, USA, December 2008

F. El Baf, T. Bouwmans, B. Vachon, “A Fuzzy Approach for Background Subtraction”, ICIP 2008, San Diego, California, U.S.A, October 2008.

F. El Baf, T. Bouwmans, B. Vachon. " Fuzzy Integral for Moving Object Detection ", IEEE-FUZZY 2008, Hong Kong, China, June 2008.

F. El Baf, T. Bouwmans, B. Vachon, “Fuzzy Foreground Detection for Infrared Videos”, CVPR 2008 Workshop, pages 1-6, Anchorage, Alaska, USA, 27 June 2008.

F. El Baf, T. Bouwmans, B. Vachon, “Foreground Detection using the Choquet Integral”, International Workshop on Image Analysis for Multimedia Interactive Services, WIAMIS 2008, pages 187-190, Klagenfurt, Austria, May 2008.

Fuzzy Background Subtraction

Publications

Journal

D. Farcas, C. Marghes, T. Bouwmans, “Background Subtraction via Incremental Maximum Margin Criterion: A discriminative approach” , Machine Vision and Applications, March 2012.

International Conferences :

C. Marghes, T. Bouwmans, "Background Modeling via Incremental Maximum Margin Criterion", International Workshop on Subspace Methods, ACCV 2010 Workshop Subspace 2010, Queenstown, New Zealand, November 2010.

D. Farcas, T. Bouwmans, "Background Modeling via a Supervised Subspace Learning", International Conference on Image, Video Processing and Computer Vision, IVPCV 2010, pages 1-7, Orlando, USA , July 2010.

Background Subtraction via IMMC

Publications Chapter

C. Guyon, T. Bouwmans, E. Zahzah, “Robust Principal Component Analysis for Background Subtraction: Systematic Evaluation and Comparative Analysis”, INTECH, Principal Component Analysis, Book 1, Chapter 12, page 223-238, March 2012.

International Conferences :

C. Guyon, T. Bouwmans. E. Zahzah, “Foreground Detection via Robust Low Rank Matrix Factorization including Spatial Constraint with Iterative Reweighted Regression”, International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan, November 2012.

C. Guyon, T. Bouwmans. E. Zahzah, “Moving Object Detection via Robust Low Rank Matrix Decomposition with IRLS scheme”, International Symposium on Visual Computing, ISVC 2012,pages 665–674, Rethymnon, Crete, Greece, July 2012.

C. Guyon, T. Bouwmans, E. Zahzah, “Moving Object Detection by Robust PCA solved via a Linearized Symmetric Alternating Direction Method”, International Symposium on Visual Computing, ISVC 2012, pages 427-436, Rethymnon, Crete, Greece, July 2012.

C. Guyon, T. Bouwmans, E. Zahzah, "Foreground Detection by Robust PCA solved via a Linearized Alternating Direction Method", International Conference on Image Analysis and Recognition, ICIAR 2012, pages 115-122, Aveiro, Portugal, June 2012.

C. Guyon, T. Bouwmans, E. Zahzah, "Foreground detection based on low-rank and block-sparse matrix decomposition", IEEE International Conference on Image Processing, ICIP 2012, Orlando, Florida, September 2012.

Foreground Detection via RPCA