1

Robust Video Stabilization Based on Particle Filter Tracking of Proj

ected Camera Motion (IEEE 2009)

Junlan Yang University of Illinois,Chicago

2

Reference

• [1]A tutorial on particle filters for online nonlinear non-Gaussian Bayesian tracking

• [4]probabilistic video stabilization using kalman filtering and mosaicking

• [5]Fast electronic digital image stabilization for off-road navigation

• [18]condensation conditional density propagation for visual tracking

3

Outline

IntroductionCamera Model Particle Filtering EstimationComplete System of Video Stabilization Simulation and ResultsConclusion

4

Introduction

• Video Stabilization– Camera motion estimation

• Particle filter– Tracking projected affine model of camera mo

tion

• SIFT algorithm (范博凱 ) – Detect feature points in both images

• Removing undesired (unintended) motion– Kalman filter

5

Outline

IntroductionCamera Model Particle Filtering EstimationComplete System of Video Stabilization Simulation and ResultsConclusion

6

Example of camera motion

Motion Camera

X

Y

Z

(x0,y0,z0)

at time t 0

P

Camera

X

Y

Z

(x1,y1,z1)

at time t 1

P

7

Generating Camera model

• Related of two vectors

3 3 3 1where R ,T are the transform of camera's

3-D rotation and translation, repectively

8

Building 2-D affine model

• Projection of P in time t0 and t1

where is the image plane-to-lens distance of the camera

Z

XY

(x0,y0,z0)

λ (u0,v0,λ)

9

Building 2-D affine model

• Rewriting the related of two projected vectors

• 2-D affine model

y123y

x113x10

)T /zλ (λsRˆt

)T /zλ (λsRˆt,/zzˆs define wewhere

10

Building 2-D affine model

3 3R is orthonrmal matrix

Global motion estimation is to determine the six parameters for every successive frame

11

Why do she use 2-D affine model to represent camera motion?

A pure 2-D model2-D translation vector and one rotation angle

3-D modelGiant complexity

12

Outline

IntroductionCamera Model Particle Filtering EstimationComplete System of Video Stabilization Simulation and ResultsConclusion

13

Particle Filtering Estimation

• Markov discrete-time state-space model state vector at time k

observations z, and the posterior density is

k 1:kp(x |z )

ik

ik

Give a set of particles x ,i = 1,...,N ,and weights

w ,i = 1,...,N ,where N is the number of particles

and k is the time step

Tkkkykxkkk RRRttsx ],,,,,[ˆ 211211

14

To approximate the posterior

i ik kWhere w are Normalized weights , particles x ~q()

are random vectors drawn from a proposal q() ,and the

q() refered as an importance density

i = 3 2 1 N

(.)~ qxik

...

15

Estimation of current state

N1 rate econvergenc and sense

squaremean in )z|p(xdensity posterior

true toconvergeion approximat, N As

:k1k

16

Importance density q(.)

• Traditionally – prior density

• This paper takes into account the current observation zk. The proposed important density whose mean vector obtained from the current observation zk

• Why do she use the particle filtering estimation ?

)x|p(x 1-kk

kx

17

Advantage of particle filtering estimation

• With Low error variance

• Proof : In large particle numbers condition, the estimation gives lower error variance than

kx kx

2k

k

k

11kik

matrix covariance diagonal with xstate trueof estimation

unbiasedan isx that estimate fine aconsider We

.estimationmotion based-feature from obtained is x

and diagonal be set to is where,),xq(~x

18

Covariance matrix of errors

assumption ssunbiasednegiven

)e,Cov(e aserror covariance ,xxˆe

)ε,Cov(ε aserror covariance ,xxˆε

2kkkkk

kkkkk

mean zero have themofBoth .xˆe and xˆε

origin in the as xstate true

set the she , prove hesimplify t order toIn

kkkk

k

19

Lemma 1:

where

20

Tkk1k

ik

ik

1k

Tikk

ikk

ik

ik

1kik

xx ]x|xE[x

]x|]E[xx|E[x]x|xE[x

),xG(~x

T

T

21

Lemma 1:

k with varing,σ varianceand mmean

with variablesrandom i.i.d as regarded and particles, i.i.d

for computed likelihood theare π where, π/πw

2ππ

ik

N

1i

ik

ik

ik

Strong law of large number

k1 2kk2π

2π

2πk c)(

N

1)ε,Cov(ε ,)/mσ(mˆc Denote

22

23

24

Outline

IntroductionCamera Model Particle Filtering EstimationComplete System of Video Stabilization Simulation and ResultsConclusion

25

Complete system of video stabilization

• At time k

Frame k

Frame k-1

SIFT algorithm

Match feature points

PFME(Particle filtering-

based motion estimation)

kxAccumulative

motion

}ˆ,ˆ,ˆ{xk kkk TRs

Kalman filter

}T,R,{s Ak

Ak

Ak

Compensate undesired motion

Stailized output

26

Getting six parameters

• SIFT algorithm – Find corresponding pairs• At time k

It needs three pairs to determine a unique solution

T21k12k11kykxkkk

T1T

]R,R,R,t,t,s[xA

YXX][XA

Y X A

27

(a) SIFT correspondence from frame 200,201 in outdoor sequence STREET

28

Generate particles

• Important density q(.) is a six-dimensional Gaussian distribution

• Particles

• In experience , N set to only 30 with better quality than prior distribution set N = 300

29

Quality of the particles

• N particles have N proposals of transformation matrix ,and N Inverse transform to frame k have N candidate image Ai

• Compare these images with k-1 frame A0

Point P at k-1framematch

Inverse transform

Point P at k frame

frame 1kat PPoint

30

Similar with A0 and Ai

• Mean square error – Difference of gray-scale from pixel to pixel

• Feature likelihood – Distance of all corresponding feature points

31

Particle filtering for global motion estimation

• Weight for each particle

• Estimation of current state

where

32

Accumulative motion

• At time k-1 to k

• At time 0 to k

Where s is scaling factor , R is rotation matrix and T is

translation displacement

22k21k

12k11kk

y

xkkk

RR

RRR,

t

tT,ss

33

Ak

0

0Ak

Ak

kA

1kk0

0A1kk

A1kk

1-k

1-kk

k

k

A1k

0

0A1k

A1k

1k

1k

Tv

uRs

TTRˆv

uRRsˆT

v

uRˆ

v

u

Tv

uRs

v

u

kkk sss

34

Intentional Motion estimation and motion compensation

• Compensate for the unwanted motion

Dk

Dk

Dk

sfactor scaling and,Tn vector translatio

,Rmatrix rotation lintentionaget filter toKalman ngImplementi

35

Complete system of video stabilization

• At time k

Frame k

Frame k-1

SIFT algorithm

Match feature points

PFME(Particle filtering-

based motion estimation)

kxAccumulative

motion

}ˆ,ˆ,ˆ{xk kkk TRs

Kalman filter

}T,R,{s Ak

Ak

Ak

Compensate undesired motion

Stailized output

36

Outline

IntroductionCamera Model Particle Filtering EstimationComplete System of Video Stabilization Simulation and ResultsConclusion

37

(a) Original image , (b) Matched-feature-based motion estimation (MFME)

(c) p-norm cost function-based motion estimation (CFME) (d) proposed method (PFME)

38(a) Original image , (b) MFME (c) CFME (d) PFME

39(a) Original image , (b) MFME (c) CFME (d) PFME

40

(a) Original video sequence (ground truth) (b) unstable video sequence (c) PFME

41(a) Motion in horizontal direction (b) Motion in vertical direction

Ty?

42

Comparison of average MSE and PSNR for stabilized output

PSNR = peak signal to noise ratio

Large PSNR has low distortion

43

Outline

IntroductionCamera Model Particle Filtering EstimationComplete System of Video Stabilization Simulation and ResultsConclusion

44

Conclusion• We demonstrated experimentally that the

proposed particle filtering scheme can be used to obtain an efficient and accurate motion estimation in video sequences.

45

Contributed of this paper

• Constraining rotation matrix projected onto the plane ?(depth change)

• Show using particle filtering can reduce the error variance compared to estimation without particle filtering

• Using both Intensity-based motion estimation method (PFME) and feature-based motion estimation (SIFT) method