dsp based motion tracking system dani cherkassky ronen globinski advisor: mr slapak alon

DSP Based Motion Tracking SystemDSP Based Motion Tracking System

Dani Cherkassky

Ronen Globinski

Advisor: Mr Slapak Alon

Presentation contents

IntroductionIntroduction

Summery

Results

Implementation

•Objective•The tracking task•Utilization potential•Design stages

•System diagram•Data flow diagram•Tracking flow chart•Motion detection algorithm•Tracking algorithm

•Conclusions •Future work

Design and implement a DSP based vision system that can detect and track a single, rigid, moving object

Design and implement a DSP based vision system that can detect and track a single, rigid, moving object

Objective

Introduction ResultsImplementation Summery

• The problem:Detecting and Tracking moving objects in a camera vision field often requires the constant attention of human operators, thus making the tracking task exposed to human errors and working hours problems

The tracking task


•The solution:We offer a DSP based machine vision application that can perform the motion detection and tracking tasks automatically.Our solution is based on an image processing algorithms that can be implemented on a digital signal processor platform and deliver a real-time performance.

Utilization potential


Motion Detection and tracking system

Security SurveillanceMilitary systems

Traffic control

Design stages


Stage CStage C Stage DStage D Stage EStage EStage AStage A Stage BStage B

PossibleAlgorithms

Examination(literature survey)

ImplementingThe selectedAlgorithms

(Matlab real time)

ImplementingThe selectedAlgorithms

(DSP platform)

Building theCamera motor

system

Tests andBenchmarks

System Diagram


Motors Pan & tilt system

B&W CMOS Camera

Analog Video

Stream

Master DSP BoardVideo Encoding & Image Processing

Slave DSP BoardVideo Decoding

PWM Motor Angle Control

Analog Video

StreamDigital Image

StreamVia SPORT

B&W TV Monitor

Data Flow Diagram


Bank 1Input

Bank 2NTSC frame 1

Bank 3NTSC frame 2

Input Image244 x 360Gray scale

Init empty NTSC frame before

transfer begins

Slave SDRAM

SPORT, DMA CoreUpdate NTSC frame with new image data

TVDMA, PPI

Software access (core)

Hardware access (DMA)

SlaveDSP

Video Data Flow on the slave DSP board


MasterDSP

Tracking Flow Chart

Motion tracking algorithms running on the master DSP board

Digital Video Stram

Separate Active Video

Current State

Idle

Tracking Detecting

∑-∆ Algorithm

EKLT Tracking Algorithm

Lost Motion Detected

Current State ß Tracking

Current State ß Detecting

NO YES

Current State ß Detecting

Current State ß Tracking

YESNO


Motion Detection Algorithm

In this section a motion detection algorithm(∑-∆ filter) proposed by A. Manzanera J. C. Richefeu is introduced.

Methods for motion detection :

• Background subtraction algorithms (the naive method)

• The background must stay constant• Irrelevant motion is not discarded

• Last K frames analysis algorithms (histogram , entropy , linear prediction)

• Consumes memory (K have to be large)

Not suitable for DSP applications !

• Recursive methods that use fixed number of estimates (Kalman filter , ∑-∆ filter)

OK! low complexity, low memory consumption, statistical measure on the temporal activity

Not compatible with noisy real world situations!

Motion detection , how ???

The principle of the motion detection method is to build a model of the static scene (i.e. without moving objects) called background, and then to compare every frame of the sequence to this background in order to discriminate the

regions of unusual motion, called foreground (the moving objects).


Motion Detection AlgorithmThe ∑-∆ algorithm – a recursive approximation of the temporal statistics, allowing a simple and efficient pixel-level change detection

0 0

t-1 t t-1

t-1 t t-1

for each pixel :

M ( ) I ( )

for each pixel :

Initializ

if M ( ) ( ) then M ( ) M ( )

if M ( ) ( ) then M ( ) M ( )

ation

for each frame t

t

t

x

x x

x

x I x x x X

x I x x x X

t t

for each pixel :

( ) M ( ) I (

for each ame

)

fr t

t

x

x x x

1 2

4

0 0

t

t-1 t t t-1

t-1 t

for each pixel :

V ( ) ( )

for each pixel such that ( ) 0 :

if V ( ) ( ) then V ( ) V ( )

if V ( ) ( ) then

Initialization

for each frame t

V

x

x x

x x

x N x x x Y

x N x

t t-1( ) V ( )x x Y

3

t

t

for each pixel :

if ( ) ( )

D ( ) 0;

else

fo

D (

r each frame t

) 1;

t t

x

x V x

x

x

t

- Input Frame

- Estimated Background value (approximation of the median of )

- First observation field / difference field (the mot )ion likelihood measure

V - Second observation fi

t

t t

t

I

M I

t

eld (the variance)

D - Motion field

Background estimation:Computation of the ∑-∆ mean (an approximation of the median)

Motion likelihood measure:Computation of the difference between the image and the median

Computation of the variance:Defined as the ∑-∆ mean of N times the non-zero differences

Motion computation:Comparison between the difference and the variance

, ,

parameters used for

sensitivity adjustments

X Y N


Motion Detection AlgorithmExample: two frames from a sequence acquired by a stationary camera

Still area

Clutter area

Motion areaAlgorithm results for 3 particular pixels from 3 different areas

The algorithm can discard irrelevant (clutter) motion - tree moving with the wind, and also has low sensitivity to noise

Moving object(foreground)

The motion field corresponds to the Boolean indicator of the condition: “the green line is over the black line” – every time the difference is greater than the variance the pixel is considered to be moving.


Motion Detection AlgorithmExample - cont’dThe motion field of the moving car

The sensitivity in this case is too high, thus the motion field is very noisy and unusable

The effect of the parameters X, Y, N on the motion field:

X=1, Y=1, N=4 X=1, Y=3, N=4 X=1, Y=3, N=4 Filtered

Changing the parameter Y affects the variance calculation and in this case lowering the sensitivity which results in a cleaner motion field

The motion field can be further cleaned by applying a 3x3 median filter to remove the “salt” noise.

Salt noise


Motion Tracking Algorithm

In this section a method proposed by J. Shi and C. Tomasi is introduced. It is an extension of a method proposed by B. D. Lucas and T. Kanade andwill therefore, from now on, be called Extended Lucas-Kanade Tracking (ELKT).

Image Motion Model (affine motion model)

The ELKT is based on the assumption that most changes between two frames are caused by image motion and affine transformations.

Moving template in Frame 1,

Moving template in Frame 2,

( , ) ( ( , , ), ) m mI t I t t x ξ x

Affine Model

image sequence

image coordinates with respect to center of template

transformation function

deformation matrix

translation (displacement) of the center of the template

the six parameters of

mI

x

D

d

the Affine motion model

T

,

xxx xy

yx yy y

xx xy yx yy x y

dd d

d d d

d d d d d d

ξ = Ax + d

A = I + D

D d

μ

( , )mI tx ( , )mI t x

Lets look at two sequential frames and express the transformation of the

moving object in mathematical terms



( ) ( ) K JAx + d x0 0( , ), ( , )m mJ I x t K I x t Affine transform

two samples of the image sequence

Image Motion Estimation

2The dissimilarity:

The area of integration (the object templat

( ) (

e spatial coordinates

)

) W

J K d

W

x Ax + d x

We cannot assume perfect correspondence between J(x) and K(Dx + d). This due to image noise and imperfections in the image motion model.

There for, some dissimilarity measure should be minimized. In ELKT the sum of squared differences (SSD), is chosen as the dissimilarity measure.

Motion estimation = minimizing the dissimilarity . The SSD method ensures that the dissimilarity function has only one minimum.

• The natural approach to minimize the dissimilarity would beto differentiate the dissimilarity function and set the results to zero.

Methods for minimizing the dissimilarity:

This equation would however be non-linear, and therefore difficult to solve. Not a good solution for our DSP Real-time system.

• The better solution is to linearize the function with a truncated Taylor expansion with respect to some good estimate and solve the new system by iterating the procedure.



The tracking algorithm

arg min ( ) μμ μ

Tracking = determining the six parameters motion vector µ.

The final equation for estimating the six parameters of the Affine motion vector:

( )1 1 ( )( 1) ( )' 'n nn n T T T

μ μ φ φ φ E

0

1

1

1

( ) ( ) ( ( ))

( ) ( ) ( )

0

0 0 0y 0

0 0 and 0

0 0 1 0

0 1

T

x J K

X x J

x

x

y

x

0

0

0

E x ξ x

φ x x

D

D X(x)

D

The tracking is performed by updating µ using the iterative calculation of µ’ for every input frame. The number of iterations can be determined in advance according to the tracking performance.

For each frame n:1.Calculate µ’(n+1) by iterating this

equation.2.Update the motion vector µ. 3.Use µ to update the position of

the target object.



The EKLT algorithm is recursive, we need to determine the number of iterations

• We have discovered that if the algorithm doesn't converge after 4-5 iterations there will be practically no convergence at all. Instead of making the exhaustive iterative process we offer a different approach:

We have found that method problematic for our system because :

• T have to be larger then some picture noise level (noise level must be measured).

• It may take too long for the algorithm to converge (not capable with RT systems).

• Run the EKLT algorithm for a fix number of iterations.

• Calculate the tracking error using the next equation :

• Check the tracking quality:

• If the previous inequality is satisfied continue tracking, else, go back to motion detection.

1

1

( 1) (n)

N

Tracking ii

Tracking Tracking

E EN

E n E

• It was suggested in [4] to continue the iterations till : ( 1)n

T

E

The average error calculated using Alfa filter

( ) (1 ) ( 1) for 0( 1)

( 1)

for 0

Tracking TrackingTracking

Tracking

E n E n nE n

E n n

The Advantages of our method are :1. No need to measure picture noise level because is an

approximation of that level.2. Tracking quality is estimated for each frame, it is easy to change

the template when that estimate goes low.

(1)TrackingE


Results

• Algorithm Complexity:

•CPU Utilization (for 10 x 20 Template size ) :a. Motion detection: 82,070,156 (Cycles)

b. Feature extraction: 20,857 (Cycles)

c. Tracking initialization: 6,550,668 (Cycles)

d. Solving EKLT equation: 5,205,380 (Cycles)

a. Motion detection: O{mxn} , mxn = Frame Size

b. Feature extraction: O {N} , N = Template Size

c. Tracking initialization – compute : O {N}

d. Solving EKLT equation: O {N}

1T φ φ φ


• System limitation : Results - cont’d

The system’s tracking speed is limited

• Tracking results :

α

y

x

Object plain Camera position

Calculation of the angular velocity of the target object:


Conclusions

• The motion detection algorithm:The ∑-∆ filter, combined with 3x3 median filter, proved to be a robust and accurate method of detection of moving objects for a small cost in memory consumption and computational complexity.

• The tracking algorithm:The EKLT tracking algorithm performed well on our system. The algorithm is efficient – the on-line calculations are relatively small and the convergence time is fairly good. The performance can be greatly improved by better selection of the initial features for the tracking process.

The good performance of the algorithms proved our concept that a real-time tracking system can be

implemented successfully on DSP platforms

The good performance of the algorithms proved our concept that a real-time tracking system can be

implemented successfully on DSP platforms


Future work

The DSP implementation must be optimized The DSP implementation must be optimized

Consider using a DSP with a dual PPI core to improve the performance Consider using a DSP with a dual PPI core to improve the performance

Improve the pan and tilt mechanismImprove the pan and tilt mechanism

Improve the system’s transition from motion detection to tracking Improve the system’s transition from motion detection to tracking


References

[1] A.Manzanera, J.C.Richefeu "A robust and computationally efficient motion detection algorithm based on Background estimation"

[2] G.Jing, C.Eng and D.Rajan. "Foreground motion detection by difference-based spatial temporal entropy"

[3] J.Shi and C.Tomashi, "Good Features to Track" Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp.593-600 IEEE CS Press 1994.

[4] D.Hager and N.Belhumeur, "Efficient Region Tracking Whit Parametric Models of Geometry and Illumination" IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 20, No.10, October 1198.

dsp based motion tracking system dani cherkassky ronen globinski advisor: mr slapak alon

Documents

motion detection method

tracking tasks

constant irrelevant

regions of unusual motion

median motion likelih

slave dsp board slide

master dsp board slide

vision system