rlreal-ti obj t t kitime object tracking system on

R l Ti Obj t T kiReal-Time Object Tracking System on FPGAsSystem on FPGAs

Su Liu1, Alexandros Papakonstantinou2, Hongjun Wang1, Deming Chen2Hongjun Wang , Deming Chen

1School of Information Science and Engineering Shandong1School of Information Science and Engineering, Shandong University, Jinan, China

2Electrical & Computer Engineering Department, University of Illinois, Urbana-Champaign

OutlineOutline

• Introduction

• Object Tracking Framework

• Searching Object Map Techniques

• Cascade Boundary Joiner

• Experiment Results

Applications of Object TrackingApplications of Object Tracking

Tasks of Object TrackingTasks of Object Tracking

Obj ki i d f id if i h• Object tracking is used for identifying the trajectory of moving objects in video frame sequences.

• Build the model of what you want to track.

• Use what you know about the previous frames to make estimation about the position of objectsto make estimation about the position of objects in current frames.

Real-time object tracking approachesReal time object tracking approaches

• Object detectionObject detection– Point-detection schemes : find interest points in images which have an

expressive texture in their respective localitiesBackground subtraction techniques : Object detection can be achieved– Background subtraction techniques : Object detection can be achieved by building a representation of the scene called the background model and then finding deviations from the model for each incoming frameS t ti titi th i i t t ll i il i– Segmentation : partition the image into perceptually similar regions

• Tracking– Point tracking : Objects detected in consecutive frames are represented

by points– Kernel tracking : Objects are tracked by computing the motion g j y p g

(parametric transformation such as translation, rotation, and affine) of the kernel in consecutive frames

– Silhouette tracking: Such methods use the information encoded insideSilhouette tracking: Such methods use the information encoded inside the object region (appearance density and shape models)

Our WorkOur Work

• Propose a highly parallel hardware implementation of an object tracking p j galgorithem

• Improve the object region identification• Improve the object region identification performance of the object tracking l halgorithm

• Hardware implementation achieves up toHardware implementation achieves up to 100X speedup over the software execution

OutlineOutline

• Introduction





Main architectureMain architecture

Camera MemoryCamera Input

Memory(DDR2)

Monitor+Pre‐ Tracking

positionMain

+processing

position outputsearching

MC Memory(DDR2)

Monitor+Camerainput

Pre‐ Tracking position

Main

+processing

position outputsearching

FPGA-based object tracking systemFPGA based object tracking system

RAW to RGB format

Pre‐processing

CCD CAM DDR2

RAW to RGB format

2D Haar Transformation

BGRND Current 1 1 1 0 0 0 0 0 0 0BRAM BRAM

Classification GeneratorMain

1 1 1 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 0 0 1 1 1 1 0 0 0 0 0 1 1 1 1 1 0

Search Map

Search Path Splitter

Dilation & ErosionSearching 0 0 0 0 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1


Path1

Single Square

Path2

Single Square

Path12

Single Square

...

Path1 BRAM Path2 BRAM Path12 BRAM

Tracking & Display

Five-level boundary joiner

Position Information Output

OutlineOutline

• Introduction





Searching Techniques IntroductionSearching Techniques Introduction

1 1 1 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 01 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0

Memory(0,0)(1 0) 001 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0

1 1 1 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 00 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1 1 1 1 0

(1,0)(0,1)(2,0)

2000

3000

40

0 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1 1 1 1 00 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 01 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1

(1,1)(0,2)

0 1000 2000 3000 4000

010

00

(2,1)1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 10 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 10 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 0

(1,2)…………0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 0

1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1

(3,3)……

0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1

Searching Map ImprovementSearching Map Improvement

• Parallel Object Map ProcessingSplit object map into 12 20x20 sub-matricesp j p

3000

4000

0 100 200 300 400

010

020

030

040

0

Index

n

……

0 1000 2000 3000 4000

010

0020

00

0 100 200 300 400

010

020

030

040

0

Index

n

• Dual- Mode Object Identification– Single Mode– Block Mode

Object Map1 1 1 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 01 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0


Path1 Path2 Path12...

1 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 01 1 1 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 00 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1 1 1 1 0

Path1 BRAM Path2 BRAM Path12 BRAM

0 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1 1 1 1 00 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 01 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1

3.8 ms1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 10 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 10 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 0

Memory

(0 0)0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 01 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 11 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1

(0,1)

(0,0)(1,0)

(1,1)

0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 12.66ms

OutlineOutline

• Introduction





Cascade Boundary JoinerCascade Boundary Joiner

Path 1 Path 2 Path 3 Path 12…

Physical information recovery

….

FORAM

5-stage boundary joiner

Cascade Boundary JoinerCascade Boundary Joiner1 2 3 4 A1 2 3 4

5 6 7 8

9 10 11 12

A

B

C9 10 11 12 C

A1’ B1’ A1’’

A2’ B2’

A3’ B3’

A2’’

A3’’

A1’’’

Complete object position

A1’’’

A2’’’A2

OutlineOutline

• Introduction





Experiment SettingExperiment Setting

AVALON

MasterTemplate

Camerainput

DDR2

NMonitor

ki

+Pre‐

processing

Tracking position output

Mainsearching

Experimental ResultsTABLE I. PERFORMANCE COMPARISON: SW VERSUS HW

Experimental ResultsObject # SW Exec. Time SW fps HW Exec. Time HW. fps Speedup

0 30ms 33.3 1.45ms 689.6 20.69X 1 79 12 66 3 53 283 3 22 38X1 79 ms 12.66 3.53 ms 283.3 22.38X2 252 ms 3.97 3.87ms 258.4 65.12X 3 392ms 2.55 5.17ms 193.4 75.82X 4 546ms 1.83 5.27ms 189.7 103.6X 5 857ms 1.17 8.73ms 114.5 98.16X 6 1489 ms 0 67 14 4 ms 69 4 103 4X6 1489 ms 0.67 14.4 ms 69.4 103.4X

TABEL II. 20-FRAME PERFORMANCE COMPARISON

Hardware Ex. Time Software Ex. Time Speedup

97.82 ms 6038ms 62X

TABEL II. 20 FRAME PERFORMANCE COMPARISON

Conclusions and Future Work

• We presented an FPGA implementation for object tracking in color videotracking in color video– a highly parallel hardware implementation

object tracking through a dual search technique– object tracking through a dual search technique– efficient boundary joining

up to 100X speedup over the software execution– up to 100X speedup over the software execution

• Future workh f h k l h h– improve the sensitivity of the tracking algorithm to the

luminance of the scene3D i t ti– 3D image reconstruction

Thank you!Thank you!

rlreal-ti obj t t kitime object tracking system on

Documents