PERFORMANCE EVALUATION OF VIDEO ANALYTICS

FOR SURVEILLANCE ON-BOARD TRAINS

Context and Objectives

ACIVS’13, October 30th 2013, Poznan, Poland

Low level Evaluation

Real-time video surveillance is used in public transportation, including on-board metro trains, to support human operators in control rooms.

Video Content Analytics (VCA) is effective when the performance is such to reduce

false alarms under appropriate acceptability thresholds. Reducing the causes of false alarms by fine tuning the VCA is particularly important in

on-board applications, that are more prone to disturbances with respect to fixed installations due to more critical operating conditions (e.g. constrained camera field of view, frequent light changes, vibrations, occlusions, etc.).

Accurate performance evaluation is essential to decide about:

Which alarms to activate depending on scenarios How to set algorithm parameters in each scenario Comparison between commercial and open-source implementations is also important to evaluate the added value provided by the former in terms of achievable performance.

Black Box Evaluation

Valentina Casola, Mariana Esposito, Francesco Flammini, Nicola Mazzocca, Concetta Pragliola

Contributions

Evaluation of Commercial Off-The-Shelf (COTS) VCA system

Low Level Black Box

Alarm Performance Evaluation

Comparison with Open source

System

Frame and Object Based Metrics

Ground Truth values compared with Algorithm Result

Computed by specific tool developed in Matlab

Object Based Metrics

Consider the whole trajectory of each

object in the scene and preserves its

lifetime

Correct Detected Track (CDT)

Track Detection Failure (TDF)

False Alarm Track (FAT)

Track Fragmentation (TF)

ID Change (IDC)Temporal Overlap

Spatial Overlap

Performance Evaluation ToolAlgorithm Result

Ground Thruth

Tot Tos

Metrics

(15%)

Frame Based Metrics

Measure the performance on individual

frames of a video stream and do not

consider the preservatio. The blobs are

evaluated in their size and location and

compared with GT.

True Positive (TP)

False Negative (FN):

Fragmentation

ecall

False Positive (FP)

Merging

Precision

0

50

100

150

200

250

300

350

400

Oc

cu

rre

nc

ies

#AR TP FN FP TF IDC

Object-Based Results

COTS

Open-Source

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

Pe

rfo

rma

nc

e In

de

x

Accuracy FAR PP FNR

Object-Based Performance

COTS

Open-Source

False Alarm Rate FPTP

FP

FAR

Detection rateFNTP

TP

DR

Positive PredictionFPTP

TP

PP

False Negative RateTPFN

FN

FNR

Fragmentation IndexFRAGM

TPFI

Merge IndexMERGE

FPOBJ AR FM

♦ 1510 objects in 600 frames, 8 GT tracks ♦ COTS system detects less objects but it is far more reliable regarding false positives (84 vs 2168) ♦ FA of 6% instead of 41% (the latter

clearly unacceptable). ♦ COTS object Fragmentation is less then one half w.r.t. Open source.

Metrics COTS Open

TP 1307 1324

FN 203 186

FP 84 2168

Metrics COTS Open

TP 7 8

FN 1 0

FP 9 213

TF 11 17

IDC 0 0

♦ 1510 objects in 600 frames, 8 GT tracks ♦ COTS FP and PP reveal higher reliability ♦ Same consideration for TF (important for event persistence).

Performance IndicesTest of COTS system on real

vehicle for light railwails and

tramways

False Alarm IndexRealAlarms#

sFalseAlarm#FAI

Probability Of DetectionPDTPFN

TP

POD

over it detects camera occlusionblurring with paint or obstacles.amper it detects the manumission of the camera by moving it.top it detects still objects in the scene.Presence it detects objectspeople moving in the scenerowd it raises an alarm when the scene is overcrowded.

ALARMS