video event recognition algorithm assessment evaluation workshop veraae etiseo – nice, may 10-11...

Video Event Recognition Algorithm

Assessment Evaluation Workshop VERAAE

ETISEO – NICE, May 10-11 2005Dr. Sadiye Guler

Sadiye Guler - Northrop Grumman IT/TASCMubarak Shah, Niels da Vitoria Lobo - University of Central Florida Rama Chellappa, Dave Doermann - University of Maryland

US Government Champions:

Terrence Adams-NSA, John Garofolo, Rachel Bowers-NIST

Advanced Research and Development Activity

TASC ProprietaryMay-05

Problem

Comparative study of Video Event Recognition (VER) algorithms to

assess applicability, usefulness and limitations of different

approaches

Motivation:- Several promising VER algorithms exist

- The algorithms have varying degrees of success with different types of

event detection

- No largely accepted criteria or data set (with ground truth) exist for VER

evaluation (few emerging studies..)

- The performance of VER algorithms is highly dependent on the results

of object detection and tracking, rendering fair comparison of just the

“event recognition” very difficult


Workshop Goals

Produce realistic operational video event data set representing

scenarios for surveillance domain

Ground truth the video event data for VER and map to suitable Event

Ontology developed in previous workshops

Annotate the data set with object detection and tracking metadata that

serves the needs of all participating/expected event recognition

algorithms

Develop evaluation criteria and metrics for quantitative evaluation of

VER algorithms and software tools for evaluation

Assess different VER approaches for the applicability to operational

scenarios by their learning/explanation/ recognition capabilities


VERAAE Approach

Evaluationmethodology

Video Event RecognitionAlgorithms

object metadataEvent

Ontology event metadata

TechnologyAssessment

VideoData

IC Event Scenarios

AnnotatedVideoData

Evaluationmethodology

Video Event RecognitionAlgorithms

object metadataEvent

Ontology event metadata

TechnologyAssessment

VideoData

Event Scenarios

AnnotatedVideoData


Content Extraction

Event Detection

Event Recognition

Video data

Object features, tracks

Behaviors, actions, events

Abnormal and suspicious eventsTrends, correlations..

Signal

Raw Information

Semantics, Ontology

Knowledge, Intelligence

Video Event Recognition and VERAAE

VERAAE

Eva

luat

ion

Pro

vid

ed


VERAAE Domain

VERAAE domain focus: surveillance

realistic scenarios of interest

Events and activities existing algorithms can detect

Realistic high level or complex events end-users want to detect

Workshop event scenarios

Data Set


Data Set Planning

Primary factors that determine the data requirements:

- Fixed camera views, no PTZ

- Color, B&W and IR

- Realistic operational scenarios

About 10 events with varying complexity, at least 10 samples per event

- The collection parameters that address the functional capabilities of the algorithms

- Annotation will include the object track data required by the participating algorithms

(automatically and manually generated) e.g.:

Silhouettes of tracked objects

Bounding boxes and centroid of objects (U Maryland ViPER tool)

Object category e.g. vehicle, person, box, animal,…

Ground truth for video events will be generated using the event ontology work

- Frame numbers (time offsets) for Event Start and End, identified simple sub events


Event Ontology (Event Taxonomy workshop)

Simple eventDomain independent action descriptorse.g. abandoning an object

Compound (complex or multi-threaded) eventMultiple simple events taking place in time and space constraints to achieve complex

activities.e.g. planting suspicious object, (if considered with below simple events

moving in the wrong direction

parked car at the curb-side

no one exiting parked car

getting in the car

Domain specific high level event- Semantic interpretation of events in a particular context, over multiple-views and

multiple data type eventse.g. sabotaging public facility


Recognizing Surveillance Events

Surveillance Event types from the user’s point of view

Violation of some rule - wrong direction (in thru the out door)- abandoned object ( suitcase left unattended for t>T)

Suspicious or Interesting activity- non exit from a parked car- repeated visits to a store shelf

Abnormal activity- approaching several cars in the lot- several somewhat suspicious events in close proximity

Naturally represented by rules and constraints

Users can easily describe them

Highly context dependent, even context from other

camera views

Users can not easily describe but know when they see it

Naturally represented by probabilistic models and

learning

Users build a sense of “normalcy”


Recognizing Surveillance Events

Knowing what can be detected we describe the events using

not only observable, but also detectable actions

Example: Shoplifting

Camera 1 in the store

•Repeated visit to an area

•Running in the store

Camera 2 in the parking lot

•Car in front of emergency exit

•No one exits from car


Rule Based Event: Violation by an activity

constraint – car parked in the

driveway


Rule Based Event: Violation by an object class

constraint


Suspicious Event: “testing” the exclusion zone


Abnormal Event: Vehicle casing the building


Abnormal Event: Large Vehicle at the Gate


Workshop Timeline

Evaluation tools development, Evaluation results, Final report

WorkshopDry-RunMeeting

October (3rd week)In Boston

Data, Evaluation criteria generation, distribution

Planning, invitations communications

FirstWorkshopMeeting

June 20/21With CVPR

Scenario FocusMeeting

EvaluationCriteria FocusMeeting

WorkshopFinalMeeting

May 05

Final report

December 05

This is a “seedling” workshop to investigate feasibility


Workshop Approach

- First Workshop Meeting (2 days, June):Purpose:

- Workshop goals and vision;

- Presentation and determination of algorithms to

participate in the workshop;

- Presentation of example data sequences.

Outcome: - Outline of the data requirements (object tracking,

data exchange protocols etc.)

- Draft a rough set of evaluation criteria

- Solicit feedback on scenario complexity and

realism


Workshop Approach

Evaluation Criteria Focus Meeting (2 days, July):

Purpose: to determine evaluation criteria best suited for VER.

Outcome: - Evaluation Criteria will be interactively developed in workshop

meetings leveraging Event Ontology, VEML and ETISEO workshop findings

Evaluation metrics at the component and system level will be defined based on

- Recognition rate

- Learning rate

Recall and Precision rates

True/False positives, True/False negatives and relevance of false detections

Event decomposition (based on the ontology defined sub event recognition rate)


Workshop Approach

Workshop Dry-Run Meeting (2 days, October 05)

Purpose and Outcomes: • Participant’s feedback on processing the sample data sets.

• Evaluation tools and methodology presentation

• Evaluating the “evaluation criteria” and finalizing all metrics to be used.

• Planning of evaluation format

• Discussion of interpretation of results


Workshop Results

- Raw and annotated (with object detection and tracking data)

video sequences for realistic operational scenarios

- Event Recognition ground truth data based on surveillance

Event Ontology

- Re-usable and extendible Evaluation Criteria suitable for VER

- Software tools for event detection evaluation

- The groundwork for a formal VER evaluation process

video event recognition algorithm assessment evaluation workshop veraae etiseo – nice, may 10-11...

Documents