national science foundation engineering research … pdfs/iwatch_session...– event detection: ram...

47
National Science Foundation Engineering Research Center

Upload: leduong

Post on 21-Mar-2018

216 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Page 2: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

iWatch: iWatch: Intelligent Surveillance at Scale byConnecting-the-Dots in Space and Time!

Farnoush Banaei-KashaniViterbi School of Engineering

University of Southern CaliforniaLos Angeles, CA 90089

[email protected]

Page 3: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Intelligent Surveillance

Surveillance [sɜːˈveɪləns]: Obtaining information and knowledge about events of interest through observation, investigation, and analysis for the sake of understanding

Intelligent Surveillance : IT + Surveillance

Pictures courtesy of rotoconcept.com

Page 4: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Intelligent Surveillance at Scale!

Home Scale

Campus Scale

National Scale

Human-in-the-Loop Surveillance

Page 5: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

• Human-Deeply-in-the-Loop Surveillance!– Efficiency: Limited human processing power limits capabilities of

the system in handling, e.g.: • Numerous heterogeneous input feeds• High input and event rate• Large number of events of interest

– Accuracy: Human is error-prone!

– Cost: Waste of resources

Limitation of the Existing Systems

Page 6: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Our Vision: Geo-Immersive Surveillance

• Support numerous input feeds of various modalities

• Let machine capture all “incidents”

• And piece together incidents in space and time to generate “event” scenarios

• Forensic analysis in historic data

• Real-time monitoring of current input feeds

• Event prediction of potential expected (and unexpected!) events

Page 7: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Applications

• Large-scale Urban Securitye.g., campus incidents

• Military Intelligencee.g., counter-insurgency (COIN) war

• Plant Disaster Managemente.g., oil spills

• National Securitye.g., terrorist attacks

Page 8: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

iWatch: A Prototype System

• Application: Campus Security Tool for USC’s Department of Public Safety (DPS)

• Platform: iCampus.USC• Multidisciplinary Team:

– App: Carol Hayes, David Beeler– Event Detection: Ram Nevatia (tracking),

Gerard Medioni (face detection)– Spatiotemporal Query and Analysis: Cyrus

Shahabi, Farnoush Banaei-Kashani– Visualization: Ulrich Neumann

Page 9: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Current Status

• Input: Video feed from 25 PTZ cameras• Mode: Forensic Analysis

Page 10: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Research Challenges

• Advanced video event detection– Allow for sparse coverage– Detect from mobile video (unknown parameters)– Devise event ontology

• Spatiotemporal Query and Analysis– Spatiotemporal modeling of the events– Spatiotemporal indexing of the events– Support for uncertain data

• Visualization– Rich and realistic modeling using multiple complementary data

sources (e.g., aerial as well as street LiDAR)

Talk by Ram Nevatia

Page 11: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Future Research Plan /Collaboration Opportunities

• Closing the loop by personalized information provision

• Mobile video: communication and spatiotemporal modeling

• Advanced video event detection:– Face recognition– Human and social behavior analysis

• Extending the supported data modalities beyond video (text, sensor readings, audio, etc.)

• Have other applications?• Have complementary ideas and technology?

Talk by Daniel Goldberg

Talk by Roger Zimmermann

Page 12: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Sustainability

• Potential Opportunities– USC– Government Research Grants

• Security and Intelligence: DHS, NGA, ARDA, NIJ

• Defense: NURI, MURI

– Industry• Cash gift• Research Proposals• Integrating industry products into the iWatch prototype• License iWatch to other universities & small cities• Spin-off company!

: Google: HP

Page 13: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Q & A

Page 14: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Telling What-is-What in Video

Ram Nevatia, Professor

University of Southern California

Page 15: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Objectives and Approach

• Objectives– Detect and track objects of interest in a video– Infer activities of interest– Detect anomalies

• Approach– Assume calibrated, stationary video camera– Detect objects based on shape and motion– Tracking based on detections– Activity inference based on structured models

Page 16: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

• Part-based representation; allows for detection in presence of occlusions– Requires joint reasoning of inter-occluding humans

• Part detectors use local features, e.g. edgelets• Detectors are cascades of classifiers learned by

AdaBoost procedure• Tracking consists of linking detection responses

– Hierarchical associations– Learning of affinity functions

Shape Based Human Detection and Tracking

Page 17: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Human Detection Examples

Page 18: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Tracking Example (iWatch)

Page 19: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Tracking Result Video (TREVid)

Page 20: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Two Approaches of Activity Detection

• Bottom-up Approach– Track pose/position, infer events from tracks– More general and easier to extend but bottom-up analysis

is not robust• Top-down Approach

– Apply event models directly to low-level features extracted from the videos

– Poses come as a by-product of event recognition• Simultaneously produces pose tracks and action recognition

(STAR)

– More robust but makes a “closed world” assumption

Page 21: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Event Detection ResultsDrop and Pick up Event

Left Luggage Event (Video)

Page 22: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Activity Detection Result Video

Page 23: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Future Direction

• Develop methods for detecting and tracking objects that humans interact with, e.g. a briefcase, left luggage etc.

• Scene objects can provide valuable context but detection of such objects is difficult– Campus models may provide some of this

• Define a class of important activities for iWatch• Integrate the object and event detection capabilities

into the iWatch system

Page 24: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Q & A

Page 25: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Daniel Goldberg, Kaveh Shahabi, John WilsonSpatial Sciences Institute

College of Letters, Arts & SciencesUniversity of Southern California

Los Angeles, CA [email protected]

http://spatial.usc.edu

Targeted Trojan Alerts: Targeted Trojan Alerts: Enabling Dynamic LocationEnabling Dynamic Location--Based Evacuation StrategiesBased Evacuation Strategies

Page 26: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Targeted Alerts Motivation

TROJANSALERT

“When an emergency occurs, authorized senders can instantly notify you using TROJANS ALERT.

TROJANSALERT is your connection to real-time updates, instructions on where to go, what to do, or what not to do, who to contact and other important information.”

Page 27: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Emergency Notification

KAP

GER

DNI

OHE

RTH

RRI

DRBSuspicious package, PSA,3rd floor NE corner.Stay away from area

Page 28: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Broadcast Messages

Suspicious package, PSA,3rd floor NE corner.Stay away from area

Glendale

Park La BreaSanta Monica

Culver City

LAXDowney

LBI New Jersey

Page 29: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Geo-Targeting Messages

KAP

GER

DNI

OHE

RTH

RRI

DRB

Personalize messages•Location of the individual•Relation to the threat

KAP•Exit building •Head north and west toward Vermont and Jefferson

OHE•Exit building •Head east toward the center of campus

CAL•Remain inside

ISI•Avoid travelling to UPC

Page 30: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Targeted Message

You are in the danger zone. Please follow the route to a safe place immediately.

User-Specific Information

Risk

High

Low

Page 31: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

System Operation

End User

Registered Smart Phone

DPS Admin

Trojan Alert Server

ThreatIdentification

Broadcast Message

Report Location

Threat Notification

Page 32: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Research Questions

• Soliciting Participation– Mobile monitoring and Privacy– Market penetration and Incentives

• Risk and Population Surface Construction– Location-based (GPS – know they are there)– Context-based (Classes – supposed to be there)

• Dynamic Routing– Real-time– Dynamic risk, environment, population– Varying spatial scales

But its for your own good.We swear..

Page 33: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

The Team

• Dan Goldberg– USC Spatial Sciences Institute

• Kaveh Shahabi– PhD Student, USC Computer Science

• John Wilson– USC Spatial Sciences Institute

Page 34: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

QuestionsConcerns

Suggestions

Thanks!

??

????

??

??

??

??

????

Page 35: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Roger ZimmermannNational University of Singapore

Singapore [email protected]

http://geovid.org

GeoVidGeoVidManaging Mobile Video in Managing Mobile Video in

Spatiotemporal SpaceSpatiotemporal Space

Seon Ho KimUniversity of Southern California

Los Angeles, CA [email protected]

http://imscwww.usc.edu

Page 36: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Motivation

• Trends– User-generated video content is growing rapidly.– Mobile devices make it easy to capture video.

• Challenge– Video is still difficult to manage and search.

• Geo-Referenced Video– Location and direction information

can now be collected through anumber of sensors(e.g., GPS and compass).

Page 37: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Overview of Approach

1. Viewable scene modeling2. Video and meta-data acquisition3. Indexing, querying, and presentation of results

1)

d2) 3)

Page 38: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Scene Modeling

• Describe the video stream through a sequence of field-of-views (FOV).

2D 3D

FOVScene(P,d,,R) FOVScene(P,d,,,R)

Page 39: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

GeoVid Acquisition

iPhone AppiPhone App Android AppAndroid App

Page 40: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Spatio-Temporal Search

<-117.010, 46.725><-117.013, 46.725><-117.013, 46.728><-117.010, 46.728>

.

.

.

.

Search for videos that capture the given trajectory

Search for the videos of

“Kibbie Dome”

<-117.010, 46.725><-117.013, 46.725><-117.013, 46.728><-117.010, 46.728>

Page 41: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Query Execution

Object X

Camera location

• Spatio-temporal queries to find relevant video segments:“Find videos that show object X from t1to t2.”

• Goal:– Reduce

irrelevant video segments (i.e., reduce user browsing time)

Page 42: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Ex.: Spato-Temporal Range Query

Video 2

Video 1

Queryarea

Page 43: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Search and Results: 2D http://geovid.org/Query.html

Page 44: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Search and Results: 3D

Page 45: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Thank YouThank You

Other Team MembersOther Team Members• Sakire Arslan Ay• Beomjoo Seo• Jia Hao• Guanfeng Wang• Ma He• Shunkai Fang• Lingyan Zhang

http://geovid.orghttp://geovid.orgFurther information at:

[email protected]@usc.edu

Page 46: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

Q & A

Page 47: National Science Foundation Engineering Research … pdfs/iWatch_Session...– Event Detection: Ram Nevatia (tracking), ... • Integrate the object and event detection capabilities

National Science Foundation Engineering Research Center

End of the iWatch SessionThanks!