1

Multimedia Systems Security:

Video Data Analysis for Security Applications and Securing Video Data

Dr. Bhavani Thuraisingham

September 2007

2

OutlineOutline

Data Mining for Security ApplicationsData Mining for Security Applications Video Analysis Suspicious Event Video Analysis Suspicious Event

DetectionDetection Access ControlAccess Control Privacy Preserving SurveillancePrivacy Preserving Surveillance Secure Third Party Publication of Video Secure Third Party Publication of Video

DataData Malicious Code DetectionMalicious Code Detection Directions and OpportunitiesDirections and Opportunities

3

AcknowledgmentsAcknowledgments

Professor Latifur Khan for data mining Professor Latifur Khan for data mining applications and Malicious Code Detectionapplications and Malicious Code Detection

Prof Elisa Bertino (Purdue) and Prof. Prof Elisa Bertino (Purdue) and Prof. Jianping Fan (UNCC) for Privacy Jianping Fan (UNCC) for Privacy Preserving Video AnalysisPreserving Video Analysis

Prof. Elisa Bertino, Prof Elena Ferrari Prof. Elisa Bertino, Prof Elena Ferrari (Milan/Como) and Prof. Barbara Carminati (Milan/Como) and Prof. Barbara Carminati (Milan/Como) for Secure Third Party (Milan/Como) for Secure Third Party PublicationPublication

Students at the University of Texas at Students at the University of Texas at DallasDallas

4Data Mining for Security Data Mining for Security ApplicationsApplications

Data Mining has many applications in Data Mining has many applications in Cyber Security and National SecurityCyber Security and National Security Intrusion detection, worm detection, Intrusion detection, worm detection,

firewall policy managementfirewall policy management Counter-terrorism applications and Counter-terrorism applications and

SurveillanceSurveillance Fraud detection, Insider threat analysisFraud detection, Insider threat analysis

Need to enforce security but at the Need to enforce security but at the same time ensure privacysame time ensure privacy

5

Problems AddressedProblems Addressed Huge amounts of video data Huge amounts of video data

available in the security available in the security domaindomain

Analysis is being done off-line Analysis is being done off-line usually using “Human Eyes”usually using “Human Eyes”

Need for tools to aid human Need for tools to aid human analyst ( pointing out areas in analyst ( pointing out areas in video where unusual activity video where unusual activity occurs)occurs)

Need to control access to the Need to control access to the video datavideo data

Need to securely publish Need to securely publish video datavideo data

Need to ensure that the data Need to ensure that the data is not maliciously corrpuptedis not maliciously corrpupted

6Video Analysis fore SecurityVideo Analysis fore SecurityThe Semantic GapThe Semantic Gap

The disconnect between the low-level The disconnect between the low-level features a machine sees when a video is features a machine sees when a video is input into it and the high-level semantic input into it and the high-level semantic concepts (or events) a human being sees concepts (or events) a human being sees when looking at a video clip when looking at a video clip

Low-Level featuresLow-Level features: : color, texture, color, texture, shapeshape

High-level semantic conceptsHigh-level semantic concepts: : presentation, newscast, boxing matchpresentation, newscast, boxing match

7

Our ApproachOur Approach Event Representation Event Representation

Estimate distribution of pixel intensity Estimate distribution of pixel intensity change change

Event ComparisonEvent Comparison Contrast the event representation of Contrast the event representation of

different video sequences to determine if different video sequences to determine if they contain similar semantic event content.they contain similar semantic event content.

Event DetectionEvent Detection Using manually labeled training video Using manually labeled training video

sequences to classify unlabeled video sequences to classify unlabeled video sequences sequences

8Event Representation, Event Representation, Comparison, DetectionComparison, Detection

Measures the quantity and type of changes occurring within a Measures the quantity and type of changes occurring within a scene scene

A video event is represented as a set of x, y and t intensity A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales.gradient histograms over several temporal scales.

Histograms are normalized and smoothedHistograms are normalized and smoothed Determine if the two video sequences contain similar high-level Determine if the two video sequences contain similar high-level

semantic concepts (events). semantic concepts (events).

Produces a number that indicates how close the two compared Produces a number that indicates how close the two compared events are to one another. events are to one another.

The lower this number is the closer the two events are. The lower this number is the closer the two events are. A robust event detection system should be able toA robust event detection system should be able to

Recognize an event with reduced sensitivity to actor (e.g. clothing or Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation.skin tone) or background lighting variation.

Segment an unlabeled video containing multiple events into event Segment an unlabeled video containing multiple events into event specific segmentsspecific segments

22 1 2

, , 1 2

[ ( ) ( )]1

3 ( ) ( )

l lk kl l

k l i k k

h i h iD

L h i h i

9

Labeled Video EventsLabeled Video Events

These events are manually labeled These events are manually labeled and used to classify unknown eventsand used to classify unknown events

Walking1 Walking1 Running1Running1 Waving2Waving2

10

Labeled Video EventsLabeled Video Events

  walkinwalkin

g1g1walkinwalkin

g2g2walkinwalkin

g3g3runninrunnin

g1g1runninrunnin

g2g2runninrunnin

g3g3runninrunnin

g4g4waving waving

22

walkinwalking1g1 00 0.276250.27625 0.245080.24508 1.22621.2262 1.3831.383 0.974720.97472 1.37911.3791 10.96110.961

walkinwalking2g2 0.276250.27625 00 0.178880.17888 1.47571.4757 1.50031.5003 1.29081.2908 1.5411.541 10.58110.581

walkinwalking3g3 0.245080.24508 0.178880.17888 00 1.12981.1298 1.09331.0933 0.886040.88604 1.12211.1221 10.23110.231

runninrunning1g1 1.22621.2262 1.47571.4757 1.12981.1298 00 0.438290.43829 0.304510.30451 0.398230.39823 14.46914.469

runninrunning2g2 1.3831.383 1.50031.5003 1.09331.0933 0.438290.43829 00 0.238040.23804 0.107610.10761 15.0515.05

runninrunning3g3 0.974720.97472 1.29081.2908 0.886040.88604 0.304510.30451 0.238040.23804 00 0.204890.20489 14.214.2

runninrunning4g4 1.37911.3791 1.5411.541 1.12211.1221 0.398230.39823 0.107610.10761 0.204890.20489 00 15.60715.607

wavingwaving22 10.96110.961 10.58110.581 10.23110.231 14.46914.469 15.0515.05 14.214.2 15.60715.607 00

11

Experiment #1Experiment #1

Problem: Recognize and classify events Problem: Recognize and classify events irrespective of direction (right-to-left, left-irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to to-right) and with reduced sensitivity to spatial variations (Clothing)spatial variations (Clothing)

““Disguised Events”- Events similar to Disguised Events”- Events similar to testing data except subject is dressed testing data except subject is dressed differentlydifferently

Compare Classification to “Truth” Compare Classification to “Truth” (Manual Labeling)(Manual Labeling)

12

Experiment #1Experiment #1

Classification: WalkingClassification: Walking

Disguised Walking 1

walking1walking1 walking2walking2 walking3walking3 running1running1 running2running2 running3running3 running4running4 waving2waving2

0.976530.97653 0.451540.45154 0.596080.59608 1.54761.5476 1.46331.4633 1.57241.5724 1.54061.5406 12.22512.225

13

Experiment #1Experiment #1

Classification: RunningClassification: Running

Disguised Running 1

walking1walking1 walking2walking2 walking3walking3 running1running1 running2running2 running3running3 running4running4 waving2waving2

1.4111.411 1.38411.3841 1.06371.0637 0.567240.56724 0.974170.97417 0.935870.93587 1.09571.0957 11.62911.629

14

XML Video AnnotationXML Video Annotation Using the event detection scheme we generate a Using the event detection scheme we generate a

video description document detailing the event video description document detailing the event composition of a specific video sequencecomposition of a specific video sequence

This XML document annotation may be replaced This XML document annotation may be replaced by a more robust computer-understandable by a more robust computer-understandable format (e.g. the VEML video event ontology format (e.g. the VEML video event ontology language). language).

<?xml version="1.0" encoding="UTF-8"?><?xml version="1.0" encoding="UTF-8"?><videoclip><videoclip> <Filename>H:\Research\MainEvent\<Filename>H:\Research\MainEvent\ Movies\test_runningandwaving.AVI</Filename>Movies\test_runningandwaving.AVI</Filename> <Length>600</Length><Length>600</Length> <Event><Event> <Name>unknown</Name><Name>unknown</Name> <Start>1</Start><Start>1</Start> <Duration>106</Duration><Duration>106</Duration> </Event></Event> <Event><Event> <Name>walking</Name><Name>walking</Name> <Start>107</Start><Start>107</Start> <Duration>6</Duration><Duration>6</Duration> </Event></Event></videoclip></videoclip>

15

Video Analysis ToolVideo Analysis Tool Takes annotation document as input and organizes the Takes annotation document as input and organizes the

corresponding video segment accordingly.corresponding video segment accordingly. Functions as an aid to a surveillance analyst searching for Functions as an aid to a surveillance analyst searching for

“Suspicious” events within a stream of video data.“Suspicious” events within a stream of video data. Activity of interest may be defined dynamically by the Activity of interest may be defined dynamically by the

analyst during the running of the utility and flagged for analyst during the running of the utility and flagged for analysis.analysis.

16Access Control: Access Control: Authorization ObjectsAuthorization Objects

Authorization objects, the actual video data to Authorization objects, the actual video data to which we wish to restrict access and represented which we wish to restrict access and represented in the form of a 7 value tuple. in the form of a 7 value tuple.

This tuple contains information about the content This tuple contains information about the content of a particular video object. Some of this content of a particular video object. Some of this content information pertains to high-level semantic information pertains to high-level semantic information such as events and objects. information such as events and objects.

This information is stored as a set of concepts This information is stored as a set of concepts taken from a “closed-world” hierarchical taxonomy taken from a “closed-world” hierarchical taxonomy which relates these concepts to one another. which relates these concepts to one another.

Other content information such as location and Other content information such as location and timestamp is represented as a special data type timestamp is represented as a special data type that allows more meaningful specification of this that allows more meaningful specification of this unique kind of content.unique kind of content.

17Access Control: Video Access Control: Video Object HierarchyObject Hierarchy

Surveillance Object

Still CameraVideo Camera

Satellite Image

Aerial ImageHallway Camera

LobbyCamera

18Access Control: Access Control: Other ConceptsOther Concepts

Events Events is the set of semantic events is the set of semantic events occurring within the video object. occurring within the video object.

Objects Objects is the set of semantic objects is the set of semantic objects contained within the video object. contained within the video object.

Location Location is the term indicating the is the term indicating the geographic earth coordinates of where the geographic earth coordinates of where the surveillance video object was captured. surveillance video object was captured.

Timestamp Timestamp is the term describing the real is the term describing the real world time when the video was capturedworld time when the video was captured. .

19Access Control: Event and Access Control: Event and Object HhierarchiesObject Hhierarchies

Video Event

Stationary Event

Mobile Event

Waving

Walking

Running

Jumping

Video Object

Vehicle Toy

Truck

Ball Frisbee

Car

20

Video Object ExpressionsVideo Object Expressions Video object expressions describe the object Video object expressions describe the object

for which access control is to be applied. for which access control is to be applied. These expressions are expanded and made These expressions are expanded and made

more robust so that a video object may be more robust so that a video object may be specified not only by its object ID but rather specified not only by its object ID but rather by any of its attributes or their combination. by any of its attributes or their combination.

This is similar to querying a relational This is similar to querying a relational database using a complex SQL query database using a complex SQL query specifying a particular set of records.specifying a particular set of records.

We use access functions to reference the We use access functions to reference the different components of our surveillance different components of our surveillance video objects for use in our expressions. video objects for use in our expressions.

21

Authorization SubjectsAuthorization Subjects We use the concept of user credentials to We use the concept of user credentials to

authorize users. authorize users. That is, each user entity, in addition to having a That is, each user entity, in addition to having a

unique user id or belonging to a group also unique user id or belonging to a group also possesses a set of credentials. possesses a set of credentials.

Each credential is an instantiation of a certain Each credential is an instantiation of a certain credential type, the template for credentials in credential type, the template for credentials in which the set of credential attributes, and whether which the set of credential attributes, and whether they are optional or obligatory is defined. they are optional or obligatory is defined.

Specific values are assigned to these attributes Specific values are assigned to these attributes when a new user instantiates the credential type. when a new user instantiates the credential type.

A subject may instantiate any number of A subject may instantiate any number of credential types. credential types.

These credential types are defined in a credential These credential types are defined in a credential type hierarchy relating each credential type to the type hierarchy relating each credential type to the other credential typesother credential types

22Access Control: Credential Access Control: Credential Type HierarchyType Hierarchy

Person

Maintenance Staff

Security Officer

Database Administrator

Police Guard

Patrolman Captain

23Access Control: Access Control: AuthorizationsAuthorizations

Authorizations are what allow us to specify Authorizations are what allow us to specify our access control policy for the objects in our access control policy for the objects in our video surveillance database. our video surveillance database.

Derived Authorizations: The properties of Derived Authorizations: The properties of the hierarchical taxonomies used in defining the hierarchical taxonomies used in defining surveillance video object types, semantic surveillance video object types, semantic event types and semantic object types can event types and semantic object types can be used to obtain implicit authorizations be used to obtain implicit authorizations from the explicit authorizations specified as from the explicit authorizations specified as a part of the access control policy base. a part of the access control policy base.

Additionally the relationships between the Additionally the relationships between the various privilege modes allow further various privilege modes allow further extrapolation of authorizations.extrapolation of authorizations.

24

Access Control AlgorithmAccess Control Algorithm User requests for surveillance video objects must User requests for surveillance video objects must

be compared to the policy base of object be compared to the policy base of object authorizations before access can be granted. authorizations before access can be granted.

Furthermore, if the user request is not for a Furthermore, if the user request is not for a specific object but rather a query for a particular specific object but rather a query for a particular set of objects the system must be able to set of objects the system must be able to successfully reconcile the query criteria with the successfully reconcile the query criteria with the objects existing in the database. objects existing in the database.

If the user request is authorized for some part If the user request is authorized for some part (but not all) of the surveillance video object (but not all) of the surveillance video object instead of denying the access entirely it is instead of denying the access entirely it is possible to post-process the data after retrieval possible to post-process the data after retrieval and release only authorized portions to the user. and release only authorized portions to the user.

Hence our access control process has three major Hence our access control process has three major components: Authorization, retrieval, post-components: Authorization, retrieval, post-processing and delivery.processing and delivery.

25Access Control Policies: Access Control Policies: ExtensionsExtensions

Policies based on content, Policies based on content, associations, time, and eventassociations, time, and event

Policy engine that evaluates the Policy engine that evaluates the policies for consistencypolicies for consistency

Enforcement engine for enforcing Enforcement engine for enforcing the policiesthe policies

Distributed policies: Objects at Distributed policies: Objects at different locations taken together different locations taken together are sensitiveare sensitive

26System Architecture for System Architecture for Access ControlAccess Control

UserPull/Query Push/result

Video XML Documents

X-Access X-AdminAdmin Tools

Policybase

Credentialbase

27

Third-Party Third-Party ArchitectureArchitecture

Credential base

policy baseXML Source

User/Subject

Owner

Publisher

Query

Reply documen

t

SE-XML

credentials

The Owner is the The Owner is the producer of producer of informationinformation It It specifies access specifies access control policies on control policies on the Video objectsthe Video objects

The The PPublisher is ublisher is responsible for responsible for managing (a managing (a portion of) the portion of) the Owner information Owner information and answering and answering subject queriessubject queries

Goal: Untrusted Goal: Untrusted Publisher with Publisher with respect to respect to Authenticity and Authenticity and Completeness Completeness checkingchecking

28

• Policy Information• Merkle Signature

XML Document

SE-XML Document

Security Enhanced Video XML document

Privacy Preserving Privacy Preserving Video Analysis Video Analysis

•A recent survey at Times Square found 500 visible surveillance cameras in the area and a total of 2500 in New York City.

•What this essentially means is that, we have scores of surveillance video to be inspected manually by security personnel

•We need to carry out surveillance but at the same time ensure the privacy of individuals who are good citizens

30

System System UseUse

Raw video surveillance data

Face Detection and Face Derecognizing system

Suspicious Event Detection System

Manual Inspection of video data

Comprehensive security report listing suspicious events and people detected

Suspicious people found

Suspicious events found

Report of security personnel

Faces of trusted people derecognized to preserve privacy

31

Detecting Malicious CodeDetecting Malicious Code✗Content -based approaches consider Content -based approaches consider only machine-codes (byte-codes).only machine-codes (byte-codes).✗Is it possible to consider higher-level Is it possible to consider higher-level source codes for malicious code source codes for malicious code detection?detection?✗Yes: Diassemble the binary executable Yes: Diassemble the binary executable and retrieve the assembly programand retrieve the assembly program✗Extract important features from the Extract important features from the assembly programassembly program✗Combine with machine-code features Combine with machine-code features ✗Extract both Binary n-gram features Extract both Binary n-gram features and Assembly n-gram featuresand Assembly n-gram features

32Hybrid Feature Retrieval Hybrid Feature Retrieval (HFR)(HFR) Training Training

TestingTesting

33

Summary and DirectionsSummary and Directions We have proposed an event representation, comparison We have proposed an event representation, comparison

and detection scheme.and detection scheme. Working toward bridging the semantic gap and Working toward bridging the semantic gap and

enabling more efficient video analysisenabling more efficient video analysis More rigorous experimental testing of conceptsMore rigorous experimental testing of concepts Refine event classification through use of multiple Refine event classification through use of multiple

machine learning algorithm (e.g. neural networks, machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine decision trees, etc…). Experimentally determine optimal algorithm.optimal algorithm.

Develop a model allowing definition of simultaneous Develop a model allowing definition of simultaneous events within the same video sequenceevents within the same video sequence

Define an access control model that will allow access to Define an access control model that will allow access to surveillance video data to be restricted based on surveillance video data to be restricted based on semantic content of video objects semantic content of video objects

Secure publishing of Video Documents Secure publishing of Video Documents Privacy Preserving AnalysisPrivacy Preserving Analysis Detecting Malicious CodeDetecting Malicious Code

34Opportunities for the Opportunities for the CommunityCommunity WeWe