A Complex Event A Complex Event Recognition ArchitectureRecognition Architecture

Will FitzgeraldWill Fitzgerald

Kalamazoo CollegeKalamazoo College

R. James FirbyR. James Firby

I/NET, IncI/NET, Inc..

A Complex Event A Complex Event Recognition Recognition ArchitectureArchitecture

Protecting us from Protecting us from the Metal Horde!the Metal Horde!

Will FitzgeraldWill Fitzgerald

R. James FirbyR. James Firby

What is …What is …

A Complex Event?A Complex Event? Complex events are hierarchical, Complex events are hierarchical,

discrete, time-stamped structures discrete, time-stamped structures inferred from multi-channel, inferred from multi-channel, asynchronous signals.asynchronous signals.

What is …What is …

A Complex Event A Complex Event Recognition Architecture?Recognition Architecture?

A description or A description or implementation of typical implementation of typical patterns and recognition patterns and recognition algorithms for complex algorithms for complex events.events.

A real example…A real example…

Water Recovery Water Recovery System at NASA's System at NASA's Johnson Space Johnson Space Center Center Four complex Four complex

subsystems,subsystems, About 200 sensors About 200 sensors

and actuators, and actuators, Each subsystem Each subsystem

asynchronously asynchronously signals data.signals data.

Detecting “Safe Mode”Detecting “Safe Mode” When a problem is detected When a problem is detected

internally, the Water Recovery internally, the Water Recovery System attempts to go into “safe System attempts to go into “safe mode,” which occurs when the mode,” which occurs when the four subsystems are safed. four subsystems are safed.

““Safing” of the four subsystems Safing” of the four subsystems happen asynchronously. happen asynchronously.

““Safing” detection for each Safing” detection for each subsystem differs from one subsystem differs from one another.another.

On recognizing that the WRS On recognizing that the WRS has gone into safe mode, signal has gone into safe mode, signal an event that all subsystems an event that all subsystems have been safed. have been safed.

Another example…Another example…

To get directions to a To get directions to a location on the on-location on the on-board map, the user board map, the user says:says: ““Go here” andGo here” and Taps the display Taps the display

locationlocation

within 200 ms.within 200 ms.(CNN photo)

Parsing the worldParsing the world

Dynamic Predictive Memory Architecture Dynamic Predictive Memory Architecture (DPMA)(DPMA) KR and Semantic parsingKR and Semantic parsing Task execution and dialogue managementTask execution and dialogue management “ “complex, dynamic environments”complex, dynamic environments”

Do similar techniques apply to …Do similar techniques apply to … multi-channel, asynchronous sensors?multi-channel, asynchronous sensors? multi-modal interface input?multi-modal interface input?

A Complex Event Recognition A Complex Event Recognition ArchitectureArchitecture

What What assumptionsassumptions are reasonable to are reasonable to make about the form of input data?make about the form of input data?

What What useful general patternsuseful general patterns are are there in the data?there in the data?

What What recognition algorithms recognition algorithms do we do we need?need?

NLP AssumptionsNLP Assumptions

Input to Natural Language Processing Input to Natural Language Processing systems are typically assumed to be:systems are typically assumed to be: Discrete events of one type (“words”)Discrete events of one type (“words”) Single channelSingle channel Totally ordered by position; duration Totally ordered by position; duration

irrelevantirrelevant

11 22 33 44 55

timetime fliesflies likelike anan arrowarrow

More generally…More generally…

Events of various typesEvents of various types Over multiple channels and asynchronousOver multiple channels and asynchronous Duration of event often importantDuration of event often important ““Hierarchical” model still usefulHierarchical” model still useful

000000 005005 010010 015015 020020 025025 030030

putput thesethese herehere

click and drag click and drag taptap

Assumptions about EventsAssumptions about Events

Discrete: Discrete: Individually distinict, non-Individually distinict, non-continuous data (could be discretized).continuous data (could be discretized).

Time-stamped:Time-stamped: Event carries the start Event carries the start and end times (defining the event and end times (defining the event durationduration, which could be instanteneous)., which could be instanteneous).

Typed:Typed: Events form distinct types (e.g., Events form distinct types (e.g., wordswords vs. vs. tapstaps).).

Structured:Structured: Event may internal, Event may internal, hierarchical structure (“complex”). hierarchical structure (“complex”).

Standard Event PatternsStandard Event Patterns

Are there patterns of events which are Are there patterns of events which are particularly useful to identify?particularly useful to identify?

Are there recognition algorithms to identify those Are there recognition algorithms to identify those patterns?patterns?

Yes.Yes. ONE and BINDING ONE and BINDING IN-ORDER, ALL, ONE-OFIN-ORDER, ALL, ONE-OF Allen patternsAllen patterns WITHIN and WITHOUTWITHIN and WITHOUT

ONEONE and and BINDINGBINDING patterns patterns

ONEONE: The simple : The simple pattern of looking for pattern of looking for a single event (of a a single event (of a particular type).particular type).

BINDING:BINDING: ONEONE pattern plus collecting pattern plus collecting and constraining and constraining state.state.

Essentially “event-Essentially “event-driven” programming; driven” programming; the “stimulus” in S-R.the “stimulus” in S-R.

ON-CLICKON-CLICK A A ONEONE pattern if just pattern if just

looking for the ‘click’looking for the ‘click’ A A BINDINGBINDING pattern if pattern if

x,yx,y coordinates are coordinates are significant.significant.

IN-ORDERIN-ORDER patterns patterns

Events will occur “in order”Events will occur “in order” That is, saying two events, That is, saying two events, AA and and BB, occur in , occur in

order, the order, the start timestart time of B is of B is the the end timeend time of of A.A.

(IN-ORDER A B C D)(IN-ORDER A B C D) First an event of type A, then B, etc. First an event of type A, then B, etc.

IN-ORDERIN-ORDER as as NLPNLP

Combined with Combined with BINDINGBINDING and signaling of and signaling of subpatterns this is essentially a classic subpatterns this is essentially a classic natural language processing pattern.natural language processing pattern.

S S NP VP NP VP

NP NP DET N DET N

VP VP V NP V NP

The boy saw the girl.

[S [NP [DET the][N boy]] [VP [V saw] [NP [DET the] [N girl]]]]

ALL ALL PatternsPatterns

Events will all occur, but in any orderEvents will all occur, but in any order With this, we leave (our) standard NLP With this, we leave (our) standard NLP

approaches.approaches. For example, user will choose from all of the For example, user will choose from all of the

sets of options.sets of options. For example, all subsystems will be “safed”, For example, all subsystems will be “safed”,

but in any order.but in any order.

ALL patterns and ALL patterns and contradictioncontradiction

The problem: user or system “undoing” an The problem: user or system “undoing” an event that has already been seen event that has already been seen (interpreting events as state changes).(interpreting events as state changes).

Example: Class will start when all the Example: Class will start when all the students, Alice, Bob, Charles, Dominique, students, Alice, Bob, Charles, Dominique, have arrived.have arrived.

Consider this sequence for Consider this sequence for (ALL A B C D):(ALL A B C D):

1.1. Charles arrives.Charles arrives.

2.2. Alice and Bob arrive together.Alice and Bob arrive together.

3.3. Alice starts to sing.Alice starts to sing.

4.4. Charles leaves.Charles leaves.

5.5. Dominique arrives.Dominique arrives.

6.6. Charles arrives.Charles arrives.

Order is not relevant; Alice’s singing is not relevant; but Charles’s leaving undoes his earlier arrival.

ONE-OF PatternONE-OF Pattern

Look for any of a set of event formsLook for any of a set of event forms Example: Office hours begin as soon as Example: Office hours begin as soon as

one of the professors A,B,C or D arrives.one of the professors A,B,C or D arrives. (ONE-OF A B C D)(ONE-OF A B C D)

Time-based patternsTime-based patterns

Allen relationshipsAllen relationships WITHINWITHIN patterns patterns WITHOUTWITHOUT patterns patterns

Allen PatternsAllen Patterns

James Allen James Allen described the described the relationships between relationships between two intervals.two intervals.

Allen patterns look for Allen patterns look for temporal relationships temporal relationships between 2 events or between 2 events or an event and an an event and an interval.interval.

1.1. containscontains2.2. finishesfinishes3.3. startsstarts4.4. beforebefore5.5. meetsmeets6.6. overlapsoverlaps7.7. equalequal8.8. overlapped byoverlapped by9.9. afterafter10.10.met bymet by11.11.started bystarted by12.12.finished byfinished by13.13.during during

A containsB

A … overlaps B

WITHIN and WITHOUTWITHIN and WITHOUT

WITHIN WITHIN patterns patterns reflect that the reflect that the duration of an event is duration of an event is no longer than a no longer than a certain amount of certain amount of time.time.

E.g.E.g., an , an ALLALL pattern pattern wrapped in a wrapped in a WITHINWITHIN pattern.pattern.

WITHOUTWITHOUT patterns patterns reflect that an interval reflect that an interval of time will pass of time will pass without the without the occurrence of an occurrence of an event.event.

E.g.E.g., Sherlock , Sherlock Holmes’s Holmes’s “significance of the “significance of the barking dog.”barking dog.”

Pattern CombinationPattern Combination

““Go here” and a tap Go here” and a tap within 200 ms.within 200 ms.

(within

(all

(in-order go here)

(tap ?x ?y))

200 ms)(CNN photo)

Safe mode recognizerSafe mode recognizer

(define-recognizer (safing-complete)(define-recognizer (safing-complete)

(pattern (pattern

'(all '(all

(safing (system pbbwp) (status on))(safing (system pbbwp) (status on))

(safing (system ro) (status on))(safing (system ro) (status on))

(safing (system aes) (status on))(safing (system aes) (status on))

(safing (system pps) (status on))))(safing (system pps) (status on))))

(on-complete (st end)(on-complete (st end)

(signal-event '(all-safed) st end))) (signal-event '(all-safed) st end)))

Some details elided…

Parsing AlgorithmsParsing Algorithms

The parsing algorithms and recognizer The parsing algorithms and recognizer semantics are more fully described in the semantics are more fully described in the paper.paper.

Implementation DetailsImplementation Details

ConclusionsConclusions

Standard patterns of events.Standard patterns of events. Standard recognizers for these patterns.Standard recognizers for these patterns. Good for monitoring complex (internal) Good for monitoring complex (internal)

system state.system state. Useful for recognizing patterns of complex Useful for recognizing patterns of complex

events over multiple modes, over time.events over multiple modes, over time.

AcknowledgmentsAcknowledgments

Work done under NASA SBIR contract Work done under NASA SBIR contract NAS9-00122.NAS9-00122.

We would like to especially acknowledge We would like to especially acknowledge collaborators at NASA, including Debra collaborators at NASA, including Debra Schreckenghost, Pete Bonasso, Carrol Schreckenghost, Pete Bonasso, Carrol Thronesbery and others.Thronesbery and others.

Pulp Images from “Pulp of the Day”:Pulp Images from “Pulp of the Day”:groups.yahoo.com/group/pulpofthedaygroups.yahoo.com/group/pulpoftheday

Questions?Questions?