multi-purpose machine-intelligence-based information ... · • machine-intelligence character of...

MIT Lincoln Laboratory2007 CBIS --1

Jerome J. Braun 12/1/2006

Multi-Purpose Machine-Intelligence-based Information Fusion (FLASH)

Dr. Jerome J. Braun,Mr. Yan Glina,

Ms. Laura J. Brattain, and Dr. Timothy J. Dasey

Lincoln Laboratory, Massachusetts Institute of Technology

2007 Chemical Biological Information Systems (CBIS) Conference

10 January 2007

This material is based upon work supported by the National Science Foundation under Grant No. 0329901. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.



Overview

• Problem– Fusion, recognition and reasoning with uncertain and disparate data (from sensors

and other information sources) for demanding decision-support applications, including complex phenomenology and absence of reliable models

• Relevance– Exploitation of multiple imperfect data sources is key to robust decision support

for CBD systems

• Solution– Advanced algorithmic methods for decision support– FLASH: novel cognitive-processing oriented machine-intelligence

decision support architecture and methods– Broad applicability to CB attack detection and characterization, facility protection,

course-of-action guidance and other decision-support tasks

Uncertain and Disparate Multisource Data (sensors and other sources)

FLASHDecision Support Architecture

Machine-Intelligence Multisource Information Fusion

Decision Support for diverse CBD applications



Motivation for FLASH Machine Intelligence Approach

• CBD domain challenges– Complex phenomenology– Limited reliability of models and processes– Data inexactness and uncertainty

• Machine-intelligence character of FLASH methods– Particularly suitable for demanding decision-support tasks such as CBD– Generality of algorithmic methods offers potential of multi-purpose functionality– Alleviate redevelopment problems and costs associated with proliferation of

single-purpose approaches lacking scalability and migration capability– Enable augmented cognition for warfighter, reduce information overload and

enhance decision quality



Machine Intelligence and Machine Learning

• Paradigm shift from coding pre-known tasks to coding learning process itself• Machine intelligence must include learning

– Automatically learn from information embedded in data, to recognize/decide cases not-seen-before

– Learning implies adaptability• Cognitive systems – high-end machine-learning systems

Training data

Automatic Learning

Classifier (Recognizer)

Military transport Civilian transport

Training

Military or civilian vehicle?

Decision

Generalization: correct decision on data significantly different from previously seen

For harder cases, classifier must generalize better

Classifier (Recognizer)

Training data

Recognition (deciding)

Training

Feature Extraction

Feature Extraction

Data to recognize

Features: salient aspects of inputs to recognizer

Decision

For CBD: “backgrounds”and “releases”

For CBD: “attack Y/N”

Machine intelligence and machine learning alleviate need for precise models, adapt better to unanticipated conditions and different applications

Machine intelligence and machine learning alleviate need for precise models, adapt better to unanticipated conditions and different applications



Hybrid Approach

• Information fusion = inference, recognition, reasoning, and decision-making– Algorithmic process from multiple data sources to decisions– Not about networking, data collection and storage, etc.

• Existent approaches to information fusion all have limitations– Sample of method areas shown below– Relative strengths and weaknesses shown do not imply method applicability or

inapplicability for particular task

We are developing hybrid approach to alleviate limitations of individual methodsWe are developing hybrid approach to alleviate limitations of individual methods

Classical signal processingBayesian methodsNon-Bayesian methodsEstimation theoryNeural networks

RelativeStrength

RelativeWeakness

Perfo

rm

infer

ence

Cope w

ith

unce

rtain

data

Cope w

ith

dispa

rate

data

Suitab

le fo

r FF -

featur

e fus

ion

Suitab

le fo

r DF -

decis

ion fu

sion

Suitab

le fo

r bot

h

FF an

d DF

Suitab

le fo

r

time-s

eries



Conceptual View of FLASH(Fusion, Learning, Adaptive Super-Hybrid)

• FLASH integrates diverse machine-intelligence supervised and unsupervised learning and reasoning methods (subsystems, agents) at multiple levels

• Cognitive-processing architecture – inspired by neuroscience, cognitive studiesDetection, characterization, tasking, COA

Multi-agent cognitive-processing architectureMulti-agent cognitive-processing architecture

Context-scope recognition hybrid

Local-scope recognition

hybrid

Feature Selection

Event simulations

Feature Extraction

Local-scope fusion

Context-scope fusion Approximate

reasoning

Action planning

Planning and goal

satisfaction

Uncertainty estimation and

adjustmentEvent

clustering State-space estimation

Shared information

Lower levels

Higher levels

Multisource data



FLASH-1 Simplified Block Diagram

InstanceLevel

Recognition(ILR)

Hybrid

Time-Series Recognition

(TSR) Hybrid

Feature Extraction

and Selection

Instance Level

Fusion

Uncertainty Management

Background clustering

Release simulators

Training set generation

TemporalFusion Reasoning

ResultsEvent detection, characterization, CoA

Multisensor and context data



Multisensor Fusion Testbed

• Source of data for CB information fusion and decision support studies

• Portable for deployments in various settings• Multiple multisensor nodes, easily reconfigurable, seamless

addition of sensors• Foundation for CB “common operating infrastructure”• Multiple deployments within MIT LL and multi-month data collection

campaigns– Office building environment– Cafeteria with complex background, including cooking facilities

operation and proximity to outdoorsNode contents: • Multiple optical particle counters• MIT LL BAWS III• Airflow sensor• Space for additional sensors• Cluster processor• Power supply

Lower levels

Higher levels Context -scope

recognition hybrid

Local – scope recognition hybrid

Feature Selection

Event simulations

Feature extraction

Local -scope fusion

Context -scope fusion Approximate

reasoning

Action planning

Planning and goal

satisfaction

Uncertainty estimation and adjustment

Shared information

Multisource data

Event clustering

State -space estimation



Testbed Data Management System

Sensor interfaces

Periodic updates

Web ServerSensorML Data Services

Multisensor Node Software

Web Server SensorMLMonitoring

Remote Server

Access via Web Services

Central Data Warehouse

Local storage

Node Web

Server

Short-term Storage

Multisensor data model

• Extensible and modular• Encapsulate and insulate sensor specifics• Data integrity and security• Data model and warehouse for sensor-independent

multisensor data management

Infrastructure suggested as blueprint for CB testbed effortsInfrastructure suggested as blueprint for CB testbed efforts

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




Background Data

• Extensive indoor-background measurement campaigns conducted • Collected data used for development of training datasets for machine learning• Enable background characterization

06/21 06/22 06/23 06/24 06/25 06/26

Diurnal Pattern Node 3

06/21 06/22 06/23 06/24 06/25 06/2606/21 06/22 06/23 06/24 06/25 06/26Time (month/date)

Diurnal Pattern Node 3

0

100

200

Part

icle

C

once

ntra

tion

(a.u

.)

21:10 21:20 21:30 21:40Time (hr:min)

Inter-node Temporal Shifts

Node 1

Node 3

Node 4

21:10 21:20 21:30 21:4021:10 21:20 21:30 21:40

Node 1

Node 3

Node 4

Part

icle

Con

cent

ratio

n (a

.u.)

0

50

100

0

50

100

0

20

40

60

Background data excerpts

Jan Feb Mar0

500

0

500

0

500

0

500

1000

Time (month)

Part

icle

Con

cent

ratio

n (a

.u.)

00:00 06:00 12:00 18:00 00:00Time (hr:min)

Multi-month campaign

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




Release Data

• Stochastic simulation approach for efficient generation of multiple release-cases to support development of training and testing datasets

• Span parameter-space of plausible releases

• Computationally efficient, rudimentary T&D (transport and dispersion)

Release simulation example

Node 1 Node 2 Node 3 Node 4 Node 5 Node 1 Node 2 Node 3 Node 4 Node 5

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




Multisensor Feature Extraction

• Feature design “from first principles” challenging in CB– Non-specific sensing modalities, complex T&D

• Diverse set of features, unconventional types and uses

Dynamic modeling

Virtual sensor features

Kalman or Particle filters estimate signal at locations without sensors

2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 1 2 0 0 1 4 0 0

1 0 2

1 0 3

Am

plitu

de (a

u)

time

2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 1 2 0 0 1 4 0 0- 1 5 0

- 1 0 0

- 5 0

0

5 0

1 0 0

1 5 0

2 0 0

5 0 0 1 0 0 0 1 5 0 0 2 0 0 0 2 5 0 00 . 5

1

1 . 5

2

2 . 5

3

3 . 5

x 1 0 - 3

TKE

(au)

Poly

(au

)

OPC signal

Temporal localization

features

Uncertainty management

features

Indicate events in sensor signal

Gauge sensor uncertainty

• Signal derivatives

• Entropy• Histogram

differences• Covariances• Fourier• Wavelet• Polynomial fit• Turbulent

kinetic energy• Eddy

dissipation• and other…

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




Feature Selection

• Classification task difficulty increases with number of features• Feature selection automatically decides feature efficacy• Classifier-independent feature selection methods developed• Exploitable for dynamic feature space selection

Information-theoretic approach (mutual information maximization) ranks features according to how well they represent the data

Classifier-independent feature selection approachClassifier-independent feature selection approach

0 5 10 15 20 25

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2x 10-3

# Features Removed From Feature Set

Mut

ual-

Info

rmat

ion-

Base

d Fe

atur

e Se

t Qua

lity

(a.u

.)

Automatic feature ranking designates features that can be removed without significant feature-space quality loss

Feature space matched to present data characteristics

Feature space matched to past data characteristics

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




Background Clustering

• Background vs. release discrimination difficult due to background dynamics

• Find background subtypes by clustering, and recast problem as set of easier tasks

– Subtype vs. release

• Temporal clustering methods developed– Enhancements over Adaptive Resonance

Theory (ART) based clustering

Automatically discover plausible and non-obvious background typesAutomatically discover plausible and non-obvious background types

1

1a1c 1b

22a

2b06/21 06/22 06/23 06/24 06/25 06/26 06/27 06/28 06/29 06/30 07/01

Cluster ID

Time

1

1a1c 1b

1

1a1c 1b

22a

2b

22a

2b06/21 06/22 06/23 06/24 06/25 06/26 06/27 06/28 06/29 06/30 07/0106/21 06/22 06/23 06/24 06/25 06/26 06/27 06/28 06/29 06/30 07/01

Cluster ID

Time

Diurnal effectsBackground clustering result Non-obvious background types

Notional 2D view of multisensor data

Feature A

Feat

ure

B

Feature A

Feat

ure

B

Backgrounds Backgrounds and releases

Temporal progression

Per-subtype discrimination and fusion

Point to classify

Per-subtype Decision boundaries

f f

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PFA

P DET

PFA

P DET

Computational experiment ROCs

Computational Experiments with FLASH-1

• Preliminary results of relative gains with respective aspects of FLASH– Simple non-FLASH fusion scheme provided for reference

• Suggestive of relative benefit of hybrid approach elements, not any specific system performance

A: Simple multisensor fusion algorithm (learning of thresholds)

B: FLASH-1 feature selection, background clustering and ILR recognition

C: FLASH-1 as in (B), plus ILR fusion and TSR hybrid

D: FLASH-1 as in (C) plus fuzzy context modification method (threat data simulated)

Preliminary results of relative gains illustrate potential of FLASHPreliminary results of relative gains illustrate potential of FLASH

More details on A, B, C, and D, follow



Local-Scope Recognition

• Instance-Level Recognition (ILR) Hybrid currently based on multiple Support Vector Machines (SVM)

• ILR outputs are functions of distance from decision boundaries of multiple SVMs

• Computational experiments on datasets from deployment 1

• Comparison with “non-FLASH”simple fusion algorithm, based on estimating thresholds

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PFA

P DET

PFA

P DET


A: Simple multisensor fusion algorithm

Lower levels

Higher levels Context-scope

recognition hybrid

Local –scope recognition hybrid

Feature Selection

Event simulations

Feature extraction

Local-scope fusion


reasoningAction

planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering State-space

estimation

Performance results showing ILR hybrid outperforming simple fusion method

Performance results showing ILR hybrid outperforming simple fusion method




Local-scope Fusion and Context-scope Recognition

• ILR fusion based on Dempster-Shafer theory of evidence

• Each recognizer in ILR hybrid constitutes separate evidence source

• Time-Series Recognition (TSR) operates on sequences of ILR fusion stage outputs

• TSR based on Hidden Markov Models– Continuous Density HMMs (CD-HMMs)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PFA

P DET

PFA

P DET

Refuting evidence

Belief Plausibility

Supporting evidence

1

Supporting or unknown

0




Performance results showing relative gains of FLASH-1 ILR fusion and TSR recognition

Performance results showing relative gains of FLASH-1 ILR fusion and TSR recognition

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local-scope fusion


reasoningAction

planning

Planning and goal

satisfaction


Shared information

Multisource data


estimation




Fuzzy Logic Methods in FLASH-1

• Fuzzy logic methods convenient for representing descriptive/vague data

• FLASH-1 fuzzy outcome modification exploits certain data as modifiers rather than inputs to classification

• Example: Perceived threat assessment– Modulate TSR outcome by perceived

threat assessment– Context simulated by Gaussian

distributions correlated with truth-values– Mamdani-type fuzzy inference

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PFA

P DET

PFA

P DET




FuzzyReasoning

ContextModifier

Feature or ILR/TSR level belief

membership functions

Perceived threat context membership

functionsContext-modified belief membership functions

D: FLASH-1 as in (C) plus fuzzy context modification method (threat data simulated)

Performance results showing further improvements with simulated “context” data

Performance results showing further improvements with simulated “context” data

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local-scope fusion


reasoningAction

planning

Planning and goal

satisfaction


Shared information

Multisource data


estimation




Belief Network Reasoning

• Knowledge representation with FLASH semantic nets• Initial results with Bayesian net, indoor protection example

TSR-level alarmTrueFalse

76.223.8

AirflowEastWestLV

90.05.005.00

Risk level categoryVery lowLowHighVery high

0100

Close Fire DoorsNoneEastWestBoth

23.14.0568.84.05

Sensor network statusOKPerturbedInop

90.05.005.00

Known interferenceTrueFalse

1.0099.0

TSR release indicationTrueFalse

90.010.0

Set HVAC ModeNormalHigh

24.375.7

Issue Warning AlertYesNo

7.6292.4 Immediate evacuation

YesNo

7.8692.1

Conditions

Actions

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local-scope fusion


reasoningAction

planning

Planning and goal

satisfaction


Shared information

Multisource data


estimation



Course-of-Action Decision Support

• Course-of-action (COA) guidance – recognizing relevant battlespace conditions and reasoning to generate available options and tradeoffs

• Cognitive processing character makes FLASH approach promising for demanding COA guidance applications

– Machine learning, uncertainty provisions, approximate reasoning

• Further options include machine learning from scenarios, past cases, “lessons learned”, desktop exercises, after-action reviews, and information acquired from tacticians and other subject matter experts



Potential of FLASH for Battlespace Management Systems

• Potential as information-fusion and machine-intelligence algorithmic backbone for CBRN Battlespace Management systems such as JWARN, JEM, and JOEF

• Intelligent decision-support technology with multi-purpose applicability– Attack detection/characterization, prediction, impact assessment, course-of-action, others

• Augmented cognition for warfighters

Point, Standoff, Biological, Chemical, etc.

Intel,Weather,Commerce,Healthcare,Other MSB, etc.

Transport,Dispersion,Effectiveness,Epidemiology,Population,etc.

Disparate Sensor & Context Data: Uncertain, inconsistent, conflicting,

incomplete, ambiguous, vague

JWA

RN

, JEM, JO

EFFLASH Cognitive-Processing Machine-Intelligence Backbone

for Information Fusion, Recognition and Reasoning

Multiple sensors Multiple systems Multiple models

Results: Event detection and characterization, effect predictions, course-of-action guidance, tradeoffs, …

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local -scope fusion


reasoning

Action planning

Planning and goal

satisfaction


Shared information

Multisource data

Event clustering




Summary

• FLASH: novel cognitive-processing machine-intelligence decision-support architecture and methods

• Performance results to date show potential of FLASH

• Broad multi-purpose applicability, including CB detection, characterization, Course-of-Action guidance, and other CBR defense decision-support applications

Lower levels


recognition hybrid


Feature Selection

Event simulations

Feature extraction

Local-scope fusion


reasoningAction

planning

Planning and goal

satisfaction


Shared information

Multisource data


estimation

multi-purpose machine-intelligence-based information ... · • machine-intelligence character of...

Documents