Moises SuditOctober 28, 2013

Ontology and Human Intelligencesin Optimization and Fusion

Slide 2

Gadenfors Conceptual Spaces

Consider a situation where you are walking through the woods:

Associationist: Travel through one small part at a time, understand features (rocks, rivers, trees, etc.), learn as we go, clear path for next time we travel…

Conceptual: “overhead view” understanding of geometry of paths as features come together (N,S,E,W)…

Symbolic: Semantic street names and directions (left, right, etc.) are given to paths, thus we gain independence from the features…

Conceptual Spaces

• Overview• A conceptual space consists of a set of geometric domains and

their associated metrics and corresponding similarity measures• A concept is a collection of property regions within these domains,

the correlations (i.e., co-occurrences) between these properties, and their salience weights

• Each concept is additionally characterized by a set of forbidden domain-property pairs

• A query is a set of points, one in each domain, describing its attributes

D1 D2 DK

ConceptualSpace

Domains

Properties

1( , )s x y 2 ( , )s x y ( , )Ns x y

Introduction to Conceptual Spaces

Armor Level

# Wheels

Amphibious

Domains•# Wheels•Armor Level•Amphibious

Concepts•Tank (0, Heavy, No)•LAV (>6, Light, Yes)•Truck (4-6, Light, No)•Jeep (>6, Light, No)

Benefits•Allows similarities btw objects to be calculated•More flexible than First Order Logic•Transparent

>6

4-6

0Light Medium HeavyNo

Yes

Properties•# Wheels(0, 4-6, >6)•Armor Level (Light, Medium, Heavy)•Amphibious(Yes, No)

Observations (Wheels, Armor, Amphibious)

Introduction To Conceptual Spaces

Armor Level

# Wheels

Amphibious

(0, Heavy, No)(4, Light, No)(4, Light, Yes)

d1d2

Slide 6

Two Models for Conceptual Spaces

• Single Observation Mathematical Model• Only one observation is made on one object• This object is compared to each individual concept in the library

(world) to determine which it is most similar to

• Multiple Observation Mathematical Model• Multiple observations are made from either a single sensor or

multiple sensors• Observations may not necessarily be of the same object

• This handles “The Association Problem” in Data Fusion

• Each observation is compared to each concept to determine which it is most similar to

Slide 7

Single Observation Model

j j k

Set of Concepts

Set of Domains

Subset of Domains of concept k for k C

Set of properties of Domain j for j D

=Set of (i,j) and (i ,j ) that mutually exclusive,

where i P and i P for j,j D

k

j

k

C

D

D

P

I

, k C

similarity of property j within domain i

1,

0, . .

ij

ij

s

if property j from domain i is consideredx

o w

Concept encoded in set of constraintsObserved Object appears only in the objective function

1 1 2 2

1

1 1 2 2

max

. . 1

1 , , ,

i

ij iji j

n

ijj

k

i j i j

s x

s t x i

x x i j i j I

x B

We prove a lemma showing that any sized finite set of mutually exclusive properties can be broken into pairs.

Slide 8

Example: Decision Variables

11 12 13 14 15 16

21 22 23

31 32 33

41

Color: red(x ), white(x ),brown(x ), black(x ), yellow(x ),grey(x )

Shape: rectangular(x ),short & round(x ), tall & thin(x )

Sound: quick "boom"(x ),explosion(x ), humming(x )

RelativeSize: small(x 42 43

51 52

61

),medium(x ), big(x )

Motion: drives(x ), walks(x )

Smell:gaseous(x )

We set up a library of 4 concepts (Bomb, Auto, Human, Gas Tank). Each utilize some of the same domains/properties and some different ones.

We run them against 4 observed objects and see how our model works.

Slide 9

Example: Concepts

Bomb:Color: red, white, brown,

blackShape: rectangular, short &

roundSound: “boom”, explosionRelative Size: small11 12 13 14

21 22

31 32

41

11 22

13 22

14 21

22 31

11 32

13 32

1

1

1

1

1

1

1

1

1

1

x x x x

x x

x x

x

x x

x x

x x

x x

x x

x x

x B

Auto:Color: black, yellowShape: rectangular, short & roundSound: hummingRelative Size: smallMotion: drives

14 15

21 22

33

41

51

14 21

15 22

1

1

1

1

1

1

1

x x

x x

x

x

x

x x

x x

x B

Slide 10

Example: Concepts (cont.)

Human:Color: white, blackShape: short & round, tall &

thinRelative Size: largeMotion: walks

Gas Tank:Color: black, greyRelative Size: mediumSmell: gaseous

12 14

22 23

42 43

52

12 42

12 23

14 43

14 22

1

1

1

1

1

1

1

1

x x

x x

x x

x

x x

x x

x x

x x

x B

14 16

42

61

1

1

1

x x

x

x

x B

Slide 11

Multiple-Observation Model

j

Set of Concepts

Set of Domains

Subset of Domains of concept k for k C

Set of Observations

Set of properties of Domain j for j D

=Set of (i,j) and (i ,j ) that mutually exclusive,

where i P a

k

j

k

C

D

D

O

P

I

j knd i P for j,j D , k C

maximum number of distinct concepts that could be observed

pedigree of domain j by observation o for j D and o O

similarity of property i of domain j by observation

jo

jio

m

p

s

jo for i P ,

j D and o O

1 if property i of Domain j is associated with observation o

for i ,

0 Otherwise

1 if observation o is associated with concept k for

0 Otherwise

j jio

ok

x P j D and o O

k C and o Oy

1 if observation o is associated with concept k for

0 Otherwisek

k Cz

Slide 12

Multiple-Observation Model (cont.)

j

j j jo io io

o O j D i P

Max p s x

. : , ,

1 , with

jk

jo

jio ok

i P

j joio

i P

st x y j k o

x j o P

2 {( , ), ( , )} , ,j j kio i o okx x y i j i j I k o

1okk C

y o

ok ko O

kk C

y O z k

z m

, , 0 1 , , ,jio ok kx y z or i j k o

Constrains the number of properties selected in each domain.

Constrains cross-domain property disallowed pairings.

Allows only one concept to be selected for each observation.

Constrains the number of objects being observed by the sensory system.

Maximizes property similarities based on sensor reports.

Slide 13

What do we have?

• A hybrid Conceptual Space/Integer Programming model that can:• Consider multiple observations by multiple sensors• Account for the pedigree of each sensor in

accordance to its ability to sense each specific property/domain

• The ability to change the number of allowed objects being observed (m)

• All of these capabilities are captured within a single, mathematical model using proven optimization techniques

• How well does it work?• Emotion Recognition (compared against Support

Vector Machine)• Automatic ICON Identification for CPOF

Emotion Recognition

Slide 15

Emotion Recognition through Conceptual Spaces

True EmotionsTrue Emotions

False EmotionsFalse Emotions

FearFear

SadnessSadness

AngerAnger

EnjoymentEnjoyment

We are taking the BB3 Data and classifying pictures into one of 8 concepts:

4 true emotions and 4 false emotions (attempted deceit)

Slide 16

Emotion Recognition

• Process• Images are obtained and analyzed automatically in terms of

facial features• Facial features are considered in classification of images into

emotions, both true emotions and falsified emotions

• Parts of the ProcessComponent Description Workload

Major Component

s (MC)

Measurements and distances between parts of the face – used to determine which Action Units exist.

CUBS

Action Units (AU)

50+ defined – technique for measurement of facial movement (Facial Action Coding System).

Ekman & Friesen

Emotions Existence and combination of AU’s help define a person’s true emotion and may be able to depict deceit as well.

Our Work

• Major Components Action Units Emotions

Emotion Recognition

wrinkles

Crows feetLips Compressed…

…

…

…

…

AUi

AUj

AUk

AUl

Anger

Enjoyment

Fear

Sadness

Measurable features calculated based on distances between certain points. Determined by automated systems.

Several Major Components combine to form Action Units

The presence of Several Action Units at the same time define emotions.

Slide 18

Conceptual Spaces – Classification Model

1 2

1

2

{ , , , ,

, ., , }

{ 1, 2, 4, 6, 12, 15, 23}

{ , },ji i

i

i

C Anger Enjoyment Fear Sadness

False Anger False Enjoy False Fear False Sadness

D AU AU AU AU AU AU AU

P x x i D

where x AU does not occur in Concept k

where x AU occurs in Concept k

AU1 AU2 AU4 AU6 AU12 AU15 AU23

Concept X11 X12 X21 X22 X31 X32 X41 X42 X51 X52 X61 X62 X71 X72

Anger .5 .5 .5 .5 1 0 1 0 1 0 1 0 0 1

Enjoy. .8 .2 .5 .5 1 0 0 1 0 1 1 0 1 0

Fear 0 1 0 1 1 0 1 0 1 0 1 0 1 0

Sad .5 .5 0 1 .5 .5 1 0 1 0 0 1 1 0

False Anger .5 .5 1 0 .7 .3 1 0 1 0 1 0 1 0

FalseEnjoy.

1 0 1 0 1 0 1 0 .75 .25 1 0 1 0

False Fear .5 .5 1 0 1 0 .75 .25 1 0 1 0 1 0

False Sad 1 0 1 0 1 0 1 0 1 0 .5 .5 0 1

Table below shows the existence of properties in concepts. There are properties that cannot exist together – the constraints handle these.

Slide 19

Observations & Model Results

• Observations• Taken from the BB3 Dataset (CUBS) – 344 images analyzed• Since many images produced the same MC values, we

consolidate into 49 observations• MC’s either occur or they do not {0, 1}• Each AU contains anywhere from 1 to 4 MC’s that suggest

the AU is occurring.

# MC's fitting property j in domain i for observation o

Total # MC's in domain ijis

• Model Results

• 49 observations out of which 7 are conflicting so we deleted them• 42 observations against the 8 concept definitions in IP through CPLEX• Objective Value = 291.50• Solution Time = 0.13 seconds• Of the 42 observations, all 42 were classified correctly!

Multi-Class SVM – Classification Model

Using “SVM Light” (Joachims, T. SVM-Light Multi-Class, 2007. Cornell U.)

42 Obs.

27 Obs. 15 Obs.

Training Set Experiment Set

Slack/Kernel

Linear 2nd Poly. 3rd Poly. Radial Basis

Sigmoid

c = 1 7 7 7 8 8

c = 10 7 7 (1.16s)

9 (2.64s)

8 7

c = 100 7 10 (4.47s)

11 (10.98s)

7 7

c = 1000 7 12 (16.78s)

12 (40.73s)

N/A 7 (2.55s)

c = 5000 12 12 (39.73s)

12 (155.94s)

N/A 7 (5.44s)

c = 10000 13 12 (39.86s)

12 (156.00s)

N/A 7 (8.08s)

Output in Table:• Value = # correct (of 15)• Time = training time (if > 1.0 sec)

Conceptual Spaces – Classification Model

1 2

1

2

{ , , , ,

, ., , }

{ 1, 2, 4, 6, 12, 15, 23}

{ , },ji i

i

i

C Anger Enjoyment Fear Sadness

False Anger False Enjoy False Fear False Sadness

D AU AU AU AU AU AU AU

P x x i D

where x AU does not occur in Concept k

where x AU occurs in Concept k

AU1 AU2 AU4 AU6 AU12 AU15 AU23

Concept X11 X12 X21 X22 X31 X32 X41 X42 X51 X52 X61 X62 X71 X72

Anger .8 .2 .4 .6 .7 .3 1 0 1 0 1 0 0 1

Enjoy. .5 .5 1 0 1 0 .25 .75 0 1 1 0 1 0

Fear .25 .75 0 1 .5 .5 .75 .25 1 0 1 0 1 0

Sad .55 .45 .3 .7 .675 .325 .85 .15 1 0 0 1 1 0

False Anger 1 0 1 0 0 1 .875 .125 .5 .5 1 0 1 0

FalseEnjoy.

1 0 1 0 1 0 .625 .375 0 1 1 0 1 0

False Fear .5 .5 1 0 .625 .375 .75 .25 1 0 1 0 1 0

False Sad 1 0 1 0 1 0 1 0 1 0 .5 .5 0 1

Used the averages of the xi1 and xi2 values to “train” the concepts below.

SVM’s v. Conceptual Spaces

• These should be used under different conditions.• SVM’s – No a priori knowledge, but trainable data is

available• Conceptual Spaces – A priori knowledge available, no need

to train Support Vector Machines

Conceptual Spaces

Parameter Choices

Kernel selectionParameters within each kernelSlack allowance (c-value)

N/A

Model Accuracy (in this example)

13/15 = 86.67% 14/15 = 93.33%

Model Speed (in this example)

0.16 seconds 0.08 seconds

Multi-Class SVM – Classification Model

Using “SVM Light” (Joachims, T. SVM-Light Multi-Class, 2007. Cornell U.)

42 Obs.

42 Obs. 42 Obs.

Training Set Experiment Set

Slack/Kernel

Linear 2nd Poly. 3rd Poly. Radial Basis

Sigmoid

c = 1 30 31 30 31 20

c = 10 31 31 (1.44s)

32 (3.62s)

36 28

c = 100 31 36 (9.75s)

37 (16.84s)

38 26

c = 1000 37 38 (48.75s)

41 (50.49s)

N/A 25 (4.95s)

c = 5000 37 41 (48.27s)

42 (121.05s)

N/A 25 (9.06s)

c = 10000 39 41 (90.83s)

42 (263.89s)

N/A 21 (13.34s)

Output in Table:• Value = # correct (of 42)• Time = training time (if > 1.0 sec)

SVM’s v. Conceptual Spaces

• These should be used under different conditions.• SVM’s – No a priori knowledge, but trainable data is

available• Conceptual Spaces – A priori knowledge available, no need

to train Support Vector Machines

Conceptual Spaces

Parameter Choices

Kernel selectionParameters within each kernelSlack allowance (c-value)

N/A

Model Accuracy (in this example)

42/42 = 100% 42/42 = 100%

Model Speed (in this example)

121.05 seconds (training) 0.11 seconds

CS v. SVM Testing (Observation Dimensionality)

For SVM’s, use 3rd deg. Polynomial and c = 1000

No. Obs. SVM training-time SVM run-time CS run-time

42 2.22 0.08 0.03

420 2,504.15 4.55 0.31

4,200 285,821.11 291.80 6.64

42,000 32,623,621.50 21,048.08 566.39

Run Time Comparison

0

5000

10000

15000

20000

25000

0 10000 20000 30000 40000 50000

Number of Observations

Tim

e (sec.)

SVM run-time

CS run-time

CS v. SVM Testing (Concept Dimensionality)

For SVM’s, use 3rd deg. Polynomial and c = 1000

No. Conc. SVM training-time SVM run-time CS run-time

8 2,504.15 4.55 0.31

24 14,568.64 16.70 2.58

80 89,759.09 138.66 59.99

120 167,631.52 255.67 174.33

Run Time Comparison

0

50

100

150

200

250

300

0 20 40 60 80 100 120 140

Number of Concepts

tim

e (s

ec.)

SVM run-time

CS run-time

CPOF ICON Example: Project Overview(Command Post of the Future)

Speech Recognition

Software

[A B C D E] 40% B2 D2

D3

Filter

Domain

Library

[A B2 C D3 E]

AeroText/ Java Class Creation

A B2 C D3 E

Field Soldier TOC Operator/Field Soldier

INFERD

Event Report

Conceptual Spaces Algorithm

INCIDENT

ICON

Known Event

Unknown Event

Key = Input

= Output

Text

CPOF Event Icons

1. Bomb2. Drive-by Shooting3. Explosion4. Grenade Attack5. IED (Improvised Explosive Device)

6. Mortar Attack7. Murder

8. Point of Impact9. RPG (Rocket Propelled Grenade)

10. Sniping11. VBIED (Vehicle-Borne IED)

12. PBIED (Person-Borne IED)

Process Flow“Shark 6, this is Oscar Two Delta. Contact Left. Over“Oscar Two Delta, this is Shark 6, over"Location - Mike Rome 05742371, over“Roger, Over One WIA from pistol shot, estimate enemy force of 5, in pursuit, overHeading south from CP1 on route 7 at high speed”Roger, 1 WIA. OverRequest QRF to location 38 SMB xxxxxyyyyy.RogerRequest immediate medevac at Checkpoint 2.Roger, deploying medical personnel. Over.

Mission background in

situ DB

In situ database•Communications ElectronicsOperations Instructions (CEOI)•Patrol Orders•Intelligence Preparation of the Battlefield •Call signs/code names•Channels•Location•Organizational constructs

Shark 6 = Fallujah TOCOscar 2 D = CINC ACF A (Lt. Wayne Demerol’s unit, 5 men)Mike Romeo 05742371 = Grid 38SMB428489021538 SMB xxxxxyyyyy = lat/long surface marker buoyCP1 = Grid abcdefg, temporary checkpoint buildingWIA = wounded in ActionQRF = quick reaction forceRemove extraneous (Over, roger, swear words)

ICON DB

PastSpot/SIGACT

Reports

• Created SPOT report

people = Wayne Demarolpeople = enemy number = 5place = 38SMB4284890215place = 38.889556, – 77.0352546organization = cinc acf aorganization = fallujah tocevent = injury number = 1event = pursuitevent = shootingrecord = firearm item = pistolevent = deploy medical personnelContext: time = 1349; date = 05092005

Infuse implicit information

Extract entities, events and relationshipsand Context

1.date = 0913492.event = direct fire3.icon =4.Attributes• affiliation = hostile• target = US• weapon class =

light• WIA=15. confidence = .856. Translated text

Fuzzy Matchingof attributes and events;Confidencelevel;Icon creation

Representative speech-to-text output, including confidence score

Fuzzy Context Search

Finishing the Example (cont.)

Snipin

g

Most Likely Icon at End of Time Four

Bomb

Snipin

g

VBIED

Conceptual

SpacesAlgorith

m

EVENT REPORT:Weapon: GunPersonnel: GroupEvent: Ambush

EVENT REPORT:Weapon: BombPersonnel: NoneEvent: Explosion

EVENT REPORT:Weapon: GunPersonnel: GroupEvent: Skirmish

EVENT REPORT:Weapon: Personnel: VehicleEvent: