Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution

Malik Magdon-IsmailRensselaer Polytechnic

Institute

2

Our Group

M. Goldberg M-I B. Szymanski A. Wallace

Students: Mykola Hayvanovich Apirak Hoonlor Stephen Kelley Konstantin Mertsalov

3

Motivation

Communications supporting IED planning have patterns and are correlated….

Analysis of the patterns can reveal the groups as well as their internal group structure.

4

CommunicationsTime: January 12, 2005, 09:35

From: joe@xyz.com

To: sue@abc.com

Subject: Hello

Message: Where have you been?

16:06:31] <FreeTrade> Republicans were the worst pacifists before ww1 and ww2[16:06:43] <SweetLeaf> France Fries[16:06:50] <FreeTrade> As a generality, of course their were Republican Hawks.[16:07:13] <FreeTrade> Sweet, good pun but bad story![16:07:18] <SweetLeaf> yup[16:07:23] <Lupine> anyways, he's perpetually tormented by presidential actions[16:07:25] <SweetLeaf> it aint good for no one[16:07:47] <SweetLeaf> I think they knew it was commiing[16:07:51] <FreeTrade> Rossevelt met monthly in New York with mostly trusted Republicans to talk about how to get america into the war.[16:08:10] <FreeTrade> and he spent 2 year with Churchill meeting him sometimes secretly in the ocean to discuss the same topic.[16:08:22] <FreeTrade> Exchanging a lot of letters.[16:08:25] <FreeTrade> telegrams[16:08:28] <Lupine> There really is nothing like a shorn scrotum. It's breathtaking, I suggest you try it.[16:08:55] <FreeTrade> Well they didnt literally meet in the ocean, they were on ships.

5

Streaming ExampleTime From To Message10:00 Alice Charlie Golf tomorrow? Tell everyone.10:05 Charlie Felix Alice mentioned golf tomorrow.10:06 Alice Bob Hey, golf tomorrow. Spread the word.10:12 Alice Bob Tee off: 8am at Pinehurst.10:13 Felix Grace Hey guys, golf tomorrow.10:13 Felix Harry Hey guys, golf tomorrow.10:15 Alice Charlie Pinehurst Tee time: 8am.10:20 Bob Elizabeth We’re playing golf tomorrow.10:20 Bob Dave We’re playing golf tomorrow.10:22 Charlie Felix Tee time 8am at Pinehurst10:25 Bob Elizabeth We tee off 8am at Pinehurst.10:25 Bob Dave We tee off 8am at Pinehurst.10:31 Felix Grace Tee time 8am, Pinehurst.10:31 Felix Harry Tee time 8am, Pinehurst.

6

Streaming ExampleTime From To Message10:00 Alice Charlie Golf tomorrow? Tell everyone.10:05 Charlie Felix Alice mentioned golf tomorrow.10:06 Alice Bob Hey, golf tomorrow. Spread the word.10:12 Alice Bob Tee off: 8am at Pinehurst.10:13 Felix Grace Hey guys, golf tomorrow.10:13 Felix Harry Hey guys, golf tomorrow.10:15 Alice Charlie Pinehurst Tee time: 8am.10:20 Bob Elizabeth We’re playing golf tomorrow.10:20 Bob Dave We’re playing golf tomorrow.10:22 Charlie Felix Tee time 8am at Pinehurst10:25 Bob Elizabeth We tee off 8am at Pinehurst.10:25 Bob Dave We tee off 8am at Pinehurst.10:31 Felix Grace Tee time 8am, Pinehurst.10:31 Felix Harry Tee time 8am, Pinehurst.

7

Streaming ExampleTime From To10:00 Alice Charlie10:05 Charlie Felix10:06 Alice Bob 10:12 Alice Bob 10:13 Felix Grace10:13 Felix Harry10:15 Alice Charlie 10:20 Bob Elizabeth10:20 Bob Dave10:22 Charlie Felix10:25 Bob Elizabeth 10:25 Bob Dave10:31 Felix Grace10:31 Felix Harry

8

Overview: SIGHTS & RDM

b u y , t r a d e . . . b u y

2 t r a d e . . . 2 t r a d e

3 , s e l l . . . 3 , h e l l

Pattern id = 2Pattern = “buy,”

Pattern id = 3Pattern = “2trade”bb

Level 0

Level 1

Level 2

Higher ranked leaders

Group leader

Subgroup leaders

Members

9

Communications

Email, Telephone, Newsgroup, Weblog, Chatrooms, …

Time: January 12, 2005, 09:35

From: joe@xyz.com

To: sue@abc.com

Subject: Hello

Message:

Where have you been lately?

Time: January 12, 2005, 09:35

From: joe@xyz.com

To: sue@abc.com

Subject: Hello

Message:

Where have you been lately?

10

Communication Graph

January 12, 2005, 09:35

sue@abc.comjoe@xyz.com

11

Communication Graph

What are the social groups/coalitions?

12

Social Groups are Clusters

Clusters may overlap.

13

Social Groups are Clusters

Clusters may overlap.A cluster is a locally defined object.

14

Social Groups are Clusters

Clusters may overlap.A cluster is a locally defined object.

Group members are more introverted than extroverted.

YES NO

15

Social Groups are Clusters

Clusters may overlap.A cluster is a locally defined object.

Group members are more introverted than extroverted.

Social groups (clusters) persist

16

SIGHTSStatistical Identification of Groups Hidden in Time and Space

- System for statistical analysis of social coalitions in communication networks

Data SourcesBlogsEmails (Enron)ChatroomSynthetic data

Coalition DiscoveryOverlapping ClusteringStreaming groupsPersistent groups.

Coalition AnalysisLeadersOpposing groupsTopic matching

VisualizationsSize-Density plotsStatic coalitionsDynamic coalitions

Groups matching analyst topic in red

Size vs. Density Plot

Visualization options

Choose time window

Groupmembers

Different analyses on dataset

Leader index

17

Examples

Two clusters: Electric circuit design; Optimization of Neural Networks:

Intersection: “Sensitivity analysis in degenerate quadratic programming”

Citeseer

ENRON

GROUND TRUTH Group A

Dog Vulture Camel Yassir Hussein Bird (6 others)

Group B Ahmet Saleh Sarwuk Shaid Pavlammed Pavlah Osan Domenik

SIGHTS Group A

Dog Vulture Camel Gopher

Group B Ahmet Saleh Sarwuk Shaid Dajik

Ali Baba Data Set (DoD)

18

Build a classifier to identify the relationship between sender and receiver of a message

EXAMPLE:“Do you have time to meet some time this week?”

“Lets meet 2pm today, ok?”

Which is advisor, which is student?

Recursive Data Mining (RDM)

19

Hierarchical Pattern Construction (recursive definition)Captures patterns; patterns of patterns; patterns of patterns of

patterns… (can even capture long-range patterns)

Pattern Definition

Larger patterns

b u y , t r a d e . . . b u y

2 t r a d e . . . 2 t r a d e

3 , s e l l . . . 3 , h e l l

Pattern id = 2Pattern = “buy,”

Pattern id = 3Pattern = “2trade”

Pattern id = 4Pattern = “3,_ell”

Level 0

Level 1

Level 2

20

Ensemble of classifiersClassifier for each level in the hierarchical approachFeatures gathered from the training messagesGlobal features include average length and number of sentencesApproximate matching allows treatment of noise

A Classifier – Joining the Pieces

21

Binary classification: for a given message m,

is m sent by a person with role r?

r є {CEO, Manager, Trader, Vice-President}

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

CEO Manager Trader Vice-pres.Roles

1 -

F_

sco

re

NB RDM_NB SVM RDM_SVM CPAR

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

CEO Manager Trader Vice-pres.

Roles

1 -

F_

Sco

re

NB RDM_NB SVM RDM_SVM CPAR

Multi-classification: for a given message m,

which role r is the most likely for the sender?

r є {CEO, Manager, Trader, Vice-President}

The bars show the error of classification. Universally RDM_SVM outperforms other classifiers

Results on Enron

22

Summing Up

SIGHTS:Structural; non-semantic; language

independentFinds groups, their dynamics and

structure; visual analytic capabilities.RDM

Uses statistical semantics; language independent

Identifies roles within the group

23

Thank You

http://www.cs.rpi.edu/~magdon