designing intelligent social systems 121205

12/5/12 1

•  Social systems rely on primi0ve technology. •  Big Data has opened Big Opportuni0es. •  Situa0on recogni0on is a key technology. •  EventShop may be useful in designing Intelligent Social Systems.

•  Comments. •  Sugges1ons. •  Collabora1on opportuni1es. •  [email protected] •  Gmail, FB, TwiBer: jain49

Send:

Intelligent: displaying or characterized by quickness of understanding, sound thought, or good judgment. Social Systems: Social systems are the paBerns of behavior of a group of people possessing similar characteris1cs due to their existence in same society.

•  Introduc1on •  Social Systems •  Intelligent Social Systems •  Designing Intelligent Social Systems •  Situa1on Recogni1on •  Concept recogni1ons •  Personalized Situa1ons •  EventShop

An Interes0ng Situa0on

When we were data poor – we searched for words in documents.

Now that we are data rich – should we s0ll search for words?

Time has come for us to stop thinking data poor; really start thinking and behaving data rich.

7

Volume

Varie

ty

Big Data offers Big Opportuni4es. But, …. ?????

Middle 4 Billion

Top 1.5 Billion

BoOom 2 Billion

Middle of the Pyramid (MOP):

Ready.

Most aOen0on by Technologists – so far.

Not Ready

9

Data is Essen0al. But, we are really interested in its products:

Informa0on, Knowledge, and Wisdom.

12/5/12 10

Knowledge Observe

Recognize

Act Big Data

Planning Control

Objects Situa0ons

12/5/12 11

Past is EXPERIENCE Present is EXPERIMENT Future is EXPECTATION

Use your Experiences In your Experiments

To achieve your Expectations

12/5/12 12

Astrology

To

Astronomical Volumes of Data

•  People •  Things •  Events

We are immersed in Networks of

It is now possible to be Pansophical. 12/5/12 13

12/5/12 Proprietary and Confiden1al, Not For Distribu1on 14

Our mobile wireless infrastructure can be “reality mined” to understand the paOerns of human behavior, monitor our environments, and plan social development. -‐-‐-‐-‐ Pentland in “Society’s Nervous System: Building Effec0ve Government, Energy, and Public Health Systems”

•  Objects -‐-‐ popular in the West. •  Rela0onships and Events – popular in the East. •  Objects and Events – seems to be the new trend.

•  The Web has re-‐emphasized the importance of every object and event being connected to others -‐-‐ East Meets West.

Geography of Thought by Richard NisbeB

•  Data •  Objects •  Rela0onships and Events

•  Take place in the real world. •  Captured using different sensory mechanism.

– Each sensor captures only a limited aspect of the event.

•  Can be used to bridge the seman1c gap.

Events: Types and Granulari1es •  Conferences

–  Days •  Sessions

–  Talks »  Purpose of the talk

•  Wedding •  An Earthquake •  The Big Bang •  World Wide Web •  Yahoo: Winter School 2012 •  Me

– My Birth, –  Being here, and –  Dying in 100 years.

People Things Places Time Experiences Events

E by Westerman and Jain

E* by Gupta and Jain

Sense making from mul1modal massive geo-‐social data-‐streams.

20

•  Introduc1on

• Social Systems •  Intelligent Social Systems •  Designing Intelligent Systems •  Situa1on Recogni1on •  Concept recogni1ons •  Personalized Situa1ons •  EventShop

Poli0cs Religion

Economics Health

Educa0on

Connec4ng People to Resources effec4vely, efficiently, and promptly

in given situa4ons.

•  Minimize hunger in the world. •  Maximize female educa1on in India. •  Minimize ‘deaths’ in the coming hurricane in Florida.

•  Minimize work-‐hours lost in traffic during week days in Bangalore.

•  System: –  A set of diverse parts forming a whole. – Parts are put together with a common objec1ve/purpose.

•  Each part could be considered a system. •  Each part plays a role towards the system objec1ve.

•  Designing the informa1on flow among parts is essen1al to make a system work apprpriately.

•  A social system is composed of persons or groups who share a common objec1ve.

•  An individual objec1ve is usually a part of the group’s objec1ve.

•  Persons •  Families •  Organiza1ons •  Communi1es: City, State, Country •  Socie1es •  Cultures

•  Top Down: – The social system determines its parts. – People’s behavior determined by society.

•  BoBom Up: – The Society is the sum of its indivduals –  Individual ac1ons determine the character of the society.

•  Each social en1ty is a holon. •  Holon: Each en1ty is simultaneously a part and a whole.

•  A social component is made up of parts and at the same 1me maybe part of some larger whole.

•  Any system is by defini1on both part and whole.

•  The primary ‘currency’ of a social system is informa1on.

•  System behavior can be understood as the movement of informa1on: – Within a system –  Between the system and its environment

•  Informa1on is used to sense as well as to control or act.

•  Introduc1on •  Social Systems

•  Intelligent Social Systems •  Designing Intelligent Systems •  Situa1on Recogni1on •  Concept recogni1ons •  Personalized Situa1ons •  EventShop

•  Systems that perceive, reason, learn, and act intelligently.

•  Adaptability to varying environmental situa1ons is a key element of intelligent systems

•  Social systems that perceive, reason, learn, and act intelligently.

•  What does ‘perceive’, ‘reason’, ‘learn’, and ‘act’ mean in the context of social systems?

•  Introduc1on •  Social Systems •  Intelligent Social Systems

• Designing Intelligent Social Systems

•  Situa1on Recogni1on •  Concept recogni1ons •  Personalized Situa1ons •  EventShop

• Desired state (Goal) •  System model and Control Signal (Ac0ons)

•  Current State (for Feedback)

Observed State

Real World Events

Observa0o

ns

Feedback

Control Signals

Social Networks

Connecting

People

Needs and Resources

Not even FaceBook!

Current Social Networks

Important Unsa1sfied Needs

12/5/12 46

•  Resources – Physical: food, water, goods, … –  Informa:onal: Wikipedia, Doctors, … – Transporta:on – Employment – Spiritual

•  Timeliness •  Efficiency

Connecting People

And Resources

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12/5/12 48

•  Introduc1on •  Social Systems •  Intelligent Social Systems •  Designing Intelligent Systems

• Situa0on Recogni0on •  Concept recogni1ons •  Personalized Situa1ons •  EventShop

Connec4ng People to Resources effec4vely, efficiently, and promptly

in given situa4ons.

•  rela1ve posi1on or combina1on of circumstances at a certain moment.

•  The combina1on of circumstances at a given moment; a state of affairs.

•  Situa1on awareness, or SA, is the percep1on of environmental elements within a volume of 1me and space, the comprehension of their meaning, and the projec1on of their status in the near future.

•  What is happening around you to understand how informa1on, events, and your own ac1ons will impact your goals and objec1ves, both now and in the near future.

•  Example 1: – A person shou1ng. – 1000 people shou1ng.

•  In a contained building •  In main parts of a city

•  Example 2: – One person complaining about flu. – Many people from different areas of a country complaining about flu.

Facebook and TwiBer (now GOOGLE +)

Massive collec1on of events. Have been repor0ng events as micro-‐blogs

Does the flap of a buEerfly’s wings in Brazil set off a tornado in Texas?

12/5/12 57

Have been repor0ng events as micro-‐blogs

Sensors and Internet of Things are crea1ng and repor1ng even more events than humans are.

FROM TWEETS TO REVOLUTIONS

Time

Atomic and Composite Events

•  Given a plethora of event data. How can we: – Disambiguate relevant and irrelevant events? – Combine events into meaningful representa1ons ? – Allow inference and cascading effects? – Support different interpreta1ons based on applica1on domain?

– Support Control & decision making?

1.  Inherent support for event-‐based (temporal) reasoning

2.  The ability of the controller to reason based on symbols (rather than just signals)

3.  Explicit inclusion of domain seman1cs (to support mul1ple applica1ons)

An ac4onable abstrac4on of observed spa4o-‐temporal characteris0cs.

62

•  Introduc1on •  Social Systems •  Intelligent Social Systems •  Designing Intelligent Systems •  Situa1on Recogni1on

• Concept recogni0ons •  Personalized Situa1ons •  EventShop

Time Line

Data Type

1950 2000

Time Line

Data Type

Character 1959

Objects 1963

Events 1986

Speech 1962

Situa0on 2010

1950 2000

66

Environments

Real world Objects

Situa1ons

Ac1vi1es

Single Media

SPACE TIME

Scenes Loca1on aware

Visual Objects

Trajectories

Visual Events

Loca1on unaware

Sta1c Dynamic

Loca1on aware

Loca1on unaware

Sta1c Dynamic

Data = Text or Images or Video

•  1963: Object Recogni1on [Lawrence + Roberts] •  1967: Scene Analysis [Guzman] •  1984: Trajectory detec1on [Ed Chang+ Kurz] •  1986: Event Recogni1on [Haynes + Jain] •  1988: Situa1on Recogni1on [Dickmanns]

1960 1970 1980 1990 2000 2010

Object Scene Trajectory

Event

Situa1on

68

Environments

Real world Objects

Situa1ons

Ac1vi1es

SPACE TIME

Loca1on aware

Loca1on unaware

Sta1c Dynamic

Heterogeneous Media

Loca1on aware

Loca1on unaware

Sta1c Dynamic

Data is just Data. Meta-‐data is also data. Caste system does not exist here. Medium and sources do not maOer.

•  Introduc1on •  Social Systems •  Real Time Social Systems •  Designing Real Time Systems •  Situa1on Recogni1on •  Concept recogni1ons

• Personalized Situa0ons •  EventShop

A) Situa0on Modeling B) Situa0on Recogni0on C) Visualiza0on, Personaliza0on, and Alerts

…

STT Stream

Emage

Situa1on

C1 ⊕

v2 v3 ⊕

v5 v6

@

∏

Δ @

i) Visualiza1on

ii) Personaliza1on

+

+ Available resources

iii) Alerts

Personal context

Personalized

situa1on

70

73

STT data

Tweet: ‘Urrgh… sinus’

Loc: NYC, Date: 3rd Jun, 2011 Theme: Allergy

Situa1on Detec1on

User-‐Feedback

‘Please visit Dr. Cureit at 4th St immediately’

Date: 3rd Jun, 2011

Aggrega1on,

1) Classifica1on 2) Control ac1on

Opera1ons

Alert level = High

•  Introduc1on •  Social Systems •  Intelligent Social Systems •  Designing Intelligent Systems •  Situa1on Recogni1on •  Concept recogni1ons •  Personalized Situa1ons

• EventShop

•  E-‐mage

–  Visualiza1on –  Intui1ve query and mental model –  Common spa1o temporal data representa1on – Data analysis using media processing operators (e.g. segmenta1on, background subtrac1on, convolu1on)

76

•  Spa1o-‐temporal element –  STTPoint = {s-‐t-‐coord, theme, value, pointer}

•  E-‐mage –  g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V = N)

•  Temporal E-‐mage Stream –  TES=((ti, gi), ..., (tk, gk))

•  Temporal Pixel Stream –  TPS = ((ti, pi), ..., (tk, pk))

77

12/5/12 78 Proprietary and Confiden1al, Not For Distribu1on


Retail Store Loca0ons

Net Catchment area

•  Humans as sensors •  Space + Time as fundamental axes •  Real 0me situa0on evalua0on (E-‐mage Streams)

(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)

d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)

81

•  Help domain experts externalize their internal models of situa1ons of interest e.g. epidemic.

•  Building blocks: –  Operators –  Operands

•  Wizard: –  A prescrip1ve approach for modeling situa1ons using the operators and operands

82 Singh, Gao, Jain: Situa:on recogni:on: An evolving problem for heterogeneous dynamic big

mul:media data, ACM Mul0media ‘12.

Growth rate (Flu reports) Feature

Thresholds (0, 50)

Data source

Meta-‐data

-‐Emage (#Reports) Representa1on level

TwiBer-‐Flu

83

Knowledge or data driven building blocks

Get_components (v){ 1)  Identify output state space 2)  Identify S-T bounds 3)  Define component

features: v=f(v1, …, vk)

•  If (type = imprecise) –  identify learning data source, method

4)  ForEach (feature vi) { If (atomic)

•  Identify Data source.

•  Type, URL, ST bounds •  Identify highest Rep. level reqd. •  Identify operations

Else Get_components(vi)

} }

84

v f1

v4

⊕

v2 v3 @

D1

Emage

Δ

D2

∏

Emage

Δ

D3

Δ

@

Emage D2

∏

Emage

Δ

f2 ⊕

v5 v6

<USA, 5 mins, 0.01x 0.01>

ϵ { Low, Mid, High}

Epidemic Outbreaks

Unusual Ac1vity? Growth Rate

⊕

Current ac1vity level

Historical ac1vity level

⊕

Emage (#reports ILI)

Δ

TwiBer-‐Flu

⊕

TwiBer.com <USA, 5 mins, 0.01x 0.01>

Emage (Historical avg)

Δ

TwiBer-‐Avg

DB, <USA, 5 mins, 0.01x 0.01>

Δ

TwiBer-‐Flu



ϵ {Low, mid, high}, <USA, 5 mins, 0.01x

0.01>

Growing Unusual ac1vity

γ 1) Model


Δ

TwiBer-‐Flu

Emage (popula1on)

Δ

CSV-‐ Popula1on

⊕

π


Census.gov, <USA, 5 mins, 0.01x 0.01>

2) Revise

Subtract

Subtract

Mul1ply

Classifica1on: Thresh (30,70)

Normalize [0,100]

3) Instan1ate

85

Level 1: Unified representa1on (STT Data)

Level 3: Symbolic rep. (Situa1ons)

Proper1es

Proper1es

Proper1es

Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, …

Level 2: Aggrega1on (Emage)

…

STT Stream

Emage

Situa1on

86

Opera1ons

87

⊕

PaBern Matching

Aggregate

ψ

@ Characteriza1on

∏ Filter

γ Classifica1on

72%

+

+

Growth Rate = 125%

Data Suppor1ng parameter(s) Output Operator Type

+

Classifica1on method

Property required

PaBern

Mask

Δ Transform … Spa1o-‐temporal

window

88

⊕ Aggregate +

γ Classifica1on Classifica1on method

@ Characteriza1on Growth Rate = 125%

Property required

PaBern Matching ψ 72%

+PaBern

∏ Filter +Mask

Φ Learn Learning method

{Features}

{Situa1on}

f f

1) Data into right representa1on

2) Analyze data to derive features

3) Use features to evaluate situa1ons

Suppor1ng parameter(s)

Data Output Operator Type

Singh, Gao, Jain: Social Pixels: Genesis and Evaluation, ACM Multimedia ‘10. 89

S. No Operator Input Output

1 Filter ∏ Temporal E-‐mage Stream Temporal E-‐mage Stream

2 Aggrega0on ⊕ K*Temporal E-‐mage Stream Temporal E-‐mage Stream

3 Classifica0on γ Temporal E-‐mage Stream Temporal E-‐mage Stream

4 Characteriza0on : @ •  Spa0al •  Temporal

•  Temporal E-‐mage Stream •  Temporal Pixel Stream

•  Temporal Pixel Stream •  Temporal Pixel Stream

5 PaOern Matching ψ •  Spa0al •  Temporal

•  Temporal E-‐mage Stream •  Temporal Pixel Stream

•  Temporal Pixel Stream •  Temporal Pixel Stream

•  Select E-‐mages of US for theme ‘Obama’. –  ∏spa1al(region=[24,-‐125],[24,-‐65]) (TEStheme=Obama)

•  Iden1fy 3 clusters for each E-‐mage above. –  γkmeans(3) (∏spa1al(region=[24,-‐125],[24,-‐65])(TEStheme=Obama))

•  Show me the speed for each cluster of ‘Katrina’ e-‐mages

–  @speed(@epicenter(γkmeans(n=3) (∏spa1al(region=[24,-‐125],[24,-‐65]) (TEStheme=Katrina)))) •  How similar is paBern above to ‘exponen1al increase’?

–  ψexp-‐increase(@speed(@epicenter(γkmeans(n=3) (∏spa1al(region=[24,-‐125],[24,-‐65])

(TEStheme=Katrina))))

90

1) Macro situa0on

Macro data-‐sources Personal

Context

Profile + Preferences

2) Personalized situa0on

User data

91

IF person ui <is-‐in> (PSj) THEN <connect-‐to> rk

Personalized situa0on: An ac4onable integra4on of a user's personal context with surrounding spa4otemporal situa4on.

3) Personalized alerts

Available resources

Resource data

Personalized Situa1on Recogni1on: Operators

⊕

PaBern Matching

Aggregate

ψ

@ Characteriza1on

∏ Filter

γ Classifica1on

+

+

Growth Rate = 125%

Data Suppor1ng parameter(s) Output Operator Type

+

Classifica1on method

Property required

PaBern

User loca1on

…

… … …

… …

…

… Match= 42%

92

•  IF 𝑢𝑖 𝑖𝑠𝑖𝑛 𝑧𝑗 𝑇𝐻𝐸𝑁 𝑐𝑜𝑛𝑛𝑒𝑐𝑡 (𝑢𝑖, 𝑛𝑒𝑎𝑟𝑒𝑠𝑡(𝑢𝑖, 𝑟𝑘))

1) 𝑖𝑠𝑖𝑛(𝑢𝑖, 𝑧𝑗) →𝑚𝑎𝑡𝑐ℎ(𝑢𝑖, 𝑟𝑘)) 𝑓:(𝑈×𝑍)→(𝑈×𝑅)

•  U = Users •  Z = Personalized Situa1ons •  R = Resources

2) 𝑛𝑒𝑎𝑟𝑒𝑠𝑡(𝑢𝑖, 𝑟𝑘)=𝑎𝑟𝑔𝑚𝑖𝑛 (𝑢𝑖.𝑙𝑜𝑐, 𝑟𝑘.𝑙𝑜𝑐𝑠)

93

12/5/12 94

Billions of data sources. Selec0ng and combining appropriate sources to detect situa0ons. Interac0ons with different types of Users

Decision Makers Individuals

12/5/12 95

Front End GUI

NewDataSource

NewQuery

E-‐mageStream

E-‐mage Stream

E-‐mage Stream

Data Cloud

Back End Controller

Stream Query Processor

Data IngestorRegisteredData

Sources

RegisteredQueries

Raw Spatial Data Stream

API Calls

Raw DataStorage

Personalized Alert Unit

AlertRequest

User Info

12/5/12 96

11/28/2012 97

+setBagOfWords()

TwitterWrapper

+setBagOfWords()+setColors()

FlickrWrapper

+setURL()+setTheme()+setParas()

Wrapper

+hasNext() : bool+next() : STTPoint

STTPointIterator

DBSTTPointIterator

+hasNext() : bool+next() : <unspecified>

Iterator

-‐timeWindow : long-‐syncTime : long-‐latUnit : double-‐longUnit : double-‐swLat : double-‐swLong : double-‐neLat : double-‐neLong : double

FrameParameters

-‐Parameterize

1 1

-‐theme : String-‐start : Date-‐end : Date-‐latUnit : double-‐longUnit : double-‐swLat : double-‐swLong : double-‐neLat : double-‐neLong : double-‐image

Emage

11..*

1..*


Iterator

+hasNext() : bool+next() : Emage

EmageIterator

11

1..*

-‐theme : String-‐value : double-‐start : Date-‐end : Date-‐latUnit : double-‐longUnit : double-‐latitude : double-‐longitude : double

STTPoint

-‐initResolution-‐finalResolution

ResolutionMapper

1 1


Iterator

+hasNext() : bool+next() : Emage

STMerger

+setURL()+setTheme()+setParas()

VisualImageIterator

CSVWrapper KMLWrapper

11/28/2012 98

Situa0onal controller

• Goal • Macro Situa1on • Rules

Micro event e.g. “Arrgggh, I

have a sore throat” (Loc=New York, Date=12/09/10)

Macro situa0on

Control Ac0on “Please visit nearest CDC

center at 4th St immediately”

Date=12/09/10

Alert Level=High

Level 1 personal threat + Level 3 Macro threat -‐> Immediate ac0on 12/5/12 99

•  What personal informa1on can be shared? •  How should it be shared to benefit the user? •  Developing an architecture for personal informa1on management.

102

Asthma Threat level

Allergy reports Pollen Count

⊕

∏

Emage (Pollen Level)

Δ

Visual-‐ Pollen level

Air Quality

∏

Emage (AQI.)

Δ

Visual-‐ Air quality

∏

Emage (Number of reports)

Δ

TwiOer-‐Allergy

c ϵ {Low, mid, high}, [USA, 6 hrs, 0.1x 0.1]

Weather.com, [USA, 6 hrs, 0.1x 0.1]

TwiOer API, [USA, 6 hrs, 0.1x 0.1]

Pollen.com, [USA, 6 hrs, 0.1x 0.1]

Macro situa1on model

103 /

Personal threat level c ϵ {Low, mid,

high} γ

Physical exer0on Asthma threat level

⊕

TPS (Funf)

Δ

Funf-‐ac0vity

Phone sensors, (relaxMinder app),

[USA, 6 hrs, 0.1x 0.1]

EventShop

∏ Normalize (0, 100)

And

Classifica0on: Thresh(30,70)


[USA, 6 hrs, 0.1x 0.1]

TPS (Asthma)

∏ UserLoc

Personal threat level c ϵ {Low, mid,

high} γ

Physical exer0on

Asthma threat level

⊕

TPS (Funf)

Δ

Funf-‐ac0vity

Phone sensors,

(relaxMinder app), [USA, 6 hrs, 0.1x 0.1]

EventShop


And

Classifica0on: Thresh(30,70)


[USA, 6 hrs, 0.1x 0.1]

TPS (Asthma)

∏ UserLoc

104

12/5/12 107

Flood level - Shelter

Flood Level Shelter

Twitter

Classify (Flood level - Shelter)

12/5/12 108

Outline •  Introduc1on •  Social Systems •  Real Time Social Systems •  Designing Real Time Systems •  Situa1on Recogni1on •  Concept recogni1ons •  Personalized Situa1ons •  EventShop

• Going Forward

•  Social observa1ons are now possible with liBle latency.

•  Now possible to design social systems with feedback.

•  Situa1on Recogni1on and Need-‐Availability iden1fica1on of resources becomes a major challenge.

•  EventShop is a step in the direc1on of implemen1ng Social Life Networks.

Useful Links •  Demo:

–  hBp://auge.ics.uci.edu/eventshop •  Data Defini1on Language Schema

–  hBp://auge.ics.uci.edu/eventshop/documents/EventShop_DDL_Schema

•  Query Language Schema –  hBp://auge.ics.uci.edu/eventshop/documents/EventShop_QL_Schema

•  Example Query in JSON –  hBp://auge.ics.uci.edu/eventshop/documents/EventShop_Example_Query

11/28/2012 112

Thanks for your 1me and aBen1on.

For ques1ons: [email protected]

designing intelligent social systems 121205

Education