8/21/2013 1

• Why am I excited about things?

• What are We doing?

• What are the challenges?

What is your research interest?

• Computer Vision

• Multimedia

• Machine Learning

• Social Media

• Intelligent Systems

• Big Data

• Stream Processing

Building a Better Society.

Events and Entities Exist in the real world.

Events and entities result in Data and Documents

What is the most Fundamental Problem in Society?

Connecting People to Resources

Effectively, Efficiently, and Promptly

in given Situations.

Hint: Economics, Health Care, Politics, Computer Science, Operations Research, …

Maslow: Basic Needs

http://www.theappgap.com/collaboration-whats-in-it-for-me.html

Maslow: Needs and Resources

http://www.theappgap.com/collaboration-whats-in-it-for-me.html

• Financial

• Natural (Food)

• Human Skills

• Health

• Rescue

• Transportation

• Education

• Production

• …

Social Life Networks

Connecting People to Resources

Modern Data

8/21/2013 10

Knowledge Data

Processing

Information/ Decision

Decisions

Food Friend

What???

Information Data

Knowledge

Observations

• Desired state (Goal)

• System model and Control Signal (Actions)

• Current State (for Feedback)

Key Steps

1. Identify Situation 2. Determine Needs 3. Determine Resources 4. Develop best resource management

approach 5. Communicate/Actuate decisions 6. Go to 1.

Social Networks

Connecting

People

Current Social Networks

Important Unsatisfied Needs

8/21/2013 17

Motivation

Location Based Mobile Applications

Ongoing Archived Database System

satellite

Environmental Sensor Devices

Internet of Things

Social Media

Social Life Network

Experts

People

Governmental Agencies

Situations

Social Life Network

Aggregation

and

Composition

Situation

Detection Alerts

Queries

Information

Fundamental Problem

Connecting People to Resources effectively, efficiently, and promptly

in given situations.

• Social observations are now possible with little latency.

• We can design social systems with feedback.

• Situation Recognition and Need-Availability identification of resources is a major challenge.

Smart Social Systems Architecture

Situation Recognition

Evolving Situations

Available Resources

Identified Needs

Need- Resource Matcher

Communication/Control

Unit

Resources

People

Data Sources

Database System

satellite

Environmental Sensor Devices

Social Media

Concept Recognition: Last Century

23

Environments

Real world Objects

Situations

Activities

Single

Me

dia

SPACE TIME

Scenes Location aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

Visual Concept Recognition: Quick History

• 1963: Object Recognition [Lawrence + Roberts]

• 1967: Scene Analysis [Guzman]

• 1984: Trajectory detection [Ed Chang+ Kurz]

• 1986: Event Recognition [Haynes + Jain]

• 1988: Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object Scene Trajectory Event Situation

Concept Recognition: This Century

25

Environments

Real world Objects

Situations

Activities

SPACE TIME

Location aware

Location unaware

Static Dynamic

Hete

roge

ne

ou

s Me

dia

Location aware

Location unaware

Static Dynamic

Data is just Data. Medium and sources do not matter.

• relative position or combination of circumstances at a certain moment.

• The combination of circumstances at a given moment; a state of affairs.

Situations: Definition

An actionable abstraction of observed spatio-temporal characteristics.

27

8/21/2013 28

• Example 1:

– A person shouting.

– 1000 people shouting.

• In a contained building

• In main parts of a city

• Example 2:

– One person complaining about flu.

– Many people from different areas of a country complaining about flu.

Micro-events: Sensors detecting and chirping

(broadcasting) events

• Billions of disparate kinds of sensors being placed everywhere.

• Each sensor detects ‘basic events’ and broadcasts it in a simple form.

• Develop a system to process these micro-events and make them useful.

Example: Cameras in a city

• ‘Chirps’ could be of different types • Define behaviors like:

– Heavy traffic – Popular event going on – People leaving X area – Violence starting – . . .

• Use for Macro-behvior analysis

From micro events to situations

Thermodynamics

provides a framework for relating the microscopic properties of individual atoms and molecules to the macroscopic or bulk properties of materials that can be observed in everyday life.

Data Types in Situation Recognition

• Static Data: – POIs (location of hospitals) – Population – Technical data of hospitals – Contact persons in committees or health organization, and – All information on support-potentials for personnel, material and

infrastructure

• Dynamic Data: – Current disease data – Twitter/Google Trends – Environmental data – Personal Individual data

Two Big Challenges

• Data Ingestion to efficiently extract data from the Web and make them available for later computation is not-trivial.

• Stream Processing Engine to bridge the semantic gap between high level concept of situations and low level data streams.

S

Social Networks

2-D spatial Grid at time T

Database Systems

Global Sensors

Phone Apps Internet of

Things

(S, T, T)

S Uses Application semantics to combine different data items.

Observed State(Situations)

Intelligent Social Systems: Spatial Perspective

Real World Events

Ob

serv

atio

ns

Control Signals

Goal

Observed State(Situations)

Intelligent Social Systems: People Perspective

Real World Events

Ob

serv

atio

ns

Control Signals

Goal

• Spatio-temporal element

– STTPoint = {s-t-coord, theme, value, pointer}

• E-mage

– g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V = N)

• Temporal E-mage Stream

– TES=((ti, gi), ..., (tk, gk))

• Temporal Pixel Stream

– TPS = ((ti, pi), ..., (tk, pk))

38

8/21/2013 39

8/21/2013 40

8/21/2013 41

Retail Store Locations

Net Catchment area

Level 1: Unified representation

(STT Data)

Level 3: Symbolic rep. (Situations)

Properties

Properties

Properties

Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, …

Level 2: Aggregation

(Emage)

…

STT Stream

Emage

Situation

42

Operations

43

Pattern Matching

Aggregate

@ Characterization

∏ Filter

Classification

72%

+

+

Growth Rate = 125%

Data Supporting

parameter(s) Output Operator Type

+

Classification method

Property required

Pattern

Mask

• IF 𝑢𝑖 𝑖𝑠𝑖𝑛 𝑧𝑗 𝑇𝐻𝐸𝑁 𝑐𝑜𝑛𝑛𝑒𝑐𝑡 (𝑢𝑖, 𝑛𝑒𝑎𝑟𝑒𝑠𝑡 𝑢𝑖, 𝑟𝑘 )

1) 𝑖𝑠𝑖𝑛 𝑢𝑖, 𝑧𝑗 → 𝑚𝑎𝑡𝑐ℎ(𝑢𝑖, 𝑟𝑘))

𝑓: (𝑈 × 𝑍) → (𝑈 × 𝑅) • U = Users

• Z = Personalized Situations

• R = Resources

2) 𝑛𝑒𝑎𝑟𝑒𝑠𝑡 𝑢𝑖, 𝑟𝑘 = 𝑎𝑟𝑔𝑚𝑖𝑛 (𝑢𝑖. 𝑙𝑜𝑐, 𝑟𝑘. 𝑙𝑜𝑐𝑠)

44

8/21/2013 45

Billions of data sources. Environment for Selecting, and Combining appropriate sources to detect situations.

Prediction for Pro-active actions

Interactions with different types of Users Decision Makers Individuals

Output Ingestor

EventSource Parser

Data Adapter

Emage Generator

(+resolution mapper)

Processing

EvShop Internal Storage

Query Parser

Query Rewriter

Event Stream Processing Executor

ᴨ

ᴨ

Action Parser

Register EventSource Register Continuous Query

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Action Control

Event Property & Other Information

(e.g., spatio-temporal pattern)

µ

Data Access Manager

Live Stream

Archived Stream

Situation Stream 46

Real-Time DataSource (e.g., sensor

streams, geo-image streams)

Near Real-Time DataSource

(e.g., preprocessing data streams, social

media streams)

Raw Event

Input Manager

External Event Preprocessing

(Collaboration)

47

Real-Time Sensor Streams e.g., Cloud Satellite Images

Real-Time Sensor Streams e.g., Wind Speed, Traffic Flow

RealTime DataSource

1D Data Wrapper

STT to Emage

2D Data Wrapper

Data Adapter Emage Generator

Emage

Emage Factory

STT

Emage

Raw Social Media Streams e.g., Twitter, News RSS Feed

NearRealTime DataSource

Event Model Wrapper

STT to Emage

Data Adapter Emage Generator

Emage

Emage Factory

STT

Topic Event Detection

Abnormal Event

Detection

Raw Sensor Streams e.g., PM2.5 data

“EventModel” Streams e.g., suddenly change

of data trend within time window

Emage Store

STT Store

Metadata Store

EventSource Parser Interface

(Optional)

RealTime Emage Streams

NearRealTime Emage Streams

Processing Manager

ES Descriptor

ES Control (Start/Stop/ View ES)

Users Input

Automatic Data/Events Flow

InitialResolution AggregationFunc

Metadata

Theme AdapterType SourceURL TimeWindow Parameters

48

Stream Processing Engine

Operators Manager

Built-in Operators

User-Defined Operators

ᴨ

ᴨ

µ

Data Access

ᴨ

ᴨ

µ

Data Access

ᴨ

ᴨ

µ

Data Access

Input Manager (Accessing

External Data)

Event Stream Executor

Operators Nodes

Internal Storage (Accessing Internal

Data)

AsterixDB, SciDB, MongoDB

Emage Store

STT Store

Metadata Store

Query Parser Interface

Query Descriptor

Query Control (Start/Stop/ View)

Real-time/ near real-time

Emage Streams

Archived Emage Streams Situation Streams

Emage Interpolation

Function

Emage Conversion

Final Resolution

Parameter Operators

Operators

Store Parameters

Retrieve Parameters

Query Rewriter

Execution Plan

8/21/2013 49

8/21/2013 51

Flood level - Shelter

Flood Level Shelter

Twitter

Classify (Flood level - Shelter)

8/21/2013 52

Personalized Alerts

Disaster Situation

Assimilation and Control

Environmental

Resources and Historic Data

Governmental Agencies

Internet of Things

Social Sources

Experts

Users

All Users are not EQUAL.

What defines a person?

Activities

Communications

Media

Persona: Turning Disassociated Data into Meaningful Information

AGGREGATED PERSONAL ANALYTICS

Sensors

55

Persona

Personal EventShop: Micro Situation Detection

HEALTH PERSONA

Logical Sensor Physiological Sensors Fitness Tracking Sensors

Life Event

Kinetic Event

Physiological Event

Food Event

Personicle

Health Persona Framework

Life Events Ontology

Meeting with development group

Yoga with Jena

Meeting with test group

Mina’s birthday party

9:30-11:00 12 - 1 4:00- 5:00

Tran

spo

rtat

ion

wal

kin

g

wal

kin

g

Tran

spo

rtat

ion

Tran

spo

rtat

ion

Tran

spo

rtat

ion

Home Home Work Caspian

Hav

ing

Bre

akfa

st

Do

ing

the

dis

hes

Co

mm

uti

ng

Co

mm

uti

ng

Wal

kin

g

Wal

kin

g

Mee

tin

g

Mee

tin

g

wo

rkin

g

Wo

rkin

g

Exer

cise

Exer

cise

Wo

rkin

g

Wo

rkin

g

Wo

rkin

g

Wo

rkin

g

Mee

tin

g

Mee

tin

g

Wal

kin

g

Co

mm

uti

ng

Co

mm

uti

ng

Co

mm

uti

ng

Sho

pp

ing

Sho

pp

ing

Co

mm

uti

ng

Par

tyin

g

Par

tyin

g

Par

tyin

g

Par

tyin

g

Par

tyin

g

Co

mm

uti

ng

Co

mm

uti

ng

Co

mm

uti

ng

Slee

pin

g

Slee

pin

g

Wat

chin

g TV

Cle

anin

g

Personicle

Commute Walk Meet work Exer. work Meet Commute shop party Commute Sleep

GP

S lo

cati

on

tra

ckin

g

Cal

end

ar

Act

ivit

y le

vel

Pers

on

icle

SLN Architecture

EventShop

Personal EventShop

Evolving Situations

Available Resources

Identified Needs

Need Resource Matcher

Communication/Control

Unit

Resources

People

Data Sources

Database System

satellite

Environmental Sensor Devices

Social Media

Research Challenges

• Situation Recognition

• Persona and Personal Context

• Chronicle Analytics and Visualization

• Massive Geo-Spatial Heterogeneous Stream Processing

• Dynamic Need-Resource Optimization

Situation Recognition

• Next Frontier in Concept Recognition

• Heterogeneous Geo-spatial Dynamic Data

• Social data and IoT become a key element

• Application and domain semantics

• Model definitions

• High dimensionality

• Unification of data: Social-Cyber-Physical

Persona and Personal Context

• Not only Logs of Keyboard and Surfing.

• You Log and explore every thing.

– Entity resolution on TURBO

• Many new data processing and unification challenges.

Single Tri-axial Accelerometer

Example: Activity Recognition Multiple

Accelerometers

Activity Classification

Features: mean, variance, standard deviation, energy, entropy Correlation between axis, discrete FFT coefficients

Activities: running, walking, lying, sitting, ascending stairs, descending stairs, cycling

Location

Calendar

History

…..

Context

Activities: running, walking, lying, sitting, ascending stairs, descending stairs, cycling, shopping, commuting, doing housework, office work, lunch routine.

MicroBlogs and Twitter: LIMITATIONS

• Very LOW Signal-to-Noise ratio: High Noise-to-Signal ratio

• Focus on being broad platform.

• Difficult to extract SIGNAL.

• Information must be extracted from limited text.

Solution: Tweeting Applications

• Develop focused Apps: Focused MicroBlogs

• Get all information from ‘motivated’ and collaborative users.

• Help them solve their problem.

WAZE: Outsmarting Traffic, Together

Chronicle Analytics

• Enterprise Warehouse were for late 20th Century – Planetary Warehouses are defining this century.

• Big data is important because it collects everything that happens to build ‘Prediction Machines’.

• Machine learning and visualization are the key tools.

Massive Geo-Spatial Heterogeneous Stream Processing

• Why does the DATA become so BIG?

• And it will keep getting BIGGER.

• We have to go beyond Batch Processing as primary computing approach.

• Should be of great interest to Social Media researchers.

Dynamic Need-Resource Optimization

• What are the fundamental problems in Computer Science? – Time and memory compexity

– Operating systems, networks, storage management, algorithms, …

• What is the main concern in – Economics?

– Healthcare?

– Politics?

– …

Live EventShop and Collaboration

• Live EventShop Demo – http://auge.ics.uci.edu/eventshop/

• Current Collaborators & Plan – Cyber-Physical Cloud Computing Project

• NICT, NIST

– SLN4MOP Project • Sri Lanka Farmers; Prof. Ginige in Sydney leading

– Open Source EventShop by mid-year of 2013 • HCL

Thanks for your time and attention.

For questions: jain@ics.uci.edu