sreekar krishna, vineeth balasubramanian , sethuraman ( panch ) panchanathan

CENTER FOR COGNITIVE UBIQUITOUS COMPUTING

CUbiC

ARIZONA STATE UNIVERSITY

Sreekar Krishna, Vineeth Balasubramanian, Sethuraman (Panch) Panchanathan


CUbiC

Enriching Social Situational Awareness in Remote Interactions -

Insights and Inspirations from Disability Focused Research

ARIZONA STATE UNIVERSITY


2

CUbiC

Recognition & Learning

Sensing &

Processing Intera

ction

&

Delive

ry

Assistive Tech.Technologies for Daily

Living

Rehabilitation Assessment & Training

Medical Decision Support


3

CUbiC Success Stories - iCARE ReaderCamera

Phase 1• 2005• 3 prototypes developed• Deployed in ASU and AzSDB

Phase 2• 2006• Personal size• Customization capabilities

Phase 3• In development• Incorporating high resolution digital cameras on the glasses



4

CUbiC Success Stories - NoteTaker

The assistive technology for low vision and legally blind students for taking notes in the classroom

Zoomed video of the lecturer’s presentation in real time

Student notes with digital ink


Multi Touch Camera ControlWinner of 2010 MS Imagine Cup

Award


5

Shopping Assistance


6

Social Assistance - Origin of the Problem

Dr. Terri Hedgpeth – Director, Disability Resource Center, ASU

Focus Group Study of Individuals who are Blind:

• “It would be nice to walk into a room and immediately get to know who are all in front of me before they start a conversation”.

• “It would be great to walk into a bar and identify a friend”.


Enactor (Encoding) Recipient (Decoding)

Interpersonal Social Interactions

7

N. Ambady and R. Rosenthal, “Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.,” Psychological Bulletin. Vol. 111(2), vol. 111, Mar. 1992, pp. 256-274.

Body

Voice

Speech

Face

27%19%

Visual

18%35%

Audio

Verb

alN

on-v

erba

l

65%

35%

Social Touch

What role does touch

play?


8

Socio-Behavior Example – Hand Shake

Social Interactions

Sensory

Perceptual

Cognitive

Motor

Handshake

Step 1: Eye Contact Step 3: Move (Proxemics)Step 2: Intent to interact

Step 4: Shake hands Step 5: Conversation distance


9

Social Interactions

Social TouchSocial SightSocial Hearing

Social

Stimulation

Social

Reciprocation

Face

Body

VoiceSocial

Cognition

Social Stimulation

Social Cognition

Social Reciprocation

Social Situational Awareness


• How many people?• Where are they located?• What are their facial

expressions?• Eye Gaze• Eye Contact• Body Mannerisms

SSA in Various Settings

10

Social Assistance Decision Making Remote Collaborations

TeamSTEPPS

• Leadership• Mutual Support• Communication• Attitude• Situation Monitoring• Patient Safety

• Expressing Opinion• Managing Conflict• Making Decision• Speed of Decision• Interaction with

Colleagues• Difficulty Establishing

Rapport


CUbiC Multidisciplinary Team

11

Technology, Psychology & Human Comm. Tech. Dissemination & Validation

Assistive TechnologyTeam

Dr. Sethuraman PanchanathanComp. Sci. ASU

Human Centered Multimedia

Dr. John Black Jr.Comp. Sci. ASUAssistive Tech.

Specialist

Dr. Vineeth Balasubramanian

Comp. Sci. ASUDecision Sys. & Risk

Analysis

Dr. Prasad BoradkarSchool of Design, ASUAssistive Tech. Design,

Interdisciplinary Design Initiatives

Dr. Jameson WetmoreConsortium of Science, Policy

& Tech., ASUAsst. Tech. Ethics, Practices &

Effective Dissemination

Dr. Terri HedgpethDir. Disability Resource

Center, ASUAsst. Tech. Usability

Expert, Early Adoption Specialist

Dr. Michele ShiotaPsychology, ASU

Interpersonal Interactions, Facial Expression, Dyadic Communications

Dr. Don HomaPsychology, ASU

Visual Perception, Working Memory, Haptic Concepts

Dr. Artemio RamirezHuman Comm., ASURemote Interactions

and Communications, Modeling Professional

Meetings, Conflict

Sreekar KrishnaElec. Engg.

ASUIntegration Engg.

Behavioral Psychology and Communication Team


12

Social Interaction Assistant

Stereotypy

Face ReadingSocial Scene Analysis


Stereotypy

Any non-functional repetitive behaviorTwo main causes for stereotypy

Lack of sensory feedbackLack of cognitive feedback

Methods of control Stereotypy

• Curtail Behavior immediately

• Reward / PunishmentIntervention

• Do not intervene directly• Develop cognitive

replacement

Self Monitoring

13

Body Rocking is the most prevalent

stereotypy for people who are

blind and visually impaired


Proposed solution

XY

Z

Rocking

Non - Rocking

14

Rocking action can be recognized with an

accuracy of 94% within 2 seconds

Behavioral Psychology literature shows that one rock action is approximately 2.2 seconds long. Effectively, recognizing a rocking behavior well within one rock cycle.


15

Dyadic Interaction – Face Reading

Camera

Social Interaction Asst.


16

Dynamic Delivery of Facial Mannerisms


17

Social Gaze & Interaction Space

IntimatePersonal

SocialPublic

1.5’ 4’ 12’ 25’0’

Interpersonal Space


18

Modeling Distance & Direction through Face Detection

Module 1: Color Analysis

Module 2: Markov Random Field LPCD

n

j

HzHzh

dopt

dk

kj

Tkj

optenh

zP1

21

2

12

)2(

1)(Module 3: Evidence

Aggregation


Structured Mode Searching Particle Filter (SMSPF)

Initial Estimat

e

Corrected Estimate

Example Search

Windows

Motivation: Weak Temporal Redundancy

Approach: Stochastic Search over a large search space (Color Histogram Comparison)

Result: Approximate Estimate

Step 1Step 2

Motivation:ComplexObject Structure & Abrupt Motion

Approach: Deterministic Search over a small probable search space (Histogram of Gradients with Chamfer Match)

Result: Accurate Estimate

19


20

Face/Person Detection/Tracking

Face Detection Person Detection

Tracking

Model

Deliver


21

Social Scene Delivery System


22

Conveying Body Mannerisms

Body Posture Body Gestures

Enactor

Recipient

Social Mirror




Disability & Deficit Inspired Computing

Disabled Population

Activities of Daily Living (ADL)

Observe

Identify Barriers to ADL

Design and Develop Assistive

Tech.

Refine

Disseminate

Extrapolate to the general population

Did you know the typewriter was invented for the blind?

Blindness is only a concept


24

Mayo Multi-disciplinary Simulation Center

Doctors, Nurses, Professionals, etc.

On-body Affect Sensors

Environment Affect SensorsVision Audio

• Automated monitoring of group dynamics to determine communication breakdowns• Automatic evaluation of the social affinity between team members• Leadership evaluation and nomination through long term monitoring of teams and

individuals


25

Assisting Remote Interactions

Se-ries

158% 60% 62% 64% 66% 68% 70% 72% 74%

73%

69%

64%

Managing Conflict

Making Decision

Expressing Opinion

Challenges in virtual teams compared to face-to-face teamsTop five challenges faced during virtual team meeting

Series1

50 55 60 65 70 75 80 85 90 95

90

80

77

76

75

Different leadership styles

Insufficient time to build relations

Colleague not participate

Method of decision making

Speed of decision making

Personal challenges during virtual team meetings

Series1

45 55 65 75 85 95 105

94

85

81

77

68

66

Difficulty seeing the whole picture

Absence of collegiality

Sense of isolation

Reliance on email and telephone

Difficulty establishing rapport and trust

Inability to read non-verbal cues

The Challenges of Working in Virtual Teams: Virtual Teams Survey Report 2010. RW3 CultureWizard, 2010.


26

Socio-Behavioral Computing

Sensor & Actuator

TechnologiesHuman

Computer Interaction

Machine Learning and

Pattern Recognition

SBC

Socio Behavioral Computing

• Affective Computing

• Social Robotics

• Human Communication Dynamics

• Human Centered Computing

• User Behavior Modeling

Questions?

sreekar krishna, vineeth balasubramanian , sethuraman ( panch ) panchanathan

Documents

glasses center

disability resource

digital ink center

eye contact step

proxemics step

hands step

social touchwhat role

assistive technology