A Path to Robot Consciousness via Social Cognition: Agency and

Intention  Peter Ford Dominey, Robot Cognition LaboratoryStem Cell and Brain Research Institute INSERM U846, France, peter.dominey@inserm.fr

ComprendreAMORCES

Outline

More questions than answers…. Motivation – Can robotics clarify our ideas about

consciousness? Context – relation of consciousness and meaning Sytem architectures Progress in Meaning and cooperation in robots

Action and language Shared plans “Meaning” Simulation as meaning

Discussion

Motivation – cooperative robots

When a robot can do things like …. Learn the name of an object, Learn a new action with that

object Tell you what it knows Ask questions when it doesn’t

understand Anticipate what you are about

to do … can it be useful in the

scientific study of consciousness? Dominey, Metta, Nori, Natale (2008)

IEEE Int. Conf. On Humanoid Robotics

Context – consciousness meaning and robotics

Jeffrey Gray: Conscious experiences are imbuded with meaning; computers cannot

(without human interpretation) compute meaning; therefore, computers cannont be conscious

Robots differ from computers in that they are endowed with just such behavioral dispositions [to compute meaning]… [but] we should remain doubtful whether they are likely to experience conscious percepts… that will depend on how the trick of consciousness is done.

Daniel Dennett: … set out to make a robot that is theoretically interesting independent

of the philosophical conundrum about whether it is conscious. Such a robot would have to perform a lot of the feats that we have typically associated with consciousness in the past, but we would not need to dwell on that issue from the outset.

Towards a System Architecture of Consciousness

Brain interacts unconsiously with

world Constructs a simulation

Simulation is consiously perceived Public cognitve space Private cognitive space Private bodily space

Real Unperceived External World

The unconsciousBrain

Experienced external world, including body

from outsidePublic cognitive

space

Inner cognitiveexperiences

(thoughts,Images…)

Private cognitive space

Inner bodily sensations, feelings

Private body space

Cybernetic interactionsConstrains

Simulates

Conscious experience

The different spaces of conscious experience, from Gray 2004

System architecture for a Cooperative Robot

Lallée, Lemaignan, Lenz, Melhuish, Natale, Skachek, van Der Zant, Warneken, Dominey (submitted)

Real Unperceived External World

The unconsciousBrain

Experienced external world, including body

from outsidePublic cognitive

space

Inner cognitiveexperiences

(thoughts,Images…)

Private cognitive space

Inner bodily sensations, feelings

Private body space

Cybernetic interactionsConstrains

Simulates

Conscious experience

External world

Cybernetic interaction

Simulates

Linking Grammar Learning with Vision for Event Description and Interrogation

Humannarrator

Vision, "Meaning"Extraction

Spoken Language Interface

(CSLU RAD)

Grammatical Construction Model: Sentence to Meaning

The moon gave the cylinder to the block.The block was gave the cylinder by the moon.The cylinder was gave to the block by the moon.

Event(Agent, Object, Recipient)

E(A,O,R) Sentence

CCD Camera

Gave(moon, cylinder, block)

Dominey & Boucher (2005) Artificial Intelligence

Meaning in Cooperation: Language-Based interaction with the Robot Apprentice

Cooperative Table Assembly Scenario Robot Helps Users to Assemble a Table

Functional Requirements - The robot should: Respond to human spoken

commands with simple behaviors

Open left hand, turn right,.. Grasp(X): X in <visible>

Learn complex behaviors constructed from the primitives

Give me the orange leg Hold the table

Kawada Industries HRP-2 PlatformCNRS-AIST Joint Robotics Laboratory

LAAS, Toulouse, France

Spoken Language Programming

Part of Joint Robotics Laboratory project, AIST/CNRSDominey, Mallet, Yoshida (2007) IEEE ICRA, IEEE Humanoids, (2009) IJHR

Spoken Language Programming

Method Hand coded « primitives » (postures) and grasp(x) procedure Sequenced together via spoken

language Macro Programming

Humanoids 2007 Procedure with Arguments

ICRA 2007 Generalizes to different tasks

Assembly, disassembly

Part of Joint Robotics Laboratory project, AIST/CNRSDominey, Mallet, Yoshida (2007) IEEE ICRA, IEEE Humanoids, (2009) IJHR

User guides action by spoken language

Pseudo-code: At each command: If current subsequence is in

InteractionHistory L1 – anticipate speech L2 – propose next action L3 – take initiative Increment L

Else get next command Execute Update Interaction History

Automatic Learning, and Anticipation

Dominey, Metta, Nori, Natale (2008) IEEE Int. Conf. On Humanoid Robotics

Progressive effects of Learning

First experieceWith leg 1

Speech anticipationWith leg 2

Action propositionWith leg 3

Robot initiativeWith leg 4

Mea

n E

xecu

tion

time

for

a si

ngle

act

ion

(sec

)

Tomasello M, Carpenter M, Call J, Behne T, Moll HY (2005) Understanding and sharing intentions: The origins of cultural cognition, Beh. Brain Sc;. 28; 675-735.Dominey PF (2005) Toward a construction-based account of shared intentions in social cognition. Behavioral and Brain Sciences 28:696-+.

But,…. Cooperation Requires Shared Plans

Lallee, Warneken, Dominey (2009) EpiRob, Humanoids Workshop

Perceive action Attribute agency Form shared plan

Ordered list of (agent, action) pairs Use it in cooperation

Role reversal Limitations: robot doesn’t know “why?”

Larry (left) Robert (right)

ToyBox

Learning Shared Plans from Observation

Approaching Meaning: Linking actions to states

Lallee et al. Submitted Frontiers NeuroRobotics

Learn to recognize action via Dynamic perceptual primitive patterns

Visible, Contact, Moving

Enrich this with knowledge of Enabling state (initial) Resulting State (final/goal) « Derived predicates »

Derived Predicates On, Under Has

Reasoning: Forward chaining from current state to

goal Backward chaining from goal to current

state

Submitted to IROS 2010

Meaning: Linking actions to states

Learn the name ofa new action

Learn the relation between “cover” and “on”

Demonstrate transferto a new enactment

Learn to recognize action via Dynamic perceptual primitive

patterns Visble, Contact, Moving

Enrich this with knowledge of Enabling state (initial) Resulting State (final/goal) « Derived predicates »

Derived Predicates On, Under Has

Reasoning: Forward chaining from current

state to goal Backward chaining from goal to

current state

« Cover Arg1 with Arg2 »

Meaning: Linking actions to states

Language and meaning

Language can augment meaning derived from vision Explaining derived statesExplaining causal relations

MeaningInitial State –Action – Final State-

Vision

Language Action

Towards a System Architecture of Consciousness

Brain interacts unconsiously with

world Constructs a simulation

Simulation is consiously perceived Public cognitve space Private cognitive space Private bodily space

Real Unperceived External World

The unconsciousBrain

Experienced external world, including body

from outsidePublic cognitive

space

Inner cognitiveexperiences

(thoughts,Images…)

Private cognitive space

Inner bodily sensations, feelings

Private body space

Cybernetic interactionsConstrains

Simulates

Conscious experience

The different spaces of conscious experience, from Gray 2004

Hybrid Embodied-Propositional System

Propositional system manipulates compact, « symbolic » representations of actions, plans Embodied system employs « situated simulations », unpacking the compact representations Language allows the speaker to « direct the film » that unfolds in the listener’s mind

Madden, Hoen, Dominey (2009) A Cognitive Neuroscience Perspective on Embodied Language forHuman-Robot Cooperation, Brain and Language

Towards Embodiment: Learning to predict the perceptual consequences of a motor action

Grasping requires vision of the handThe hand has “infinite” posturesHow to reduce the visual recognition

space?

VisionProprioception(joint angles)

…Hand Posture

3Hand Posture 2Hand Posture

1

Perceptual-Motor Learning

Area 5 MMCM

Multi Modal Convergence Maps (MMCM) Topographical Organisation (Kohonen SOM like)

Associates Distinct Patterns of Joint Angles with the Corresponding Image of the hand

Training: Vision-Proprioception pairs every

100ms for 8 min 6 joints moved in cyclic pattern ~16 cycles

Me

an

Re

cog

ntin

tim

e (

+ S

D,

SE

)0,005

0,010

0,015

0,020

0,025

0,030

0,035

0,040

0,045

0,050

LEFT OFF RIGHT OFF LEFT ON RIGHT ON

Vis-Motor LearningOFF ON

Experimental Effects on Performance

Using Vis-Motor Learning has a significant effect on visual recognition time F(1,39) = 418, p < 0.0001

iCub (Robotcub project)

S. LALLEE1, G. METTA2, L. NATALE2, U. PATTACINI2, *P. F.DOMINEY1; (2009) Proprioception of the hand contributes to visual recognition speed andaccuracy: Evidence from the Multi-Modal Convergence Map model of ParietalCortex Area 5, Society for Neuroscience Abstract

Return to the neurophysiology: of language, action and cooperation

Ventral stream (green) phonological and lexical processing (STS, MTG, PFCv)

Dorsal stream (Blue) grammatical integration/unification and sensorimotor interface, simulation (TPJ, PFCdl, PPC)

Complex Event Recognition (Orange) social cognition, cooperation (STS), Agency, Simulation, Intention,Teleological reasoning, Perspectivie taking

Discussion

A Path to Robot Consciousness via Social Cognition: Agency and Intention Manipulates representations of self, other Perspective taking Recognition of agency Designation of intention based on action recogntion Use of language to express beliefs

Can robot studies be used to address any aspects of consciousness? What is the roadmap?

Action and language Shared plans “Meaning” Simulation as meaning Self – body scheme

How would we define robot consciousness?

Acknowledgements

Collaborators Jocelyne Ventre-Dominey Michel Hoen Carol Madden Felix Warneken Toshio Inui Frank Ramus Anthony Mallet Eiichi Yoshida Giorgio Metta Giulio Sandini Francesco Nori Lorenzo Natale Ugo Pattacini

Research Organizations CNRS INSERM

Funding CHRIS (EU FP7) Organic (EU FP7) French ANR

Amorces (PsiRob) Comprendre (Blanc)

RobotCub (EU FP6) iCub Open Call

Students/ PostDocs Jean-David Boucher Stephane Lallee Mehdi Khamassi Xavier Hinaut Anne-Lise Jouen

The Cube Game

Fagel, Bailly (GIPSA-Lab, Grenoble)Boucher,Ventre-Dominey, Dominey(INSERM-RCL, Lyon)

The different spaces of conscious experience

Real Unperceived External World

The unconsciousBrain

Experienced external world, including body

from outside

Public cognitive space

Inner cognitiveexperiences

(thoughts,Images…)

Private cognitive space

Inner bodily sensations, feelings

Private body space

Cybernetic interactionsConstrains

Simulates

Conscious experience

(From Gray, 2004, Fig 1.1)

Integration: Hybrid Telelogical/Embodied Cognitive System

Madden, Hoen, Dominey (2009) Brain and LanguageiCub project – Lyon, June 2009

Teleological reasoningEmbodied representation/simulation

Learning Actions and their Consequences

Learn to recognize action via Dynamic perceptual primitive

patterns Visble, Contact, Moving

Enrich this with knowledge of Enabling state (initial) Resulting State (final/goal) « Derived predicates »

Derived Predicates On, Under Has

Reasoning: Forward chaining from current

state to goal Backward chaining from goal to

current state

« Cover Arg1 with Arg2 »

Meaning: Linking actions to states

RobotCub – A Great Achievement

Distinctions and Definitions

Public vs Private Public – « the red

book on the shelf » Private – thoughts,

feelings

Inner vs External

Replace Explicit Programming

Use On-line, automatic learning of behavior via continuous comparison with the Interaction History

Learning from Experience:Automatic Learning, and Anticipation

Dominey, Metta, Nori, Natale (2008) IEEE Int. Conf. On Humanoid Robotics

Real Unperceived External World

The unconsciousBrain

Experienced external world, including body

from outside

Public cognitive space

Inner cognitiveexperiences

(thoughts,Images…)

Private cognitive space

Inner cognitiveexperiences

(thoughts,Images…)

Private body space

Cybernetic interactionsConstrains

Simulates

Conscious experience

Perceptual-Motor Learning

Multi Modal Convergence Maps (MMCM) Topographical Organisation (Kohonen SOM like)

Associates Distinct Patterns of Joint Angles with the Corresponding Image of the hand

VisionProprioception(joint angles)

…Hand Posture

3Hand Posture 2Hand Posture

1

Area 5 MMCM

We want to cooperate with this guy… But How?

Plan Define a Context for Cooperation Build some basic tools: Spoken Language Programming Learning

Automatically from one’s own experience Shared plans from Observation The meaning of actions

Towards Embodyment A Hybrid Propositional & Embodied Cognitive System