prof. chrystopher l. nehaniv dr. joe saunders prof. …...uh - the team principal scientists prof....
TRANSCRIPT
![Page 1: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/1.jpg)
Inaugural Meeting
7th April 2008Adaptive Systems Research Group
Prof. Chrystopher L. NehanivDr. Joe Saunders
Prof. Kerstin Dautenhahn
![Page 2: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/2.jpg)
UH - Where are we?
![Page 3: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/3.jpg)
Human-Robot Interaction Research @ UH
• Current European FP6 Robotics Projects:– Cogniron (IP): Cognitive
Robot Companion– Robotcub (IP):
Developmental Robotics (iCub)
– IROMEC (STREP): Robotic Toys for Children
• New FP7 projects: – LIREC (IP)– ITalk (IP)
• Networks: Euron, euCognition
• Researchers:– Academics:
• Prof. Kerstin Dautenhahn• Prof. Chrystopher L. Nehaniv• Dr. Rene te Boekhorst• Dr. David Lee
– Postdocs:• Dr. Ben Robins• Dr. Ester Ferrari• Dr. Joe Saunders• Dr. Hatice Kose-Bagci• Dr. Aris Alissandrakis• Dr. Khenglee Koay• Dr. Nuno Otero
– Research Assistants/PhD students• Michael L. Walters• Assif Mirza• Qiming Shen• Dag Syrdal• Josh Wainer
![Page 4: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/4.jpg)
UH - Related EU Projects
RobotCub – Social Interaction, Developmental Robotics (FP6 IP, September 2004 – August 2009)
Cogniron – Robot Social Spaces, Proxemics, Learning and Development, Imitation
(FP6 IP, Jan 2004 – Feb 2008)
LIREC - LIving with Robots and intEractive Companions (FP 7 IP, March 2008 – June 2012)
![Page 5: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/5.jpg)
UH - The Team
Principal Scientists
Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn
Researchers
Dr. Joe Saunders
+
* Own Effort
Research Assistant/
FellowNew Hire
Dr. Caroline Lyon*
Hello
Frank FörsterPhD Student*
![Page 6: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/6.jpg)
UH - Approach to ITALK
A robot will use what it learns individually and from others sociallythrough grounded sensorimotor interaction
to bootstrap the acquisition of language via aspects of ideomotor theory,
and will use its language abilities in turn to drive its learning of social and manipulative abilities.
This creates a positive feedback cycle between using language and developing other cognitive abilities.
Like a child learning by imitation of its parents and interacting with the environment around it, the robot will master basic principles of structured
grammar, like negation, by using these abilities in context.
“In the beginning, there was the act.”- Goethe
![Page 7: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/7.jpg)
UH - Approach to ITALK
Major Open Areas for Robot/Agent Development of Language:
1) deixis, gesture and reference;2) predication;3) negation;4) emergence of syntactic categories;5) compositionality
Current Work and Open Problems: A Roadmap for Research into the Emergence of Language of Communication. In C. Lyon, C.L. Nehaniv, A. Cangelosi (editors), Emergence of Communication and Language, pages 1-27, Springer-Verlag, 2007
![Page 8: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/8.jpg)
UH - Viewpoint
Extending C.S. Peirce, latter L. Wittgenstein, & R. Garrett Millikan
To understand (and create!) language use capacity bottom up:(proto-)representations need NOT separate “what-to-do” from “what is-the-case”
(pushmi-pullyu reps.)activity of manipulating and communicating socially in a context of embodied action is
necessarymajor purpose of language is to manipulate the environment, including the social
environment, via communication(other purposes: to bootstrap learning, to regulate interaction, encoding
temporally/spatially remote experience stories – cf: Ho et al. 2008,Mirza et al., 2008)to understand the grammar of a word/linguistic construction look how it is usedmeaning is created in an embodied process of semiosis with interpretant, signal (sign),
signified.
![Page 9: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/9.jpg)
Outline and UH Role
Five Major Work Packages
Action DevelopmentConceptualisationSocial Interaction – Work Package LeaderLanguage EmergenceIntegration and Bootstrapping of Cognition
Robotic Experiments and Demonstrations
Based on iCub robotSimulation (where necessary)Prior to iCub on Kaspar
![Page 10: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/10.jpg)
Outline and UH Role
• Development of artificial embodied agents that are able to acquire complex behavioural, cognitive and linguistic skills through individual and social learning.
Specifically•To learn to handle and manipulate objects and tools autonomously
•To cooperate and communicate with other robots and humans
•To adapt their abilities to changing internal, environmental and social conditions
Aim
UH will build upon our social learning architecture - ROSSUM (RObot Self-Imitation and Scaffolding Utility Mechanism)
Saunders J, Nehaniv C.L, Dautenhahn K. and Alissandrakis A.,(2007), Self-imitation and Environmental Scaffolding for Robot Teaching, International Journal of Advanced Robotics Systems, Vol.4, Issue 1, pp.109- 124
Scaffolded Skills <-> Scaffolded Language
![Page 11: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/11.jpg)
Outline and UH Role
AbilitiesAgents learn to handle objects
UH:
Investigate, by extending existing studies on Ideomotor theory applied to physical robots, how to develop more complex compositional and hierarchical actions via scaffolded learning.
![Page 12: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/12.jpg)
Outline and UH Role
AbilitiesAgents learn to handle objectsAgents create and use embodied concepts
UH:
Our current studies on robot teaching and interaction ensure that the robots experiences are grounded in its own internal and external perceptual states. The crucial aspects of these states are selected using information theoretic approaches.
![Page 13: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/13.jpg)
AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skills
UH:
Using our existing social learning and teaching robotic architecture (called ROSSUM) which is now operating on our humanoid robot (called Kaspar) we intend to study relationships between deixis, gesture and reference. Exploit concept of shared meaning arising through similar gestures and also study ideas of ‘negation’.
Outline and UH Role
![Page 14: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/14.jpg)
AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skillsAgents develop linguistic abilities
UH:Investigate relationships between actions and objects via hierachical scaffolding
Outline and UH Role
![Page 15: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/15.jpg)
AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skillsAgents develop linguistic abilitiesAgents transfer knowledge between cognitive domains
UH:
Study the HRI aspects of verbal or gestural communication.
Outline and UH Role
![Page 16: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/16.jpg)
UH Existing Work on ROSSUM and its Relation to Language
Teaching via Assisted/self
ImitationImitation
CorrespondenceProblem
Affordances andEfffectivities
EmbodiedGrounded
Sensory-motorConcepts
Steps toLanguage Acquisition
![Page 17: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/17.jpg)
How can a child discover affordances and effectivities for action?:
• by trial-and-error – may be slow• by observational imitation – but correspondence problem exists• by assisted/self-imitationCaregivers invite infants to imitate – by putting infants on both sides of the correspondence problem
Scaffolded Teaching via Assisted Imitation
Assisted/Self-Imitation
• Putting through of infants movements• Acting in tandem• Linking effectivities of the infants body with affordances for action• Scaffolding taught competencies
(* Affordances, Effectivities and Assisted Imitation - Patricia Zukow-Goldring, Michael Arbib (2007).)
UH Existing Work on ROSSUM and its Relation to Language
![Page 18: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/18.jpg)
Correspondence Problem and Difficulties in Observational Learning
Matching Movements?
Matching Goals/Effects/States?
Matching Combinations of Movements?
What does matching mean?
The Correspondence Problem*
*C.L.Nehaniv & K.Dautenhahn (2002) – The Correspondence Problem
UH Existing Work on ROSSUM and its Relation to Language
![Page 19: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/19.jpg)
The Correspondence Problem in Imitation
Relative Position
Demonstrated Effect Corresponding Effects
Relative Displacement
Absolute Position
(also see Aris Alissandrakis, Chrystopher L. Nehaniv, Kerstin Dautenhahn and Joe Saunders (2005))
UH Existing Work on ROSSUM and its Relation to Language
![Page 20: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/20.jpg)
UH Existing Work - Robots Learning Socially
using ROSSUM
![Page 21: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/21.jpg)
Teaching via Assisted/self
Imitation
Inverse andForward models
ExtendedIdeomotor
Theory
Imitation
CorrespondenceProblem
Affordances andEfffectivities
ReafferencePrinciple
EmbodiedGrounded
Sensory-motorConcepts
Steps toLanguage Acquisition
UH Existing Work on ROSSUM and its Relation to Language
![Page 22: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/22.jpg)
From Analyse der Empfindungen – Ernst Mach - 1922
Body is visually perceived in exactly the same way as the environment – representation
is the same
Distinction between body and environment must be
based on a non-visual criteria
Distinction achieved by voluntary control
Actions are considered to be creations of the will
UH Existing Work – Ideomotor Theory
![Page 23: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/23.jpg)
Ideomotor Theory : Lotze (1852), James (1890) and Műnsterberg (1888)
How can voluntary behavior or ‘will’ be explained?
All actions are held as ‘images’ of the sensory feedback they produce.
Sensory stimulations result from internal feedback they produceplus
external feedback from the effects on the environment
Actions are generated based on imagining their sensory consequences
UH Existing Work – Ideomotor Theory
![Page 24: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/24.jpg)
Why is this important for a robotic learning system?
One powerful way of learning is through imitation
Extend the Ideomotor Idea to allow imitation – Extended Ideomotor Theory*
If an action can be invoked from thinking about it…..
then
the same action could be invoked by seeing it performed by someone else
*An Ideomotor Approach to Imitation - Wolfgang Prinz (2005)
Extended Ideomotor theory thus supports the idea of learning through experience and
learning through observation
UH Existing Work – Ideomotor Theory
![Page 25: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/25.jpg)
UH New Work – extending ROSSUM
Teaching via Assisted/self
Imitation
Inverse andForward models
ExtendedIdeomotor
Theory
Imitation
CorrespondenceProblem
Affordances andEfffectivities
ReafferencePrinciple
EmbodiedGrounded
Sensory-motorConcepts
Learning viaObservational
Imitation
“Putting oneself In the shoes of
Another”TOM?
Learning viaAssociation,
Inference andReferential intent
Adultto
Child?(Teaching)
Childto
Adult?(Learning) Steps to
Language Acquisition
![Page 26: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/26.jpg)
Perceived State Action State Resulting From Action Execution
Action State Resulting From Action Execution
Action State Resulting From Action Execution
Action State Resulting From Action Execution
.
.
.
ROSSUM Behavioural Component / Memory Model
‘Ideas of Movement’From Previous Learning Effects of Movement
Perceived State
Perceived State
Perceived State
Acts as an Inverse Model Acts as a Forward Model
Similarity
Matching
Current State
Similarity
Matching
Model State
![Page 27: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/27.jpg)
Goal-directed Task
Sequence
Primitive
Behaviour
Primitive Primitive Primitive
Goal-directed Task
Primitive Primitive
Example of a Trained Hierarchy
State
Primitive
Behaviour
Using task scaffolding to build competencies
![Page 28: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/28.jpg)
UH The First Year – Experimental and Theoretical
Research Questions What is the relationship between deixis, gesture and reference ----------- D3.1 (M24)
- Related to hypothesis on the emergence of language based on mirror neurons- Shared meaning arises through similar affordant gestures- the human tutor reveals to the robot via deixis, gesture and reference relevant effectivities and affordances
How can negation arise though the mechanisms of communicative social interaction?---D3.2 (M48)
-Experiments to be carried out where facial and manual gestures used in social interaction to indicate negation- This would involve inhibition signals operating on existing scaffolded behaviours
![Page 29: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/29.jpg)
UH The First Year – Technical Considerations for ROSSUM
Implement Forward ModelForward model side of ROSSUM to be implemented (relates to D1.1)
Gaze Following and Observational ImitationImplement direct imitation via human body tracking either as combination of TOF depth sensor/magnet field trackers/ eye gaze tracker (as Yu/Ballard, S. Knoop)
Object DetectionExtend capabilities for enhanced object recognition by robot using colour segmentation, shape, texture (as Yu/Ballard, Deb Roy).
Word/Phoneme Detection?Study possibility of word/phoneme recognition (as per Deb Roy)
Assisted or Self ImitationExtend current model to further process dynamic movements
![Page 30: Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. …...UH - The Team Principal Scientists Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn Researchers Dr. Joe Saunders + *](https://reader030.vdocuments.net/reader030/viewer/2022040910/5e842b74508a8045cd067fc6/html5/thumbnails/30.jpg)
Thank you
Adaptive Systems Research Group