emergence of cognitive grasping through emulation, introspection and surprise grasp eul 7 th...

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP

EUl 7th Framework Program

GRASPEmergence of Cognitive Grasping

through Emulation, Introspection and Surprise

Danica Kragic

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


GRASP

Challenge 2: Cognitive Systems, Interaction, Robotics

March 2008 – February 2012

Project officer: Mr Mariusz Baldyga

Coordinator: Danica Kragic (KTH)

Co-coordinator: Markus Vincze (TUW)

Administrator: Jeanna Ayoubi (KTH)

Project number: 215821

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Consortium

University of KarlsruheKarlsruhe, Germany

T. Asfour, R. Dillmann

Kungliga Tekniska Högskolan

Stockholm, SwedenD. Kragic, P. Jensfelt

Lappeenranta University of Technology, Lappeenranta,

FinlandV. Kyrki & H. Handroos

Technische Universität München,

MunichGermany

D. Burschka, H. Deubel

Technische Universität Wien, Vienna, Austria

M. Vincze

Universitat Jaume I, Castellón, SpainA. Morales, P. Sanz

Otto Bock,Austria OBH. Dietl

GRASPGRASPFoundation for Research and Technology - Hellas,

GreeceA. Argyros & M. Lourakis

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Introduction GRASP is a project with a focused target

Objectives• Theory of grasp modeling• Self and context-awareness• Curiosity and surprise driven behavior• New grasping strategies• Exploitation and evaluation for future

prosthesis, industrial and service markets

Grasping and manipulation as a control problem have been studied since the beginning of robotics: HOWEVER- very little has been done in terms of cognitive aspects of grasping, implementation and evaluation of systems - this is what GRASP is about

Still a huge gap between visual servoing and grasping

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


GRASP: Concept and Objectives “Robots handling individually or jointly ....”

GRASP proposes a generic predict-act-perceive paradigm for grasping and manipulation of objects, dealing with static and dynamic properties of objects, the environment and the task.

Cognitive approach: predict – act – perceive• Generic principle of human brain

Paradigm open to development• Unsupervised, fully autonomously, new objects

Clear application domain: grasping Measurable benchmark: empty basket, fill pallet Expected results:

• Grasping ontology• Proof or disproof of paradigm

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Worpackages

1. Learning to Observe Human Grasping and Consequences of Grasping

2. Representations and Ontology for Learning and Abstraction of Grasping

3. Self-experience of Grasping and Multimodal Grounding

4. Perceiving Grasping Context and Interlinking Contextual Knowledge

5. Surprise: Detecting the Unexpected and Learning from it

6. Introspection and Prediction through Simulation

7. Cognitive Robotic Grasping - Integration and Applications

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Relationship between WPs

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Major Milestones

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Platforms, sensory feedback

No new hardware development in

GRASP

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


From simulation to real world and back

WP1

What/Where/How

WP7

Integration and

Applications

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Research objectives Goal:

• filter action relevant from action-irrelevant stimuli to enhance the manipulation performance;

• analyse, for a variety of increasingly complex sensorimotor tasks, the spatial and temporal properties of selective visual processing before and during manual reaching and grasping movements of human actors.

WP1: Kinematics and grasping forces, together with the eye gaze behaviour during the grasp will be measured/tracked.

WP2, WP7 – the definition of the ontologies requires definitions of action specific, selective visuo-spatial processes

WP3, WP4, WP6 – representations of control parameters, object and body (hardware) attributes

WP5 - eye-gaze positions: What to look for? What to disregard?

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Major Milestones – month 12

1 Spatio-temporal representation of human body and hands, mapping to different robot embodiments; initial evaluation on a humanoid platform.

WP1, WP2,WP7

FORTH

2 Definition of initial ontology based on human studies; acquisition (perception and formalisation) of knowledge through hand-environment interaction.

WP1, WP2,WP4, WP5

KTH

3 Initial design of the control and engine architectures; evaluation of the perception-action planning cycle in simulation.

WP3, WP4,WP6, WP7

LUT

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP



4 Analysis of action-specific visuo-spatial processing, vocabulary of human actions/interactions for perception of task relations and affordances.

WP1, WP2,WP4, WP5

TUW

5 Implementation of high-level controllers including a global uncertainty model, integration and evaluation in the simulator and experimental platforms, grounding grasping primitives.

WP3, WP6, WP7 UniKarl

6 Integration and evaluation of human body and hand tracking on active robot heads, demonstration of a grasping cycle on the experimental platforms.

WP1, WP2,WP4, WP7

TUM

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP



7 Observing consequences of grasping; vocabulary of robot action/interactions and definition of a hierarchical structure of features.

WP1, WP2,WP4, WP5

FORTH

8 Implementation of hybrid controllers for on-line adaptive primitive grounding; evaluation in the simulator and on experimental platforms.

WP3, WP6,WP7

LUT

9 Integrating contextual representation in the ontology and development of the attention system with view planning; implementation on active head.

WP1, WP2,WP5, WP7

TUM

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP



10 Linking structure, affordances, actions and tasks; evaluation of representations defined by the ontology.

WP2, WP4 TUW

11 Integration and evaluation of scenarios on multiple experimental platforms, demonstration of cognitive capabilities of robots.

All UniKarl

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


Everything will be open source

• The simulator• Hand models • Tracking algorithms• Vision algorithms• Object and action representations • Database with human data• Software framework for testing and benchmarking

Em

erg

ence

of

Cognit

ive G

rasp

ing

th

roug

h E

mu

lati

on

, In

trosp

ect

ion

and

Su

rpri

seG

RA

SP


THE END

emergence of cognitive grasping through emulation, introspection and surprise grasp eul 7 th...

Documents