embodied cognition - booch hicss50

IBM Confidential | Do Not Distribute © 2016 IBM Corporation 1

Embodied Cognition

Grady Booch


Imagine unleashing Watson in the physical world. Give it eyes, ears, and touch, then let it act in that world with hands and feet and a face, not just as an action of force but also as an action of influence. This is embodied cognition: by placing the cognitive power of Watson in a robot, in an avatar, an object in your hand, or even in the walls of an operating room, conference room, or spacecraft, we take Watson's ability to understand and reason and draw it closer to the natural ways in which humans live and work. In so doing, we augment individual human senses and abilities, giving Watson the ability see a patient's complete medical condition, feel the flow of a supply chain, or orchestrate the tasks in a day in the life of an individual.


Embodied Cognition Defined

§ An embodied cognition must– Be in and of the world– Reason– Learn– Have identity

§ An embodied cognition is not– Simply an STT -> NLC -> action pipeline


Robotics

Raw curiosity

More computational

power

Advances in AI algorithms and architectures

Better/smaller sensors and

actuators

Better batteries

3D printing

Advances in neural implants

New military doctrine

Demographic changes

Economic need

Advances in neural implants Technology push

Demand pullRaw curiosity


An Agent May Be Embodied In A Robot…


Dependent Autonomous


Fixed place Mobile


Thing-facing Human-facing


Predictable Responsive Social


Architecturally, robots may be classified along one of several dimensions


Biologically-organized

Symbolically-organized


Thin Client Thick Client


Thin Client Thick Client

By way of context, Watson is largely a symbolically-organized thin client cognitive system

Biologically-organized

Symbolically-organized


An Avatar…


A Space…


Or An Object


EmbodiedCognition

17

UseCases

Concierge(Robot, Avatar,

Space)

Whereistheelevator?

Retail(Robot, Avatar,

Space)

Doyouwannabuildasnowman?

Elder Care(Robot, Avatar,

Space)

I’vefallenandIcan’tgetup!

Cobot(Robot)

Getmeascrewdriver.Manufacturing

(Robot)

Watchmedothis.

Transportation(Robot, Avatar,

Space)

Openthepodbaydoors,Watson.

Boardroom(Avatar, Space)

Helpmedecide. Companion(Avatar, Device)

Let’splayagame.


Self

18

Sensors&Percep,on

Actuators

Agents

Goals&Planning

Pla5ormManagement Infrastructure

MetaManagement

Models

VoluntaryBehavior&Skills

InvoluntaryBehavior&Reflexes

Others

World

Self

Architecture


PointsofTechnicalConfluence

19

Collaboration

Whosaidthat?

Sensory Fusion

Ifeelsad.

Theory of Mind

IneedthePhillipsheadscrewdriver.

Context

Givemethatone;no,Imeanthatone!

Embodied Conversation

It’sgoodtoseeyouagain,Sandia!

Learning

I’vejustpickedupafaultintheAE35unit.

Devices

InwhatroomwasAlyssaworkingyesterday?

Knowledge Representation


EmbodiedCognition

§ Team‒ Leader:GradyBooch;Co-Leader:JeffKephart‒ KeyGlobalTeamMembers:SamAdams,RachelBellamy,RayChancey,JonConnell,StefanHamer,Kohichi Kajitani,SkyMatthews,NatMills,RobertMoore,AntonRiabov,FrancescaRossi,MartinRufli,Ryuki Tachibana,HironobuTakagi,JohnVergo

§ Theme‒ DeliveraplatformthatmakesthecognitivepowerofWatsonmanifestintherealworldwithsensorsandactuators,servingasanembodiedpersonathatorchestratestheagencyofthatsystemtoaugmentandcollaboratewithhumansusingvisual,vocal,andtactileformsofcommunication

‒ Deploythisplatformintheformofrobots,avatars,spaces,andobjects‒ Focusonhigherlevelsofreasoningandlearning‒ Enablepluggableagents,goals,plans,skills,reflexes,classifiers,andmodelssoastofacilitateavibrantdeveloperandsolutionecosystem

§ ShortTermGoal(oneyear)‒ GotomarketwiththeWatsonGroupandWatsonIoT,deliveringtheSelfkernel,services,andexamplar solutions.

§ LongTermGoal(threeyears)‒ Becomethedefactoindustrystandardforhigherorderembodiedcognitivesolutions

§ Gaps&Issues‒ There’smoreworktobedonethanIBMhasskillsorresourcestodoalone‒ Maniacalfocusisessential:therearemanymovingpartsthatcutacrossorganizationsboundaries

20

ExecutiveSummary

Sensors&Percep,on

Actuators

Agents

Goals&Planning

Pla5ormManagement Infrastructure

MetaManagement

Models

VoluntaryBehavior&Skills

InvoluntaryBehavior&Reflexes

Others

World

Self

Self


EmbodiedCognitionCore Technology Topic Project Name Contact/Lab ~ Team SizePlatform Self Booch/Almaden & Chancey/Austin 1 + 4

Goals & Planning

Planning Riabov/Yorktown 2

Embodied Action Moore/Almaden 1

Multi-modal Contextual Interaction Bellamy/Yorktown 5

Embodied Decision Support Systems Rossi/Yorktown 5

Models of the World, Others, and Self

Spatial Intelligence Rufli/Zurich 4

People Intelligence Hoory/Haifa 5

Spatial Awareness Kephart/Yorktown 4

Theory of Mind TBD

Learning Learning by Example Connell/Yorktown 2

Sensors & Perception/ActuatorsPhysical Interaction Tachibana/Tokyo 5

Edge Intelligence Hamer/Australia 2

Platform & Meta Management Tooling TBD

21

Solution Area Project ContactSpaces Watson for Buildings/Watson for Retail TBD/cross-laboratory

Robots, Avatars, & Spaces Watson Labs Chancey/Austin

Robots & Spaces Watson IoT Matthews/Durham

Robots Industrial Robots Kajitani/Tokyo

Spaces Elder Care Takagi/Tokyo & Adams/Raleigh

Spaces M&A Vergo/Yorktown

Spaces Boson Mills/Cambridge

Objects Cognitive Objects Bellamy/Yorktown

ProjectPortfolio


EmbodiedCognition

22

Collaborators

Solution Area Project ContactRobots Cooperative Robots Carnegie Mellon University

Spaces Multimodal Interaction Rensselaer Polytechnic Institute

Avatars Baby/X University of Auckland

Robots TBD University of California, San Diego

Robots TBD University of California, Berkeley

Robots TBD Massachusetts Institute of Technology

Robots TBD Stanford Research Institute


Background

§ In November 2015, Rob High, Ray Chancey, and I were brainstorming about a suitable grand challenge for the 2016 Embodied Cognition GTO.

§ We settled on a very grand one: build a system to support mission control, flight operations, and the scientific agenda of NASA’s Mission to Mars; the requirements of this mission lead to the concept of Watson in the walls (an embodied cognition for the Orion spacecraft and the Mars habitat) together with robots and other devices powered by Watson (namely, Robonaut and its peers).

§ Recognizing this was a 20+ year research effort, I set out to devise an architecture that would not only support this Mission to Mars but the journey of which would yield a middleware that would attend to a variety of use cases, from concierge to retail to elder care and many others.

§ Ray named this architecture “Self”.


Self Basic Principles

§ Augment human capabilities

§ Learn, don’t program

§ Theory of mind and social intelligence

§ Self understanding

§ Platform agnostic

§ Embodiment as a robot, an avatar, a device, or a space

§ Deployment as middleware with microservices in the cloud


Self Significant Design Decisions

§ Self is a hybrid architecture, encompassing explicit symbolic computation in the center together with neural networks at the edges.

§ Inspired by Minsky’s Society of Mind, behavior takes place in the context of multiple concurrent agents that communicate opportunistically via blackboards and deterministically via peer to peer connections.

§ Inspired by Brooks’s subsumption architecture, behavior takes place in a hierarchy of cognition, from involuntary reflexes to voluntary skills to goals and planning.

§ We maintain a clear separation of concerns among perception, actuating, models, and behavior.

§ As much as possible, behavior is either taught or is learned, not programmed.

§ As much as possible – driven by these separation of concerns, the needs of packaging, and performance – all components are made manifest as RESTful microservices.

§ As much as possible, plans, skills, and reflexes are extensible.

§ Self is intentionally full of strange loops: components of Self are also parts of the models of itself.

§ Self is intentionally fractal: an instance of Self may have models of others, which themselves are other instances of Self.


Self Markitecture

IBM Confidential 26

Sensors & Perception

Actuators

Agents

Goals & Planning

Platform Management Infrastructure

Meta Management

Models

Voluntary Behavior & Skills

Involuntary Behavior & Reflexes

Others

World

Self


Embodied cognition represents the next generation of Watson


Embodied cognition represents the extension of IT into the physical world, bringing cognition to the edge of the Internet of Things


At this touch point we encounter the intersection of big data and human presence, mediated by cognitive systems that reason


Text

Watson can be the back end to a variety of embodied cognitive systems.


Text + Audio


Text + Audio

This is the current state of an embodied Watson(the low hanging fruit, wherein platforms such as Pepper serve as animated speakers)


Text + Audio

There is much, much more that is possible(if we extent the embodiment to encompass greater perception and action in the world)


Text + Audio + Vision


Text + Audio + Vision + Emotion


Text + Audio + Vision + Emotion + Motion


Text + Audio + Vision + Emotion + Motion + Physical interaction


Text + Audio + Vision + Emotion + Motion + Physical interaction

Socially-intelligent cooperative robot