reverse engineering brain

Author: francis-clinton-prashanth

Post on 04-Apr-2018

220 views

Category:

Documents


0 download

Embed Size (px)

TRANSCRIPT

  • 7/29/2019 Reverse Engineering Brain

    1/52

    Krasnow Institute for Advanced Studies -- George Mason University

    James Albus

    Senior FellowKrasnow Institute for Advanced Studies

    George Mason University

    Founder & Chief (Retired) Intelligent Systems DivisionNational Institute of Standards and Technology

    [email protected]

    An Engineering Perspectiveon

    Reverse Engineering the Brain

  • 7/29/2019 Reverse Engineering Brain

    2/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Outline

    An Engineering Viewpoint

    The Neuroscience

    Reverse Engineering the Brain

  • 7/29/2019 Reverse Engineering Brain

    3/52

    Krasnow Institute for Advanced Studies -- George Mason University

    65-75 NASA-NBS -- Cerebellum model for learning control (CMAC neural net)73-85 Navy/ NBS Robot control, Automated Manufacturing Research Facility86-87 DARPA -- Multiple Unmanned Undersea Vehicles (MAUV)88-89 DARPA -- Submarine Operational Automation System (SOAS)90-92 GD Electric Boat -- Next generation nuclear submarine control86-88 NASA -- Space Station Flight Telerobotic Servicer (NASREM)

    87-89 Bureau of Mines -- Coal mine automation87-91 U.S. Postal Service -- Stamp distribution center, General mail facility86-08 Army -- TEAM, TMAP, MDARS, Picatinny Arsenal UGV, Demo I and III ARLCollaborative Technology Alliance, JAUGS, VTA, FCS-ANS96-97 Navy Double Hull Robot, Multiple UAV SWARM94-95 DARPA / General Motors Enhanced CNC & CMM Control99-01 Boeing Cell Control, Riveting, Hi Speed machine tool92-01 Commercial CNC - plasma & water jet cutting96-98 DARPA MARS, PerceptOR02-04 Boeing/SAIC FCS Autonomous Navigation System, Integrated Combat Demo02-07 AirForce RoboCrane Paint Stripping Robot for Large Aircraft08-09 DOT Intelligent vehicles, Foveal-Peripheral Vision for Driving06-07 DARPA Learning Applied to Ground Robotics (LAGR)

    08-10 DARPA EATR Foraging Robot

    Intelligent Systems EngineeringIntelligent Control Projects ~ $100M total over 43 years

  • 7/29/2019 Reverse Engineering Brain

    4/52

    Krasnow Institute for Advanced Studies -- George Mason University

    HWS workstation

    Intelligent Machining Workstation

    circa 1981

  • 7/29/2019 Reverse Engineering Brain

    5/52

    Krasnow Institute for Advanced Studies -- George Mason University

    CDBWS workstation

    Intelligent Cleaning andDeburring Workstation

    circa 1982

  • 7/29/2019 Reverse Engineering Brain

    6/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Intelligent Coal Mining Machine

    circa 1988

  • 7/29/2019 Reverse Engineering Brain

    7/52Krasnow Institute for Advanced Studies -- George Mason University

    MAUV pics

    Multiple Autonomous Undersea Vehicles

    circa 1989

  • 7/29/2019 Reverse Engineering Brain

    8/52Krasnow Institute for Advanced Studies -- George Mason University

    Intelligent Vehicle Control

    circa 1993

  • 7/29/2019 Reverse Engineering Brain

    9/52Krasnow Institute for Advanced Studies -- George Mason University

    NIST Autonomous Mobility Team

  • 7/29/2019 Reverse Engineering Brain

    10/52Krasnow Institute for Advanced Studies -- George Mason University

    4D/RCS Reference Model Architecturefor Unmanned Vehicle Systems

    Hierarchical structureof goals and commands

    Representation of the worldat many levels

    Planning, replanning,and reacting at many levels

    Integration of many sensorsstereo CCD & FLIR, LADAR,radar, inertial, acoustic, GPS,internal

    Adopted by GDRS for FCS Autonomous Navigation SystemAdopted by TARDEC for Vetronics Technology Integration

  • 7/29/2019 Reverse Engineering Brain

    11/52Krasnow Institute for Advanced Studies -- George Mason University

    4D/RCS Reference Model

    O P E R A T O R I N T E R F A C E

    SPWM

    BG

    SP WM BG

    SP WM BG

    SP WM BG

    Points

    Lines

    Surfaces

    SP WM BG SP WM BG

    SP WM BG

    0.5 second plansSteering,velocity

    5 second plansSubtask on object surfaceObstacle-free paths

    SP WM BGSP WM BG

    SP WM BGSP WM BG SP WM BG

    SERVO

    PRIMITIVE

    SUBSYSTEM

    SURROGATE SECTION

    SURROGATE PLATOON

    SENSORS AND ACTUATORS

    Plans for next 2 hours

    Plans for next 24 hours

    0.05 second plansActuator output

    SP WM BGSP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG

    Objects of attention

    LocomotionCommunication Mission Package

    VEHICLE Plans for next 50 secondsTask to be done on objects of attention

    Plans for next 10 minutesTasks relative to nearby objects

    Section Formation

    Platoon Formation

    Attention

    Battalion Formation SURROGATE BATTALION

    6

    Intelligent System Architecture

    Spinal MotorCenters

    MidbrainCerebellum

    PrimarySensory-MotorCortex

    CorticalSensory-MotorHierarchy

    Small groups

    SituationsEpisodes

  • 7/29/2019 Reverse Engineering Brain

    12/52Krasnow Institute for Advanced Studies -- George Mason University

    A 4D/RCS Computational Node

    4D/RCSnode

    FrontalCortex

    PosteriorCortex

    Mapping to the Brain

  • 7/29/2019 Reverse Engineering Brain

    13/52Krasnow Institute for Advanced Studies -- George Mason University

    What is the Goal?

    The Engineering GoalTo build machines that DO what the brain does

    A Scientific GoalTo understand HOW the brain does what it does.

    A second Scientific Goa lTo understand how the brain LEARNS to do what it does

  • 7/29/2019 Reverse Engineering Brain

    14/52Krasnow Institute for Advanced Studies -- George Mason University

    Front to back:Behavior generation in frontSensory processing in back

    Overall Structure of Brain

    Side to side:Representation of right egosphere on left sideRepresentation of left egosphere on right side

    Top to bottom:Conscious self at topSensors and muscles at bottom

    At the center:Emotions, Appetites, & Internal state

  • 7/29/2019 Reverse Engineering Brain

    15/52Krasnow Institute for Advanced Studies -- George Mason University

    What is the brain for?

    Fine manipulation, language, and reasoningare recent developments

    Early evolution => control of locomotion

    Swimming motion & gait generation coordination of actuators

    Path planning how to get from A to B

    Decision making where to go, when, why, how

    Tactical behaviors hunting for food, evading predators, . . .

    Strategic behaviors migrating, establishing territory, mating, . . .

    E v ol u

    t i on

    The brain is first and foremost a control system

  • 7/29/2019 Reverse Engineering Brain

    16/52Krasnow Institute for Advanced Studies -- George Mason University

    Gravity sensors establish the horizontal planefor an internal egosphere representation

    Body kinematics measured by proprioception

    Tactile input

  • 7/29/2019 Reverse Engineering Brain

    17/52Krasnow Institute for Advanced Studies -- George Mason University

    What are the Outputs?

    forces and velocities in the limbs and torso

    Behavior consisting of:

    goal-driven tasks and subtasks on objects in the world

    control signals to muscles

    Behavior that has many levels of resolution in: feedback error correction feed-forward control

    planning and coordination

    Behavior consistent with goals that aregenerated in the frontal cortex by processes that use:

    a rich internal model of the external world an internal model of body kinematics and dynamics an internal representation of needs and desires

  • 7/29/2019 Reverse Engineering Brain

    18/52Krasnow Institute for Advanced Studies -- George Mason University

    Hierarchical ArchitectureBrain is organized hierarchically

    Frontal hierarchy: decision making, goal selection,priority setting, planning and execution of behavior

    Posterior hierarchy: attention, segmentation, grouping,computing attributes, classification, establishing relationships

    Unitary SELF at topMillions of sensors and actuators at bottom

    Complex strategies at topSimple actions at bottom

  • 7/29/2019 Reverse Engineering Brain

    19/52Krasnow Institute for Advanced Studies -- George Mason University

    Cortex and Mind

    Joaquin Fuster

    The brain is ahierarchical

    signal processing

    & control system

    Cortical Architecture

  • 7/29/2019 Reverse Engineering Brain

    20/52Krasnow Institute for Advanced Studies -- George Mason University

    Cortex and Mind

    Joaquin Fuster

    Moreneurons

    at the top

    Hierarchical Architecture

    Brainhierarchy

    is not apyramid

  • 7/29/2019 Reverse Engineering Brain

    21/52Krasnow Institute for Advanced Studies -- George Mason University

    Computational Mechanisms

    Synapse is an electronic gate-- complex biochemistry, site of long-term memory

    Neuron is a computational element-- non-linear processes on many inputs, & decide

    Neural Cluster is a functional unit

    -- arithmetic or logical operations, correlation, convolution-- coordinate transformation-- finite-state automata-- rules, grammar, direct and indirect addressing

  • 7/29/2019 Reverse Engineering Brain

    22/52Krasnow Institute for Advanced Studies -- George Mason University

    Neural Clusters in Spinal Cord

    Anteriorhorn

    Posteriorhorn

    Posteriornucleus

  • 7/29/2019 Reverse Engineering Brain

    23/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Input OutputCommand & feedback

    AddressAddress

    If (Situation)

    ActionContentsPointerThen (Consequent)

    Anatomical structure

    Random access table-look-up computationwith generalization

    Marr 1969, Albus 1971

    Functional structure

    Neural Clusters in Midbrain(e.g. Cerebellum)

  • 7/29/2019 Reverse Engineering Brain

    24/52

    Krasnow Institute for Advanced Studies -- George Mason University

    General Functional Model

    S(t) P(t + D t) = H(S(t))

    memory storage & recall, arithmetic or logical functions,IF/THEN rules, goal-seeking reactive control,

    forward & inverse kinematics, direct & indirect addressing

    Input vector

    or array

    Output vector

    or array

    Feedback forLearning

    Neural Cluster

  • 7/29/2019 Reverse Engineering Brain

    25/52

    Krasnow Institute for Advanced Studies -- George Mason University

    differential and integral functions, dynamic models, phase-lock loops,time and frequency analysis, recursive estimation, Kalman filtering

    S(t) P(t + D t) = H(S(t))

    Functional Model + FeedbackFeedback for

    Learning

    Neural Cluster

  • 7/29/2019 Reverse Engineering Brain

    26/52

    Krasnow Institute for Advanced Studies -- George Mason University

    A Neural Finite State Automaton

    Markov processes, scripts, plans, behaviors, grammars,Bayesian networks, semantic nets, narratives

    S(t) P(t + D t) = H(S(t))

    State Next state

    Feedback for

    Learning

    Neural Cluster

  • 7/29/2019 Reverse Engineering Brain

    27/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Cortical Structure

    Cortex is a 2D sheet 2000 cm 2 area x 3 mm thick

    Each region is segmented into arrays of columns

    Regions are arranged in hierarchical layers

    Cortical sheet is partitioned into functional regions

    Each column has capabilities of a fsa + memory

  • 7/29/2019 Reverse Engineering Brain

    28/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Circuit diagramof visual system

    in brain12 layers32 areas

    Each area is an array of

    Cortical Columns

    Felleman &van Essen 1991

    Each area represents theVisual Field of Regard

  • 7/29/2019 Reverse Engineering Brain

    29/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Cortical Column Structure

    Microcolumns 100 250 neurons30 50 diameter, 3000 long

    Posterior: detect patterns, compute attributes

    Frontal: evaluate alternatives, recommend actions

    Hypercolumns (a.k.a. columns)100+ microcolumns in a bundle

    500 in diameter, 3000 long Posterior: segmentation, grouping, classification, relationships Frontal: set goals, make plans, control action

    There are about 10 6 hypercolumns in human cortex

  • 7/29/2019 Reverse Engineering Brain

    30/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Communication in the BrainAxon is an active fiber connecting one neuron to others

    (transmits a scalar variable on a publish-subscribe network with bandwidth ~ 500 Hz)

    Two kinds of axons:

    Drivers Preserve topology and local sign Data vectors or arrays of attributes and state-variablesi.e., images, objects, events, attributes and state -- e.g., color, shape, size, position, orientation, motion

    Modulators Dont preserve topology or local sign Context & broadcast variables, addresses, and pointerse.g., select & modify algorithms, set parameters, define relationships

    Exploring the Thalamus

    Sherman & Guillery 2006

  • 7/29/2019 Reverse Engineering Brain

    31/52

    Krasnow Institute for Advanced Studies -- George Mason University

    EarlyCorticalVision

    ProcessesCorticalColumns

    in V1

    +LateralGeniculate

    in Thalamus

    Hypercolumn

    Modulator Input

    Modulator Output toother cortical regions

    Input from lgn

    Driver Outputto Higher Level & superior colliculus

    Modulator OutputBack to lgn

    Microcolumn

    Receptivefield

    DiffuseFibers(Modulators)

    PixelAttributes(Drivers)

    C i l H l Th l i l

  • 7/29/2019 Reverse Engineering Brain

    32/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Cortical Hypercolumn + Thalamic loop

    windowing, segmentation, grouping, computing group attributes &state, filtering, classification, setting and breaking relationships

    CorticalComputational

    Unit

    (CCU)

    drivers = attribute vectors& arrays

    modulators = broadcast variables,& address pointers

    Receptive field

  • 7/29/2019 Reverse Engineering Brain

    33/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Drivers(data)

    Modulators(addresses)

    CCUOutputs

    CCU Data Structure Hypothesis

  • 7/29/2019 Reverse Engineering Brain

    34/52

    Krasnow Institute for Advanced Studies -- George Mason University

    drivers = attribute vector array

    Cortico-Thalamic Loop

    12

    3456

    t

    modulators = address pointers

    windowingsegmentation & groupingcompute group attributes

    recursive filteringclassification

    corticalhypercolumnthalamic

    nucleus

    A Cortical

    ComputationalUnit(CCU)

  • 7/29/2019 Reverse Engineering Brain

    35/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Posterior

    Cortico-ThalamicLoop Hierarchy

    windowingsegmentation & groupingcompute group attributes

    recursive filteringclassification

    at each level

  • 7/29/2019 Reverse Engineering Brain

    36/52

    Krasnow Institute for Advanced Studies -- George Mason University

    1. Receptive field hierarchies

    Two types of hierarchies

    Receptive field hierarchies are defined by driveranatomical connectivity and are relatively fixed

    2. Entity and Event hierarchies

    Entity and event hierarchies are defined bymodulator activity that can establish or break

    belongs-to and has-part pointers in ~ 10 ms

  • 7/29/2019 Reverse Engineering Brain

    37/52

    Krasnow Institute for Advanced Studies -- George Mason University

    CCU Receptive Field Hierarchy

    Defined by driver neurons flowing up the processing hierarchy

  • 7/29/2019 Reverse Engineering Brain

    38/52

    Krasnow Institute for Advanced Studies -- George Mason University

    CCUEntity/Event

    HierarchyDefined by pointers

    result of segmentation& grouping processes

    Pointers link pixelsto symbols & vice versa

    Provides symbol grounding

    Pointers resettop to bottom within

    a saccade ~ 150 ms

  • 7/29/2019 Reverse Engineering Brain

    39/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Segmentation& Grouping

    ProcessEach level detects

    patterns within itsreceptive field&

    sets or breaks pointers

    This produces anEntity/Event

    Hierarchy

    CCU C di t F

  • 7/29/2019 Reverse Engineering Brain

    40/52

    Krasnow Institute for Advanced Studies -- George Mason University

    CCU Coordinate Frames

    Each level has multiplecoordinate frames

    Oculomotor signals,vestibular inputs, &

    range estimates providetransform parameters

    Coordinate transformscomputed in parallel

    at each level in ~ 10 ms

  • 7/29/2019 Reverse Engineering Brain

    41/52

    Krasnow Institute for Advanced Studies -- George Mason University

    World CenteredHierarchy of

    Entity PointersCoordinate frame is fixed

    in the world

    Entities have continuityin space & time

    Entities have state position, orientation, velocity

    Entities have attributes-- size, shape, color, texture, behavior

    Entities have relationships-- class, rank, spatial, temporal, causal

  • 7/29/2019 Reverse Engineering Brain

    42/52

    Krasnow Institute for Advanced Studies -- George Mason University

    NAMEattributesstate

    pointers

    9876543210123456789

    belongs-tohas-part1has-part2class1class2

    shape

    sizecolor behavior

    positionmotion

    observed states expected states

    EntityFrame

    Temporal Continuity for an Entity

    Each CCU containsan entity frame

    Each entity frame

    contains:a Trace of O bserved States&a Prediction of E xpected States

  • 7/29/2019 Reverse Engineering Brain

    43/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Building computational machines that are

    functionally equivalent to the brain in their ability to perceive, think, decide, and act in a purposeful

    way to achieve goals in complex, uncertain, dynamic, and possiblyhostile environments, despite unexpected events and unanticipated

    obstacles, while guided by internal values and rules of conduct.

    Reverse Engineering the Brain

    Functional equivalence ::= producing the sameinput/output behavior

    What does that mean?

    R E i i th B i

  • 7/29/2019 Reverse Engineering Brain

    44/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Will require a deep understanding of how thebrain works and what the brain does

    How does the brain generate the incredibly complexcolorful, dynamic internal representation that

    we consciously perceive as external reality?

    Reverse Engineering the Brain

    How are signals transformed into into symbols?

    How are images processed?How are messages encoded?

    How are relationships established and broken?

    What are the knowledge data structures?

    What are the functional operations?

    How is information represented in the brain?How is computation performed?

  • 7/29/2019 Reverse Engineering Brain

    45/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Reverse Engineering the BrainCited as a Grand Challenge by

    U.S. National Academy of Engineering

    Decade of the Mind InitiativeA proposed 10 year $4B program

    to understand the mechanisms of mind Krasnow Lead

    Steering Committee of Top ScientistsRecent workshops at:

    Sandia National Labs Jan 13 15, 2009Berlin Sept 10 12, 2009

    Singapore October 18-20, 2010

    Focus of DARPA SyNAPSE & NEOVISION2, IARPA ICArUS,& European Union FACETS & POETICON Programs

    Wh N ?

  • 7/29/2019 Reverse Engineering Brain

    46/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Why Now?

    Neurosciences computation and representation in the brain- biochemistry, synaptic transmission, brain imaging, neuron modeling- neuroanatomy, neurophysiology, network & whole brain models

    Cognitive Modeling representation and use of knowledge- mathematics, logic, language, learning, problem solving- psychophysics, cognitive psychology, functional brain modeling

    Intelligent Control making machines behave appropriately- control theory, cybernetics, AI, knowledge representation, planning- manipulation, locomotion, manufacturing, vehicles, weapons

    Computational Power speed and memory that rival the brain- supercomputer = 1015 ops today, laptop > 10 15 ops in 20 years

    The science & technology is ready

    Progress is rapid, an enormous literature

  • 7/29/2019 Reverse Engineering Brain

    47/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Computational Power isApproaching a Critical Threshold

    ComputingPower

    (ops)

    Estimatedcomputing power

    of human brain:~10 13-10 16 ops

    2005 2010 2015 2020 2025 2030 203510 1010

    11

    10 1210 1310 14

    10 1510 16

    Moores LawComputational power

    x10 every 5 years

    Single chip

    MultiCore

    Supercomputer

    teraflop chip

    Road Runner

  • 7/29/2019 Reverse Engineering Brain

    48/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Military FCS, UGV, UAV, UUV, USV, UGSCommercial manufacturing, transportationEntertainment video games, cell phonesAcademic neuroscience, computer science

    Intelligent Machines Will Be Critical forMilitary Security and Economic Prosperity

    Money Is Flowing

    Billions of $ will be invested over the next decade

    h h h

  • 7/29/2019 Reverse Engineering Brain

    49/52

    Krasnow Institute for Advanced Studies -- George Mason University

    overall system level (central nervous system) arrays of macro-computational units (e.g., cortical regions) macro-computational units (e.g., cortical hypercolumns & loops) micro-computational units (e.g., cortical microcolumns & loops) neural clusters (e.g., spinal and midbrain sensory-motor nuclei) neurons (elemental computational units) input/output functions synapses (electronic gates, memory elements) synaptic phenomena membrane mechanics (ion channel activity) molecular phenomena

    What is the path to successfor reverse engineering the brain?

    Pick the right level of resolution

    AI and Cognitive Psychology

    Mainstream Neuroscience & Neural Nets

    CCUs

  • 7/29/2019 Reverse Engineering Brain

    50/52

    Krasnow Institute for Advanced Studies -- George Mason University

    State of art supercomputer running at 10 15 fops

    provides 10 million fops per CCU modeling cycle

    106 CCUs running at 100 Hz requires108 CCU modeling cycles per second

    Estimated communication load between CCUs106 bytes per second for each CCU, or

    1012

    bps for full brain modelThis appears to be within the state of the art

    for current supercomputers

    Computational Requirementsfor Engineering Human Brain at CCU level

    S & C l i

  • 7/29/2019 Reverse Engineering Brain

    51/52

    Krasnow Institute for Advanced Studies -- George Mason University

    Summary & Conclusions

    Near term success will require selecting the rightlevel of resolution, e.g. CCU level

    The science and engineering can support it

    Reverse Engineering the Human Brain

    appears feasible in the near term

    Real-time modeling at CCU level of resolution

    appears achievable now with supercomputersand in ~ 20 years with laptop class computers

    The impact will be revolutionary

    The benefits will justify the investment

  • 7/29/2019 Reverse Engineering Brain

    52/52

    Thank you

    Questions?

    [email protected]