![Page 1: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/1.jpg)
Complementary Learning Systems in Natural and Artificial Intelligence
James L. McClellandDepartment of Psychology &
Center for Mind, Brain and Computation
Stanford University
![Page 2: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/2.jpg)
Tom’s questions for me
• What sort of NN architectures could serve an automated programmer in constructing a program?
• How do you imagine different memory systems working in a human programmer?
![Page 3: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/3.jpg)
Outline for the session
• Complementary learning systems
– The basic theory
– Rapid schema consistent learning
– Comparison of the two learning systems
• Deep learning and complementary learning systems
– Rehearsal buffer in the DQN
– Memory based parameter adaptation
• Revisiting Tom’s prompt and a response
![Page 4: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/4.jpg)
Your knowledge is in your connections!
• An experience is a pattern of activation over neurons in one or more brain regions.
• The trace left in memory is the set of adjustments to the strengths of the connections.
– Each experience leaves such a trace, but the traces are not separable or distinct.
– Rather, they are superimposed in the same set of connection weights.
• Recall involves the recreation of a pattern of activation, using a part or associate of it as a cue.
• The reinstatement depends on the knowledge in the connection weights, which in general will reflect influences of many different experiences.
• Thus, memory is always a constructive process, dependent on contributions from many different experiences.
![Page 5: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/5.jpg)
Effect of a HippocampalLesions
• Intact performance on tests of intelligence, general knowledge, language, other acquired skills
• Dramatic deficits in formation of some types of new memories:– Explicit memories for
episodes and events– Paired associate learning– Arbitrary new factual
information
• Spared priming and skill acquisition
• Temporally graded retrograde amnesia:– lesion impairs recent
memories leaving remote memories intact.
Note: HM’s lesion was bilateral
![Page 6: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/6.jpg)
![Page 7: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/7.jpg)
Key Points
• We learn about the general pattern of experiences, not just specific things
• Gradual learning in the cortex builds implicit semantic and procedural knowledge that forms much of the basis of our cognitive abilities
• The Hippocampal system complements the cortex by allowing us to learn specific things without interference with existing structured knowledge
• In general these systems must be thought of as working together rather than being alternative sources of information.
![Page 8: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/8.jpg)
Effect of Prior Association on Paired-Associate Learning in Control and Amnesic Populations
Cutting (1978), Expt. 1
-20
0
20
40
60
80
100
Very Easy Easy Fairly Easy Hard Very Hard
Category (Ease of Association)
Pe
rce
nt
Co
rre
ct
Control (Expt)
Amnesic (Expt)
Base rates
![Page 9: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/9.jpg)
Kwok & McClelland Model ofSemantic and Episodic Memory
• Model includes slow learning cortical system and a fast-learning hippocampal system.
• Cortex contains units representing both content and context of an experience.
• Semantic memory is gradually built up through repeated presentations of the same content in different contexts.
• Formation of new episodic memory depends on hippocampus and the relevant cortical areas, including context.
– Loss of hippocampus would prevent initial rapid binding of content and context.
• Episodic memories benefit from prior cortical learning when they involve meaningful materials.
ContextRelation Cue
Target
Neo-Cortex
Hippocampus
![Page 10: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/10.jpg)
Simulation Results From KM Model
Cutting (1978), Expt. 1
84
0
70
9
0
68
-20
0
20
40
60
80
100
Very Easy Easy Fairly Easy Hard Very Hard
Category (Ease of Association)
Pe
rce
nt
Co
rre
ct
Control (Model)
Amnesic (Model)
Control (Expt)
Amnesic (Expt)
Base ratesin model
![Page 11: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/11.jpg)
Emergence of Meaning in Learned Distributed Representations through
Gradual Interleaved Learning
• Distributed representations (what ML calls embeddings) that capture aspects of meaning emerge through a gradual learning process
• The progression of learning and the representations formed capture many aspects of cognitive development
Progressive differentiation
– Sensitivity to coherent covariation across contexts
– Reorganization of conceptual knowledge
![Page 12: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/12.jpg)
![Page 13: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/13.jpg)
The Rumelhart Model
![Page 14: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/14.jpg)
The Training Data:
All propositions true of items at the bottom levelof the tree, e.g.:
Robin can {grow, move, fly}
![Page 15: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/15.jpg)
![Page 16: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/16.jpg)
Experience
Early
Later
LaterStill
![Page 17: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/17.jpg)
What happens in this system if we try to learn something new?
Such as a Penguin
![Page 18: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/18.jpg)
Learning Something New
• Used network already trained with eight items and their properties.
• Added one new input unit fully connected to the representation layer
• Trained the network withthe following pairs of items:
– penguin-isaliving thing-animal-bird
– penguin-cangrow-move-swim
![Page 19: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/19.jpg)
Rapid Learning Leads to Catastrophic Interference
![Page 20: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/20.jpg)
A Complementary Learning System in the Medial Temporal Lobes
colorform
motion
action
valance
Temporal
pole
name
Medial Temporal Lobe
![Page 21: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/21.jpg)
Avoiding Catastrophic Interference with Interleaved Learning
![Page 22: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/22.jpg)
Initial Storage in the Hippocampus Followed by Repeated Replay Leads to the Consolidation of
New Learning in Neocortex, Avoiding Catastrophic Interference
colorform
motion
action
valance
Temporal
pole
name
Medial Temporal Lobe
![Page 23: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/23.jpg)
Rapid Consolidation of Schema Consistent Information
RichardMorris
![Page 24: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/24.jpg)
Tse et al (Science, 2007, 2011)
During training, 2 wellsuncovered on each trial
![Page 25: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/25.jpg)
Schemata and Schema Consistent
Information
• What is a ‘schema’?– An organized knowledge
structure into which existing knowledge is organized.
• What is schema consistent information?– Information that can be
added to a schema without disturbing it.
• What about a penguin?– Partially consistent– Partially inconsistent
• In contrast, consider– a trout– a cardinal
![Page 26: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/26.jpg)
New Simulations
• Initial training with eight items and their properties as before.
• Added one new input unit fully connected to the representation layer also as before
• Trained the network on one of the following pairs of items:
– penguin-isa & penguin-can– trout-isa & trout-can– cardinal-isa & cardinal-can
![Page 27: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/27.jpg)
New Learning of Consistent and Partially Inconsistent Information
INTERFERENCELEARNING
![Page 28: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/28.jpg)
Connection Weight Changes after Simulated NPA, OPA and NM Analogs
Tse Et al 2011
![Page 29: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/29.jpg)
How Does It Work?
![Page 30: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/30.jpg)
How Does It Work?
![Page 31: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/31.jpg)
Comparison of the two learning systems
![Page 32: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/32.jpg)
Dense vs Sparse Coding
• Pattern separation:
– Sparse randomconjunctive coding
![Page 33: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/33.jpg)
Similarity Based Representations in Cortex
![Page 34: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/34.jpg)
In more detail…
• Input from neocortex comes into EC; EC projects to DG, CA3, and CA1
• Drastic pattern separation occurs in DG
• Downsampling in CA3 assigns an arbitrary code
• Invertable somewhat sparsifiedrepresentation in CA1
• Fewish-shot learning in DG, CA3, CA3->CA1 allows reconstruction of ERC pattern from partial input.
• Other connections shown in black are part of the slow-learning neocortical network.
• Recurrence within CA3, through the hippocampal circuit shown, and through the outer loop also involving the rest of the neocortex
![Page 35: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/35.jpg)
Two modes of generalization
• Parametric vs. Item-based
• As long as the embeddings are already known, these modes can both support generalization
• The hippocampus can do so without requiring interleaved learning
• Adapting the embeddings may be relatively hard
ContextRelation Cue
Target
Neo-Cortex
Hippocampus
![Page 36: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/36.jpg)
How might hippocampus support inference and generalization?
‘Inference’
• Finding missing links in the transitive inference task
![Page 37: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/37.jpg)
Complementary Learning Systems in AI
• DQN • MBPA
![Page 38: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/38.jpg)
Tom’s questions for me
• What sort of NN architectures could serve an automated programmer in constructing a program?
• How do you imagine different memory systems working in a human programmer?
• My version of the question:
– What additional form of memory do intelligent agent’s need?
![Page 39: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/39.jpg)
Working Memory
• Is there a special working memory system in the brain?
• Or do we learn connection weights that sustain information an active state in memory?
• RNNs and LSTMs provide forms of working memory
• What is exciting about these models is that they learn what to retain
– We learn to retain the information that will be useful later
![Page 40: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/40.jpg)
The Differentiable Neural Computer
![Page 41: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/41.jpg)
Learning what to store – in two senses
![Page 42: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/42.jpg)
Memory Augmented Neural Networks
Santoro et al (2016) One-shot learning with MANNs
![Page 43: Complementary Learning Systems in Natural and Artificial ... · Complementary Learning Systems in Natural and Artificial Intelligence James L. McClelland Department of Psychology](https://reader033.vdocuments.net/reader033/viewer/2022042220/5ec60b2d2618c5026750beec/html5/thumbnails/43.jpg)
Some closing comments
• Cognitive Science, Neuroscience, and AI now have increasingly powerful ideas that we can use to help us understand learning and memory
• AI has expanded the space of what we can consider to be learned rather than innate
• But currently, AI breakthroughs are drastically over-compartmentalized
• We can use meta-learning to teach a neural network just about anything
• But there’s little generalization outside of a limited meta-task space
• And there’s very little fully integrative work going on, allowing a single integrated learner to acquire a range of skills all of which can be brought together to solve the problem of general artificial intelligence