adaptive cooperative systems chapter 8 synaptic plasticity 8.11 ~ 8.13 summary by byoung-hee kim...
TRANSCRIPT
Adaptive Cooperative SystemsAdaptive Cooperative Systems Chapter 8 Chapter 8
Synaptic PlasticitySynaptic Plasticity
8.11 ~ 8.13
Summary by Byoung-Hee KimBiointelligence Lab
School of Computer Sci. & Eng.
Seoul National University
(C) 2009 SNU CSE Biointelligence Lab
ContentsContents
8.11 Principal component neurons Introductory remarks Principal components and constrained optimization Hebbian learning and synaptic constraints Oja’s solution / Linsker’s model
8.12 Synaptic and phenomenological spin models Phenomenological spin models Synaptic models in the common input approximation
8.13 Objective function formulation of BCM theory Projection pursuit Objective function formulation of BCM theory
2
Goals and ContentsGoals and Contents
Goal: the information-processing functions of model neurons in the visual system
Contents Principal component neurons Special class of synaptic modification models Relation to phenomenological spin models Objective function formulation of BCM theory
(C) 2009 SNU CSE Biointelligence Lab 3
Introductory RemarksIntroductory Remarks
Images are highly organized spatial structures – some common statistical properties
Development of the visual system is influenced by the statistical properties of the images
knowledge of the statistical properties of natural scenes ~ understanding the behavior of cells in the visual system
(C) 2009 SNU CSE Biointelligence Lab 4
Scale Invariance in Natural ImagesScale Invariance in Natural Images
Studies of image statistics reveal non-preferrence of angular scale Decimation procedure with the grey-valued pixels of the image
assuming the role of the spins p.d.f. of image constrasts and image gradients are unchanged
(Field 1987), (Ruderman and Bialek 1994), (Ruderman 1994)
Representing the scale invariance through the covariance matrix Gives a constraint on the form of the covariance matrix Starting point for the PCA (Hancock, et al. 1992), (Liu and Shouval 1995), (Liu and Shouval
1996)
(C) 2009 SNU CSE Biointelligence Lab 5
Principal ComponentsPrincipal Components We are rotating the
coordinate system in order to find projections with desirable statistical properties
Projections: maximally preserve information content while compressing the data into a few leading components
(C) 2009 SNU CSE Biointelligence Lab 6
Variance of the data projected onto the axis is maximal
Principal Components and Constrained Principal Components and Constrained OptimizationOptimization
(C) 2009 SNU CSE Biointelligence Lab 7
n-component random vector
correlation matrix
If <X>=0, then the covariance matrix
Introducing a fixed vector that satisfies the normalization condition
use this to help us find interesting projections
Variance after operation:
• Optimization problem: find the vector a that satisfies the normalization condition, and maximizes the variance
The variance is equal to the eigenvalue
The maximum variance is given by the largest root
Hebbian Learning and Synaptic ConstraintsHebbian Learning and Synaptic Constraints
(C) 2009 SNU CSE Biointelligence Lab 8
T Ti i
i
c m d m d d m
The simplest form Hebb’s rule for synaptic modification
[Problem]Unstable. The synaptic weights would undergo unbounded growth
c: output activitym: synaptic weight vectord: input activity vector
On reaching a fixed point
m is an eigenvector of the input correlation matrix with eigenvalue equal to zero
Solutions for the Unbounded Growth ProblemSolutions for the Unbounded Growth Problem
Oja’s solution
Linsker’s model
(C) 2009 SNU CSE Biointelligence Lab 9
On reaching a fixed point
Results in a synaptic vector m for whichthe projection of the input activity has a maximum variance
Tm d
The synaptic system may be characterized as performing a principal component analysis of the input data
constraint on the total synaptic strength
Clipping- The sum of the synaptic weights are kept constant- each synaptic weight lies within a set range
Q kE E E
EQ: the variance in the input activity Ek: constraint
Properties of the Linsker’s ModelProperties of the Linsker’s Model
Stability corresponds to a global near minimum of the energy function Equivalent to the maximum in the input variance subject to the
constraint
Dynamics of the model system In different regimes for the parameters k1 and k2, different receptive field
structures dominate As k1 and k2 are varied, particular eigenvectors other than the principal one
gain in relative importance
(C) 2009 SNU CSE Biointelligence Lab 10
Q kE E E
Synaptic and Phenomenological Spin Synaptic and Phenomenological Spin ModelsModels
Theory on synaptic modification Model to explain the emergence of
these highly ordered repeating structures
(C) 2009 SNU CSE Biointelligence Lab 11
Phenomena Cells in the primate visual cortex self-organize onto ocular dominance
columns and iso-orientation patches The patterns observed experimentally are highly ordered
Phenomenological Spin ModelsPhenomenological Spin Models
2D Ising lattice of eye-specificity encoding spints (Cowan and Friedman 1991) Coupling strengths
If we take with , this type of coupling generates a short-range attraction plus a long-range repulsion between terminals from the same eye
Hamiltonian for iso-orientation
(C) 2009 SNU CSE Biointelligence Lab 12
2 2
| | | |exp expij
i j i jJ a a
2 2/ /a a
| || | cos( )ij i j i ji j
E J s s
(C) 2009 SNU CSE Biointelligence Lab 13
Synaptic Models in the Common Input Synaptic Models in the Common Input ApproximationApproximation Consider an LGN-cortico-cortico network with modifiable
geniculocortico synapses and fixed cortico-cortico-connections
Design of an energy function s.t. the fixed point of the network correspond to the minima of the energy function
The common input model by Shouval and Cooper hamiltonian in this model:
(C) 2009 SNU CSE Biointelligence Lab 14
T Ti i
i
c m d m d d m
general form
correlational hamiltonian
Information-processing Activities by Information-processing Activities by Common Input NeuronsCommon Input Neurons For exclusive excitatory connections
symmetry breaking does not occur all receptive fields have the same orientation selectivity
Inhibition affects both the organization and structure of the receptive fields If there is sufficient inhibition, the network will develop
orientation selective receptive fields
The cortical cells self-organize into iso-orientation patches with pinwheel singularities
(C) 2009 SNU CSE Biointelligence Lab 15
Objective Function Formulation of BCM Objective Function Formulation of BCM Theory - IntroTheory - Intro Distinguishment between information preservation
(variance maximization) and classification (multimodality)
(C) 2009 SNU CSE Biointelligence Lab 16
Projection PursuitProjection Pursuit
Projection pursuit a method for finding the most interesting low-dimensional features
of high-dimensional data sets The objective is to find orthogonal projections that reveal
interesting structure in the data PCA is a particular case with the proportion of total variance as
the index of interestingness Why is it needed? High-dimensional spaces are inherently sparse,
or “curse of dimensionality”
For classification purpose Interesting projection is one that departs from normalcy
(C) 2009 SNU CSE Biointelligence Lab 17
Objective Function Formulation of BCM Objective Function Formulation of BCM Theory (1/3)Theory (1/3) In the objective (energy) function formulation of BCM
theory, a feature is associated with each projection direction
A one-dimensional projection may be interpreted as a single feature extraction
Goal: to find an objective (loss) function whose minimization produces a one-dimensional projection that is far from normal
(C) 2009 SNU CSE Biointelligence Lab 18
Objective Function Formulation of BCM Objective Function Formulation of BCM Theory (2/3)Theory (2/3)
(C) 2009 SNU CSE Biointelligence Lab 19
Redefining the threshold function
Synaptic modification functions
How? Introduce a loss function
With some assumptions
Objective Function Formulation of BCM Objective Function Formulation of BCM Theory (3/3)Theory (3/3)
(C) 2009 SNU CSE Biointelligence Lab 20
The risk, or expected value of the loss,which is continously differentiable
We are able to minimize the risk by means of gradient descent w.r.t. mi
Slightly modified, deterministic version of the stochastic BCM modification equation
A BCM neuron is extracting third-order statistical correlates of the dataThis would be a natural extension of principal component processing in the retina
2 2
( )( ( ), ( )) ( ) (8.19)
( ) ( ( )) ( ( ) ( )) (8.22)
d tc t t t
dt
t c t t t
md
m d
M
M
Take-Home MessageTake-Home Message
(Tomasso Poggio, NIPS 2007 tutorial)
(C) 2009 SNU CSE Biointelligence Lab 21