(!!!!9PergamonNeuralNetwork,Vol. 10,No.4, pp. 705–720,1997

021997ElsevierScienceLtd.AllrightsreservedPrintedin GreatBritain

0893–6080/97$17.00+.00PII: S0893-6080(97)00020-8

CONTRIBUTEDARTICLE

Linsker-typeHebbianLearning:A QualitativeAnalysison the ParameterSpace

JIANFENGFENG1, HONG PAN2 ANDVWANI P. ROYCHOWDHURY2

‘Biomathematics Laboratory, The Babraham Institute, and ‘School of Electrical and Computer Engineering, Purdue University

(Received 30 March 1995; accepted 9 December 1996)

Abstract—Wedeveloped a new method to relate the choice of systemparameters to the outcomes of the unsupervisedlearning process in Linsker’smulti-layernetworkmodel.The behaviorof this model is determinedby the underlyingnonlineardynamicsthatareparameterizedby a set ofparameterson”ginatingfiomtheHebbruleandthearbordensityof thesynapses.Theseparametersdeterminethepresenceorabsenceofa specificreceptivefield (orconnectionpattern)as a saturatedfxedpoint attractorof themodel.Wederiveda necessa~ andsu$cient conditionto test whethera givensaturatedweightvectoris stableor notfor anygivenset of systemparameters,andusedthisconditionto determinethewholeregimein theparameterspaceoverwhichthegivenconnectionpatternis stable.Theparameterspaceapproachallows us to investigatethe relativestability of the major receptivefields reportedin Linsker’ssimulation,and todemonstratethe crucial roleplayed by the localizedarbordensi~ of synapsesbetweenadjacentlayers. The methodpresentedhere can be employedto analyzeother learningand retrievalmodels that use the limiterfinction as theconstraintcontrollingthe magnitudeof the weightor statevectors. O 1997ElsevierScienceLtd.

Keywords—UnsupervisedHebbianlearning,Networkself-organization,Linsker’sdevelopmentalmodel,Ontogenesisofprimaryvisualsystem,Afferentreceptivefield,Synapticarbordensity,Limitedfunction,Parameterspace.

1.1. INTRODUCTION

1.1. The Problem

Linsker(1986)proposeda multi-layernetworkmodelofselforganizationin theprimaryvisualsystem.The simu-lation results reported in Linsker (1986, 1988a,1988b)showthat: withoutany structuredinputand with appro-priate choicesof parameters,severalstructuredconnec-tionpatterns(e.g.,center-surroundandorientedreceptivefields)emergeprogressively,as theHebbianevolutionofthe weightsis carriedout layerby layer.To understandthisunsupervisedlearningscheme,severalmajoraspects

Acknowledgements: Some of these results were reported at the 1993IEEE International Conference on Neural Networks IProc. 1993 L?EEInt. ConJ onNeural Networks - SanFrancisco, VO1.111,pp. 1516–1521]and at the 1994 NIPS Conference, G. Tesauro, D. S. TouretzJcyand T.K. Leen (Eds.), (pp. 319–326) In Advances in Neural InformationProcessing Systems 7, (1995)]. The work of V.P. Roychowdhury andH. Parrwas supported in part by the General Motors Faculty Fellowshipand by the NSF Grant No. ECS-9308814. J. Feng was partiallysupported by the A.v. Humboldt Foundation. We thank theanonymous reviewers, David MaeKay and Zifi Liu for helpfulcomments on the manuscript.

Requests for reprints should he sent to Dr J. Feng, BiomathematicsLaboratory, The Babralmrn Institute, Cambridge CB2 4AT, UK;E-mail: jt218@cam.ac.uk.

705

of the dynamicalmechanismof Linsker’snetworkhavebeen establishedin Linsker(1986,1988b),MacKayandMiller (1990a, 1990b),Miller and MacKay (1994) andTang (1989)by conventionalapproaches.Going a stepfurther,we introducea dynamicallyinformativemethodthat aimsat studyingthe parameterspaceof this type ofmodel.

In fact, thebehaviorof the modelis determinedby theunderlyingnonlineardynamicsin the followingfor

w,+l(i)=f{wr(i) +kl + ~ [~ +Mr(j)w,(j)) U)j= 1

‘.f[wAO+~i(~7)l,Vi= 1,...,N~, (2)

wherefl.) is the limiterfinction (or the saturatinglinearfunction,or the piecewiselinear sigmoidalfunction)inthe form:

{

wmaxv if x > w-

f(x)= x, if 1x1= w~m , (3)

—‘max! if x < —w~~

and /Zi(.)is a continuousfunctionparametrized by a setof parametersoriginatingfrom the Hebb rule and thearbor density of the synapses (see Section 2.1 for a

706 J. Feng et al.

detailed explanationof the model). As establishedinLinsker(1986),the Hebb rule ensuresthat the elementsof theweightvectorwillincreaseordecreaseto theupperor the lowerbounds,therefore,the saturatedfiedpointattractors (i.e., the saturatedstableequilibriums)of eqn(2) represent the outcomes of Linsker-typeHebbianlearning.

The presenceor absenceof a specificreceptivefield(orconnectionpattern)asa saturatedfixedpointattractorof themodelis determinedbythechoiceof all thesystemparametersinvolved.Topredictwhichoneor whichonesof thesaturatedweightvectorswillbe attractor(s)underagivenchoiceof the systemparametersembodiedin hi(.),one has to study the structureof parameter space ofdynamics (2). In this paper (a detailed version of ashortletter(Fenget al., 1996)),we exploittheparameterspaceof system(2)to elucidatethebehaviorofdynamics(2) in generaland the outcomesof Linsker’snetworkin(1)as a casestudy.The parameterspaceapproachcan beappliedto analyzingotherlearningand memorymodels(e.g., Miller’s self-organizationmodel of the visualsystem (Miller, 1990a, 1990b, 1994, 1996; Miller &Stryker, 1989,Miller et al., 1989)and the Brain-State-in-a-Box(BSB) model (Andersonet al., 1977;Golden,1986,1993;Greenberg,1988;Hui&Zak, 1992))thatusethe limiterfunctionin (3).

1.2. The Approaches

Theconfigurationof theweightspaceofdynamicsin eqn(2) is determinedby the systemparametersincorporatedin hi(.).As observedin varioussimulationsof thistypeofmodel (Erwin et al., 1994;Linsker, 1986;Miller, 1996;Swindale, 1996)and as to be expectedfor a nonlineardynamicalsystem,a modelof the class statedin (2) canhave a group of coexistingattractorsfor a given set ofsystemparameters,and havedifferentgroupsof coexist-ing attractors for different sets of system parameters.Thus, a given connectionpattern could be an attractorof thedynamicsundercertainchoicesof thesystempara-meters, and could be unstableunder other choices.Thesystem parameters, thus, have state-relatedthresholds(also referred to as critical values of the parameters),which definethe stableand unstableparameterregimesin the parameter space for the correspondingweightpatterns. Hence, a characterizationof the parameterspace would enable one to predict the relationshipbetweenthe differentsets of systemparametersand thecorrespondingfixedpoint attractors.

Followingtheaboveidea,wedevelopa newmethodtorelatethe setof saturatedfixedpointsto the setof systemparametersof eqn (2). In particular,we showthat it ispossibleto derivea necessaryand sufficientconditiontotest whether a given weight vector is a saturatedfixedpoint attractorfor any given set of systemparameters,without loss of mathematicalrigor. In terms of thiscondition,one can assert the potential occurrence of a

fixedpointattractorwhenthe systemparametersarecho-sen in its stableparameterregime, or the instability ofthisfixedpointwhentheparameterslie outsidethatpara-meterregime.Usingthisscheme,we derivea newcriter-ionfor the divisionof stableparameterregimesin whichLinsker’snetworkhas the potentialto developspeciallydesignatedconnectionpatterns(alsoreferredto as affer-ent receptivefields(afferentreceptivefield(aRFs)),anduse this criterionto investigatethe relative stabilityofseveral types of dominantaRFs reported in Linsker’ssimulations or those which appeared as the majoreigenfunctionsstudied in MacKay and Miller (1990a,1990b).

Before we present the details of our method, it isinstructive to review two kinds of conventionalapproachesthathavebeenemployedin the stabilityana-lysisof this classof models:

(i) Liapunov’sdirectmethod(Cohen&Grossberg, 1983;Linsker, 1986, 1988b;Golden, 1986, 1993; Gross-berg, 1988;Feng, 1997).But, usually, an InvariantSet Theoremtypeof analysisis not practicallyinfor-mative for our purposes,in the sense that it barelyprovides any prediction about the relationshipbetween the system parameters and the set ofattractors.

(ii)The linearizationapproachin termsof the propertiesof the eigenvectorsand eigenvaluesas presentedinDayan and Goodhill (1992), MacKay and Miller(1990a,1990b)and Miller(1990b,1996).By assum-ing that the principal features of the dynamicsareestablishedbeforetheweightboundariesare reached,theshort-timeevolutionof weightvectorsin Linsker’s(andsimilarlyMiller’s)weightdynamicscanbe char-acterizedin termsof the propertiesof theeigenectorsand eigenvaluesassociated with the linear systemw,+l(i) = w,(i) + hi(wr), V i = 1,...fi~. Withthis approach, a number of strikingly informativeresults that explain Linsker’s and Miller’s simula-tions(forexample,a generalspectrumof eigenvaluesand a few types of dominanteigenfunctions)havebeen obtained.

Thisapproach,however,doesnotclearlyshowtheeffects of all the parametersknown to play crucialroles in the models, e.g., for Linsker’snetwork theroleplayedby thesynapticarbordensitybetweentwoconsecutive layers has not been addressed. Theexplicitstabilityanalysisin the parameterspace, aspresentedin thispaper,can providesuchinformationand could be complementaryto the linearizationmethod(see furtherdiscussionsin Section4).

Basicpropertiesof the genericoutcomesof system(2)are presentedin Section2.2. A necessaryand sufficientconditionis derivedto determinewhethera given con-nectionpattern will emerge in Section 2.3. Interpreta-tions of non-generic cases of Linsker’s model arediscussedin Section2.4.A generalstructureof principal

Analysisof HebbianLearning 707

0’/ .4B

0’/4 /A ‘B

(a)

i

& -.-..--,,,,’

,’.’

4 --------- --.

(b)

9GridRadhm

(c)

11181

(d)FIGURE1.An Illustrationof Lineker’snetwork.(a) Theflrstthrea Iayereof Llnaker’anetwork.EachneuronIneach Iayerrecaiveeaynapticinputs from a number of cella in the precedinglayer. The denaity of eynepeeaaccepted by an !Woeiidecreasesmonotoniceiiywith arange rMfromthe aite underlyingthe %60aii’aposition.(b) The denaityof aynapeeareceivedby an Mceii haea rangerM.For a GaussianSDF AU, Ir)-exp[ – (Ij– k12/2&)], j GL, k G M, the rangerxiaitsatendarddeviation.(c) in our numericeiex~rimente, each neuron iaaasumedto be on a grid site, and receivesaynaptic inputsfrom2S3aiteawithina circiewith grid radius9 in the precedingiayer. (d) Thegridsites are numberedintheorderaeshown. Thia numberingechemetrsnaformsatwodimensionai aynapticconnectionpetternintoaone-dimenaionaiweight vector.

parameter regimes is summarized in Section 2.5followed by detailed examples of configurationsofparameter space for several major aRFs in Section 3.Finally,in Section4, weremarkonthemeritsofutilizingdifferentmethods (for the analysisof system(2)) in acomplementaryfashion.

2. THE PARAMETER SPACE APPROACH

2.1. The Linsker Model

Linsker’snetwork is essentiallya feed-forwardneuralnetworkin whichneuronsare arrangedin layersletteredfrom fl to 3, C, D etc., and synaptic strengths aremodifiedaccordingto the Hebb rule until they arriveatboundary(seeFigure 1).In Linsker’snetwork,each cellin the present layer 94 receives synapticinputs from anumberof cells in the precedinglayerL. The densityofthesesynapticconnectionsdecreasesmonotonicallywithdistancefromthe pointunderlyingtheM-cell’sposition.Moreprecisely,assumethat $,(i) is the activityof the ithneuronin layerL = {1,...,N~}at time 7, where 1 s i sNz, and q,(k) is the activityof the kthneuronin the nextlayerM = {1,...,N~}at time 7, and 15 k 5 NM.Then

q,(k) is a linearweightedsumof inputs$T(i)’s:

NL

W)= ~ L(WA Mk i) + al, (4)iz 1

Wherew,(k,i) is the connectionstrengthbetweenthe ithL-cellandthekthM-cellat time7, r(,) is a non-negativesynapticdensityfunction(SDF)that satisfies

~ r(k,i)=l, VI sksNM,iEL

and al is a constant.Thus each !Wcell(e.g., the kthM-cell here) has a bundle of synapticconnectionswith anumber of L-cells through a weight vector or a corz-nectionpattern w~k) = {wr(k,i), i G L}, where i isnumberedas shownin Figure 1.

For the developmentprocessof w,(k,i), the Hebb-typelearningrule is used in Linsker(1986):

w,+ l(k, i) = w,(k, i) + a2+ a3(q,(k) – ad)(t,(i) – as),

(5)

wherea3 >0 is the stepsizeof Hebbianlearning,anda2,a4, and a5 are all constants.Sincethe synapticstrengthschangeon a longtimescalecomparedto the variationofrandom inputs, by averaging the Hebb rule over the

708 J. Fenget al.

(a)

k2 k2

c(w) >0 c(w) <0

FIGURE2. The parameterregimeof (k,, k2),In which w is a saturatedfixed point attractorof eqn (2). Tha regimeawith gray texture areMermjned by d,(w) < k, + C(w)k2< d~w). (a) The caae (RegimeE) with a &aitive eiopsfunction(i.e., with i poaitiveaverageaynapticstrength). For example, the ON-centeraRF with a large centrai excitatoryregion or the Off-center aRF with a smeii inhibitorycentralregion. (b) The 0sss (Regime F) with a negativealope function (i.e., with a negativeaverageaynaptic strength).For axampie, the OFF-centar aRF with a iarge cantml inhibitoryregionor tha ON-centeraRF with a amail excitatorycentrai region.

ensembleof input activitiesin layer L, the dynamicalequation for the developmentof the synaptic strengthw,(k,i) betweenthe kthM-celland the ith .&cellat timeTis

(w,+ ~(k,i))=(W,(k,i))+ k~

NL

+ ~ [~ +~21~(kj)(w,(k.0),(6)j=1

where (o)representsthe average over the ensembleofactivitypatternst,(i)’s in layerL, and kl, k2are systemparametersthat are particularcombinationsof the con-stantsof the Hebb rule:

kl = a2+ as(al – al)[(g,(i)) – a51,

k2 = a3[(&(i))– a51(.f,(D).

Here(<,(i))is a constantlindependentof i and 7,andthefull-raik covariancematrix

{~} = {a3([W-(&(0)l[tr@-(trO))l)j

1 = i,j = NL} (7)

of layerL describesthe correlationof activitiesof the ithand the jti L-cells. Actually, the covariancematrix ofeach layeris determinedby SDFSr(.,.)’sof all precedinglayers, and it is independentof r as well. To avoid theunboundednessof (w,(k,i)),upper and lowerboundsforsynapticstrengthsin eqn(6)areaddedin Linsker’ssimu-lation i.e. – w.~ s (w.(k,i)) s w-, whereW.W>0.Therefore,we could rewrite the dynamicalequationin(6) as it appeared in (l), where we have omitted theargumentk whichdenoteskth%t-cell,and havedenoted

1In Linsker’s network, it is assumed that the input signal at the firstlayer u) is random, spontarumus,and spatially uncorrelated, i.e. {~~(i)i= 1, ...,NX) - N(O,E), E= {Uti), au=0 for i #j and Uj = 1 for i=j,where i,j = 1, ...,NX.The development process of connection strengthsis mn layer by layer, one layer at a time, so this assumption is readilyverified.

(w,(i))as w.(i) for the simplicityof notation.The limiterfunctional.) in (3) definesa hypercubeQ = [ – w~~,ww]~~, theweightvectorw, = {w,(i),i= 1,...,N~) G Qand hi(.) in (2) is a continuousparametrized function.

REMARK 1 (A).From the procedures of the proofs inlatersections,onecanverifi thattheresultsof thispaperarestillvalidfor thecasewheref (.) is de$nedas:flx) =nEMif x > nE,$r;= x lfnEM-l S x < nEM;= nEM-Il~x <nw-l for O< nEM<1, whichis the limiterfinction usedin Linsker’ssimulations(Linsker,1986).

REMARK 1 (B). TheSDF is explicitlyincorporatedintothe dynamics(1) whichis equivalentto Linsker’sformul-ation. A rigorousexplanationfor this equivalenceisgiven in MacKay and Miller (1990a;see their Section3.2 andAppendixA).

2.2. TheoremaboutGenericSaturatedFixed Pointsand TheirStability

The key observationin the parameterspaceapproachisfairlydirect,and is basedon the specialformof the non-linear functionfi.) in (3), which is a strictlyincreasingfunctionin its linearregion.It is wellknownthat afiedpoint or an equilibriumstateof eqn (2) satisfies

w,(i) =f[w,(i)+ hi(wr)], Vi= 1,...,NL (8)

The set of all fixed points of eqn (2) can be furtherdividedinto two subsets:

(i) Generic: The set of fixed points satisfying thecondition

hi(w) # O, Vi= 1,...,N~ (9)

Accordingto the fixedpoint equationin (8) and theconditionin (9),onecanverify(seeTheorem2):(a)wis a saturatedfixedpoint;(b)w(i) + hi(w) > w-, or

Analysis of Hebbian Learning

w(i) + hi(w) < – w- shouldbe satisfiedfor all i =

1,...,IVz.That is, any genericfixedpointw mustbe asaturatedfixedpoint,and mustsatisfythathi(w) >0V w(i) = w.~, and hi(w) <0 V w(i)= – Wmm. Thisgives a necessaryand sufficientconditionfor check-ing wether a given weight vector is a generic fixedpointof eqn (2).

(ii)Nongenetic:The set of fixed points, which may ormay not be saturated, satisfying the followingcondition

3i EL suchthat

hi(~)=kl + ~ [~ +k2]~~)w~)=0. (lo)j= 1

In this subsectionand the followingone we concen-trateon characterizingthe setof genericfixedpoints.Asthe nomenclaturesuggests,the nongeneticfixed pointsdo not play any significantrole in the systemdynamics;we providesuchjustificationslater in Section2.4.

THEOREM 2:The whole set of generic jixed pointattractors(FPA) of the dynamicsin eqn (2) is given by

!J~P~= {wI w(i)hi(w)>0, 1s i ~ N~, and

w e V(Q)={– w-, Wm }N’ ] (11)

whereNL

hi(w)=kl + ~ [@+kz]r(j)w(j).j= 1

Proof. For the ‘if part, suppose that w is in the setdefinedby (11) and for somei G 1,...JVL,w(i) = w~~.Thus,hi(w) >0 for this i. Thenwe havew(i) + hi(w) >w.= whichimpliesthat

f(w(i) + hi(w))= Wm = w(i).

The casewhenw(i) = – w- followssimilarly,Hencew is a genericfixedpointof the dynamics(2).

For the ‘onlyif’ part, let w be a genericfixedpointofthe dynamics(2). Then w satisfiesthe fixedpointequa-tion (8), and hi(w)# OVi= 1,...,N~.Thus w must be asaturatedfixedpoint,otherwisew is not a solutionof thefixed point equation (8) since hi(w) # OVi= 1,...,N~.Because w is all saturated and the functionH.) is alinear and strictlyincreasingfunctionfor x G { –w-,w~w},w should satisfy that w(i) + hi(w) > w-, orw(i) + hi(w) < –Wm. Therefore, any generic fixedpoint w must satisfythat hi(w) >0 Vw(i)= Wmm, adhi(W) <0 Vw(i) = –w~~, which is equivalentto thedescriptionof the set definedby(11).

The followingCorollaryshowsthat the dynamicsys-tem defined by (2) is dissipative(Feng and Hadeler,1996)under condition(9), and verifiesthe stabilityofthe genericfixedpointsagainstsmallperturbations.

709

COROLLARY3.lf

(i) limm.w, exists (denotew = limp~w,) and(ii) Vi=i,...,N~, hi(w)=kl + ~~?!l[~ -t- lq]r(j)wti)

# O, then l~such that for all r > Y

w,= w and w E V(Q).

Proo$ Since

NL

hi(w)=kl + ~ [~ +k2]~~)w~) # 0,j= 1

for i = 1,..., N~,we observethat there existsa neighbor-hood ~i(w)of w such that for ally G ~i(w),

NL

hi(y)=kl + ~ [@j+kz]r~)y(j)j= 1

is definitely positive or negative. Denoten~~~~i(w)which is not empty,then we have

a(w) =

a = min inf Ihi(y)/>0.i yE6(w)

On the other hand, the existenceof the limit lim-~w,impliesthat 3T’ with the propertythat, as ~> T’

w, E a(w).

For a fixed i, withoutloss of generality,we assumethat

hi(y)>0.

Hence,for all ~> Y’+ [2wm=/a]+ 1‘~fT,we obtain

w~(i)= ~Q wt+~(i),and w~(i)= Wma.

Therefore,we assertthat w = w, and w C V(Q).Without loss of generalitywe assume from now on

that W.u = 1, i.e., we considereqn (2) with the weightspace Q= [ – 1, I]N’ and the set of verticesV(O)= { – 1,1}NL.For w G QFPA,let B(w) be the attrac-tive basin of w definedby

B(w)= {~1~ G Q = [ – 1,1]N’,

W,=o = ~, and ~1 W,= W}.

Hence,if w, startsat a state ~ in B(w)when 7 = O,thenw, willbe attractedto w undercondition(9).Theorem2togetherwithCorollary3 claimthat as longas the dyna-mical system (2) starts at a state in the vicinity of aw ~ ~~pA, then it willfindy stopat w withinfinitetime.

We next derive an explicit necessary and sufficientconditionfor the emergenceof various aRFs, i.e., wederive conditions to determine whether a given wbelongs to ~FPA.Using this condition,we can obtainthe wholeparameterregime of (kl,kz,r(0)’s)in which agivenconnectionpatternwill bean attractorof eqn (2).Our examplesin Section3 fully explainthis point.

710

2.3. The criterion for the divisionof parameterregimes

DEFINITION4. For any w C Q, define l+(w)={ilw(i)= 1}as the indexset of cellsat layer~ withexci-tatoryweightfor a connectionpatternw betweenlayerstiand_L, and.l-(w)= {i Iw(i)= – 1}as the indexsetofL-cells with inhibitoryweightfor w (see Figure3).

Notefromthe propertyof saturatedfixedpointattrac-tors (see Theorem2) that a connectionpattern w is anattractorof eqn (2) if and only if we have for all i =1,...,N~.

{w(i) kl + ~ [tij + k21rWW)

j= 1{

=w(i) kl + ~ [@ +k2]r(j)w(j)j=J+(w)

For all i C .l+(w), the above inequalityreduces to (bythe definitionof J+ (w) and .1- (w))

(13)

Equivalently,

Theaboveinequalityis satisfiedforall i inJ+(w),andthelefthand sideis independentof i. Takingmaximumoverthe set i G .l+(w) on both sides of this inequality,wethus obtain

On theotherhand,for i G l(w), we cansimilarlydeducethat

J. Fenget al.

DEFINITION5.We &fine the slope~nction:

whichis thedifferenceof sumsof theSDFr(.)overJ+(w)and J-(w), and is also the average synaptic strengthof the connection pattern w; and two kl-intercept

,%+ %>4 (+) lkZ Regime A

(a)

Q

(b)

a

(c)

o+‘k,

Reaime B I k2 kl-k fid~-),

k,

k,

o+

Regime DFfGURE3. Parameter regimes in the (kl, k2) subspsce for theernerg8ncSof SlkXCttStOfy (light gmy tSXtUIWregime A) Mdell-inhibitory (middle grey regime B) connection petterne. Theclerkgrey regime(RegimeC) Is the coexistenceregimefor bothsll-excitatoryand ell-lnhlbltoryconnection pettems. The fourthregimewithout sny texture (I.e., Regime D) Is where neitherthesll+xcltstory nor the ell-inhlbttory connection psttems srestsbkh Lineker’eelmuletion resufteon the emerWnce of struc-tured eRFs ere obteined in RegimeD.

—-———.

711Analysis of Hebbian Learning

jimctions:

[1- ~ Q@) ,

jEJ+ (w)

—co

if J+ (w) # @

if J+ (w)=@,

and

The definitionof the slope functionc(w) impliesthat itonlydependsonthe SDFr(.) betweenthe two successivelayers under considerationand does not relate to SDFSr(.)’s of the other preceding layers. Two /cl-interceptfunctionsall(w) and d2(w) embody the dependenceofeqn (2) on the covariancematrix Qz of the precedinglayer, and the SDF r(.) of the layer underconsideration(i.e., layer 94). Therefore, these two kl-interceptfunc-tions are determinedby the SDFSof all the precedinglayersandthepresentlayer.Thereasonto definethemasthe slopefunctionand the kl-interceptfunctionsis madeclear in the following theorem, which gives the newrigorous criterion for the division of stable parameterregimesto ensurethe emergenceof variousaRFs.

THEOREM 6. For every layer of Linsker’s network, aconnectionpatternw is a saturatedfiedpoint attractorof eqn (2) if and only if

d2(w) > kl + c(w)k2 > all(w). (17)

Hence,t~dz(w)< all(w),thentheaRF w isnotan attrac-tor of eqn (2).

Proof.Asderivedin eqns(12)–(16),a connectionpatternw is an attractorof eqn (2) if for i C l+(w):

> k2 ~ r(j)+ ~ djfi”),jGJ- (w) jGJ-(w)

kl+k~w)= d,(w) k,+k~w)=b

W

,’ kz,’

,’,’ 1’

,’,’ /’

,’ ,’1’ ,’

,’ /

,kl+k~w) =if,(W) kl+k~w) = if2(W)I

FIGURE4. Thepammaterraglmeof an all-but-oneaaturatadflxadpoint in the (k,, k2) plane ia the atralght Iina k, + &zG(w) = bwithinthe banddaflnedby k, + k@(w)> d,(w) and k, + k2~w) <

k, +k2 ~ r(j)+ ~ (&o)jGJ+ (w) jEJ+(w)

< k2 ~ r(j)+ ~ @r(j),jEJ- (w) jGJ - (w)

Now,the resultof the theoremfollowsdirectlydefinitionsof c(w),all(w), and d2(w).

fromthe

The above theorem shows that for any given set ofSDFS,the parameterregime of (/c1,k2),to ensure that astructuredaRF w is an attractor of eqn (2) is a bandbetween two parallel lines kl + c(w)k2 > all(w) andkl + c(w)k2 < d2(w) (see RegimeE and RegimeF inFigure2).2Note that if d2(w) < all(w),then there is noregimeof (k1,k2)for the occurrenceof that aRF w as anattractorof eqn (2). Therefore,betweenany two conse-cutive layers in Linsker’snetwork, the existenceof astructuredaRFw as an attractorof eqn (2) is determinedby kl-intemeptfunctionsall(.)andd2(.),andthereforebythe SDFSr(.)’sof all theprecedinglayersandthepresentlayer. Using this condition,one can obtain the wholeparameterregime of (k1,k2,rL,rL-,,..., rh) for a n-layer

Linsker’snetwork in which a given connectionpatternbetweenlayerLn- ~and layerL. will be an attractorofeqn (2).

Unlikefor any otheraRF, therealwaysexistsa stableparameter regime for the all-excitatory and the all-inhibitoryconnectionpatterns.We denote the kl-inter-cept functionall(w)for the all-excitatoryaRF as all(+)and d2(w) for the all-inhibitoryaRF as d2(–) respec-tively. From the above theorem and the definitionofkl-interceptfunctions,the all-excitatoryaRFis an attrac-tor of eqn (2) when kl + k2 > all(+), and so is the all-inhibitoryaRFwhenkl – kz < d2(–) (see RegimeA and

2We shsll catl any afferent receptive field, except for tbe all-excitatory and the all-inhibitory connection patterns, a struc?uredaRF.

712 J. Feng et al.

Regime B in Figure 3). Thus the parameter plane of where(kl,kz)is dividedinto four regimesby thesetwo criteria,and the regime(see RegimeD in Table 1 and Figure3)

I

?(w)= ~ r(j) – ~ r(j)+w(l)r(l)determinedby kl + kl < d,(+), and kl-kz > dz(–) is j~J - (w) jEJ+(w)

where neither the all-excitatorynor the all-inhibitoryconnectionpatterns are stable. Althoughthe values of

[‘z(w)= ,=~~w, ,=~w,~r~)

all(.) and d2(.) (–1 s all(.), d2(0)s 1) may change Jfromlayerto layer,thedivisionof theparameterregimesshownin Figure3 remainsinvariant. - ~ Q@)-Qf’~(l)r(l)

jCJ+ (w) 1(19)

2.4. Nongenetic Cases

[~I(w)= ,~~w) ,=~w,Q$rO)

Thenongeneticsetof fixedpointsof dynamics(1)canbestudiedusingthe followingthreecases:all-but-onesatu-rated patterns, all-but-two(and all-but-more-than-two)saturated patterns, and nongenetic saturated patterns.Linsker (1986) proved and observed that the onlypossibleoutcomesof the dynamics(2) are two kindsoffixedpoint attractors:all saturatedand all-but-onesatu-ratedpatterns.So far we have studiedthe situationof allsaturated fixed point attractorsunder condition(9). Inthis subsection,we providemoreinsightson all possible

I -~QtjW-f2f,wU)r(l)jCJ+ (w) 1

The quantitiesCal, 22 are slightlydifferentfrom c, dl,d2 defined in Definition5. While in Definition5 themaximumand minimumare taken overN~weights,themaximumand minimumin eqn (19)are takenovera setof N~-1weights(i.e., excludingw(l)).

nongenetic cases from the perspective of parameterspace.These resultsjustify the exclusionof all possible THEOREM7.w = {w(i), i = 1,...,N~}isafiedpoint of the

nongeneticfixedpointsin our analysis. dynamics (2), where w(i) = ? I,i # 1 and w(l) G(–1, 1), if and only ~ (k1,k2) is in the set defined by

eqn (18).

2.4.1.All-But-OneSaturatedPatterns.Assumethatw ={w(i), i = 1,...,}.} is an attractorof thedynamics(2)andw(1)G (–1, 1)is the onlyunsaturatedweight.Thenby arudimentary checking of the arguments for saturatedattractorsin Section2.3, we concludethat w satisfies

(18)

Theorem7 indicatesthat the parameterregimeof (kl,k2) in which an all-but-onesaturatedpattern is a fixedpoint of dynamics(2) is a straightline that lies in theband determinedby 22(w)> kl +k2i5(w)> al(w) (seeFigure 4). Similarly, this parameter regime is notempty if and only if ~2(w)> b > al(w). Therefore,thefixedpointswithallbutoneof theweightssaturatedwillbe sufficientlyaddressedby the analysisof their genericsaturatedcounterparts,due to the followingreasons:

I- ~ ~jro)-f?flw(l)r(l)~b 1. The stable parameterregime for a given all-but-one

jGJ+ (w) saturated pattern is approximatelya subset of the

TABLE 1

Type of Regime Parameter Regime Attractors

Regime A (Figure3(a)) k, + IQ > d,(+) and k,-lrz > dz(–) The all-excitatoryconnectionpatternisthe onlyattractorexceptfor Regime G.

Regime B (Figure3(b)) k,-kz < dz(–) and k + I@ < all(+) The all-inhibitoryconnectionpatternis the onlyattractorexceptfor Regime G

Regime C (Figure3(c)) k, + IQ > all(+) and k-k< cM-) The all-excitatoryand the all-inhibitoryconnectionpatternscoexistwitheach otherand withotherstable structuredaRFs.

Regime D (Figure3(c)) k, + k2 < d,(+) and k-k > old–) The all-excitatoryand the all-inhibitoryconnectionpatternsareunstable.The structuredaRFs may have separateparameterregimeswhen IG is largeand negative. Lir?skef’ssimulationresultson the emergenceof structuredaRFs areobtainedin Regime D

Regime E (Figure2(a)) d2(w) > kI + c(w)kz > d,(w) Stable structuredaRFs withpositiveaverage.

.—

Analysis of Hebbian Learning

stableparameterregime of its correspondinggenericsaturatedpattern.

2. The stableparameterregime for an all-but-onesatu-rated pattern is a line kl + kz~(w)= b, which is a setwitha Lebesguemeasurezero in the (IcI,k2)subspace.

2.4.2. All-But-Two Saturated Pattern. Assume w ={w(i),i = 1,...,N~}with w(l), w(2) E (–1, 1) and w(i)E {–1, 1}for i # 1,2. Afterpursuingan analysissimilarto the one for all-but-onesaturatedpattern,we easilyseethat a necessaryand sufficientconditionfor w to be afixedpointof dynamics(2) is

( ~,(w)> k, +k,t(w’) > a,(w)

- ~ QfjW)Wj-, j#l,2

)r(l) – (&w(2)r(2) ~f b, (20)

- ~ @jwO)Wj=, j#l,2

)r(l) – @w(2)r(2) ~f b2

Iz(w)= ~ r-(j)– X ‘ti)+w(1)r(1)+w(2)r(2)

jGJ- (w) jCJ+ (w)

{ 1- X @rO)-dlW(l)r(l)-d2W(2)r(2)

jGJ+ (w)

[aI(w)=,g!~w) j,~w)tirti)

\ 1- x @rO)-@~W(l)r(l)-d2W(2)r(2)

jEJ+(w)

(21)

Noticethat the two straightlinesassociatedwith the twounsaturatedweights(definedby the last two equationsin(20))areparallel.It impliesthat as long as bl # b2, theparameter regime in which w is a fixed point of thedynamics (2) is empty. For more general nongeneticcaseswherean arbitrarynumberofconnectionsareunsa-turated, in general, the parameter regimeswill also beempty. This argumentprovidesan alternativeexplana-tion of Linsker’s results that only all and all-but-onesaturatedpatternsare observed.

2.4.3. NongeneticAll SaturatedPatterns.. For a givennongenetic saturatedfixed point, the set of parameters(k,, kz) satis@ingh,(W) = O (for some i) is at most astraight line in the (kl, k2) space. Since there are atmost finitely many nongenetic saturated fixed points,

713

lrp&0e@tw8

FFD A E #

FIGURE5. A generaldivialon of the stable parameterraghnaainthe(kl, IQ)plane.Eachtypsof raglmeIadascribad In Table 1.TheaagmsntatlonIs determinedbythe nacasssryand sufficientcon-dition (stated in Theorem6) that every stable saturatedconnac-tlon pattern between any two consecutive Iayera of Llnaker’anatwork should aatlafy: The boundaries of the stable regimefor any given saturated pattern between any two consecutiveIayeracan be exactly calculated(uaing the nsossssryand auffl-cient condition)once the eat of SDFa is chosen. Noticethat, forttre weight developmentbatwean any two conasoutlvelayers Land M the numberof the bandeof E or Ftype In the (/rI, k2) planecan range from Oto 2N’‘i and will be dstennlnsd by the set ofSDFSunderconaktsrstion(sss Sactlon3).

the stable regime of all possible such fixed points hasLebesguemeasurezero on the (kl, k2)space.

2.5. The General Principal Parameter Regimes

We summarizetheprincipalparameterregimesfromourgeneral theoremapplicableto all layers in Table 1 andFigure5.

In summary,Figure5 providesa generaland yet pre-cisepictureon the (kl, k2)planefor the stableregimesofaRFs between any two consecutivelayers of Linsker’snetwork.Clearly,to obtainexact informationaboutandto manipulatethe stableregimes,one needs to incorpo-rate the rangesof SDFSas well; this is demonstratedinSection 3. Figure 5 also underscores the need forapproaches such as linearization (see MacKay andMiller, 1990b), which enables one to identify thedominantaRFs and thus obviatesthe need to study thestabilityof all possibleaRFs.

3. PARAMETER REGIMES FOR ARFS IN THEFIRST THREE LAYERS

Basedon ourgeneraltheoremapplicableto all layers,weconfine ourselves to synaptic connectionsin the firstthree layers of Linsker’s netsvork.Denote the SDFSfrom layerA to 5 and from 9 to C as ~(.,”) with ranger-zand r%c(.)with range rc,respectively. The emergenceof various aRFs in the first three layers have been

714

previously studied in the literature (Linsker, 1986,1988a, 1988b; MacKay and Miller, 1990a, 1990b;Tang, 1989),and in this paper we mentiononly the fol-lowingnew resultsmadepossibleby our approach.

3.1. Developmentof Comections BetweenLayersfl and !3

As in Linsker’ssimulations,we assumethat the randominputat firstlayerfl has an independentnormaldistribu-tion withmeanOand variance1.That is,@= 1if i =j,and @ = Oif i #~. Hence,applyingTkOrem 6, onecmverify that:

1.

2.

The stableparameterregimefor all-excitatorypatternsatisfies: kl + kz > all(+) = ‘mini= ~s(i) ~– l/Nfl; and the stable parameter regime for theall-excitatorypattern satisfies:kl – kz < dz(–) =m.ini~ ~B(i) ~ l/NA.If the SDF, fl~(.), is positive (e.g., Gaussian withranger3 likein Linsker(1986)),theneverystructuredaRF w has a correspondingstableparameterregimethat satisfies:dz(w) = tini~~-(WJ#g(i) > kl +C(W)/C2> – dli~~+(W) #s(i) = all(w). That is, theexistence of a stable parameterregime for any aRF isindependentof the thirdparameterr~,and all 2N~–‘possible structured aRFs coexist. Therefore, it isimpossible to obtain any structured aRF betweenlayersfi and 5 withoutregardto the initialcondition.But, for a developmentalmodel like Linsker’snet-work, it is expected that the different aRFs shouldemerge under different sets of parameters andshould be relatively insensitiveto the initial con-ditions.In the deeperlayersof Linsker’snetwork,asdemonstratednext, the incorporationof more para-meters (i.e., more ranges of SDFS),allows one tomake selected sets of SRFS unstable for certainchoices of parameters, and thus can avoid thecoexistenceof majoraRFs.

3.2. Developmentof ConnectionsBetweenLayersB and C

For the weightdevelopmentfromlayer%to C,as shownin Theorem 6, the existenceof a weight pattern as anattractor of the dynamics (2) is solely determinedbythe ranges rc and r~ of the SDFSracand ~. Withoutloss of generality, we assume that the connectionstrengths from layer fl to 9 are all-excitatoryas inLinsker’s simulations. Hence the covariance matrixQ3 = {Q;,M ~ ~} Of layer ~ iS a Gaussi~ finctionwith range firm if the SDF ~(.,.) between layersAand 9 is Gaussianwith range r3. Linsker (1986)usedvariouschoicesof the ratio r&9 ranging from 3–lDto101fi,mostly r&g= Win his simulations.MacKayandMiller(1990a,MacKayand Miller, 1990b)left thisratioas a free parameter,andusedthe ratioof the rangeof Qz

J. Fenget al.

Zzisc

●

0)FIGURE 6. (a) An ON-oenteraRF between Iayara 9 and c. Thesynaptic atrangtha between a c*II and a number of zxellswithinthecirclewith radiusr~naraexcitatory, andthoaa outaidethe circle are all Mlbftory. (b) other major aRFa (from left toright): OFF-eantarcall with radius rCm,oriented (tri-lobed) callwith radlue r~lm, and bi-lobadcell with radiua rSL.

to the range of the SDF r%c,C/A= 2~/& = 2/3 in theirexamples,which is equivalentto the ratio rc/r9= &.The role playedby the SDFShas not been addressedinliterature. We will use the ratio rc/r%to show thesectional drawings of the parameter subspace of (rc,r~).We use a grid systemin our examples.We assumethateachC-cellreceivessynapticinputsfrom253sitesinlayer!3,where253 is the totalnumberof sitesinsidethecircle with grid radius9 (see Figure 1). We denote theradiusof the centralcore of a ON-centeror OFF-centeraRFas r~~~, one half of the maximumwidthof the exci-tatoryareain a bi-lobed~ as rBL,onehalfof thewidthof the excitatorycentralstripin an orientedaRFas rwidth,and then label the corresponding aRFs simply asON(rCOE,9),OFF(rco%,9),BL(rBL,9) ~d OR(rWid~,9)(see Figure6).

First, as we have noticedin Section2.3, the all-exci-tatory or the all-inhibito~ weightpatternswill emergebetweenlayersB and C when

kl + kz> dl( + ) =

or

respectively.Next, in Figs7 and 8, we show exhaustively the para-

meter subspace of (rc, r-g)for four groupsof connectionpatterns, i.e., ON-center, OFF-center, bi-lobed, and

—

Analysisof HebbianLearning 715

tntuoopt Funenons of ON-co- cdl (1,0)

0.01

0

-am

a.oa

h

I

-0.0s

0.001

0

-0.001

aooa

.O.000

to*4F3snooor B90CDs

fntuwpt FunEtlone of ~ntu cdl (s,0)

-am : I4al

RuleoorB* c’&rWayFuna&n*OO”

tntuwpt Funoum9010wcontu cd (7,s)

‘1-%.o.oom

-o.ooi

4.001s

.o.Oos:RMeworm90 c’Ln91rv Funeaon

$00

FIGURE 7. Calculetlng the stable ~ regknee for !sto c ON-center SRFSfrom the k,-interw pt frmotlons(here O represents

rh(w,r~,r,) with rclr~=d, and A“mpmaemed~w, r. r$ withrdr, = ~). For any sRF, w, the existence of a stable parameterregime ie determined by the two kl-inkroept functions, d,(w, r. r~ and d~w, ro rJ. For the osfouktions of this figure, we fixed theratio rdr.to ~(aedone in Linsker’eeirnuiations),end dl(w, r“ rJand d~w, r. r~ereoaiouiated eetwofunctione of re AnaRF ieaatebieattractor if and oniy if d2(w,r. rJ > dl(w, r“ r~. We ahati caii the vaiue of rcet whkh d~w, ra rJ-dl(w, ra rJ turns from positivetonegativeas the crlflcal valueof re Speo9oonatminaus to show hereonly 4 caaeaout of the 8 kinds of zHo-cON-canterceiis studied inFigure8. When rcis Iergerthan theoritioel vaiueforan aRF (rcz 1.541fOrON1,9), rc= 2.356f0r OM2,9), rc= 4.221 for OM3,9),rC~ 6.-for OF44,9),and rc= 14.925for OKS,9)), - the eorreapondingaRFwili no Iongerbean atfraotorofeqn (2). Notethetthere ISno crftioelvaiue of rcfor 0#47,9) (aieo for -6,9) and -6,9)), i.e., a etebla regime of (kj, /r2)alwaysexists forthese sRFe when rjr~ = Wandf.cs 1~.

oriented (tri-lobed)cells, which are either observedinLinsker’ssimulationsor recognizedas major eigenvec-tors of the linearizeddynamicsin MacKay and Miller(1990a,1990b).3Basedon theseexamples,we makethefollowingobservations:

1. When r%is sufficientlysmall,the situationfor 2Lto-Cweightdevelopmentwilldegenerateto thecasefor%to-~ where every aRF has a stableregime.But if theconnectivitybetweenlayersfl and 3 is all-to-allwithconstantSDF,then theredoesnotexistanyregimeofE or F type in the (kl, kz)plane for !B-to-Cat all. In

3By MacKay and Miller’s notation for the eigenvectom., the all-positive (or all-negative) pattern, the centre-summnd pattern and thebi-lobed patfern are labeled with ‘1s’, ‘2s’ and ‘2p’ respectively. Theoriented pattern does not belong to circularly symmetric systems intro-duced in MacKay and Miller (1990a, 1990b), and is not observed inLinsker’s 9-to-C simulation, but is shown to emerge in deeper layers.

general, for each kind of connection pattern, theranges rc and r$ have pattern-relatedcritical valueswhich define the stable parameter regimes for thecorrespondingpatterns.

2. For circularlysymmetricON-center(or OFF-center)cells,4 those patterns with large ON-center (or

4Notice that for every ON-center cell, w, with excitatory connectionsinaide the circle with radius r- and inhibitory connections outside thatcircle, there exists a corresponding OFl%enter cell, w’, with inhibitoryconnections inside the circle with the same radius r,~ and excitatoryconnections outaide that circle. Since dl(w’) = - cfz(w),and cfz(w’)=–d,(w), then d2(w’) – dl(w’) = dz(w) – d,(w). Thatis, the stable andunstable parameter regimes for w and w’ are the same in the (rc, r~)subspace.If w and w’ are stable (i.e., dz(w’) – cfl(w’) = dz(w) – all(w)> O),then they will also apperuinthe(kl, k2)plane, asanE-type band andanF-~ban~respectively. Note tbatthesetwobands will have the samewidth but opposite slope values (since c(w’) = – c(w)). Therefore, weonly need to consider ON-center cells because of the symmetry betweenthe slope and intercept functions of OFF-center and ON-center cells.

716 J. Fenget al.

% Hlaerow(l#) ~ ●1

D,,’

,.’

,’.’ #’,’

.’ .“1 q“% ,A’o-

,,’,“/’ ,,’

.’ ,’

10 1 ‘; O.noo rc

~B fJN(S#)orOWW@ o

Q

1

*-

10 rc

“ Oqw)orofr(wl@ o1

Elli

1 (14.esqa 17)

1 0 rc

‘t w7#e)*-(7$) ●o1

m

i

rc

FIGURE8. The parametereubapaca of (r. r%)for verlous connectionpatterne.The textured area In eech plot is the stable parameterregimeof (rOr~ in whkh d~w, r. riJ > dj(w, r. rJ, Le.,there ie a correspondingstabieregimein (Irj, I@ aubspace.Iftha choiceof rcandrmlieaoutaidethe shadedregion,than the correspondingpatternia unstabieand it cannotbeobaerwadin simulationsirreapaotfveofthachoicesof (kI, I@ and the initialconditions.in Lineker’ssimuiationa,the ratio r$rg waa moatiyfixad to W and was aiao varlad to apanthe rangefrom 3“1nto 10In; t~ corresponding~ngeh ahown ae the area betweantwo daahed lhIeSh the(r~ rd PteneforOM1$) or’OFF(1,9),and the oftan used ratio r~r, = A is shown as a dotted line withinthe above area. The figures In Figure 7 are the cutawsyvieweaiongtha dottadlines inthe (rOr.Jptanaahownhere.The criticalvaluesof (r~ rJforthe cceewhen rjr~ = @are merkadfor ON-oanterand OFF-canteraRFs. For other weight petternscorrespondingto the eigenfunctionsIabeiedwith ‘3s’, ‘3p’, ‘3d’, etc. in MacKsyand Miller (1880s, 1890b),their stable regimesin the (r~ ra pianeare foundto be in the same regimeas shownin the case of 0~,9) andon occasion much smsiler (i.e., oniy stable when r, is sufficientlysmall).

OFF-center)core (e.g., ON(6,9), ON(7,9), ON(8,9) in the regimesE n D of the (kl, kz)plane (i.e., withand their OFF-centercounterpartsin our examples) positive c(w)), in the sense that other major aRFsalways have a stableparameterregime. That is, the (including‘2s’ with small-core,‘2p’, tri-lobed,etc)emergence of these patterns is insensitive to the can be made unstable by choosing appropriatechoice of r-cand r$. But for those ON-center (or values of rc and r%.Similarly,the large-coreOFF-OFF-center) patterns with smaller ON-center (or center patterns are dominant in the regimes H n DOFF-center) core, their stable parameter regimes of the (kl, kz)plane (i.e., with negativec(w)) in thein the (rc, rfi) plane decrease in size with r~~~. samesense.Thus,betweenlayers9 andC,the regimeThus the large-coreON-centerpatternsare dominant H n D (withnear-zeroc(w))is theonlywindowin the

-..

Analysis of Hebbian Learning 717

3.

r, ontl,o)@1

m

1

,

Wrcr, -at@1

m

1

10 rc

1

M1 0 rc

r, m#!@1

1

‘L4Wrc

r, ,mw)@1

m

1

10 rc

r, Otwca... -1

Lila1

1 rcr,1

@W#J@

T1

1

1u1 0 rc

FIGURE8. Continued.

(kl, k2) plane where there exists the opportunityfor althoughtheymay coexistwith thosecircularlysym-emergingweightpatternsother than the ON–(OFF-) metricpatternsin certainparameterregimes.But it iscenter type. importantto noticethat there indeedexist parameterFor the bi-lobed and the non-circularly-symmetric areas of (rc, rB)in which the bi-lobedpatterns(e.g.,oriented(tri-lobed)patterns,only thosepatternswith BL(4,9)and BLJ5,9)in our examples)becomedomi-small average weight strength c(w) have a wide nant,whilethecentre-sun-oundandtheorientedpatternsstable parameterregime in the (rc, r%)plane which are unstable.Figure8 alsoshowsthat thischaracteriscan match the regimes for the relatively dominant relativelyinsensitiveto theratior&9 (e.g.,from3’12ON-center (or OFF-center) patterns in size, and to 101’2)but dependson the specificvaluesof rc and

occupy the different regimes of (rc, r$) from ON- r~chosenin simulation.Moreover,since the averagecenter (or OFF-center)patternsin most areas of the synapticstrengthc(w) of these bi-lobed patterns is(r., rfl)planeexceptfor the case when rg is small.At near zero, their stable parameter regimes can bethe same time, the oriented (also referred to as tri- exclusivefrom others’in the (kl, k2, rc, rn) space.lobed,grating-like)patternsare alwaysovershadowedbyeithercentre-surroundpatternsorbi-lobedpatterns Wehavesofardiscussedtherelativestabilityofonlyaandhaveno exclusivestableareain the (rc,rg)plane, few typesof aRFs;however,for the weightdevelopment

718 J. Feng et al.

i11L-4mrc

1I--A@mlf rcr, W*@Git1

—m11 rcr, uca@1

T11

L4W20rc

r, .Urwc)1

r, ut4c9c21m1 rcr, Ww)g1

n1

IUre

1L1 0

FIGURE8. Continued.

N --1 ~~& ofpo55iblebetweenlayers%andC,thereare2 “saturatedstructuredaRFs. Obviously,it is unrealistictotest all of themto ensurethat the designatedpatternhasan exclusivestableparameterregime.Fortunately,it isunnecessaryto do so becauseof previousresultson thedynamics(Linsker,1986,1988a),whichshowthat thereare onlya few typesof saturatedattractorsthat are domin-ant for certain choices of parameters.With the sameprocedure shown in Figs 7 and 8, we have tested allpossible variations of the other major eigenfunctions‘3s’, ‘3p’ and ‘3d’ mentioned in MacKay and Miller(1990a, 1990b),and found that not one of them has astable area in the (rc, rg) plane beyondthe area wherer~ is small. It is not farfetchedto predictthat otherpat-terns will be unlikelyto have stableareas in the (rc, rj)plane sharedwith the majorpatterns,exceptfor the area

with smallr$.Evenif somepatternsotherthan thosewehavestudieddo haveoverlappingstableregimeswiththemajor patterns, the results from the linearizationapproachshowthat it is unlikelyto observethe formerin simulations,sincethe latterwill be dominant.

We concludeour experimentswith a brief descriptionof the (kl, k2) parameter subspace. As illustrated inFigure5, in the(kl,kz)subspacefor the%-to-Cdynamics,the presence or absence of a band correspondingto astructuredaRF (Regimetype E or F) is determinedbythe choicesof (rc, rg).Clearly,the rangesof SDFShaveto be chosenappropriately,so that a RegimeE or F for adominantstructuredaRF with a certainslopevaluemayno longer coexistwith any other major aRF. Next, theappropriatechoicesof the parameterskl andk2 to ensurethepotentialemergenceofdesignatedaRFs,canbebased

719Analysis of Hebbian Learning

on the followingobservations:

1.

2.

3.

(kl, /cz)mustbe in RegimeD: OnlywithinRegimeDwill various Regime E’s and F’s correspondingtovarious stable structured aRFs be removed fromRegimes A, B, C, and G where all-excitatoryandall-inhibitoryaRFs are dominantor many kinds ofattractorscoexist.kz mustbe largeandnegative:Whenkz is chosento belarge and negative, these Regime E’s and F’s withdifferent slope values will be separate from eachotherin RegimeD.– (kl/k2) = c(w): To ensurethe potentialoccurrenceof a desiredstructuredaRF w, the choiceof kl andk2mustbe in the band correspondingto w(i.e.,dz(w)>kl + c(w)k2 > all(w)). Since – [dl(w)/k2] +c(w) <–(kl/k2) < C(W) – [d2(w)/k2], – 1< all(W)<

d2(w) <1, andk2 is large, – (kl/k2) is approximatelyequal to the average synaptic strength c(w) (thatis a relationshipobserved empiricallyby Linsker’ssimulation).

Theprecedingdescriptionsaboutthechoicesof kl andk2havealsobeenderivedfromthe linearizationapproachin MacKay and Miller (1990a, 1990b).The analysisinthis paper, however, allows one to explore the wholeparameter space, includingthe critical roles played bythe rangesof SDFS.

4. CONCLUDING REMARKS

In practice, the parameterspaceapproachintroducedinthispaperandtheconventionalmethodscouldbe usedina complementaryfashion.For example,the linearizationmethod can be employedto identifya set of dominantpatterns (see Miller et al., 1989; MacKay & Miller,1990a, 1990b;Miller, 1996). If two or more patternsdo coexistfor a given set of parameters,then the linear-izationmethodcouldbe usedto derivesomeinformationon the dynamical process of their occurrence. Themethodon the parameterspace, on the other hand, cantest the stabilityof any designatedpatternfor any givenchoiceof parameters,andprovidethe stableandunstableparameterregimesfor any set of patternsunderconsid-eration.This informationcouldbe used,for example,toresolvethe coexistenceamonga setof dominantpatternsand to chooseparametersto ensure the potentialoccur-rence of the designated patterns without performingextensive simulations. Furthermore, the parameterspace approach cart also be adopted to study otherneural models that can be written in the form of (2)and where the saturatedstate or weightpatternsare thegenericoutcomes.

REFERENCES

Anderson, J.A., Silveratein, J.W., Ritz, S.A., & Jones, R.S. (1977).Distinctive features, categorical perception, and probability learning:

Some applicationsofaneural model. PsychologicalReview, 84,413-451.

Cohen, M.A.,& Grossberg, S. (1983). Absolute stabifity of globalpattern formation and parallel memory storage by competitiveneural networks. IEEE Transach”onson Systems, Man, andCybernetics, SMC-13, 815-826.

Dayan, P. S., & Goodfrill, G. J. (1992). Perturbing Hebbian rules. In J.E. Moody, S. J. Hanson & R. P. Lippman (Eds.),Advances in neuralinformationprocessing systems, 4 (pp. 19–26). San Mateo, CA:Morgan Kaufrnann.

Erwin, E., Oberrnayer, K., & Schulten, K. (1994). Models of orientationand ocular dominance columns in the visual cortex: A criticalcomparison. Neural Computation, 7, 425–468.

Feng, J. (1997). Lyapunov function for neurat nets with nondif-ferentiable input–output characteristics. Neural Computation, 9,45-51.

Feng, J., & Hadeler, K.P. (1996). Qualitative behavior of some simplenetworks. 1 Phys.A, 29, 5019–5033.

Feng, J., Pan, H., & Roychowdfmry, V.P. (1996). On neurodynamicswith limiter function and Linsker’s developmental model. NeuralComputation,8(5), 1003-1019.

Golden, R.M. (1986). The ‘Brain-State-in-a-Box’ neural model is agradient descent algorithm. Journal of Mathematical Psychology,30,73-80.

Golden, R.M. (1993). Stability and optimization analysis of the general-ized Brain-State-in-a-Box neural network model. Journal of Math-ematical Psychology, 37, 282–298.

Greenberg, H.J. (1988). Equilibria of the Brain-State-in-a-Box (BSB)neuraf model. Neural Networks, 1,323–324.

Grossberg, S. (1988). Nonlinear neural networks: Principles, meehan-isma, and architectures. Neural Networks, 1, 17–61.

Hui, S., & Zak, S.H. (1992). DynarnicaJ analysis of the Brain-State-in-a-Box (BSB) neural models. LEEE Transactions on NeuralNetworks, 3(l), 86-94.

Linsker, R. (1986). From basic network principle to neural architwture(series). Proc. Natl Acad. Sci. USA, 83, 7508-7512, 8390-8394,8779-8783.

Linsker, R. (1988a). Development of feature-anatyzing cells and theircolumnar organization in a layered self-adaptive network. In R.Cotterill (Ed.), Computer simulation in brain science (pp. 416-431). Cambridge: Cambridge University Press.

Linsker, R. (1988). Self-organization in a percepturd network.Computer, 21(3), 105-117.

MacKay, D., & Miller, K.D. (1990). Anafysis of Linsker’s applicationof Hebbian rules to linear networks. Network: Computation inNeural Systems, 1,257-297.

MaeKay, D., & Miller, K.D. (1990). Anafysis of Linsker’s simulationsof Hebbian rules. Neural Computation, 2, 173–187.

Miller, K. D. (1990a). Correlation-based mechanisms of neurafdevelopment. In M. A. Gluck and D. E. Rumelhart (Eds.), Neuro-scienceand Connectionist Theory (pp. 267–353). Hillsdale, NJ:Lawrence Erlbaum.

MiJler, K.D. (1990). Derivation of linear Hebbian equations from anonlinear Hebbian model of synaptic plasticity. NeuralComputation,2,321-333.

Miller, K.D. (1994). A model for the development of simple cell recep-tive fields and the ordered arrangement of orientation columnsthrough activity-dependent competition between On- and OFF-center inputs. J. Neuroscience, 14(1), 409-441.

Miller, K. D. (1996). Receptive fields and maps in the visual cortex:Models of ocular dominance and orientation columns. InE. Domany, J. L. van Hernmen and K. Schulten (Eds.),Models of neural networks III (pp. 55–78). New York: Springer-Verlag.

MilJer, K.D., & MacKay, D.J.C. (1994). The role of constraints inHebbian learning. Neural Computation,6, 100-126.

Miller, K. D. and Stryker, M. P. (1989). The development of oculardominance columns: Mechanisms and models. In S. J. Hanson and

—.

720 J. Fenget al.

C. R.Olson (E&.), Connectionist modeling and brain~ction: Tke Swindale, N.V. (1996). The development of topography in the visualdeveloping irrterjaces(pp. 255-350). Cambridge, MA: MIT Press. cortex: a review of models. Network: Computation in Neural

Miller, K.D., Keller, J.B., & Stryker, M.P. (1989). Ocular dominance Systems,7, 161–247.column development: Analysis and simulation. Science, 245, 605– Tang, D.S. (1989). Information-theoretic solutions to early visual infor-615. mation processing: Analytic resuIts. Wrys.Rev. A, 40, 6626-6635