Connectionist Units, Probabilistic and Biologically Inspired

Psych 209January 11, 2013

Neurons and Units• At rest, neurons are negatively polarized compared to the

surrounding medium.

– Excitatory inputs cause them to become depolarized– Inhibitory inputs cause them to become hyperpolarized.

• A neuron’s tendency to emit action potentials increases as it becomes less and less polarized, and correspondingly decreases as it becomes more an more polarized.

• Firing rate levels off at 0 at the bottom and ~100 spikes per second at the top.

• In PDP models our units correspond to notional populations of neurons (maybe 10,000 per unit).

• The continuous valued output of a unit, which usually ranges from 0 to 1, can be thought of as approximating the proportion of neurons in the population emitting an action potential in a small time interval, divided by the maximum value this proportion could take.

• For example, if we choose the time interval to be one msec, the maximum proportion of neurons firing per msec would be .1. So an output of .2 would correspond to .02 of the 10,000 neurons firing (or 200 of the 10,000 neurons) firing per millisecond.

Neuro-similitude or implementation of a computational theory?

• Early models (Grossberg’s, the iac model you will explore in the first homework) embodied attempts to capture characteristics of real neurons in a simplified way.– Simulations were used to show how a process could be

modeled.• Later models are more grounded in theory

– Some of the key features of the physiology are still captured

– And the formulation allows us to relate what neurons are doing to a theory of what they should be doing.

• For many purposes, the two formulations work very similarly, even if one is not exactly valid from a probabilistic inference perspective.

Probabilistic Formulation: Review and Discussion of Probabilistic Concepts

• Two different sources of evidence e1 and e2 are conditionally independent given hi, iff

p(e1&e2|hi) = p(e1|hi)p(e2|hi)

• If this holds for all i we can say the different elements of evidence are ‘conditionally independent’

• In case of N sources of evidence, all conditionally independent under h, then we get:

p(e|hi) = Pj p(ej|hi)

• Combining this with the prior we get a quantity I call the Support for hypothesis i:

Si = p(hi) Pj p(ej|hi)

• Taking logs we get:

log(Si) = log(p(hi)) + Sj log(p(ej|hi))

• In the case of a single h and its alternative ~h we get:

log(S/(1-S)) = log(p(h)/(1-p(h)) + Sj log(p(ej|h)|p(ej|~h))

How this relates to connectionist units (or populations of neurons)

• The baseline activation of the unit is thought to depend on a constant background input called its ‘bias’.

• When other units are active, their influences are combined with the bias to yield a quantity called the ‘net input’.

• The influence of a unit j on another unit i depends on the activation of j and the weight or strength of the connection to i from j.

• Connection weights can be positive (excitatory) or negative (inhibitory).

• These influences are summed to determine the net input to unit i:

neti = biasi + Sjajwij

where aj is the activation of unit j, and wij is the strength of the connection to unit i from unit j.

unit i

Input fromunit j

wij

A Unit’s Activation can Reflect P(h|e)• The activation of unit i given its net

input neti is assumed to be given by:ai = exp(neti) 1 + exp(neti)

• This function is called the ‘logistic function’. It is usually written in the numerically identical form: ai = 1/[1 + exp(-neti)]

• In the reading we showed thatai = p(hi|e) iffaj = 1 when ej is present, or 0 when ej is absentwij = log(p(ej|hi)/p(ej|~hi))biasi = log(p(hi)/p(~hi))

• This assumes the evidence is conditionally independent given the state of h.

ai

neti

Posterior probability when there are N alternatives

• In this case, the probability of a particular hypothesis given the evidence becomes:

P(hi|e) = p(e|hi)p(hi) Pi’p(e|hi’)p(hi’)

• The normalization implied here can be performed by using a ‘net input’ as before but now setting each unit’s activation according to:

ai = exp(neti) Si’exp(neti’)

• In this case, ai = p(hi|e) iff

aj = 1 when ej is present, or 0 when ej is absent

wij = log(p(ej|hi))

biasi = log(p(hi))

h

e

The Grossberg Unit(Biologically Inspired)

Unit i

Output fromunit j

wij

max = 1

min = -.2

rest

a

0ei = Sj+outputj wij ; ii = Sj-outputj wij

Dai = (max –ai)ei – (ai-min)ii – decay(ai-rest)

outputi = [ai]+

The IAC Unit

Unit i

Output fromunit j

wijmax = 1

min = -.2

rest

a

0

Suppose the net input to a unit is constant (and positive). What is its

equilibrium activation value?

How Competition Works

Jets & Sharks Model (IAC)

• Allows you to explore a simple localist / hard-wired PDP type model that has been applied to many problems in perception, cognition and social psychology.

• In the Jets and Sharks case, we will explore:– Information retrieval by name or attributes– Assignment of plausible default values– Spontaneous generalization