Multisensory Integration - ExamplesI. Posture – maintain center of gravity above the feet

A) Vestibular

B) Visual

C) Proprioception – ankle sway

McCollum et al. J Theor Biol 180: 257-270, 1996

D) Other Proprioceptive Cues – arm posture

Rabin, E. et al. J Neurophysiol 82: 3541-3549 1999

1) Fingertip contact attenuates sway

II. Ilusions

A) Proprioception – vibration produces illusion of muscle stretching

1) Example – vibrate biceps, feel arm extend

2) Pinocchio effect

3) Multimodal effects

a) Overrides vision

B) Weight and Pressure Cues

Lackner and DiZio, Trends Cogn Sci 4:279-288, 2000

C) But vision can also override proprioception

Lackner and DiZio, Trends Cogn Sci4:279-288, 2000

III. Sensing Heading – Neural Correlates in MST

A) Optic Flow

1) The focus of expansion (FOE) defines heading and it depends on gaze direction

2) Neurons in MST are tuned to optic flow and the locus of the FOE

Page and Duffy, J Neurophysiol 81: 596-610, 1999

B) Influence of motion cues on MST neuron activity1A) Monkey rides a cart (CW or CCW) and views lights on a wall

1B & 1C) – a heading selective neuron

1D) But not all neurons are heading selective – some encode place and some responses depend on the path taken

Froehler and Duffy, Science 295: 2462-2465, 2002

“Heading” Cell

“Place” Cell

Froehler and Duffy, Science 295: 2462-2465, 2002

C) Vestibular contributions to MST activity1) The task – discriminate heading2) Behavior Intact

Vestibulectomy

Gu et al. Nature Neurosci 19: 1038-1047, 2007

Optic flow

Motion cues

3) MST neurons respond to vestibular only cues (without optic flow)

a) A sample neuronb) Tuning to optic flow (•), vestibular aligned on best direction (•) or on visual (•)c) Same, after vestibular lesion

Some MST neurons have same tuning to optic flow and vestibular, others have tuning 180o out of phase

Gu et al. Nature Neurosci 19: 1038-1047, 2007

D) MST neurons encode optic flow in space rather than the motion on the retina:

pursuit stimulus (see eye velocity trace) to right or left

optic flow stimulus for 200 ms

but MT neurons encode retinal image velocity

E) Summary – MST neurons respond to:

optic flowvestibular signalsextraretinal signals (efferencecopy)

Inaba et al. J Neurophysiol 97: 3463-3483, 2007

IV The weightings given to different sensory modalities depends on how reliable they are

As vision becomes progressively less reliable, it is weighted less and tactile cues are weighted more

Ernst and Banks, Nature 415: 429-433, 2002