onr muri 3/10/03 sajda lab … intelligent image integration through probabilistic inference in...

ONR MURI 3/10/03

Sajda LabSajda Lab… Intelligent Image Integration Through Probabilistic … Intelligent Image Integration Through Probabilistic

Inference in Sparsely Connected “Hypercolumn” Inference in Sparsely Connected “Hypercolumn” Networks…Networks…

Mechanisms for surround suppression in a spiking-neuron model of cortical hypercolumns in V1 (Objective A)

Perceptual salience as novelty detection in cortical pinwheel space (Objectives A & B)

A Bayesian network for cue integration: Application to figure-ground segmentation and tracking (Objectives B & C)

Prioritization of image search via integration of spatial EEG signatures (Objective C)

ONR MURI 3/10/03

Mechanisms for surround suppression in a Mechanisms for surround suppression in a spiking-neuron model of cortical spiking-neuron model of cortical

hypercolumns in V1hypercolumns in V1 Surround suppression in V1 neurons is typically

characterized as an extra-classical receptive field phenomena. There is significant controversy regarding the neural mechanisms responsible for surround suppression.

Simoncelli and Schwartz have shown that surround suppression is consistent with a statistically-derived normalization model of neuronal activity—i.e. early visual processing is well-matched to the statistics of natural images and surround suppression is a mechanism for probabilistically “factoring” the input space.

from Simoncelli and Schwartz, 1998

from Walker, Ohzawa & Freeman, 2000

ONR MURI 3/10/03

Development of a spiking-neuron model of Development of a spiking-neuron model of an “effective layer” in V1an “effective layer” in V1

Macaque V1, input or “effective” layer 8 ocular dominance columns, 64

orientation hypercolumns (pinwheels) (Blasdel, 1992; Bonhoeffer & Grinvald, 1991)

16 mm2 cortical area, 1-25 deg2 visual field

4 cell populations: 75% excitatory cells, 25% inhibitory cells (Beaulieu et al., 1992) and 50% simple cells, 50% complex cells (De Valois et al., 1982)

Anatomically “correct” LGN input and retinotopic map (Croner & Kaplan, 1995; Reid & Alonso, 1995, Blasdel & Lund, 1983; Freund et al., 1989, Conolly & van Essen, 1984, Malpeli et al, 1996, van Essen et al., 1984, Tootell et al., 1988)

Surround suppression in a model of V1Surround suppression in a model of V1

ONR MURI 3/10/03

Development of a spiking-neuron model of Development of a spiking-neuron model of an “effective layer” in V1an “effective layer” in V1

Anatomically “correct” cortical length scales:

DEaxon=200um, DI

axon=100umDE

dend=50um, DIdend=50um

(Fitzpatrick et al. 1995; Callaway & Wiser, 1996; Lund & Wu, 1997; Yoshida et. al. 1996)

First order LGN temporal kernel (Gielen et al., 1981, Benardete, 1994)

Cortical time scales : AMPA (5 msec) NMDA (50 msec), GABA (10,100 msec) (Koch, 1999; Reyes, 2000)

Anatomically “correct” long range lateral connections (Bosking et al., 1997; Malach et al., 1993; Schmidt et al., 1997; Stettler et al., 2002)


Long range lateral connections (model)

ONR MURI 3/10/03

““Simulated” optical imaging of V1 modelSimulated” optical imaging of V1 model


Model

ONR MURI 3/10/03

Simulations of surround suppressionSimulations of surround suppression

neuron 1

neuron 2

Stimulus Responses (F1 component of firing rates)


ONR MURI 3/10/03

Multiple mechanisms for surround Multiple mechanisms for surround suppressionsuppression

… surround suppression naturally emerges from the cortical … surround suppression naturally emerges from the cortical architecture…architecture…

Suppression via inhibition of cortical excitationSuppression via direct inhibition

neuron 1 neuron 2


ONR MURI 3/10/03

Perceptual salience as novelty detection Perceptual salience as novelty detection in cortical pinwheel spacein cortical pinwheel space

Q: Why is the perceptual salience of the horizontal bar so different for each figure?

A: Bar is somehow “novel” compared to the background

Q: What does it mean to be novel?

ONR MURI 3/10/03

The negative log likelihood The negative log likelihood as a measure of noveltyas a measure of novelty

Mahalanobis distance:

To model , one approach (Rosenholtz, 1999) is to consider the population distribution to be Gaussian. The negative log likelihood is then given by the Mahalanobis distance:

In general, however, the assumption of a Gaussian distribution will not accurately model the population distribution.

)( Pfp

)()())(log( 1 ffPfp T

Our Approach: A mixture model of population responses in cortical “pinwheel space”

Perceptual salience as novelty detectionPerceptual salience as novelty detection

ONR MURI 3/10/03

Integrating input from CRF and nCRF to Integrating input from CRF and nCRF to construct cortical responses in “pinwheel construct cortical responses in “pinwheel

space”space”

Retinal/LGN input (quadrature mirror filters):

Cortical interaction:

Pinwheel activity:

122 ))(())(()( nIRIRf oer

otherwise

nfCf jiiri

c 0

)()()( 2

))(1)(()( cr fff

Cortical interaction from Kapadia[4]

Model cortical interactions


ONR MURI 3/10/03

Estimating Estimating ))(log( Pfp i

Estimating population distribution:

A pinwheel’s response can be represented as a coordinate in an N (number of orientations) dimensional space. The population distribution is estimated via fitting data with a mixture of multi-variant Gaussians using the EM algorithm.

Perceptual salience is computed as the novelty of a pinwheel response relative to the population:

c

PcpPcfpPfp )(),()(

))(log( PfpS ii


ONR MURI 3/10/03

Results: Perceptual salience in the face of Results: Perceptual salience in the face of contextual cues contextual cues


ONR MURI 3/10/03

Results: Detecting salient structure in Results: Detecting salient structure in natural imagesnatural images


ONR MURI 3/10/03

A Bayesian network for cue integration: A Bayesian network for cue integration: Application to figure-ground Application to figure-ground segmentation and trackingsegmentation and tracking

Vision is fundamentally a problem of making inferences about the world from uncertain information

Link computational theories of vision problems to models of visual information processing in biological systems (e.g. figure-ground discrimination, tracking, recognition etc). from Zhou et al., 2000

Neurons in V2 selective to “direction of figure”

ONR MURI 3/10/03

A Bayesian network model for inferring direction-of-figure (DOF) from multiple cues

Bayesian network for cue integrationBayesian network for cue integration

Local message passing Network “chain”

ONR MURI 3/10/03

Integrating Concavity/Convexity with Integrating Concavity/Convexity with Similarity/Proximity to compute the DOFSimilarity/Proximity to compute the DOF

Use convexity as local cue (Sajda and Finkel, 1995; Weiss 1997, 1999)-Weiss (1997, 1999) Probabilistic formulation-Xi: 2D vector representing DOF at point i (hidden variable)-Yi: determined by local angle at point i (observation)

-Maximize P(X|Y) = c exp(-CF) Similarity/Proximity cue (Sajda and Finkel, 1995)

- Influence decreases as distance increases

Linear cue combination (weak fusion model, Clark & Yuille, 1990)

- A separate estimate is made using independent observations

- Combine with a weighted average

- P(Y|X) = w1P(Yconvexity|X) + w2P(Ysimilarity|X)

1ii

ii

ii XXλYX CF


ONR MURI 3/10/03

Results: Integrating local observations via Results: Integrating local observations via message passingmessage passing


ONR MURI 3/10/03

Ambiguous Figure-Ground


ONR MURI 3/10/03

Results: Cue integration consistent with Results: Cue integration consistent with human perceptionhuman perception


ONR MURI 3/10/03

Perceptual shift induced by prior information


ONR MURI 3/10/03

Rubin’s vase (with face prior)


ONR MURI 3/10/03

Rubin’s vase (with vase prior)Rubin’s vase (with vase prior)


ONR MURI 3/10/03

Bayesian Integration of multiple cues for target tracking

Color:

Form:

Motion:


ONR MURI 3/10/03

Capturing statistical dependencies across time


ONR MURI 3/10/03

Results: Target tracking in the face clutter Results: Target tracking in the face clutter and occlusionand occlusion


ONR MURI 3/10/03

Prioritization of image search via spatial integration of EEG signatures

… on-line triage of imagery using Rapid Serial Visual Presentation…

Evidence for trial-averaged EEG signature in Rapid Serial Visual Presentation RSVP. Can be detected in 1/3 of the time compared to press of button (Thorpe et al, Nature, 1996).

Signature in neurophysiological recordings demonstrates detection of target images for presentation rates of up to 72 images/sec. (Keysers et al, JCN, 2001).

After exploiting EEG signature

stack 1 stack 2 stack 3

5% targets 5% targets90% targets

stack 1 stack 2 stack 3

33% targets 33% targets 33% targets

Before

ONR MURI 3/10/03

Spatial Integration of EEG SignaturesSpatial Integration of EEG Signatures

)|( xtP

)|( xtPt

Prioritization of image search via spatial integration

ONR MURI 3/10/03

Experimental Design An experimental session consists of 3 blocks of 85 trials

Image duration varies in each block - 200ms, 100ms, 50ms

Each trial consists of a sequence of 100 natural images

Each image sequence has a 50% chance of containing 1 target image

Target images contain a person in a natural scene

Non-target distractor images are natural scenes

1 trial - 100 images20 sec (200ms/image)10 sec (100ms/image) 5 sec ( 50ms/image)

targetdistractor

+fixation

50% chance a sequencecontains a target image

5 sec

. . .85 trials per block


ONR MURI 3/10/03

Demonstration: Real-time on-line single-Demonstration: Real-time on-line single-trial detection of visual targetstrial detection of visual targets

Frames selected based on EEG signature

Visual target detection events


ONR MURI 3/10/03

Performance at different frame rates


ONR MURI 3/10/03

Performance comparison between Performance comparison between detected signature and overt response detected signature and overt response

(button release)(button release)


ONR MURI 3/10/03

Conclusions and Future WorkConclusions and Future Work

ONR MURI 3/10/03

Backup Slides

ONR MURI 3/10/03

Model of V1: Spiking NeuronsModel of V1: Spiking Neurons

ONR MURI 3/10/03

Characterization of V1 model without Characterization of V1 model without contextual interactionscontextual interactions

Characterization of Simple and Complex Cells in layer 4C

black curve : Ringach et al., J. Neurosci, 2002blue curve: Model

Model

C S

Ringach et al., J. Neurosci, 2002

Orientation tuning of Simple and Complex Cells in layer 4C

ONR MURI 3/10/03


Carandini and Ferster., J. Neurosci, 2000 Model

Characterization of Simple and Complex Cells using membrane potential in layer 4C

ONR MURI 3/10/03


Dynamics of orientation tuning

McLaughlin et al., PNAS 2000

.0.5 .1 .15

Model

ONR MURI 3/10/03


from Sceniak et al, Nat. Neuro, 1999

Contrast-dependent Receptive Fields

Model

ONR MURI 3/10/03

Distribution of simple Distribution of simple and complex cellsand complex cells

ONR MURI 3/10/03

Development of an Intelligent Development of an Intelligent Image Integration Inference Image Integration Inference

Engine (IEngine (I44E)E)Objectives

A. Investigate the hypothesis that the visual system uses probabilistic inference to integrate bottom-up, top-down and “horizontal” information.

B. Develop an intelligent image analysis framework which exploits the integration mechanisms and strategies used in biological vision systems.

C. Demonstrate for specific applications the advantages of I4E, in particular when considering the human analyst as part of the I4E system (I4E = biomimetic algorithms + human visual system)

ONR MURI 3/10/03

Development of an Intelligent Development of an Intelligent Image Integration Inference Image Integration Inference

Engine (IEngine (I44E)E)Project Objectives Satisfied Project Leader

Mechanisms for surround suppression in a spiking-neuron model of cortical hypercolumns in V1

A Columbia

Perceptual salience as novelty detection in cortical pinwheel space

A,B Columbia

A Bayesian network for integrating cues for inferring intermediate-level scene attributes: Application to figure-ground segmentation and tracking

B,C Columbia

Prioritization of image search via spatial integration of EEG signatures

C Columbia

A fast maximum-likelihood approach to spectral unmixing

C Columbia

onr muri 3/10/03 sajda lab … intelligent image integration through probabilistic inference in...

Documents