ISETBio: A Computational Engine for Modeling the Early Visual SystemJames Golden1, David Brainard2, E.J. Chichilnisky1, Fred Rieke3, Joyce Farrell1, Nicolas Cottaris2, Haomiao Jiang1, Xiaomao Ding2, Ben Heasley2, Jonathan Winawer1, Brian Wandell1; 1. Stanford University, 2. University of Pennsylvania, 3. University of Washington

8/8/

15, 1

1:53

AM

Page

1 o

f 3ht

tp:/

/ww

w.ib

.cne

a.go

v.ar

/~re

dneu

/201

3/B

OO

KS/

Prin

cipl

es%

20of

%2…

w2/

ovid

web

.cgi

sidn

jhko

algm

eho0

0dbo

okim

ageb

ookd

b_7c

_2fc

~26.

htm

Back

V Pre

cep

tio

n

A s

che

ma

tic

dia

gra

m o

f a

ra

dia

l se

ctio

n s

ho

win

g t

he

va

rio

us

lay

ers

of

the

re

tin

a p

ub

lish

ed

in

18

87

by

Fe

rru

cio

Ta

rtu

feri

. Ta

rtu

feri

was

Pro

fess

or

of

Oph

thalm

olo

gy

at

the

Un

iver

sity

of

Mes

sin

a i

n S

icil

y an

d w

as

inte

rest

ed p

rim

ari

ly i

n c

lin

ical

eye

dis

ease

s. T

his

stu

dy,

“S

ull

an

ato

mia

del

lare

tin

a”

(In

tern

ati

on

al

Mon

ats

sch

rift

An

ato

mie

Phys

iolg

ie 4

:42

1-4

41

),appea

red a

yea

r bef

ore

Ram

ón

y C

aja

l's

firs

t w

ork

on

th

e re

tin

a.

Cou

rtes

y of

Rober

t R

odie

ck.

…on

e day

in

win

ter,

on

my

retu

rn h

om

e, m

y m

oth

er,

seei

ng t

hat

I w

as

cold

, off

ered

me

som

e te

a,

a t

hin

g I

did

not

ord

inari

ly

physiological !optics

image!formation

retinal!processing inferencephoto!

transduction

8/8/

15, 1

1:53

AM

Page

1 o

f 3ht

tp:/

/ww

w.ib

.cne

a.go

v.ar

/~re

dneu

/201

3/B

OO

KS/

Prin

cipl

es%

20of

%2…

w2/

ovid

web

.cgi

sidn

jhko

algm

eho0

0dbo

okim

ageb

ookd

b_7c

_2fc

~26.

htm

Back

V Pre

cep

tio

n

A s

che

ma

tic

dia

gra

m o

f a

ra

dia

l se

ctio

n s

ho

win

g t

he

va

rio

us

lay

ers

of

the

re

tin

a p

ub

lish

ed

in

18

87

by

Fe

rru

cio

Ta

rtu

feri

. Ta

rtu

feri

was

Pro

fess

or

of

Oph

thalm

olo

gy

at

the

Un

iver

sity

of

Mes

sin

a i

n S

icil

y an

d w

as

inte

rest

ed p

rim

ari

ly i

n c

lin

ical

eye

dis

ease

s. T

his

stu

dy,

“S

ull

an

ato

mia

del

lare

tin

a”

(In

tern

ati

on

al

Mon

ats

sch

rift

An

ato

mie

Phys

iolg

ie 4

:42

1-4

41

),appea

red a

yea

r bef

ore

Ram

ón

y C

aja

l's

firs

t w

ork

on

th

e re

tin

a.

Cou

rtes

y of

Rober

t R

odie

ck.

…on

e day

in

win

ter,

on

my

retu

rn h

om

e, m

y m

oth

er,

seei

ng t

hat

I w

as

cold

, off

ered

me

som

e te

a,

a t

hin

g I

did

not

ord

inari

ly

physiological !optics

image!formation

retinal!processing inferencephoto!

transduction

8/8/

15, 1

1:53

AM

Page

1 o

f 3ht

tp:/

/ww

w.ib

.cne

a.go

v.ar

/~re

dneu

/201

3/B

OO

KS/

Prin

cipl

es%

20of

%2…

w2/

ovid

web

.cgi

sidn

jhko

algm

eho0

0dbo

okim

ageb

ookd

b_7c

_2fc

~26.

htm

Back

V Pre

cep

tio

n

A s

che

ma

tic

dia

gra

m o

f a

ra

dia

l se

ctio

n s

ho

win

g t

he

va

rio

us

lay

ers

of

the

re

tin

a p

ub

lish

ed

in

18

87

by

Fe

rru

cio

Ta

rtu

feri

. Ta

rtu

feri

was

Pro

fess

or

of

Oph

thalm

olo

gy

at

the U

niv

ers

ity

of

Mess

ina i

n S

icil

y an

d w

as

inte

rest

ed p

rim

ari

ly i

n c

lin

ical

eye

dis

ease

s. T

his

stu

dy,

“S

ull

an

ato

mia

dell

are

tin

a”

(In

tern

ati

on

al

Mon

ats

sch

rift

An

ato

mie

Phys

iolg

ie 4

:42

1-4

41

),appeare

d a

year

befo

re R

am

ón

y C

aja

l's

firs

t w

ork

on

th

e r

eti

na.

Cou

rtesy

of

Robert

Rodie

ck.

…on

e d

ay i

n w

inte

r, o

n m

y re

turn

hom

e,

my

moth

er,

seein

g t

hat

I w

as

cold

, off

ere

d m

e s

om

e t

ea,

a t

hin

g I

did

not

ord

inari

ly

physiological !optics

image!formation

retinal!processing inferencephoto!

transduction

Scene representations Physiological optics Photo transduction Retinal processingInference

Biophysical model captures cone nonlinearities

I (pA

)

-130

-110

-90

-70

86420time (s)

data full model linear model

stimulus dR(t)/dt = s(t)� �R(t)

dP (t) = �R(t) + ⌘ � �P (t)

.

.

.

dG(t)/dt = s(C)� P (t)G(t)

KEY: model provides tool to determine how nonlinearities in cone responses affect coding

see also Clark at el., 2013

F

L-coneM-cone

S-co

ne

Decision variable

Prob

abili

ty

Gabor in RF(signal)

Gabor absent(noise)discriminant

vector

Gabor in RF(signal)Gabor absent

(noise)

e

minadiiscrictorve

F

S-co

ne

MM

Sco

ne

L-L cocone-coconen

d

GaboGabor in RF(sig(s(s nalnal)Gabor abbssentt

(noisse)(noisse)

G

Figure 1

10 100

0.1

Frequency (Hz)

Cone

noi

se

(pA2 p

er H

z)

E

C

Frequency (Hz)

Wei

ght f

unct

ion

(arb

itrar

y un

its)

10 100

0.01

1

10– 6

10– 5

Cone

sign

al

(pA2 p

er H

z)

1

0 0.1 0.2 0.3

0

20

40

80

Time (s)

fA p

er R

*

0 0.2 0.4 0.6

8.0

6.5

7.0

7.5

Time (s)

R* p

er se

c x1

000

D

B

60

A

Time (s)

Gabor

FP

Targs

Targets onStimulus

666 msFixation500 ms

Delay100-600 ms

0 0.5 1.0 1.5

Classifier weights(eye movements)

Computational observer accuracy: 75%Viewing distance (m)

Photocurrent (with noise)

Photon absorptions

Cla

ssifi

catio

n ac

cura

cy (

%)

The Primate Early Visual System

The primate early visual system is comprised of the cornea, the lens and the retina. The retina has a number of layers, including the photoreceptors, bipolar cells and retinal ganglion cells.

ISETBio: Image Systems Engineering Toolbox for Biology

ISETBio is based on ISET, the Image Systems Engineering Toolbox. The processing pipeline consists of computational simulations of a hyperspectral scene and a particular display device, followed by the optics of the cornea and the lens, the transduction in the photoreceptors and further processing in the bipolar cells and retinal ganglion cells. The final stage in the pipeline is the computational observer, a linear classification that separates signal trials from noise trials.

The Scene and Display: Changing the Illuminant Spectrum

ISETBio allows for modeling of the spectra of the reflectance properties of object surfaces as well as the illuminant. Here, the spectrum of the illuminant is changed, and the reflected light from each object changes as a consequence.

Cone Optics:The Point Spread Function

The hyperspectral representation of the stimulus on a display device is projected upon the surface of the retina through the cornea and lens. ISETBio simulates the PSF for different cone types, which results in S cones with wider PSFs than L and M cones.

Cone Mosaic:Isomerizations and Photocurrent

The cone mosiac is generated with the appropriate ratio of L, M and S cones, and the proper spacing for a given eccentricity. We employ a biophysical model of the cone photocurrent generation that captures nonlinear dynamics over time across image patches of greatly varying mean luminance.

The Bipolar and RGC Mosaics

Fig. 3: Failures of linear integration in Off parasol RGCs is due to nonlinear subunit RF structure. (A) Example natural image used in the modeling and experiments below. Diagram shows the spatial arrangement of subunits used in the RF models outlined in (B). (B) Two RF models that exhibit different modes of spatial integration. In both models, the RF center is composed of subunits, each of which is described by a linear Gaussian. The output of each subunit is weighted by a larger Gaussian representing sampling by the RF center. The linear-nonlinear (LN) RF model (left) takes the linear sum of subunit outputs and passes that sum through a rectified-linear output nonlinearity to yield the response. The nonlinear subunit RF model (right, “subunit”) applies the rectified-linear nonlinearity at the output of each subunit before summation at the ganglion cell. (C) Model outputs for 10,000 randomly-selected patches (gray points) from the image in (A). Black line indicates unity. (D) Histogram of the model response differences (subunit model response minus LN model response) for randomly-selected image patches (gray line). For the following experiments, we sampled these image patches in order to uniformly span the range of model differences (black line). (E) Spike rasters from an example On parasol RGC in response to a flashed stationary natural image patch (top) and its linear equivalent disc (bottom). Each stimulus was presented for 200 ms. Colored outlines indicate the region of the scene in (A) from which this patch was drawn. (F) Same as (E) for an example Off parasol RGC. (G) Mean spike counts (over five presentations of each patch) in response to 40 different patches from a natural image, for the example On parasol RGC (top) and Off parasol RGC (bottom). Dashed line indicates unity. (H) We subtracted the response to each linear equivalent disc from the response to its associated image, giving us a measure of the strength of nonlinear integration (“measured difference”) and compared that to the difference in model outputs for each image patch presented. In contrast to the On parasol RGC (top), the image patches that drive nonlinear responses in the Off parasol RGC (bottom) are well-predicted by the difference in RF model outputs. Solid lines show linear fits, with linear correlation coefficient indicated at top left. (I) Population data showing the coefficients of variation (standard deviation between image and disc responses divided by average image response) for On and Off parasol RGCs. Open symbols correspond to individual cells, closed symbols denote mean +/- S.E.M. (n = 10 On parasol RGCs, 12 Off parasol RGCs). (J) Population data for On and Off parasol RGCs showing linear correlation coefficients between experimentally measured differences and model response differences.

20

10

020100

15

10

5

0

0.40.30.20.10.0 Model difference (a.u.)

15

10

5

0

0.40.30.20.10.0

20

10

020100200 ms

200 ms

1.0

0.5

0.0

Corre

latio

n co

effic

ient

On Off

0.6

0.4

0.2

0.0 Coe

fficie

nt o

f var

iatio

n

On Off

2.0

1.5

1.0

0.5

0.0

LN re

spon

se

2.01.00.0Subunit Response

1

10

100

1000

Imag

e pa

tche

s

0.40.20.0Subunit - LN

Random Presented

Mea

sure

d di

ffere

nce

(spi

kes)

Res

pons

e to

disc

(spi

kes)

Response to image (spikes)

AModel responses Diff. histogram

CB

CV = 0.17

CV = 0.47 ρ = 0.79

ρ = -0.06On parasol!RGC

Off parasol!RGC

E G H

F

I

J

D

RF center

Subunit

On parasol RGC

Off parasol RGC

On parasol RGC

Off parasol RGC

Nonlinear subunitLinear-nonlinear

RGC

∑

RGC

∑

The cone photocurrent is fed into a model of the bipolar cell mosaic, which captures spatial nonlinearities that improve the accuracy of responses to natural scenes. The RGC mosaic is modeled with a coupled linear-nonlinear-Poisson model that outputs spikes. ISETBio captures a range of nonlinearities through the early visual system.

Threshold Measurements with the Computational Observer

The Computational Observer

The computational observer method has some similarity to the ideal observer, but is more empirical. Noise is simulated for each trial, and summary measurements of each trial can be fed into a linear classifier to find the threshold given the structure of the model.

References1. Brainard, Wandell, et al. (2015, June). Isetbio: Computational tools for modeling early human vision. In Imaging Systems and

Applications (pp. IT4A-4). Optical Society of America.2. Farrell, Wandell et al. (2014). Modeling visible differences: the computational observer model. In Proc. 2014 Soc. Inf.

Disp.(SID) Int. Symp.3. Turner & Rieke, (2016). Synaptic Rectification Controls Nonlinear Spatial Integration of Natural Visual Inputs. Neuron.4. Pillow, Chichilnisky, et al.(2008). Spatio-temporal correlations and visual signalling in a complete neuronal

population. Nature, 454(7207), 995-999.5. Angueyra & Rieke, (2013). Origin and effect of phototransduction noise in primate cone photoreceptors. Nature

neuroscience, 16(11), 1692-1700.

A classical experiment in the measurement of perceptual thresholds is that of the Vernier stimulus. We can simulate the experimental pipeline by measuring the responses to a Vernier stimulus with and without an offset. The accuracy of the linear classifier can be used to generate a prediction for physiological or psychophysical experiments.

Bipolar

RGC

http://webvision.med.utah.edu/

No Vernier Offset (noise)

Vernier Offset (signal)

github.com/isetbio; jgolden1@stanford.edu