1

A biologically useful memory mechanism for the rapid deploymentof visual attention

Ken Nakayama

2

seeing = visibility X attention

+

R

3

Metaphors for vision

• Camera: It’s like a picture• Hand: it’s more active

Attention is the hand

How is it controlled?

4

Backgroundsome robust examples of attention

• Change blindness

• Inattentional blindness (even at fovea)

• Attentive tracking “hand and fingers”

5

Rensink flicker experiment• method: alternate two pictures

– Ask subjects to identify changes

• If we were aware of everything in picture, should be easy

• raise your hand when you see the change don’t tell others

At Nissan CBR

6

QuickTime™ and aAnimation decompressor

are needed to see this picture.

Rensink et al., Simons and Levin Change Blindness

7

QuickTime™ and aAnimation decompressor

are needed to see this picture.

8

summary of Rensink flicker experiment

contrary to our phenomenological experiencewe are not aware of everything in our visual world

large changes can escape our notice

Conclusion: seeing requires attention

9

~ 0.5 seconds

attention asan inertia-lesshand

can change size and positionapprox 4-6 times/sec

10

Other functions of attention

• Guidance of motor behavior

• Foraging for food

11

12

13

The intensity of predation depends . .. on the use of specific searching images. This implies that the birds perform a highly selective sieving operation on the visual stimuli that reach their retina . . . birds can only use a limited number of different search images at the same time.

L. Tinbergen(1960)

14

Marian Dawkins (1971)Shifts of ‘attention’ in chicks during feeding

15

Search image widespread?

• Chicks (Dawkins, 1971)

• Pigeons (Reid and Shettleworth, 1992))

• Blue Jays (Bond)

• Bumble Bees (1992)

• Butterflies (Stanton, 1984)

Is it a mental image or could it be something else?

16

Claim

There exists a primitive distributed memory system (seen in our human experiments) that could account for shifts of attention attributed to search images

17

It is part of a fast transient attentional system

(Nakayama and Mackeben, 1989

Vision Research)

18

msec

msec

19

Isolation of thetransient component

Keep location constant

20

Is the transient component due to its activation by a

sensory transient ?

NOsuch sensory transients not

necessary

analogy to “action potential ?”

21

Deploy transient attentionwithout a localsensory transient

Decoys

22

decoy cueing

normalcueing

23

Transient attention is very fast, rises to peak within 100

msec.

Can transient attention learn to go quickly to the appropriate position

on a larger object ?

Is it fast AND flexible

Kristanjansson, Mackeben & Nakayama, 2001

24

cue

stimulus display

fixat ion

cue target relation

fixed

variable

25

50

60

70

80

90

rommsubjects

fixedvariable

learning,

Attention can be effectively deployed toa location within anobject

26

Is the learning a property ofsustained or transient attention ?

27

50

60

70

80

90

0 100 200 300 400 500 600

Cue lead time

AK

C ue lead tim e

Pe

rce

nt

corr

ect

vary cuelead time

Keep target position within the cue constant

28

Learning

• How fast does it occur ?• How stable is it ?

• Method: use quasi random streaks of cue target regularities

29

1

2

3

4

5

time

sequence of cues

50

60

70

80

90

0 1 2 3 4 5 6 7 8 9

P o s itio n in "s tre a k"

A K

A MA M H

% c

orre

ct

Build-up of learningst

reak

30

+

+

+

40

50

60

70

80

90

1 2 3 4 5 6 7 8

NH

AK

position in sequence

color ?st

reak

31

20

30

40

50

60

70

0 1 2 3 4 5 6 7 8 9

NH

AK

Position in streak

perc

ent

corr

ect

local shape ?

32

Learning (summary so far)

• Is very rapid and is temporary• Can be linked position within an object• Can linked to a color within an object• Can linked to a local shape within an object

Are there things “attention” can’t learn ?

33

can it learn a 2nd order relation?

+

+

color and position

++

shape and position

34

0

10

20

30

40

50

60

random consistent

++

+

++

+

random

consistent

AKAMH

2 subjects

35

50

60

70

80

90

0 100 200 300 400 500 600

Cue lead time

AK

C u e le a d t im e

Pe

rce

nt

corr

ect

Part of a fast mechanism of Attentional deployment , reaches peak with 100 msec

36

+

+

+

n-1

n

Identify shape ofthe odd colored target

Maljkovic and Nakayama

Different paradigm to study the same process

position and colorof the targetcan change

37

.

baseline

1 2 3 4 5 6 7 8 9 10 11480

500

520

540

560

580

600

620

640

660

KN

+ +

+. .1 2 nrepeat color

38

4 65321530

550

570

590

610

Order in Sequence

Repeat targetposition

++ ++

1 2 n

. .

39

Attention can learn colors, shapes and locations

Is attention, (or are we)

learning what to expect, thenforming a search image in ourminds ?

Approach:manipulate target color uncertainty

40

prediction

.

0 .2 .4 .6 .8 1

changeeverytrial

nochange

Probability of color change

?

41

VM

400

420

440

540

560

580 KN

0 .2 .4 .6 .8 1

changeeverytrial

nochange

Probability of color change0 .2 .4 .6 .8 1

42

make expectancy very explicitmaximizing the possible use ofsearch images, pitting it againstrepetition

••

•

1 2 1 2 1 2 1 2

Double alternation paradigm

Target color over trials

43

••

•

21

Trialsequence:

order in sequence

repetition

reac

tio

n t

ime

predicted outcomes

1 2 1 2 1 2 1 2

Search image(expectancy)

44

1 2 1 2 1 2 1 2

45

Learning is passive, mechanistic, piecemeal

(color, position)Its not expectancy,

Its not a search image

Its not under conscious control

46

Its not just a linkage to the previous targets but active

inhibition to non-targets

47

.

+ +

distractor color constant

.

+ +

distractor color varies

560

580

600

620

640

654321654321

660

620

640

KN

Distractor varies

distractorsame

SS

48

Fine grain temporal analysis of the learning mechanism

~ 2nd order reverse correlation

What is the influence of a singletrial in the past ?

49

+

n

+

+

n-1

+

sameor different ?

+

n- i

sameor different ?

What is influence ofa single trial inthe past ?

current trial

past future

nn-i

time

50

.

+7+5+3+1-1-3-5-7-9-11-13-15550

560

570

580

590

600

610

FuturePast

EFB

Same Color

Different Color

Same Color

Different Color KN

+7+5+3+1-1-3-5-7-9-11-13-15610

620

630

640

650

660

670

51

.

FuturePast

+7+5+3+1-1-3-5-7-9-11-13-15

EFB

-50

-40

-30

-20

-10

0

10

-15 -13 -11 -9 -3

KN

-50

-40

-30

-20

-10

0

10

+7+5+3+1-1-5-7

52

Linkage is not confined to color

Probably any salient feature

(spatial frequency, for example), will do

53

.

-35

-45

-25

0

15

5

-5

-15

-25

-20

-15

-10

-5

10

0

10

5

+7+5+3+1-1-3-5-7-9-11-13-15

-15 -13 -11 -9 -3

VM

GP

FuturePast

+7+5+3+1-1-5-7

54

Learning restricted to what attracts attention

(features, positions)

Not the fine details that attention allows one to

process

55

.

FuturePast

-15 -13 -11 -9 -3

KN

-50

-40

-30

-20

-10

0

10

+7+5+3+1-1-5-7

-25

-20

-15

-10

-5

10

0

10

5

-15 -13 -11 -9 -3

VM

+7+5+3+1-1-5-7

56

.

1614121086420620

640

660

680

700

720

740

3 5421620

640

660

680

700

720

740

Time (sec) Order in Sequence

KN KN

1sec 2sec3sec

Is the build up and decay over timeor over events?

57

Independence of features/location

58

Trial n - i

trial n

Effect of target and distractor position in the past

59

KN

560 580 620 560600 580620

Previous Position

600

influence of trial n-1

60

Independence of color and location learning

61

hypothesized properties of the memory system

Time (trials) Time (trials)

gradedsummateslinearity (superposition)has independent components (features and locations)

62

RC

location1

focalattention

location2

++

-

++

leakybucket

63

What is learned ?What is the reinforcer ?

Simple identifiers of places where attention just wentand didn’t go ?

Memory kernel function

64

Generalizations beyond the measure of attention

• Speeds eye movements (human and monkey)

• Speeds motor behavior (manual pointing)

65

Saccadic eye movements(in human and in monkey)

66

+

+

+

time+

trial n

trial n - 1

Task: make a saccade to the odd colored target

Human eye movements

67

sacc

adic

late

ncy

54321200

210

220

230VM

position in samecolor sequence

McPeek, Maljkovic & Nakayama

Cumulative effects of learning

-15

-10

-5

0

5

-10 -8 -6 -4 -2 0 2 4

late

ncy

diff

eren

ce

past future

Memory kernel

68

Learning of speeded saccades in monkey

Position in same color sequence

Sac

cadi

c la

tenc

y

Robert McPeek and Ed KellerSmith Kettlewell

69

Learning generalizes to manual pointing

+

until the response

+

Trial 1 1200ms

Measure RT withTouch sensitive screen

(Song & Nakayama, 2003)

70

Learning generalizes to manual pointing

+

until the response

+

Trial 1 1200ms

+

Trial 2 1200ms

+

until the response

Measure RT withTouch sensitive screen

(Song & Nakayama, 2003)

71

Touching target with finger

1 2 3 4 51 2 3 1 2 ….. …..

525

550

575

600

625

650

675

700

1 2 3 4 5 6 7

sequence in order

AH

baseline

425

450

475

500

525

550

1 2 3 4 5 6 7 8

sequence in order

JHS

baseline

RT

(ms)

Position within “Same color” Sequence

Target:

72

Implications for foraging

• Don’t need a cognitive concept like a search image

• Low level temporary, passive, graded connection strengths (plus and minus) may be sufficient

73

Marian Dawkins (1971)Shifts of ‘attention’ in chicks during feeding

74

Anatomical locale/mechanism?

Object centerednot retinotopic cortex

Independent featuressimple 2 layered network

75

Relationship to other learning systems

• Is it a completely specialized sub-system for attention ?

• OR

• Is it the germ for short term memory more generally ?

X

Hypothesis: it’s a biologically conserved memorysystem for the rapid deployment of visualattention.

76

A biologically useful and conservedmemory mechanism for the rapid deploymentof visual attention (and other possible functions

Ken Nakayama

Manfred MackebenVera MaljkovicRobert McPeekArni KristjanssonJoo-Hyun Song