HOW ARE COGNITIVE MAPS HOW ARE COGNITIVE MAPS REPRESENTED IN THE BRAIN?REPRESENTED IN THE BRAIN?

VISUAL VS. PROPOSITIONAL VISUAL VS. PROPOSITIONAL REPRESENTATIONS AND IMPLICATIONS FOR REPRESENTATIONS AND IMPLICATIONS FOR

GPS TECHNOLOGYGPS TECHNOLOGY

Serj MooradianSerj MooradianAlison WheatleyAlison Wheatley

Becky WrightBecky WrightJane EmertonJane Emerton

Penelope DimitrakopoulouPenelope Dimitrakopoulou

INTRODUCTIONINTRODUCTION

Behaviourism: from the 1920’s onward, Behaviourism: from the 1920’s onward, behaviour was explained in terms of simple behaviour was explained in terms of simple stimulus-responsestimulus-response mechanisms. Rats learnt mechanisms. Rats learnt mazes by linking rewards (stimulus; i.e. food) mazes by linking rewards (stimulus; i.e. food) with muscle movements (response; i.e. with muscle movements (response; i.e. running in particular directions).running in particular directions).

Tolman (1932) cast doubt on this theory – rats Tolman (1932) cast doubt on this theory – rats in mazes: swimming versus running.in mazes: swimming versus running.

The researchers concluded The researchers concluded that rats that rats construct a ‘construct a ‘cognitive cognitive mapmap’ in their ’ in their brains, and use the brains, and use the internal internal representation to representation to solve mazes.solve mazes.

MORE ABOUT COGNITIVE MORE ABOUT COGNITIVE MAPPINGMAPPING

A cognitive map is a cognitive representation of A cognitive map is a cognitive representation of an environment, that is built up whenever a an environment, that is built up whenever a person encounters a novel environment (Lynch, person encounters a novel environment (Lynch, 1960). It can comprise of landmarks, nodes, and 1960). It can comprise of landmarks, nodes, and paths.paths.

The more experience a person has with an The more experience a person has with an environment, the better their cognitive map environment, the better their cognitive map (Jacobson et al 2001). (Jacobson et al 2001).

The type and content of The type and content of howhow this information is this information is represented in the brain is still a prominent and represented in the brain is still a prominent and controversial debate.controversial debate.

There appears at present to be two main There appears at present to be two main arguments – those who believe cognitive maps arguments – those who believe cognitive maps are represented are represented pictoriallypictorially vs. those who believe vs. those who believe they are represented they are represented propositionallypropositionally..

HOWHOW ARE COGNITIVE MAPS ARE COGNITIVE MAPS REPRESENTED IN THE BRAIN?REPRESENTED IN THE BRAIN?

1. PICTORIAL vs. PROPOSITIONAL1. PICTORIAL vs. PROPOSITIONAL Type and content of cognitive maps is still Type and content of cognitive maps is still

being debatedbeing debated Pictorial – Pictorial – KosslynKosslyn Depiction of environmentDepiction of environment Implicates visual system in production Implicates visual system in production

and processing – the ‘minds eye’and processing – the ‘minds eye’ Propositional – Propositional – PylyshynPylyshyn Similar to languageSimilar to language Syntactical representationSyntactical representation

PROPOSITIONAL vs. PICTORIALPROPOSITIONAL vs. PICTORIAL

North, (TREE, HOUSE)North, (TREE, HOUSE) South (LAKE, HOUSE)South (LAKE, HOUSE)

-Linguistic description of -Linguistic description of relationships relationships

-Discrete symbols-Discrete symbols

-Requires a symbols to describe -Requires a symbols to describe the relationshipsthe relationships

-Follows grammatical rules-Follows grammatical rules

-Abstract symbols-Abstract symbols

-Unambiguous-Unambiguous

-Multiple modes of perception-Multiple modes of perception

-More easily accessible-More easily accessible-Does not require knowledge of -Does not require knowledge of languagelanguage-Not discrete symbols-Not discrete symbols-Does not require a symbol to -Does not require a symbol to describe describe the the relationshipsrelationships-Concrete symbols-Concrete symbols-Ambiguous-Ambiguous-Only visual mode of perception-Only visual mode of perception

EARLY COGNITIVE MAP EARLY COGNITIVE MAP EXPERIMENTSEXPERIMENTS

Images cannot be directly observed so study brain at Images cannot be directly observed so study brain at functional levelfunctional level

Kosslyn et al (1978)– Mental Image Scanning TasksKosslyn et al (1978)– Mental Image Scanning Tasks MethodMethod

Memorise a map of 7 locationsMemorise a map of 7 locations Focus on a locationFocus on a location Decide if a second location was on the mapDecide if a second location was on the map Picture a black speck moving between the two locationsPicture a black speck moving between the two locations

Results Results Scanning time increased linearly with the distance Scanning time increased linearly with the distance

between the two locationsbetween the two locations Conclusion Conclusion

It takes longer to scan larger distances compared to It takes longer to scan larger distances compared to small if images are depictionssmall if images are depictions

Images are depictive rather than descriptive and Images are depictive rather than descriptive and have spatial propertieshave spatial properties

EARLY EXPERIMENTS cont.EARLY EXPERIMENTS cont.ALTERNATIVE MODELALTERNATIVE MODEL

Pylyshyn (1981) – Critique of Kosslyn’s ExperimentsPylyshyn (1981) – Critique of Kosslyn’s Experiments Participants asked to Participants asked to seesee a scene a scene Use knowledge about really world to simulate eventsUse knowledge about really world to simulate events Does not tell us about inherent nature of imagingDoes not tell us about inherent nature of imaging

ExperimentExperiment Scanning phenomenon vanishes when asked to judge Scanning phenomenon vanishes when asked to judge

directiondirection Can switch attention between locations without simulating Can switch attention between locations without simulating

moving moving Distance between locations had no effect on response time Distance between locations had no effect on response time

of directionof direction ConclusionConclusion

Unclear which properties are due to the inherent Unclear which properties are due to the inherent nature of the mental imaging system and its nature of the mental imaging system and its mechanismmechanism

Or which properties are ‘cognitively penetrable’ by the Or which properties are ‘cognitively penetrable’ by the person’s own knowledge and beliefs during imagery.person’s own knowledge and beliefs during imagery.

NEW TECHNOLOGY GIVES US NEW TECHNOLOGY GIVES US A ‘WINDOW INTO THE MIND’A ‘WINDOW INTO THE MIND’

Attempts to solve debate via Attempts to solve debate via behavioural methods had led behavioural methods had led researchers down a ‘cul-de-sac’.researchers down a ‘cul-de-sac’.

However…. Advances in technology However…. Advances in technology offered a new opportunity.offered a new opportunity.

Opened up a “window into the mind”.Opened up a “window into the mind”.

PET (POSITRON EMISSION PET (POSITRON EMISSION TOMOGRAPHY)TOMOGRAPHY)

~ ~ ²/²/3 3 of activation shared by vision and imagery.of activation shared by vision and imagery.

(Kosslyn, Thompson & Alpert, 1997)(Kosslyn, Thompson & Alpert, 1997)

However, PET not very precise.However, PET not very precise.

rTMS (Repetitive Transcranial rTMS (Repetitive Transcranial Magnetic Stimulation)Magnetic Stimulation)

Applying rTMS to the occipital cortex causes temporary impairment of visual mental imagery (Kosslyn et al., 1999).

Pylyshyn (2002): Possible visual system Pylyshyn (2002): Possible visual system can be involved in both vision and can be involved in both vision and mental imagery but without generating a mental imagery but without generating a picture-like representation.picture-like representation.

““It is important not only that such … areas It is important not only that such … areas be involved in imagery, but also that be involved in imagery, but also that their involvement be of the right sort – their involvement be of the right sort – that the way their topographical that the way their topographical organization is involved reflects the organization is involved reflects the spatial properties of the image…”spatial properties of the image…”

(Pylyshyn, 2002, p.175)(Pylyshyn, 2002, p.175)

PYLYSHYN’S RESPONSEPYLYSHYN’S RESPONSE

RETINOTOPICALITYRETINOTOPICALITY

Area 17:Area 17:

Located in the occipital lobe. Involved in early visual processing e.g. shape, location.

Activity is distributed according to a ‘functional space’

i.e. the spatial properties of the visual scene are reflected in the spatial patterns of the brain activity (Kosslyn, 1994).

TOP

RIGHT

SIDE

BOTTOM

Visual Spatial Spectrum:

FUNCTIONAL SPACE: FUNCTIONAL SPACE: Different spatial areas within the visual Different spatial areas within the visual field reflected onto the retina….field reflected onto the retina….

… result in activation in different primary visual areas.

Adapted from Slotnick et al (2005)

Q. DOES THE RETINOTOPICAL Q. DOES THE RETINOTOPICAL ACTIVITY FOUND IN VISION ALSO ACTIVITY FOUND IN VISION ALSO

OCCUR DURING MENTAL IMAGERY?OCCUR DURING MENTAL IMAGERY?

Klein Klein et alet al (2004), Slotnick (2004), Slotnick et alet al (2005); (2005);• fMRI study (Functional Magnetic fMRI study (Functional Magnetic

Resolution Imaging)Resolution Imaging)• Used simple visual stimuli:Used simple visual stimuli:

Asked participants to both visualise and imagine the stimuli.

Vertical-axis space = red & green areas Horizontal-axis space = blue & yellow areas

Vision Imagery

Adapted from Slotnick et al (2005)

PYLYSHYN’S RESPONSE TO PYLYSHYN’S RESPONSE TO KOSSLYNKOSSLYN

Kosslyn’s findings have not been Kosslyn’s findings have not been replicatedreplicated

They are inherently biasedThey are inherently biased Participants were asked to visualize an Participants were asked to visualize an

imageimage By visualizing an image, they are being By visualizing an image, they are being

asked to ‘see it’ and hence their visual asked to ‘see it’ and hence their visual cortex cortex mustmust be stimulated be stimulated

PYLYSHYN’S RESPONSE TO PYLYSHYN’S RESPONSE TO KOSSLYNKOSSLYN

cont.cont. Visual images are caused by a Visual images are caused by a

response to light on the retina and response to light on the retina and mental imagery is a top-down process mental imagery is a top-down process from higher cortical processesfrom higher cortical processes It is unlikely that the same response will It is unlikely that the same response will

occur to both types of imageryoccur to both types of imagery Kosslyn: literally pictures in the brainKosslyn: literally pictures in the brain

If it takes longer to ‘scan’ greater If it takes longer to ‘scan’ greater imagined distance (time=distance/speed)imagined distance (time=distance/speed)

i.e. simulation of ‘real space’i.e. simulation of ‘real space’

PYLYSHYN’S RESPONSE TO PYLYSHYN’S RESPONSE TO KOSSLYNKOSSLYN

cont.cont. Other differences in how visual and Other differences in how visual and

mental images are accessed:mental images are accessed: Signature propertiesSignature properties Manipulations of mental images not accurateManipulations of mental images not accurate Cortical images fade more quicklyCortical images fade more quickly Emmert’s LawEmmert’s Law Two forms of images not connected to the Two forms of images not connected to the

motor system in the same way (reaching for an motor system in the same way (reaching for an object)object)

Retinal images are pre-interpretation; mental Retinal images are pre-interpretation; mental images images are are the interpretationthe interpretation

PYLYSHYN’S RESPONSE TO PYLYSHYN’S RESPONSE TO KOSSLYNKOSSLYN

cont.cont. Cognitive maps are developed by being Cognitive maps are developed by being inin the the

environment, receiving cues, interpreting route environment, receiving cues, interpreting route cues as opposed to visual, pictorial cues on a cues as opposed to visual, pictorial cues on a mapmap

Use propositional knowledge to navigate, as it is Use propositional knowledge to navigate, as it is neededneeded Can be pictorial, but isn’t by natureCan be pictorial, but isn’t by nature

Blind people are able to form cognitive maps Blind people are able to form cognitive maps (Landau, 1986)(Landau, 1986) The experience of moving through an environment is The experience of moving through an environment is

enough to form a cognitive map, without actually enough to form a cognitive map, without actually navigating the environment (Millar, 1994)navigating the environment (Millar, 1994)

PYLYSHYN’S RESPONSE TO KOSSLYNPYLYSHYN’S RESPONSE TO KOSSLYNcont.cont.

When one is driving down the M4, one When one is driving down the M4, one does not necessarily visualize a map of does not necessarily visualize a map of the M4 going East-West and taking exit the M4 going East-West and taking exit 18 south to bath, one simply exits at 18 south to bath, one simply exits at exit 18 and drives (subconscious exit 18 and drives (subconscious behaviour)behaviour) One accesses one’s cognitive map of the One accesses one’s cognitive map of the

region without stimulating the visual cortex region without stimulating the visual cortex (however it is (hopefully) stimulated by the (however it is (hopefully) stimulated by the road ahead – but not terms of the cognitive road ahead – but not terms of the cognitive map)map)

GLOBAL POSITIONING GLOBAL POSITIONING SYSTEMSSYSTEMS

GPS’s are a network of satellites and GPS’s are a network of satellites and computers that can locate, track and computers that can locate, track and monitor any mobile object/individual on monitor any mobile object/individual on earthearth

GPS in cars provide the user with a GPS in cars provide the user with a pictorial map and auditory directionspictorial map and auditory directions

GPS is normally a navigation toolGPS is normally a navigation tool However - if we regard GPS as simply However - if we regard GPS as simply

an aid for cognitive map development…an aid for cognitive map development…

IMPLICATIONS FOR THE DESIGN IMPLICATIONS FOR THE DESIGN OF GLOBAL POSITIONING OF GLOBAL POSITIONING

SYSTEMS (GPS)SYSTEMS (GPS)… … if Kosslyn is right:if Kosslyn is right: GPS technology should display visual GPS technology should display visual

information….information…. However, a visual display may turn out However, a visual display may turn out

to be more of a hindrance than a help. to be more of a hindrance than a help. Kosslyn (1985) found evidence that Kosslyn (1985) found evidence that visual information similar to that of a visual information similar to that of a mental representation can actually mental representation can actually interfere with the cognitive map.interfere with the cognitive map.

IMPLICATIONS FOR THE DESIGN IMPLICATIONS FOR THE DESIGN OF GLOBAL POSITIONING OF GLOBAL POSITIONING

SYSTEMS (GPS)SYSTEMS (GPS)……if Pylyshyn is right:if Pylyshyn is right: GPS technology should provide GPS technology should provide

auditory informationauditory information

GPS TECHNOLOGY cond.GPS TECHNOLOGY cond.

London cabbiesLondon cabbies Accuracy/reliability of GPS??Accuracy/reliability of GPS?? Knowledge of road closures, etc.Knowledge of road closures, etc.

GPS instead of training or to aid training?GPS instead of training or to aid training? Navigation vs. cognitive mapsNavigation vs. cognitive maps

Pictorial vs Propositional debate Pictorial vs Propositional debate has not been resolved yethas not been resolved yetAlthough taxi drivers may not find Although taxi drivers may not find GPS useful on the job, there is GPS useful on the job, there is much agreement that it can be much agreement that it can be useful to develop cognitive maps in useful to develop cognitive maps in training.training.Cognitive maps prove to be Cognitive maps prove to be pictorial or propositional in nature pictorial or propositional in nature (or a combination of both). This will (or a combination of both). This will have implications as how best to have implications as how best to develop future technology.develop future technology.

REFERENCESREFERENCES Jacobson, D., Lippa, Y., Golledge, R.G., Kitchin, R., & Blades, M. (2001). Jacobson, D., Lippa, Y., Golledge, R.G., Kitchin, R., & Blades, M. (2001).

Rapid development of cognitive maps in people with visual impairments Rapid development of cognitive maps in people with visual impairments when exploring novel geographic spaces. when exploring novel geographic spaces. Bulletin of people-environment Bulletin of people-environment studies,studies, 1-8. 1-8.

Klein, I., Dubois, J., Mangin, J-F., Kherif, F., Flandin, G., Poline, J-B., Denis, Klein, I., Dubois, J., Mangin, J-F., Kherif, F., Flandin, G., Poline, J-B., Denis, M., Kosslyn, S. M., Le Bihan, D. (2004). Retinotopic organization of visual M., Kosslyn, S. M., Le Bihan, D. (2004). Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging. mental images as revealed by functional magnetic resonance imaging. Cognitive Brain Research, 22,Cognitive Brain Research, 22, 26-31. 26-31.

Kosslyn, S. M. (1983). Kosslyn, S. M. (1983). Ghosts in the Mind’s Machine. Creating and using Ghosts in the Mind’s Machine. Creating and using images in the brainimages in the brain. George J. McLeod Limited, Toronto.. George J. McLeod Limited, Toronto.

Kosslyn, S. M., Ball, T.M. and Reiser, B. J. (1978). Visual images preserve Kosslyn, S. M., Ball, T.M. and Reiser, B. J. (1978). Visual images preserve metric spatial information: Evidence from studies of image scanning. metric spatial information: Evidence from studies of image scanning. Journal Journal of Experimental Psychology: Human Perception and Performance, 4,of Experimental Psychology: Human Perception and Performance, 4, 47-60. 47-60.

Kosslyn, S. M., Pascual-Leone, A., Felician, O., Camposano, S., Keenan, J. P., Kosslyn, S. M., Pascual-Leone, A., Felician, O., Camposano, S., Keenan, J. P., Thompson, W. L., Ganis, G., Sukel, K. E., and Alpert, N. M. (1999). The role Thompson, W. L., Ganis, G., Sukel, K. E., and Alpert, N. M. (1999). The role of area 17 in visual imagery: Convergent evidence from PET and rTMS. of area 17 in visual imagery: Convergent evidence from PET and rTMS. Science, 284,Science, 284, 167-170. 167-170.

Kosslyn, S. M., Thompson, W. L., and Alpert, N. M. (1997). Neural systems Kosslyn, S. M., Thompson, W. L., and Alpert, N. M. (1997). Neural systems shared by visual imagery and visual perception: A positron emission shared by visual imagery and visual perception: A positron emission tomography study. tomography study. NeuroImage, 6,NeuroImage, 6, 320-334. 320-334.

Landau, B. (1996). Early map use as an unlearned ability. Landau, B. (1996). Early map use as an unlearned ability. Cognition, 22,Cognition, 22, 201-223.201-223.

Lynch, K. (1960). Lynch, K. (1960). The image of the city.The image of the city. Cambridge, MA: M.I.T. Press. Cambridge, MA: M.I.T. Press. Millar, S. (1994). Millar, S. (1994). Understanding and representing space theory and Understanding and representing space theory and

evidence from studies with blind and sighted children.evidence from studies with blind and sighted children. New York: New York: Oxford University Press.Oxford University Press.

Pylyshyn, Z. (2000). Is the "imagery debate" over? If so, what was it Pylyshyn, Z. (2000). Is the "imagery debate" over? If so, what was it about? In: E. Dupoux (ed.). about? In: E. Dupoux (ed.). Cognition: a critical look. Advances, Cognition: a critical look. Advances, questions and controversies in honor of J. Mehler.questions and controversies in honor of J. Mehler. Cambridge, MA, Cambridge, MA, MIT Press. Available online at: MIT Press. Available online at: http://ruccs.rutgers.edu/ftp/pub/papers/pylyshyn-mehler.pdfhttp://ruccs.rutgers.edu/ftp/pub/papers/pylyshyn-mehler.pdf

Pylyshyn, Z. (2003). Return of the mental image: are there really Pylyshyn, Z. (2003). Return of the mental image: are there really pictures in the brain? pictures in the brain? Trends in Cognitive Sciences, 7(3),Trends in Cognitive Sciences, 7(3), 113-118. 113-118.

Pylyshyn, Z.W. (2002). Mental Imagery: In search of a theory. Pylyshyn, Z.W. (2002). Mental Imagery: In search of a theory. Behavioral Brain Science, 25,Behavioral Brain Science, 25, 157-238. 157-238.

Slotnick, S. D., Thompson, W. L., and Kosslyn, S. M. (2005). Visual Slotnick, S. D., Thompson, W. L., and Kosslyn, S. M. (2005). Visual Mental Imagery Induces Retinotopically Organized Activation of Early Mental Imagery Induces Retinotopically Organized Activation of Early Visual Areas. Visual Areas. Cerebral Cortex, 15,Cerebral Cortex, 15, 1570-1583. 1570-1583.

Tolman, E.C. (1932). Tolman, E.C. (1932). Purposive behavior in animals and men.Purposive behavior in animals and men. Berkeley, CA: University of California Press.Berkeley, CA: University of California Press.

REFERENCES cont.REFERENCES cont.