synaesthesia: a new perspective for understanding the...

Daphne MaurerMcMaster University

Synaesthesia: A New Perspective for Understanding

the Development of Perception and Even Language

Synaesthesia

Greek: syn (union) + aesthis (sensation)

Synaesthesia: ~54 types

Sound of C# on a flute

taste of oranges not quite ripe⇒

⇒

⇒

Carol Steen 2003

Runs Off In Front,GoldOil on paper, 105 x 70 cm

...while I listened to Santana’s version of a song called Adouma. ...I played this song over and over again as I painted the moving colours.

Coloured Hearing

Synaesthesia: ~54 types

⇒ ⇒ ⇒

2 W 5 P 4 H

Coloured letters and digits

Adapted from Pat Duffy, Blue Cats and Chartruese Kittens

Coloured letters

and digits

SynaesthesiaSound of a trumpet

• Automatic

• “All their lives”

• Mappings are consistent over years.

• Runs in families

taste of oranges not quite ripe⇒

⇒ ⇒

Spector & Maurer, 2009

Runs in families

4-year-old

Her mother

Synaesthesia: Incidence

4% - 5%

Simner et al., 2006Day, 2008

=

.

Dixon et al. 2000Mattingley et al. 2001Mills et al. 2002Myles et al. 2003Palermi et al. 2002

If 2 induces5 induces

2 5 2 5 5 2 2 5 Easy Hard

Synaesthesia: Stroop effect

Ramachandran & Hubbard, 2001Laeng et al. 2004Palermi et al., 2002Smilek et al. 2003Hubbard et al., 2005

Synaesthesia: Pop-out

Ramachandran & Hubbard, 2001Laeng et al. 2004Palermi et al., 2002Smilek et al. 2003Hubbard et al., 2005

Synaesthesia: Pop-out

• Same patterns of interference & facilitation

• Same brain areas

Synaesthesia: works like typical perception

Coloured hearing

Gray et al. 1997Nunn et al., 2002Paulesu et al., 1995Aleman et al., 2001

Sound

Sounds+

cortical pathways

+Colour V4/V8

Synasethete

ControlColour

Word-tone

Both

Coloured hearing

Nunn et al., 2002

Coloured graphemes: seeing letters in colour

Hubbard et al., 2005Sperling et al., 2006Esterman et al., 2006

Letters

Letters

Colour V4Binding colour to shape

+

+

cortical pathways

Coloured graphemes: seeing letters in colour

Sperling et al., 2006

Colour

Black letters

Synaesthesia: result of normal developmental process

Transient connections between sensory cortices

• Kitten

• Auditory ⇔ visual ⇔ tactile ⇔ motor cortex

• Infant monkey

• Auditory cortex ⇒ visual cortex V4 (colour)

Kennedy et al. 1997Dehay et al., 1984, 1988

Transient connections: Visual cortex

Huttenlocher, 1990

Adults’ event-related potentials:

Auditory cortex

Neville, 1995

Other cortical areas

Babies’ event-related potentials:

Visual cortexAuditory cortex

Neville, 1995

Neville 1995

Response to SpeechVisual Cortex Auditory Cortex

Adults

Auditory

Sound

⇐Neville, 1995

Response to SpeechVisual Cortex Auditory Cortex

Infants

Neville, 1995

Visual

Sound

⇐

Auditory

Sound

⇐

Wolff et al., 1974Somatosensory cortex

Adults’ event-related potentials:

Wolff et al., 1974Somatosensory cortex

Newborns’ event-related potentials:

Are transient connections functional in human infant?

-

PET

R inferior temporal gyrus (FFA)

L auditory temporal cortexL Broca’s area

Tzaourio-Mazoyer, de Schonen et al. 2002

Response to Faces at 2 months

-

PET

Neural Development

• Functional Connections between sensory areas

• Modified throughout development by pruning and inhibition.

Extra functional connections among cortical areas

Early childhood Synaesthesia

Experience prunes connections

Visual cortex

VisionHearingTouch

Visual cortex

VisionHearingTouch

Language

Language

Extra connections in sensory cortex

EyesEarsSkin

Visual cortexHearingTouchLanguage

Gizewksi et al. 2003Burton et al.. 2002, 2004

Melzer et al.2001Sadato et al. 1998, 2002

Röder et al. 1999, 2000, 2002

Leclerc et al. 2000Liotti et al. 1998

Kujala et al. 1995 Burton et al. 2003Amedi et al. 2003

Visual cortex

VisionHearing

Experience prunes connections

Touch

Language

Any remaining connections inhibited

Burton et al. 2002Kauffman et al. 2002Sadato et al. 2004Pascual-Leone & Hamilton 2001

Five days:

Visual cortexBraille

Sound locationVibration

Tone frequency

Sightedadults

Non-Synaesthetes

Extra functional connections among cortical areas

Pruning

Inhibition

Remnants Synaesthesia

Infants

Children

Adults

Lead to natural associations

• Less pruning

• Colour grapheme synaestheses: More white matter beside V4/V8 and parietal cortex

Synaesthesia: What’s different?

Rouw & Scholte, 2007

• Less pruning

• Colour grapheme synaestheses: More white matter beside V4/V8 and parietal cortex

• Less effective inhibition

• Acquire new synaesthetic connections when learn new alphabet, try new food, learn new language, after hypnosis

• LSD

• Enhanced with cortical depressants

Synaesthesia: What’s different?

Mills et al., 2002Ward & Simner, 2003Winawer & Witthoft, 2004

Rouw & Scholte, 2007Cohen-Kadosh, 2009

Extra functional connections among cortical areas

Pruning Less pruning

Inhibition Less inhibition

Remnants Synaesthesia

Infants

Children

Adults

Lead to natural associations

Pre-literate

Natural associations in toddlers

Learning language

Letters ~ colour

Consistencies:Colour/letter Associations

(Day, 2001, 2005; Spector, 2003; Rich, Bradshaw, & Mattingly, 2005;Simner et al., 2005)

Non-synaesthetes ~ synaesthetes

Red/Green White/Black

Replication Results

0

1

2

Toddlers Children Adults0

1

2

Toddlers Children Adults

1

.5

0

Prop

ortio

n of

exp

ecte

d re

spon

ses

**

** *

Toddlers Children Adults Toddlers Children Adults

O/X A/G

Spector & Maurer, 2008

Prop

ortio

n of

Exp

ecte

d R

espo

nse

1

.5

07-9 year old

•Based on experience

•Change with reading

Shape - colour

A G B Y

Learned associations in non-synaesthetes

Spector & Maurer, 2008Spector & Maurer, submitted

Replication Results

0

1

2

Toddlers Children Adults0

1

2

Toddlers Children Adults

1

.5

0

Prop

ortio

n of

exp

ecte

d re

spon

ses

**

** *

Toddlers Children Adults Toddlers Children Adults

O/X A/G

Spector & Maurer, 2008

7-9 year old

Colour Responses to Letter Sound and Shape

*

0

0.5

1

1.5

2

sound shape

Prop

ortio

n of

Exp

ecte

d Re

spon

ses

*

0

.5

1

Sound Shapens p=.017

Spector & Maurer, 2008

O/X: Toddlers

Natural associations in non-synaesthetes

•Not based on learning

• Like synaesthetes

•Reflect initial brain organization

Shape - colour

Spector & Maurer, 2008Spector & Maurer, submitted

Natural associations in non-synaesthetes

•Not based on learning

• Like synaesthetes

•Reflect initial brain organization

Shape - colour

smooth? jagged?

Spector & Maurer, submitted

Natural associations in non-synaesthetes

•Not based on learning

• Like synaesthetes

•Reflect initial brain organization

Shape - colour

jaggedsmooth

CSpector & Maurer, 2008

Spector & Maurer, submitted

Pitch ~ lightness

Which ball makes this noise?Which one of these balls is making this noise?

“pong”

Which one goes ping? Which goes pong?

Mondloch & Maurer, 2004

Ask toddlers,“Which makes the

sound?”

Toddlers ~ synaesthetes

Like synaesthetes who see pitch

Lighter = higher pitch

common underlying mechanism. This is illustratedin Figure 4. There was no effect of timbre andtimbre did not interact with any other factors (all F’s< 2), and so this dimension is not graphicallydisplayed. There was a highly significant effect ofpitch [F (9, 162) = 56.32, p < .001], but nodifference between synaesthetes and controls [F (1,18) = 2.51, ns] although a significant group X pitchinteraction [F (9, 162) = 3.62, p < .05] reflects a

Sound-colour synaesthesia 269

tendency for synaesthetes to select somewhat lightercolours at the highest pitches.

The same analysis was conducted on theMunsell chroma values, although we did not havespecific predictions about this dimension3. The

Fig. 3 – An example of the colours selected (on 2 occasions) for the 10 single piano, sine (or pure tone) and string notes for asynaesthete (LHM, top) and a control subject (CE, bottom).

3 Note, hue is a circularly varying dimension and cannot be analysed in thesame way.

wave

Ward et al., 2006

Insert graph from Ward et al.

Ward et al., 2006

chroma values were analysed in a 10 ! 2 ! 3ANOVA contrasting pitch (! 10), group(synaesthete, non-synaesthete) and timbre (puretone, piano, string), and averaging across thedifferent testing sessions. Pitch exerts an effect onchroma [F (9, 162) = 9.28, p < .001] and not justlightness. However, chroma differs from lightnessin two key respects. Firstly, there is no monotonicrelationship. Instead, chroma peaks at mid-rangepitches (in our experiment, a semitone below‘middle C’). Secondly, there was a highlysignificant main effect of timbre on chroma [F (2,36) = 16.89, p < .001] that was not observed forlightness. Thus, musical notes from the piano andstrings are, literally, more colourful than puretones. This is illustrated in Figure 5. Importantly,there was no group difference between synaesthetesand controls [F (1, 18) = 1.26, ns] and none of theinteractions approached significance. In sum,different aspects of auditory stimuli (pitch, timbre)appear to map on to different aspects of colour(lightness, colourfulness) in systematic ways.

270 Jamie Ward and Others

However, these regularities appear to be commonto synaesthetes and control subjects alike.

Finally, we investigated differences in lightnessand chroma across the 10 tones taken fromdifferent instruments (bagpipes, flute, etc.) thatwere matched for pitch of the fundamentalfrequency (at middle C). Given the resultspresented above, one might expect differences tobe manifest on the chroma but not lightnessdimension. Two separate 2 ! 10 ANOVA’s wereconducted with group (synaesthete, control) andinstrument (n = 10) as independent variables andchroma and lightness as dependent variables.Contrary to our prediction, there were main effectsof timbre on both lightness [F (9, 162) = 11.56, p < .001] and chroma values [F (9, 162) = 3.29, p < .001]. Yet again, however, there was nodifference between synaesthete and control groupsand no interactions between group and timbre (allF’s < 2). This is illustrated in Figure 6. Eventhough the frequency of the fundamental wasmatched, it is quite conceivable that differences in

Fig. 4 – The relationship between pitch and lightness for synaesthetes and controls for the 30 single notes (collapsed across timbresand testing sessions): 0 = darkest, 10 = lightest.

common underlying mechanism. This is illustratedin Figure 4. There was no effect of timbre andtimbre did not interact with any other factors (all F’s< 2), and so this dimension is not graphicallydisplayed. There was a highly significant effect ofpitch [F (9, 162) = 56.32, p < .001], but nodifference between synaesthetes and controls [F (1,18) = 2.51, ns] although a significant group X pitchinteraction [F (9, 162) = 3.62, p < .05] reflects a

Sound-colour synaesthesia 269

tendency for synaesthetes to select somewhat lightercolours at the highest pitches.

The same analysis was conducted on theMunsell chroma values, although we did not havespecific predictions about this dimension3. The

Fig. 3 – An example of the colours selected (on 2 occasions) for the 10 single piano, sine (or pure tone) and string notes for asynaesthete (LHM, top) and a control subject (CE, bottom).

3 Note, hue is a circularly varying dimension and cannot be analysed in thesame way.

Synaesthetes

Non-synaesthetes

Ligh

tnes

s

Non-synaesthetes ~ synaesthetes

Lighter

Visual discrimination

Marks

Non-synesthetic

Adults

Speed

Errors

Influences perception unconsciouslyNon-synaesthetic adults

DarkerMarks, 1987

Natural associations in non-synaesthetes

Pitch-lightness

•Not based on learning

Dark animals don’t consistently make lower pitched sounds.

• Like synaesthetes

•Reflect initial brain organization

Sound ~ Shape

Infant as Synaesthete

• Ramachandran: Influences evolution/development of language by linking:

• Visual shape

• Lip shape

• Feeling in mouth during production

• Gesture

• Connections adjacent sensory and motor cortical areas

• Mirror neurons: producing, seeing, hearing action

Maluma

Takeeti

Match the words on the left to the figures on the right.

Köhler, 1947

Ramachandran & Hubbard, 2005

~black

~white

Rounded vowelsvs. non-rounded vowels

bamu kutay

bouba kayki

goga titay

mabuma taketee

Non-synaesthetes: vowel sound/shape

Shapes that activate different V4 neurons

Maurer, Pathman, & Mondloch, 2006

2

1

4

3

kikibouba

21 3 4Overall Pairing

Perc

enta

ge M

atch

ing

Maurer, Pathman, & Mondloch, 2006

Rounded vowelsvs. non-rounded vowels

gigi gogo

bibi bobo

kiki koko

didi dodo

Spector & Maurer, in prep

Toddlers

Spector & Maurer, in prep

Prop

ortio

n M

atch

ing

Approximate versus Stop Consonants

roro bobo

lolo gogo

wowo koko

yoyo dodo

Rounded vowels

riri bibi

lili gigi

wiwi kiki

yiyi didi

Non-rounded vowels

Toddlers

Prop

ortio

n M

atch

ing

Spector & Maurer, in prep

Rounded shapes ⇒ rounded vowels (ah, oh)

Jagged shapes ⇒ unrounded vowels (ee, i)

No effect for stop/approximant consonants

Maurer et al., 2006

Spector & Maurer, in prep

Natural associations in non-synaesthetes

• Learning statistics of language or

....helping to learn language?

Sound-shape

Natural associations in non-synaesthetes

Sound-shape: what is effective cue?

•Heard sound?

• Sight of lips moving to make sound?

• Feeling of making sound oneself?

Infants link sound (pitch) to shape

300 Hz1700 Hz

300 Hz 1700 Hz

3- to 4-month-olds

Walker et al., 2010Up......................... Down

Infants link sound to lip movements

Yeung & Werker, 2010

4-month-olds

Matching

Infants linkmouth movements to shape?

Contrast effectYeung & Werker, 2010

Adults linkmouth movements to shape

headhad

UpDown

Ito, Tide, & Ostry, 2009

Natural associations in non-synaesthetes

Sound-shape links could be based on

•Heard sound

• Sight of lips moving to make sound

• Feeling of making sound oneself

Synaesthetic influences in toddlers and adults

Shape

colour jaggedsmooth

C

“pong”

Which one of these balls is making this noise?

“pong”

“ping” “pong”

“kiki “bouba

Hearing

vision

Synaesthetic influences in toddlers and adults

Synaesthesia: A New Perspective for Understanding

the Development of Perception and Even Language

Language learning alters synaesthetic matching (e.g., Smith & Sera)

2-year-olds 3-year-olds

Comprehension big/little, loud/quiet, dark/light

Inconsistent Good

Visual matching big/little to dark/light

Big=darkLittle=light

Inconsistent

Visual matchingbig/little to loud/quiet

InconsistentBig=loud

Little=quiet

•May influence evolution/development of language

• “round”

• “spikey”

• Intersensory metaphors parallel synaesthesia

• loud necktie (from hearing to vision)

• sharp cheese (from touch to taste)

•Can guess meanings of words in foreign languagesSean Day

Non-synaesthetes: vowel sound/shape

Ramachandran & Hubbard, 2001

Berlin, 1964