synaesthesia: a new perspective for understanding the...
TRANSCRIPT
Daphne MaurerMcMaster University
Synaesthesia: A New Perspective for Understanding
the Development of Perception and Even Language
Synaesthesia
Greek: syn (union) + aesthis (sensation)
Synaesthesia: ~54 types
Sound of C# on a flute
taste of oranges not quite ripe⇒
⇒
⇒
Carol Steen 2003
Runs Off In Front,GoldOil on paper, 105 x 70 cm
...while I listened to Santana’s version of a song called Adouma. ...I played this song over and over again as I painted the moving colours.
Coloured Hearing
Synaesthesia: ~54 types
⇒ ⇒ ⇒
2 W 5 P 4 H
Coloured letters and digits
Adapted from Pat Duffy, Blue Cats and Chartruese Kittens
Coloured letters
and digits
SynaesthesiaSound of a trumpet
• Automatic
• “All their lives”
• Mappings are consistent over years.
• Runs in families
taste of oranges not quite ripe⇒
⇒ ⇒
Spector & Maurer, 2009
Runs in families
4-year-old
Her mother
Synaesthesia: Incidence
4% - 5%
Simner et al., 2006Day, 2008
=
.
Dixon et al. 2000Mattingley et al. 2001Mills et al. 2002Myles et al. 2003Palermi et al. 2002
If 2 induces5 induces
2 5 2 5 5 2 2 5 Easy Hard
Synaesthesia: Stroop effect
Ramachandran & Hubbard, 2001Laeng et al. 2004Palermi et al., 2002Smilek et al. 2003Hubbard et al., 2005
Synaesthesia: Pop-out
Ramachandran & Hubbard, 2001Laeng et al. 2004Palermi et al., 2002Smilek et al. 2003Hubbard et al., 2005
Synaesthesia: Pop-out
• Same patterns of interference & facilitation
• Same brain areas
Synaesthesia: works like typical perception
Coloured hearing
Gray et al. 1997Nunn et al., 2002Paulesu et al., 1995Aleman et al., 2001
Sound
Sounds+
cortical pathways
+Colour V4/V8
Synasethete
ControlColour
Word-tone
Both
Coloured hearing
Nunn et al., 2002
Coloured graphemes: seeing letters in colour
Hubbard et al., 2005Sperling et al., 2006Esterman et al., 2006
Letters
Letters
Colour V4Binding colour to shape
+
+
cortical pathways
Coloured graphemes: seeing letters in colour
Sperling et al., 2006
Colour
Black letters
Synaesthesia: result of normal developmental process
Transient connections between sensory cortices
• Kitten
• Auditory ⇔ visual ⇔ tactile ⇔ motor cortex
• Infant monkey
• Auditory cortex ⇒ visual cortex V4 (colour)
Kennedy et al. 1997Dehay et al., 1984, 1988
Transient connections: Visual cortex
Huttenlocher, 1990
Adults’ event-related potentials:
Auditory cortex
Neville, 1995
Other cortical areas
Babies’ event-related potentials:
Visual cortexAuditory cortex
Neville, 1995
Neville 1995
Response to SpeechVisual Cortex Auditory Cortex
Adults
Auditory
Sound
⇐Neville, 1995
Response to SpeechVisual Cortex Auditory Cortex
Infants
Neville, 1995
Visual
Sound
⇐
Auditory
Sound
⇐
Wolff et al., 1974Somatosensory cortex
Adults’ event-related potentials:
Wolff et al., 1974Somatosensory cortex
Newborns’ event-related potentials:
Are transient connections functional in human infant?
-
PET
R inferior temporal gyrus (FFA)
L auditory temporal cortexL Broca’s area
Tzaourio-Mazoyer, de Schonen et al. 2002
Response to Faces at 2 months
-
PET
Neural Development
• Functional Connections between sensory areas
• Modified throughout development by pruning and inhibition.
Extra functional connections among cortical areas
Early childhood Synaesthesia
Experience prunes connections
Visual cortex
VisionHearingTouch
Visual cortex
VisionHearingTouch
Language
Language
Extra connections in sensory cortex
EyesEarsSkin
Visual cortexHearingTouchLanguage
Gizewksi et al. 2003Burton et al.. 2002, 2004
Melzer et al.2001Sadato et al. 1998, 2002
Röder et al. 1999, 2000, 2002
Leclerc et al. 2000Liotti et al. 1998
Kujala et al. 1995 Burton et al. 2003Amedi et al. 2003
Visual cortex
VisionHearing
Experience prunes connections
Touch
Language
Any remaining connections inhibited
Burton et al. 2002Kauffman et al. 2002Sadato et al. 2004Pascual-Leone & Hamilton 2001
Five days:
Visual cortexBraille
Sound locationVibration
Tone frequency
Sightedadults
Non-Synaesthetes
Extra functional connections among cortical areas
Pruning
Inhibition
Remnants Synaesthesia
Infants
Children
Adults
Lead to natural associations
• Less pruning
• Colour grapheme synaestheses: More white matter beside V4/V8 and parietal cortex
Synaesthesia: What’s different?
Rouw & Scholte, 2007
• Less pruning
• Colour grapheme synaestheses: More white matter beside V4/V8 and parietal cortex
• Less effective inhibition
• Acquire new synaesthetic connections when learn new alphabet, try new food, learn new language, after hypnosis
• LSD
• Enhanced with cortical depressants
Synaesthesia: What’s different?
Mills et al., 2002Ward & Simner, 2003Winawer & Witthoft, 2004
Rouw & Scholte, 2007Cohen-Kadosh, 2009
Extra functional connections among cortical areas
Pruning Less pruning
Inhibition Less inhibition
Remnants Synaesthesia
Infants
Children
Adults
Lead to natural associations
Pre-literate
Natural associations in toddlers
Learning language
Letters ~ colour
Consistencies:Colour/letter Associations
(Day, 2001, 2005; Spector, 2003; Rich, Bradshaw, & Mattingly, 2005;Simner et al., 2005)
Non-synaesthetes ~ synaesthetes
Red/Green White/Black
Replication Results
0
1
2
Toddlers Children Adults0
1
2
Toddlers Children Adults
1
.5
0
Prop
ortio
n of
exp
ecte
d re
spon
ses
**
** *
Toddlers Children Adults Toddlers Children Adults
O/X A/G
Spector & Maurer, 2008
Prop
ortio
n of
Exp
ecte
d R
espo
nse
1
.5
07-9 year old
•Based on experience
•Change with reading
Shape - colour
A G B Y
Learned associations in non-synaesthetes
Spector & Maurer, 2008Spector & Maurer, submitted
Replication Results
0
1
2
Toddlers Children Adults0
1
2
Toddlers Children Adults
1
.5
0
Prop
ortio
n of
exp
ecte
d re
spon
ses
**
** *
Toddlers Children Adults Toddlers Children Adults
O/X A/G
Spector & Maurer, 2008
7-9 year old
Colour Responses to Letter Sound and Shape
*
0
0.5
1
1.5
2
sound shape
Prop
ortio
n of
Exp
ecte
d Re
spon
ses
*
0
.5
1
Sound Shapens p=.017
Spector & Maurer, 2008
O/X: Toddlers
Natural associations in non-synaesthetes
•Not based on learning
• Like synaesthetes
•Reflect initial brain organization
Shape - colour
Spector & Maurer, 2008Spector & Maurer, submitted
Natural associations in non-synaesthetes
•Not based on learning
• Like synaesthetes
•Reflect initial brain organization
Shape - colour
smooth? jagged?
Spector & Maurer, submitted
Natural associations in non-synaesthetes
•Not based on learning
• Like synaesthetes
•Reflect initial brain organization
Shape - colour
jaggedsmooth
CSpector & Maurer, 2008
Spector & Maurer, submitted
Pitch ~ lightness
Which ball makes this noise?Which one of these balls is making this noise?
“pong”
Which one goes ping? Which goes pong?
Mondloch & Maurer, 2004
Ask toddlers,“Which makes the
sound?”
Toddlers ~ synaesthetes
Like synaesthetes who see pitch
Lighter = higher pitch
common underlying mechanism. This is illustratedin Figure 4. There was no effect of timbre andtimbre did not interact with any other factors (all F’s< 2), and so this dimension is not graphicallydisplayed. There was a highly significant effect ofpitch [F (9, 162) = 56.32, p < .001], but nodifference between synaesthetes and controls [F (1,18) = 2.51, ns] although a significant group X pitchinteraction [F (9, 162) = 3.62, p < .05] reflects a
Sound-colour synaesthesia 269
tendency for synaesthetes to select somewhat lightercolours at the highest pitches.
The same analysis was conducted on theMunsell chroma values, although we did not havespecific predictions about this dimension3. The
Fig. 3 – An example of the colours selected (on 2 occasions) for the 10 single piano, sine (or pure tone) and string notes for asynaesthete (LHM, top) and a control subject (CE, bottom).
3 Note, hue is a circularly varying dimension and cannot be analysed in thesame way.
wave
Ward et al., 2006
Insert graph from Ward et al.
Ward et al., 2006
chroma values were analysed in a 10 ! 2 ! 3ANOVA contrasting pitch (! 10), group(synaesthete, non-synaesthete) and timbre (puretone, piano, string), and averaging across thedifferent testing sessions. Pitch exerts an effect onchroma [F (9, 162) = 9.28, p < .001] and not justlightness. However, chroma differs from lightnessin two key respects. Firstly, there is no monotonicrelationship. Instead, chroma peaks at mid-rangepitches (in our experiment, a semitone below‘middle C’). Secondly, there was a highlysignificant main effect of timbre on chroma [F (2,36) = 16.89, p < .001] that was not observed forlightness. Thus, musical notes from the piano andstrings are, literally, more colourful than puretones. This is illustrated in Figure 5. Importantly,there was no group difference between synaesthetesand controls [F (1, 18) = 1.26, ns] and none of theinteractions approached significance. In sum,different aspects of auditory stimuli (pitch, timbre)appear to map on to different aspects of colour(lightness, colourfulness) in systematic ways.
270 Jamie Ward and Others
However, these regularities appear to be commonto synaesthetes and control subjects alike.
Finally, we investigated differences in lightnessand chroma across the 10 tones taken fromdifferent instruments (bagpipes, flute, etc.) thatwere matched for pitch of the fundamentalfrequency (at middle C). Given the resultspresented above, one might expect differences tobe manifest on the chroma but not lightnessdimension. Two separate 2 ! 10 ANOVA’s wereconducted with group (synaesthete, control) andinstrument (n = 10) as independent variables andchroma and lightness as dependent variables.Contrary to our prediction, there were main effectsof timbre on both lightness [F (9, 162) = 11.56, p < .001] and chroma values [F (9, 162) = 3.29, p < .001]. Yet again, however, there was nodifference between synaesthete and control groupsand no interactions between group and timbre (allF’s < 2). This is illustrated in Figure 6. Eventhough the frequency of the fundamental wasmatched, it is quite conceivable that differences in
Fig. 4 – The relationship between pitch and lightness for synaesthetes and controls for the 30 single notes (collapsed across timbresand testing sessions): 0 = darkest, 10 = lightest.
common underlying mechanism. This is illustratedin Figure 4. There was no effect of timbre andtimbre did not interact with any other factors (all F’s< 2), and so this dimension is not graphicallydisplayed. There was a highly significant effect ofpitch [F (9, 162) = 56.32, p < .001], but nodifference between synaesthetes and controls [F (1,18) = 2.51, ns] although a significant group X pitchinteraction [F (9, 162) = 3.62, p < .05] reflects a
Sound-colour synaesthesia 269
tendency for synaesthetes to select somewhat lightercolours at the highest pitches.
The same analysis was conducted on theMunsell chroma values, although we did not havespecific predictions about this dimension3. The
Fig. 3 – An example of the colours selected (on 2 occasions) for the 10 single piano, sine (or pure tone) and string notes for asynaesthete (LHM, top) and a control subject (CE, bottom).
3 Note, hue is a circularly varying dimension and cannot be analysed in thesame way.
Synaesthetes
Non-synaesthetes
Ligh
tnes
s
Non-synaesthetes ~ synaesthetes
Lighter
Visual discrimination
Marks
Non-synesthetic
Adults
Speed
Errors
Influences perception unconsciouslyNon-synaesthetic adults
DarkerMarks, 1987
Natural associations in non-synaesthetes
Pitch-lightness
•Not based on learning
Dark animals don’t consistently make lower pitched sounds.
• Like synaesthetes
•Reflect initial brain organization
Sound ~ Shape
Infant as Synaesthete
• Ramachandran: Influences evolution/development of language by linking:
• Visual shape
• Lip shape
• Feeling in mouth during production
• Gesture
• Connections adjacent sensory and motor cortical areas
• Mirror neurons: producing, seeing, hearing action
Maluma
Takeeti
Match the words on the left to the figures on the right.
Köhler, 1947
Ramachandran & Hubbard, 2005
~black
~white
Rounded vowelsvs. non-rounded vowels
bamu kutay
bouba kayki
goga titay
mabuma taketee
Non-synaesthetes: vowel sound/shape
Shapes that activate different V4 neurons
Maurer, Pathman, & Mondloch, 2006
2
1
4
3
kikibouba
21 3 4Overall Pairing
Perc
enta
ge M
atch
ing
Maurer, Pathman, & Mondloch, 2006
Rounded vowelsvs. non-rounded vowels
gigi gogo
bibi bobo
kiki koko
didi dodo
Spector & Maurer, in prep
Toddlers
Spector & Maurer, in prep
Prop
ortio
n M
atch
ing
Approximate versus Stop Consonants
roro bobo
lolo gogo
wowo koko
yoyo dodo
Rounded vowels
riri bibi
lili gigi
wiwi kiki
yiyi didi
Non-rounded vowels
Toddlers
Prop
ortio
n M
atch
ing
Spector & Maurer, in prep
Rounded shapes ⇒ rounded vowels (ah, oh)
Jagged shapes ⇒ unrounded vowels (ee, i)
No effect for stop/approximant consonants
Maurer et al., 2006
Spector & Maurer, in prep
Natural associations in non-synaesthetes
• Learning statistics of language or
....helping to learn language?
Sound-shape
Natural associations in non-synaesthetes
Sound-shape: what is effective cue?
•Heard sound?
• Sight of lips moving to make sound?
• Feeling of making sound oneself?
Infants link sound (pitch) to shape
300 Hz1700 Hz
300 Hz 1700 Hz
3- to 4-month-olds
Walker et al., 2010Up......................... Down
Infants link sound to lip movements
Yeung & Werker, 2010
4-month-olds
Matching
Infants linkmouth movements to shape?
Contrast effectYeung & Werker, 2010
Adults linkmouth movements to shape
headhad
UpDown
Ito, Tide, & Ostry, 2009
Natural associations in non-synaesthetes
Sound-shape links could be based on
•Heard sound
• Sight of lips moving to make sound
• Feeling of making sound oneself
Synaesthetic influences in toddlers and adults
Shape
colour jaggedsmooth
C
“pong”
Which one of these balls is making this noise?
“pong”
“ping” “pong”
“kiki “bouba
Hearing
vision
Synaesthetic influences in toddlers and adults
Synaesthesia: A New Perspective for Understanding
the Development of Perception and Even Language
Language learning alters synaesthetic matching (e.g., Smith & Sera)
2-year-olds 3-year-olds
Comprehension big/little, loud/quiet, dark/light
Inconsistent Good
Visual matching big/little to dark/light
Big=darkLittle=light
Inconsistent
Visual matchingbig/little to loud/quiet
InconsistentBig=loud
Little=quiet
•May influence evolution/development of language
• “round”
• “spikey”
• Intersensory metaphors parallel synaesthesia
• loud necktie (from hearing to vision)
• sharp cheese (from touch to taste)
•Can guess meanings of words in foreign languagesSean Day
Non-synaesthetes: vowel sound/shape
Ramachandran & Hubbard, 2001
Berlin, 1964