![Page 1: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/1.jpg)
Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music
Conductors
W. David Hairston, Ph.D
Advanced Neuroscience Imaging Research LabDepartment of Radiology; Wake Forest University School
of Medicine, Winston-Salem, NC 27157
![Page 2: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/2.jpg)
Multisensory enhancement within a number of paradigms, including:
• Simple reaction times
• Saccadic response latencies
• Signal detection
• Orientation/localization behavior in cat
General conditions for enhancements to be observed
-spatial alignment
-temporal congruence
-minimal efficacy (“inverse effectiveness”)
![Page 3: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/3.jpg)
Stimuli:
Vis (LED)
Aud (broadband)
Vis-Aud
50 ms, 30 reps
What about human localization ability
Hairston et al, J. Cog. Neuro, 2003
![Page 4: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/4.jpg)
0
10
20
30
40
50
60
70
80
-15 -13 -11 -9 -7 -5 -3 -1 1 3 5 7 9 11 13 15
Error (degrees)
Pe
rce
nt
Visual
Multisensory
Auditory
0
5
10
15
20
25
30
35
-15 -10 -5 0 5 10 15
Error (degrees)
Pe
rce
nt
Evaluating Precision
Precision ~ Std Dev
![Page 5: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/5.jpg)
0
2
4
6
8
10
12
-40 -20 -10 0 10 20 40
Target Position (deg.)
Lo
cali
zati
on
Var
iab
ilit
y (d
eg.)
Visual
Auditory
Multisensory
![Page 6: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/6.jpg)
Hmmmm…. Why?
A
C
B
S1 S2 M
The total amount of gain/ enhancement observed is determined by the relative contribution of each sense, based upon its own perceptual acuity
How do you increase auditory acuity?
![Page 7: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/7.jpg)
Conductors as auditory localizers
Daily, career experience requires rapid and accurate assessment of auditory scene
Also requires coordination of multisensory information – “who” played “what” wrong note, etc
20 conductors, min 7 years “podium experience” (ave 10.2)
Matched on age, education, sex, etc
(Hodges, Hairston & Burdette ’06)442.8
454.2
440
445
450
455
460
465
470
440 Hz Base Tone
Fre
quency
(H
z)Conductors
Non-Conductors
A#
A
![Page 8: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/8.jpg)
0
1
2
3
4
5
6
7
8
0 10 20 30 40
Target Location (+/- deg.)
Pre
cisi
on
(+
/- d
eg.)
Auditory
Multisensory
Visual
Visual and multisensory performance very similar
Non-musicians
![Page 9: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/9.jpg)
Multisensory performance enhanced
0
1
2
3
4
5
6
7
8
0 10 20 30 40
Target Location (+/- deg.)
Pre
cisi
on
(+
/- d
eg.)
Auditory
Multisensory
Visual
Control
0
1
2
3
4
5
6
7
8
0 10 20 30 40
Target Location (+/- deg.)
Pre
cisi
on
(+
/- d
eg.)
Auditory
ControlImproved auditory performance
0
2
4
6
8
10
12
-7 -2 3 8 13
% Auditory Improvement (> Control)
% M
ulti
sen
sory
E
nh
ance
men
t (>
Vis
ual
)
10
40
3020
![Page 10: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/10.jpg)
Conclusions from this…
• While untrained subjects gain little from an additional auditory cue, music conductors appear to benefit from additional auditory signals, for which their spatial acuity is much better.
• The degree of multisensory gain in this unique population appears to be tied to their improved auditory performance
• These results suggest that the specialized training and experience of these individuals has a profound affect within both the auditory and multisensory realms
![Page 11: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/11.jpg)
But what if…
• Multisensory “enhancements” are beneficial when the task can make use of additional information
• BUT - When do not match or are not relevant – can be detrimental or inhibiting….
-Slower RT
-Biased localization
-Illusions and misperceptions
Many times the task at hand required focusing on one sense and ignoring other.
![Page 12: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/12.jpg)
“Tuning out” irrelevant information…
![Page 13: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/13.jpg)
Current studies…
• Physiological effects of “tuning in” to one sense over another
• “Cross-modal deactivation”: Decreased activity within one sensory cortex related to stimulation of another
E.g., decreased BOLD signal in occipital cortex during auditory task
• Some evidence for relation to selective attention
Unclear whether the extent of deactivation observed is related to the difficulty of the task at hand
![Page 14: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/14.jpg)
MethodsGoals:
Does changing task difficulty affect cross-modal deactivation?
Does extreme acuity within one modality, and unique multisensory training affect this process?
Subjects- Non-musicians – various careers, no formal musical training, 25-40 y/o- Conductors – min. 5 yrs podium experience
Matched on age, gender, education, etcWhy conductors?
- Heightened auditory acuity - Conducting activities require NOT inhibiting other
(e.g., visual) information
![Page 15: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/15.jpg)
Methods: Tasks
Pitch Discrimination
440 Hz 660 Hz
440 Hz 660 Hz
Easier
Difficult
60 ms
20 ms
Temporal Discrimination
440 Hz 456 Hz
440 Hz 443 Hz
Easier
Difficult(Time)
Base Test
Subjects’ thresholds for each task acquired prior to fMRI scanning
![Page 16: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/16.jpg)
Methods: Thresholding
Acquired threshold
0
20
40
60
80
100
120
1 6 11 16 21 26 31 36 41 46 51
Presentation orderS
OA
(m
s)
Allows compensation for variability in perceptual acuity between subjects and groups
2 down/1 up rule
![Page 17: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/17.jpg)
Perceptual acuity, non-musicians vs. conductors
430
440
450
460
470
Conductors Non-musicians
Fre
qu
ency
(H
z)
0
20
40
60
80
100
Conductors Non-musicians
SO
A (
ms)
A
A#
Base comparison
Conductors
Non-Musicians
Non-musicians need a significantly larger difference to discriminate tones than conductors.
![Page 18: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/18.jpg)
Methods: fMRI
Each task performed at 2 levels
- at threshold (“difficult”)- above threshold (“moderate”)
Also performed visual temporal discrimination (2 circles) Visual
Temporal discrimination
silence TRTR(Scanner ON) (Scanner ON)(Scanner OFF)
Trial Trial Trial
Sparse sampling (10 s pause), block design
![Page 19: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/19.jpg)
Schmuel et al, Nature Neuroscience, 2006
Positive BOLD
Positive BOLD
![Page 20: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/20.jpg)
Analysis
• Activity in task (ON) blocks contrasted against resting baseline (OFF)
Baseline: eyes open on fixation, no task• ROI of visual-responsive occipital cortex
2.78 7.0
-2.78 -7.0 Used to generate summary stats
![Page 21: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/21.jpg)
Non-musicians: Moderate
Deactivation of visual cortex
+/-3.95 +/-8.0
Vis ROI
![Page 22: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/22.jpg)
Non-musicians: Difficult
Robust deactivation of visual cortex
+/-3.95 +/-8.0
Vis ROI
![Page 23: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/23.jpg)
Non-musicians: Difficult vs. Moderate
EasierDifficult
-3
-2
-1
0Voxel (Y coord)
Me
an
Sig
na
l
-3
-2
-1
0Voxel (Y coord)
Me
an
Sig
na
l
![Page 24: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/24.jpg)
-1
-0.5
0
0.5
1
Mea
n S
igna
l (%
)
-650
-450
-250
-50
150
Tota
l Sig
nal M
agni
tude
0
200
400
600
800
1000
# of
Vox
els
Non-musicians: stats
ROI A.Cort.Post. Cing.
V Cort.
ROI
A.Cort.
Post. Cing. V Cort.
# significant voxels
Total Signal Magnitude
Mean signal
![Page 25: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/25.jpg)
Conductors
DifficultModerate
Vis ROIVis ROI
+/-3.95 +/-8.0
Only slight deactivation of visual cortex in both cases
![Page 26: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/26.jpg)
Conductors: Difficult vs. Moderate
Easier Difficult
0
200
400
600
800
1000
# o
f V
ox
els
-650
-450
-250
-50
150
To
tal S
ign
al M
ag
nit
ud
e
ROI A.Cort.Post. Cing.
V Cort.
# significant voxels
Total Signal Magnitude
![Page 27: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/27.jpg)
Non-musicians vs. Conductors
Difference seen when task is difficult
Non-Musicians Conductors
0
40
80
120
160
200
Easier Difficult
N S
ign
if V
oxe
ls
Non-musicians
Conductors
n.s.
**
Difficult Task
0
20
40
60
80
High M oderate
Acc
ura
cy(%
)
![Page 28: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/28.jpg)
Conclusions
• The degree and extent of cross-modal deactivation observed specifically depend of the task difficulty; when the same task is easier, cross-modal deactivations are attenuated.
• Contrary to non-musicians, conductors show only minor cross-modal deactivation, even when the task is very difficult to perform.
• This suggests that the role of functional deactivations is dynamic, and adapts to fit the needs of the individual and situation at hand.
![Page 29: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/29.jpg)
Conductors
• Conductors are highly-trained individuals with unique daily activties and specialization
• This experience leads to not only high auditory acuity, but altered interactions when dealing with multiple senses
• Two theories:
- High auditory acuity negates the need for visual suppression.
- Daily experience has led to familiarity with completing complex auditory tasks while also monitoring visual information.
![Page 30: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/30.jpg)
![Page 31: Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors](https://reader035.vdocuments.net/reader035/viewer/2022070415/56814f60550346895dbd1587/html5/thumbnails/31.jpg)
0
15
30
45
60
75
90
Conductors Non-musicians
0
20
40
60
80
High M oderate
Acc
ura
cy(%
)