1
Current Topics in PerceptionPSYD51
Multisensory IntegrationSession 1
January 6, 2009
2
Current Topics in PerceptionPSYD51
Blurb: multisensory integrationRecent research topicMultimodal information in the worldMaking sense out of it requires multisensory integration
3
Current Topics in PerceptionPSYD51
Multisensory Integration (PSYD51), LEC 01 Place: HW 309 Time: Tuesdays, 1–3pm Intranet
The Course:
Dr. Matthias Niemeier Office: S-572 Phone: 416-287-7466email: [email protected] hours: Tuesdays, 12–1pm, or per appointment
The Instructor:
4
Current Topics in PerceptionPSYD51
45% Thought papers (9 x 5%)20% Presentation15% In-class participation20% Research proposal
Grading
PrerequisitesPSYB51 – Perception and CognitionPSYC08 – Advanced Data Analysis in Psychology
No exams, active participation
ReadingsNo textbookResearch papers (see reading list)
5
Grading System PSYD51Paper Presentations
20% of the total mark1 paper from reading listFirst come first serve – please email mePdf’s on the intranetFree presentation (notes)Three parts:
a) Brief intro (~5 min)b) Summary of methods, results & conclusions. Figures on
the intranet (~25 min)c) Group discussion led by presenter (~15 min)
6
Grading System PSYD51 Thought Papers
45% of the total mark (9 papers x 5%)Read both research papers of the week, thought paper on one“How to write a thought paper” => intranetScoring systemDue for the respective session of the seminar
1-2 pages, double-spaced2 sentences summary maximumYour own thoughtsIntro (1 paragraph, “Tell them what you are going to tell them”) –body (2-4 paragraphs, “Tell them”) – conclusion (1 paragraph, “Tell them what you told them”)
7
Grading System PSYD51 In-class Participation
15% of the total mark – so please be there and be prepared to say something in class!!Help the presenterDo you agree with the authors? Are you impressed? Something that you didn’t understand? Can you help clarifying things? Is there a reason to believe that the paper is invalid? Can you play devil’s advocate? Suggestions for future research?
8
Grading System PSYD51 Research Proposal
20% of the total mark “How to write a research proposal” => intranetScoring systemDue for the last day of the seminar8 pages, double-spacedTitle page – Abstract – Introduction – Methods –Predicted Results – References
9
Comparing Senses
aroma (flavour)dissolved chemicals
odour (scent)airborne
chemicals
deformationphysical objects
soundpressure
wave
lightelectromagnetic
wave
physical quality
of stimuli
tonguenoseskineareyeorgantastesmelltouchhearingvisionsense
Other channels of sensory input? Temperature, pain? Balance? other senses? Perhaps a better physical classification: electromagnetic, mechanical, chemical stimuli
10
Combining and integrating sensory information
Combining vs. integrating
Example: knock on woodSensory combination: interactions between non-redundant sensory signalsSensory integration: interaction between redundant ones
V: vision, N: neck muscle info, A: audition, P: proprioception from arm, L:location signal
11
Combining and integrating sensory information
Why combining/integrating sensory sources?Combined alarm systems (jungle, darkness, braking)Object identification: information from different modalities support each otherVisual-auditory localisation (twilight, peripheral visual field, occlusion, cluttered scenes)Reliable info for control of locomotion(e.g. visual-proprioceptive cross-adaptation: Pelah and Barlow 1996)Coherence of the world: object unity, perceiving one’s own presence in space (e.g. cue conflict > motion sickness)
Large interest from engineering: sensory fusion in robots, VR, video gamesHow do nervous systems combine sensory information ?
12
Multisensory neuronesSingle cell recordings in cat superior colliculusInput from various cortical regions (visual, auditory, somatosensory, motor...)
Alignment of receptive fields >> integrated multisensory mapsSpatial coincidence detectors
Multisensory receptive fields
13
Multisensory areasin the cortex
How/where are the different senses integrated?fMRI, a tool to study some cases of multisensory integration: most prominent, e.g., integration of visual & auditory info (Calvert et al. 1998)
red: viewing mouth movements without sound blue: listening to speech yellow: activated by both signals
Regions for crossmodal identification
Interaction of light and sound
An experiment
14
to demonstrate visuo-auditory integration: ‘streaming and bouncing’ (Sekuler et al. 1997)
Two objects moving straight across each other are perceived as moving on intersecting linear trajectories
A sound at the intersection point changes perception dramatically: the objects appear to collide and bounce
15
Reading emotions
Observers asked to rate the sadness of a series of faces with different expressions between happy and sad
Sadness ratings follow a psychometric function: from low to high for happy to sad facesCombining the face display with a sad or happy voice shifts the psychometric curve to higher or lower sadness ratings
16
Preliminary Schedulefor the Course
6 Jan Introduction13 Jan Neural mechanisms of multisensory integration20 Jan Neural mechanisms of multisensory integration II:
Supramodal areas (?)27 Jan Multisensory object perception3 Feb Optimal multisensory integration10 Feb Optimal sensorimotor combination24 Feb Vision and sound in space3 Mar Vision and touch in space10 Mar Body schema17 Mar Learning & plasticity24 Mar Neuropsychological deficits31 Mar Synaesthesia
17
Outlook on the Course Topics
Jan 13: Neural mechanisms of multisensory integrationLateral connections: Anatomy: cortical auditory input into V1 (Falchier et al. 2002).fMRI: Polymodal motion processing in posterior parietal and premotor cortex (Bremmer et al. 2001).
Jan 20: Crossmodal cueing of attentionSpence & Driver (1997): 1-way cross-modal dependence in covert orienting; audition influences visionWard et al. (2000): auditory cues fail to orient visual attention
18
Outlook on the Course Topics
Jan 27: Multisensory object perceptionViewpoint dependence in visual and haptic object recognition (Newell et al. 2001).Visuo-haptic object-related activation in the ventral visual pathway (Amedi et al. 2001).
19
Outlook on the Course Topics
Feb 3: Optimal multisensory integrationIntroduction to MLEHumans integrate visual and haptic information in a statistically optimal fashion (Ernst et al. 2002).
20
Outlook on the Course Topics
Optimal transsaccadic integration (Niemeier et al. 2003).Statistical decision theory and rapid, goal-directed movements (Trommershauseret al. 2003).
Feb 10: Optimal sensorimotor combination
21
Outlook on the Course Topics
Optimal ventriloquist effect (Alais et al. 2004).http://www.youtube.com/watch?v=hsfnP9DK-ZgRecalibration of audiovisual time perception (Fujisaki et al. 2004).
Feb 24: Vision & sound in space
Coordinate transformations for representing visual and tactile locations in parietal cortex (Avillac et al. 2005).Influences of gaze direction on crossmodal effects (Macaluso et al. 2002).
Mar 3: Vision & touch in space
22
Outlook on the Course Topics
Mar 17: Learning & plasticityExperience for the development of multisensory integration (Wallace et al. 2004).
Plasticity in sound localization induced by compressed spatial vision (Zwiers et al. 2003).
Out of body experience & rubber gloves (Pavani et al. 2000).fMRI and my hand (Ehrsson et al. 2004).
Mar 10: Body schema
23
Outlook on the Course Topics
Mar 24: Neuropsychological deficitsSplit-brain: Remapping of space (Spence et al. 2001).Impaired crossmodal interactions in a patient with Balintsyndrome (Valenza et al. 2004).
Mar 31: SynaesthesiaPriming and colour-number synesthesia (Mattingley et al. 2001).Neurocognitive mechanisms of synesthesia (Hubbard et al. 2004).
Doorbell ringing Dog barking