Multisensory Integration at Peak Limb Velocity

by

Tristan Defrancesco Loria

A Master’s thesis submitted in conformity with the requirements

for the degree of Master of Science

Exercise Sciences

University of Toronto

© Copyright by Tristan Loria 2015

ii

Abstract

Multisensory Integration at Peak Limb Velocity

Tristan Defrancesco Loria

Master of Science

Graduate Department of Exercise Sciences

University of Toronto

2015

The current thesis focused on multisensory integration during rapid upper-limb

movements, specifically at peak limb velocity (PLV). Participants were required to “fling” their

limb through a virtual target while aligning PLV with the target. In Experiment 1, participants

were provided with augmented: auditory, visual, audiovisual, or no feedback at peak limb

velocity. In Experiment 2, participants completed the same flinging task. At PLV, participants

were exposed to either two: auditory beeps, visual flashes, or audiovisual events. Participants

then indicated which sensory cue was presented first (i.e., left or right of the target) in a within

modality temporal order judgment task. This judgment was also made following stimuli

presentation at rest. Across both experiments, the results failed to reveal an audiovisual (i.e.,

bimodal) advantage over unisensory stimuli presentation (i.e., auditory or visual cues alone),

suggesting that the optimal integration of audiovisual information may not occur at peak limb

velocity.

iii

Acknowledgments

Without the support of my parents Rosy and Tony this work would not have been

possible. I am grateful everyday for their love and support. In addition, Dr. Luc Tremblay has

provided invaluable guidance throughout this process. Thank you Luc for your patience, hard

work, and suggestions over the last two years. The growth I have experience both personally and

academically under your supervision has not gone unappreciated. To my committee members Dr.

Tim Welsh and Dr. Matthias Niemeier, your guidance and suggestions during the early phases of

this project were greatly appreciated. To my fellow graduate students in the PMB and AA labs,

thank you for the countless hours of conversation, good times, and support. Thank you to

everyone at Positano Restaurant. Lastly, I’d like to say a very special thank you to Shanna. Time

will pass and it may go fast, but we’ll still be together.

iv

Table of Contents

Table of Contents ........................................................................................................................... iv

List of Figures ................................................................................................................................ vi

Chapter 1 – Multisensory Integration and Goal-directed Action .................................................... 1

Introduction .............................................................................................................................. 1 1

Overview of the Current Thesis ....................................................................................................... 1 1.1

Chapter 2 – Literature Review .................................................................................................. 3 2

Processing Sensory Information ...................................................................................................... 3 2.1

Multisensory Processing .................................................................................................................. 5 2.2

2.2.1 Multisensory Processing at the Cortical Level ......................................................................... 6 2.2.2 Multisensory Integration and Perception .................................................................................. 8 2.2.3 Multisensory Integration During Action .................................................................................. 9

2.2.3.1 Sensory Gating During Goal-directed Action .................................................................................. 9 Experimental Aims and Rationale ................................................................................................. 11 2.3

Chapter 3 – Common Methods ............................................................................................... 12 3

Participants ..................................................................................................................................... 12 3.1

Apparatus ....................................................................................................................................... 12 3.2

General Procedure .......................................................................................................................... 14 3.3

3.3.1 Data Analysis Experiment 1 ................................................................................................... 15 3.3.2 Experiment 2 Data Analysis ................................................................................................... 15

Chapter 4 – Can you hear that peak? Utilization of auditory and visual feedback at peak limb 4

velocity .......................................................................................................................................... 17

4.1.1 Abstract ................................................................................................................................... 18 4.1.2 Introduction ............................................................................................................................ 19 4.1.3 Methods .................................................................................................................................. 23 4.1.4 Participants ............................................................................................................................. 23 4.1.5 Apparatus ................................................................................................................................ 24 4.1.6 Procedure ................................................................................................................................ 25 4.1.7 Data Analysis .......................................................................................................................... 27 4.1.8 Results .................................................................................................................................... 28 4.1.9 Discussion ............................................................................................................................... 31

v

4.1.10 What Does This Article Add? .............................................................................................. 34 4.1.11 References ............................................................................................................................ 35 Multisensory Integration at Peak Limb Velocity in a Within-Modality Temporal Order Judgment 4.2

Task.. ....................................................................................................................................................... 39 4.2.1 Abstract ................................................................................................................................... 40 4.2.2 Introduction ............................................................................................................................ 41 4.2.3 Methods .................................................................................................................................. 46 4.2.4 Participants ............................................................................................................................. 46 4.2.5 Apparatus ................................................................................................................................ 46 4.2.6 Procedure ................................................................................................................................ 48 4.2.7 Results .................................................................................................................................... 50 4.2.8 Discussion ............................................................................................................................... 53

Chapter 5 – General Discussion ............................................................................................. 61 5

Experiment 1 Summary ................................................................................................................. 61 5.1

Experiment 2 Summary ................................................................................................................. 61 5.2

General Discussion Overview ....................................................................................................... 62 5.3

5.3.1 Critical Temporal Window For Visual Processing ................................................................ 63 5.3.2 Multisensory Integration At Peak Limb Velocity .................................................................. 65 5.3.3 The Advantage of Auditory Feedback ................................................................................... 68 5.3.4 Limitations of the Current Thesis ........................................................................................... 69 5.3.5 Concluding Remarks .............................................................................................................. 70

Chapter 6 – References ........................................................................................................... 71 6

Appendices ............................................................................................................................. 77 7

Appendix A – Block Analysis of Experiment 1 ............................................................................ 78 7.1

Appendix B – Block Analysis of Experiment 2 ............................................................................ 80 7.2

vi

List of Figures

Figure 1. A depiction of the experimental apparatus used in the feedback and TOJ experiments

with the virtual target shown.........................................................................................................12

Figure 2. An example of the flinging movement performed in both experiments........................14

Figure 3. Participants completed the experiment while their heads rested in a chin rest and a

virtual target was displayed in front of them.................................................................................23

Figure 4. An example of the flinging movement performed by participants. The velocity profiles

correspond to the approximate limb velocities illustrated above………………………………..25  

Figure 5. Variability in resultant Displacement. Also shown is the MLE predicted variability in

the audiovisual condition………………………………………………………………………...31  

Figure 6. A depiction of the experimental apparatus.....................................................................46  

Figure 7. An illustration of the flinging movement performed in the experiment………………48  

Figure 8. Number of correct responses as a function of sensory condition……………………..53  

1

Chapter 1

Multisensory Integration and Goal-directed Action

Introduction 1

We inhabit a dynamic world, rich in sensory information. Considering that we often

actively engage with multisensory stimuli through goal-directed action, it is critical to understand

how the central nervous system processes and utilizes this information. In the context of goal-

directed action, the major empirical focus to date has largely concerned how sensory information

is utilized at the critical moment in the trajectory, specifically movement end (Woodworth,

1899). However, for various motor skills such as overarm throwing, tossing, and putting, the

critical moment in the trajectory is not movement end, but rather at or near peak limb velocity.

How is sensory information used for accurate motor execution when peak limb velocity is the

critical kinematic moment? How are multisensory cues integrated at this kinematic marker?

Further, can augmented sensory feedback be employed to facilitate the spatial and temporal

occurrence of peak limb velocity? In line with these questions, the current thesis sought to

elucidate some of the underlying mechanisms governing sensory utilization, integration, and

localization when the limb is traveling at peak velocity.

Overview of the Current Thesis 1.1

The overarching goal of the current thesis is to understand how sensory information from

multiple modalities is integrated and utilized during goal-directed actions where peak limb

velocity is critical in determining the outcome. What follows is a review of literature pertaining

2

to processing sensory information, how cues from multiple modalities are combined and

integrated on the behavioural as well as neurological level, and a brief overview of sensory

gating during goal-directed action. Emphasis will be placed on how sensory information is used

throughout an ongoing movement to facilitate accurate execution of a task.

Following this literature review, two experiments are reported. In the first experiment

(i.e., feedback experiment), participants were required to fling their limb through a virtual target

with the goal of aligning their peak limb velocity with the intersection of a virtual target. At peak

limb velocity (i.e., as measured by the velocity of the index finger), participants were presented

with auditory, visual, or audiovisual feedback from a piezo-LED device affixed to the right index

finger. The results of this experiment showed that auditory feedback significantly reduced the

variability in peak limb velocity occurrence when compared to movements performed with no

feedback. When analyzing performance in the visual and audiovisual conditions in contrast, no

significant reduction in constant error or variable error was found. This pattern of results was

interpreted as evidence that participants failed to reweight sensory information in statistically

optimal fashion.

In the second experiment (i.e., TOJ experiment) participants performed the same flinging

movement with the goal of reaching peak limb velocity (i.e., as measured by the velocity of the

wrist) when the index finger intersected the virtual target. At peak limb velocity, participants

were presented with two auditory, visual, or audiovisual cues via two piezo-LED devices

positioned on either side of the virtual target. Following the trial, participants reported which side

of the virtual target the first sensory cue was presented, in a temporal order judgment (TOJ).

Participants also completed the TOJ following stimuli presentation at rest. The results indicated

that participants were more accurate in the TOJ task when viewing the stimuli at rest compared

3

to presentation at peak limb velocity. When TOJs were completed at rest, participants were more

accurate in the auditory, and audiovisual conditions relative to the visual condition. Critically,

when comparing between TOJs made at rest or following presentation at peak limb velocity, it

was found that participant’s accuracy significantly decreased only in the audiovisual condition.

Overall, the results of this experiment suggest that at peak limb velocity, visual information is

not particularly salient. In addition, the results of this experiment further suggest that the optimal

integration of audiovisual information may not occur when the limb is travelling at peak

velocity.

Chapter 2 – Literature Review 2

Processing Sensory Information 2.1

Precision in aiming movements is facilitated when afferent information is available.

During instances of rapid goal-directed action, the window for processing incoming sensory

feedback is considerably narrow (i.e., Elliott, Carson, Goodman, & Chua, 1991; Elliott, Hansen,

& Khan, 2010). Because visual information is rapidly processed (Lesevre, 1982), it provides

critical feedback as a movement unfolds. Woodworth conducted one of the first studies on the

topic of visual processing during action in 1899. In one experiment, participants were required to

trace a pencil along a rotating drum at various speeds with the lights in the room illuminated or

extinguished. Based on the participant’s performance in this task, Woodworth (1899) concluded

that visual information could be processed and used to inform movement corrections within 450

ms. However, various experiments conducted since have shown that Woodworth’s reciprocal

aiming task induced an overestimation of visual processing times.

4

Keele and Posner (1968) had participants perform a discrete movement task in which a

stylus was moved from a home position to a target under a specific movement time bandwidth.

By measuring whether participants were hitting the target zone with the lights in the room

illuminated or extinguished, Keele and Posner (1968) proposed the minimum duration for

processing visual feedback was between 190-260 ms. However, Zelaznik, Hawkins, and

Kisselburgh (1983) reduced the required movement time bandwidth, and when analyzing

endpoint variability, found that visual feedback could be utilized in as little as 100-250 ms. In

light of this, the generally accepted visual processing delay has been estimated at 100 ms

(Carlton, 1992). In addition to vision, afferent sensory feedback from the remaining modalities

can also provide valuable information relating to an ongoing movement.

Because of the rapid processing of auditory information within the central nervous

system (Celesia, 1976; Warren, Wise & Warren, 2005), this information can be used with

temporal and spatial precision. In rhesus monkeys for example, behavioural responses to

auditory stimuli have been reported to be as short as 60 ms for a saccade to an auditory target

(Russo & Bruce, 1994). In addition, reaching movements performed to auditory targets have

been shown to have endpoint errors of less than 0.5 cm (e.g., Levy-Tzedek, Hanassy, Abboud,

Maidenbaum, & Amedi, 2012). Overall, when the critical moment in the trajectory is movement

end, both visual and auditory information can be utilized during movement execution. However,

what remains to be determined is how the use of visual and auditory information changes when

the critical moment is not movement end, but rather peak limb velocity.

Besides sensory processing specifically during action, another approach often utilized

when examining how sensory information is processed is the temporal order judgment task

(TOJ). In this task, participants are presented with auditory and visual cues. These cues are not

5

always presented simultaneously; rather their presentation is staggered according to

predetermined stimulus onset asynchronies (SOA). The participant’s task is to indicate which

sensory cue they perceived to have been presented first (e.g., Hirsh and Sherrick 1961; Kanabus,

Szelag, Rojek, Pöppel, 2002; Spence, Shore, & Klein, 2001). A general finding from this

literature is that for auditory and visual information to be perceived as simultaneous, the visual

stimulus must lead the auditory stimulus by ~ 75-100 ms (e.g., Zampini, Shore, & Spence, 2003;

Jaekl & Harris, 2007). It is critical to note however that many of the TOJ studies previously

conducted presented auditory and visual cues from different spatial locations while the

participant remained at rest (e.g., Bushara, Grafman, & Hallett, 2001; Jaskowski, Jaroszyk, &

Hojan-Jezierska, 1990). As such, the processing of spatially compatible visual and auditory

information in the TOJ paradigm remains to be determined.

Although the above literature has focused on the processing and utilization of unimodal

information, it is also important to consider that sensory cues rarely exist independently of one

another. At any given time, multiple sources of sensory information are available to an individual

and can be used to guide action. Specifically, the central nervous system (CNS) is to able to

formulate a stable percept of the environment as well as plan movements by concurrently

utilizing sensory cues from the vestibular, visual, proprioceptive, and auditory modalities (for

reviews see van Atteveldt, Murray, Thut, & Schroeder, 2014; Driver & Noesselt, 2008).

Multisensory Processing 2.2

When sensory cues from multiple modalities are presented concurrently, the central

nervous system is typically able to formulate a more robust estimate of a sensory event through

multisensory integration. The Maximum Likelihood Estimation Model (MLE) put forward by

6

Ernst and Bülthoff (2004) provides a theoretical framework for how the CNS accomplishes this

task. In this model, the CNS integrates sensory information across multiple modalities in a

statistically optimal way. Depending on the context of a particular event, the quality of the

sensory information varies. For example, when reaching for a glass of water under very low

luminance conditions, the visual cue can be less reliable than the proprioceptive cue associated

with limb position. As such, proprioception would be more heavily relied upon given its

reliability relative to vision (e.g., van Beers, Sittig, & Denier van der Gon, 1999; van Beers,

Wolpert, & Haggard, 2002). However, if the lights in the room were fully illuminated, a reliance

on vision would be expected. This variability in stimulus reliability affects the weights given to

each sensory modality during the integration process, with lesser weight assigned to unreliable

cues. By taking a weighted average during the sensory integration process, sensory information

from differing modalities is combined such that ambiguity is reduced, and a more stable percept

of a sensory event is formed. In addition, numerous experiments focused on the neuronal basis of

sensory processing have shown that the CNS is largely organized for multisensory integration.

2.2.1 Multisensory Processing at the Cortical Level

Early physiological work as well as more recent neuroimaging data has shed light on the

implicit nature of multisensory processing. Seminal work conducted by Meredith, Nemitz, and

Stein (1987) as well as Stein, Meredith, Hunneycutt, and McDade (1989) showed visual and

auditory information converging on single neurons within the superior colliculus of cats. Stein et

al. (1989) observed that multisensory stimuli yielded a greater magnitude of firing (i.e., a

superadditive effect) among this population of neurons relative to unisensory stimuli, as well as a

greater number of neural responses overall. Critically, it was observed that this effect is time

sensitive, with the superadditive effect only occurring if auditory and visual information were

7

presented within a temporal window, spanning approximately 150 ms (Meredith, Nemitz, &

Stein, 1987). This convergence of multisensory information in the cortex has also been supported

in experiments involving humans.

Audiovisual convergence and multisensory neurons have been observed throughout the

cortex as well as in areas once considered to be unisensory. Bremmer et al. (2001) exposed

participants to moving auditory, visual, and tactile stimuli while measuring neural activity using

functional magnetic resonance imaging (fMRI). Of particular interest in this experiment was

neural activity within the motion sensitive areas of the cortex (i.e., the intraparietal sulcus and

ventral premotor cortex). The authors report the results of a conjunction analysis that revealed

significant activation common to all three modalities within the posterior parietal cortex (PPC),

right ventral premotor cortex, and lateral inferior postcentral cortex.

In addition, multisensory information has been shown to influence processing within the

primary sensory areas. For example, Calvert et al. (1999) exposed participants to bimodal (i.e.,

audiovisual) and unimodal (i.e., auditory or visual) stimuli in an fMRI study. Under bimodal

stimulus presentation, Calvert et al. (1999) reported significantly greater activity in the primary

auditory (i.e., A1) and visual cortices (i.e., V1) when compared to unimodal presentation. This

finding led the authors to suggest that the primary sensory areas may also process information in

the non-preferred modality. Further work also supports activation within A1 and V1 from the

non-preferred modality. In a study conducted by Lakatos et al. (2009), macaques performed an

intermodal selective attention task where they were required to attend to either auditory of visual

cues on alternating trials. While performing this task, recordings were taken from groups of

neurons in A1 and V1. When analyzing phase coherence across trials, as measured by intertrial

coherence, it was found that neuronal activity within A1 and V1 was influenced specifically

8

when the monkey attended the non-preferred modality. As such, it appears that the primary

sensory cortices may be multisensory in nature (for reviews see Ghazanfar, & Schroeder, 2006;

van Atteveldt et al., 2014).

Overall, the neuronal evidence suggests that multisensory cues are inherently processed

as such, and that the primary sensory areas are organized to process multisensory information.

How does the robust representation of multisensory stimuli in the cortex manifest in the realm of

perception and action? Multisensory perception and the integration of sensory cues during

voluntary action will be discussed in the following sections.

2.2.2 Multisensory Integration and Perception

As discussed above, the superadditive effect of multisensory input is strongly represented

at the neuronal level. Perceptually, multisensory stimulus presentation can at times be

facilitative, whereas in other instances bias perception creating multisensory illusions. One

example of facilitation is when compared to unisensory stimulus presentation, multisensory

information can yielded shorter reaction times (e.g., Hershenson, 1962). Beyond this facilitative

effect, the presence of multiple sensory modalities can influence the perception of a sensory

event as evidenced by the McGurk effect (McGurk & MacDonald, 1976). Additionally, the

ventriloquist effect (e.g., Alais & Burr, 2004) is a well-known perceptual illusion that occurs

independent of laboratory manipulations. This effect can be characterized by the example of

watching television. In this instance, the voice of an actor on screen appears to emanate

seamlessly from the actor’s mouth, despite the fact that the auditory information stems from the

television speakers. Alais and Burr (2004) examined the spatial nature of this effect by

investigating the localizability of auditory and visual stimuli. The authors report that the illusion

arises from near optimal sensory integration, with the more reliable sensory input (i.e., vision in

9

this case) biasing the spatial position of the lesser reliable sensory cue (i.e., audition).

Interestingly, when the spatial localizability of the visual cue was degraded, the effect is reversed

such that audition biases the spatial position of vision. As such, sensory information is optimally

integrated when an individual observes an audiovisual event.

2.2.3 Multisensory Integration During Action

It is important to note that the studies discussed above most often consider the

integration and utilization of multisensory information when the participant is a passive observer,

and the motor responses are only made after stimulus presentation. The results of the

experiments discussed in the previous section strongly suggest that at rest, optimal integration

across sensory modalities occurs (e.g., Alais & Burr, 2004; Ernst & Banks, 2002; Ernst &

Bültoff, 2004). It is important to note here that humans hardly remain stationary in their

multisensory environments. Interestingly, studies examining multisensory integration during

action have revealed that the integration of sensory information across multiple modalities differs

during goal-directed action. The discussion below pertains to instances of multisensory

integration during goal-directed action.

2.2.3.1 Sensory Gating During Goal-directed Action

The perception of tactile stimulation can be gated as function of movement. Chapman,

Bushnell, Miron, Duncan, and Lind (1987) noted a reduction of tactile sensitivity as a function of

passive and active limb movements. Further work conducted by Juravle, Deubel, Tan, and

Spence (2010) sought to determine the changes in tactile sensitivity throughout the time course

of a goal-directed movement. Of particular interest are the results from Experiment 1 in which

participants performed reaching movements between two effectors (i.e., computer mice), with

10

tactile stimulation being applied to the moving hand prior to and during the movement.

Specifically, discrete tactile stimulation was applied to the hand at: movement preparation,

initiation (i.e. immediately following the ‘go’ signal), movement execution, and finally after

movement completion. Juravle et al. (2010) reported decreased tactile sensitivity during the

movement with the largest reduction occurring when stimulation had been applied during

movement preparation and execution. Thus, the data strongly suggests that tactile gating is

modulated as a function of movement phase, with the strongest suppression occurring following

movement initiation. When taken as whole, the results of Chapman et al. (1987) and Juravle et

al. (2010) suggest that sensory processing at rest fundamentally differs from that during

movement preparation and execution.

More pertinent to the current thesis is the sensory gating that occurs for audiovisual

processing during goal-directed action. In an experiment conducted by Tremblay and Nguyen

(2010), participants performed a 30 cm reaching movement. Under the terminal point (i.e.,

target) of this movement, participants were exposed to the fission/ fusion illusion (Shams,

Kamitani, & Shimojo, 2000; Shams, Kamitani, & Shimojo, 2002). In this illusion, the presence

of auditory beeps biases the number of perceived flashes. For example, when two flashes are

present and one beep is heard, these events are fused such that the participant will erroneously

report seeing only one flash. However, Tremblay and Nguyen (2010) reported that participants

became less susceptible to the fusion illusion at the highest velocity stages of their upper-limb

movement. Tremblay and Nguyen (2010) suggested that this modulation in multisensory

integration might be a reflection of the task-relevant optimal integration of multisensory cues, or

sensory gating of auditory information.

11

Experimental Aims and Rationale 2.3

The empirical focus throughout the current literature has been on sensory processing

during movements where the critical moment in the trajectory is movement end (Keele &

Posner, 1968; Carlton, 1981; Ernst & Bülthoff, 2004). Unfortunately, this focus has neglected

movements where peak limb velocity is the critical moment in the trajectory such as overarm

throwing. As such, how sensory information is processed at this specific kinematic marker has

remained largely understudied. This approach is certainly warranted in order to fully understand

how sensory information is processed across the entire spectrum of motor skills. As such, the

current thesis sought to understand how sensory information was utilized with emphasis on

movements where peak limb velocity is the critical kinematic moment, as opposed to movement

end. To this end, participants in Experiments 1 and 2 were tasked with aligning the moment peak

limb velocity was reached with the intersection of a virtual target. Sensory processing and

utilization was assessed through augmented sensory feedback as well as within a within-modality

temporal order judgment.

12

Chapter 3 – Common Methods 3

Participants 3.1

Participants in Experiment 1 (N = 13) and Experiment 2 (N = 13) were recruited from the

graduate and undergraduate population at the University of Toronto. Inclusion criterion was

being right-hand dominant with normal or corrected to normal visual acuity. Participants were

financially compensated for participation (i.e., $10).

Apparatus 3.2

Participant’s heads rested on a chin rest (38 cm tall) fastened to a 61 cm by 75 cm table

for the duration of the experiments (see Figure 1). A custom built wooden frame (48 cm tall)

with a reflective surface 12 cm in diameter was positioned 13 cm away from the home position.

Attached to the upper most portion of the chin

rest was a yellow LED light (1 cm in diameter),

which served as the target for the experiment.

The participant viewed the target via the

reflective surface. As such, the target appeared

elevated from the surface of the table (i.e., a

virtual target). Each trial began with the

participant’s finger on a 1.5 cm by 1.5 cm piece

of Velcro™, which served as the home position

for the experiments. Participants were cued to

begin their movement when the target became visible via the reflection. Participant’s limb

Figure 1. A depiction of the experimental apparatus used

in the feedback and TOJ experiments with the approximate

position of the virtual target shown. Note: This figure is

not to scale.

13

movements were measured along the Y-axis (i.e., primary movement axis), as and the X and Z-

axes (i.e., secondary movement axes). These movement axes are plotted in Figure 1.

The participant’s movements were recorded throughout the experiments using an

Optotrak Certus (Northern Digital Inc., Waterloo, ON, Canada) motion capture system. An

infrared emitting diode (IRED) was used to measure the participant’s movement. The Optotrak

real-time sampling rate was set at 500 Hz and controlled using a custom Matlab script (The

Mathworks, Inc., Natwick, MA, USA) to trigger real-time stimuli presentations through an

analog-to-digital board (PCI-6024E, National Instruments Corp., Austin, TX, USA).

In both experiments, instantaneous velocity was calculated using the position and time

data collected from the Optotrak system while recording from an IRED positioned on the index

finger (e.g., the feedback experiment) and on the wrist (i.e., TOJ experiment). Movement start

location was defined when the limb first exceeded a velocity of 0.03 m/s. Peak velocity (i.e., as

measured by the unique IRED position in each experiment) was selected as two consecutive

samples with a velocity decrease, following a minimum velocity of 0.8 m/s. Lastly, the resultant

displacement between movement start and peak limb velocity occurrence was calculated and

analyzed. In order to ensure that peak limb velocity occurred at the same moment across

experiment (i.e., differing IRED locations), a pilot study was carried out with three participants

who did not participate in either the feedback or the TOJ experiment. In this pilot study,

participants performed the flinging movement while limb velocity was measured via an IRED

positioned on the index finger (i.e., experiment 1) as well as the wrist (i.e., experiment 2). Using

a Pearson correlation, the sample at which peak limb velocity occurred for both IREDs was

compared. This analysis revealed an average correlation of .99 between the samples in which

peak limb velocity occurred between the two IREDs. As such, although limb velocity was

14

measured at different locations on the moving limb, peak velocity occurrence was consistent

across both experiments.

General Procedure 3.3

Before the experimental trials, the participant estimated the target and home position. The

home position was estimated by having the participant rest their index finger on the home

position for a trial. When estimating the target position, the participant placed their finger at the

position of the virtual target. Participants were instructed to “fling” their index finger through the

target while moving as quickly and accurately as possible during the experimental trials (see

Figure 2). As such, the participant aimed to reach their peak limb velocity while passing through

the centre of the virtual target. After, participants returned to the home position and awaited the

signal for the next trial.

Figure 2. An example of the flinging movement performed in both experiments. Participants were instructed to align PLV

occurrence with the intersection of the target while propelling their limb to its terminal position.

15

3.3.1 Data Analysis Experiment 1

Performance was assessed across feedback conditions by analyzing: movement time (i.e.,

MT), average peak limb velocity, constant error (CE), variable error (VE), and resultant

displacement (i.e., R). Constant error was defined as the sum of the differences between the

position of peak limb velocity occurrence and target position divided by the number of trials.

Variable error was calculated as the square root of the sum of the squared differences between

the position of peak limb velocity occurrence and mean peak limb velocity location divided by

the number of trials. Resultant displacement refers to the distance from the home position

travelled when peak limb velocity was reached.

To determine the effects of feedback on performance accuracy, separate one-way

repeated measures ANOVA’s were conducted for constant and variable error, as well as resultant

displacement for all movement axes. Significant main effects were followed up with Tukey HSD

post hoc comparisons. Violations of spherecity were corrected using Hyunh-Feldt corrections

and the degrees of freedom reported to two decimal places.

3.3.2 Experiment 2 Data Analysis

As in experiment 1, constant error, variable error, and resultant displacement were

calculated as dependent measures. Although participants completed their entire flinging

movement, for the purposes of constant error and variable error calculations, peak limb velocity

position was utilized. In addition, the main dependent variable in this experiment was the

accurate reporting of which sensory cue was presented first. To determine how response

accuracy varies as a function of movement condition and sensory modality, a 2 Presentation (at

rest, at peak limb velocity) by 3 Sensory Cue (Auditory, Visual, Audiovisual) repeated measures

16

ANOVA was used. Tukey’s HSD post hoc comparisons were conducted on any main effect

associated sensory cue and stimulus presentation. Deviations from sphericity were corrected

using Hyunh-Feldt corrections, with the degrees of freedom reported to two decimal places.

17

Chapter 4 – Can you hear that peak? Utilization of 4auditory and visual feedback at peak limb velocity

18

4.1.1 Abstract

Purpose: At rest, the central nervous system combines and integrates multisensory cues to yield

an optimal percept. When engaging in action, the relative weighing of sensory modalities has

been shown to be altered. Because the timing of peak velocity is the critical moment in some

goal-directed movements (e.g., overarm throwing), the current study sought to test whether

visual and auditory cues are optimally integrated at that specific kinematic marker when it is the

critical part of the trajectory. Methods: Participants performed an upper-limb movement in

which they were required to reach their peak limb velocity when the right index finger

intersected a virtual target (i.e., a flinging movement). Brief auditory, visual, or audiovisual

feedback (i.e., 20 ms in duration) was provided to participants at peak limb velocity.

Performance was mainly assessed by analyzing the spatial bias and variability of the limb’s

position when it reached peak velocity. Results: Relative to when no feedback was provided,

auditory feedback significantly reduced the spatial variability of the finger position at peak limb

velocity. However, no such reductions were found for the visual feedback condition. Further,

providing both auditory and visual cues concurrently also failed to yield the predicted

improvements in endpoint variability. Conclusions: Overall, the central nervous system can

make significant use of an auditory cue, but may not optimally integrate a visual and auditory

cue at peak limb velocity, when peak velocity is the critical part of the trajectory.

19

Can you hear that peak?

Utilization of auditory and visual feedback at peak limb velocity

4.1.2 Introduction

Our environment is filled with multisensory stimuli. Incoming sensory information from

the various modalities creates a cohesive perception of our environment and, as such, is critically

relied upon for activities of daily living. Within the context of goal-directed action, studies have

primarily focused on skills where movement end was the critical moment of the trajectory (i.e.,

reach-to-grasp: e.g., Jeannerod, 1984). However, there are voluntary motor skills, such as

throwing a baseball or hitting a golf ball, for which the critical moment is ideally peak limb

velocity (PLV). In research contexts, these movements include tossing (e.g., Fleischauer &

Sherwood, 2008), flicking/flinging (e.g., Dulberg, Amant, & Zettlemoyer, 1999), and punching

(e.g., Cavanagh & Landa, 1976). Such movements can provide novel perspectives on the use of

sensory feedback during action because the predominantly utilized sensory modality (i.e., vision)

is not particularly salient at the critical kinematic marker (i.e., PLV). Indeed, the percept of the

limb when it reaches peak limb velocity is a blur, at best. As such, how can sensory feedback

from the non-visual modalities (e.g., audition) be utilized when the limb is travelling at high

velocities? Prior to addressing this question, selected studies were reviewed to ascertain current

knowledge on the use of vision, audition, and audio-visual information, specifically for

movements where the limb must stop onto a target (e.g., discrete reaches/ reaching).

During rapid upper-limb reaches, visual information gathered during the movement can be

used to perform online trajectory amendments (e.g., Keele & Posner, 1968). For example,

Proteau, Roujoula and Messier (2009) had participants perform a video-aiming task in which

vision of a cursor representing the participant’s limb was manipulated. Proteau et al. (2009)

20

reported that participants were able to initiate a correction soon after the cursor jumped, leading

them to suggest that visual information is monitored throughout discrete reaching movements

(see also Saunders & Knill, 2003). However, in these studies, the cursor jumps were

implemented early in the trajectory (i.e., soon after movement initiation). To test when visual

feedback typically contributes to online control during discrete reaches, Kennedy, Bhattacharjee,

Hansen, Reid, and Tremblay (2015) asked participants to perform 30 cm movements while

vision was provided during different windows along real-time limb velocity profiles.

Specifically, vision was provided early (i.e., limb velocity of 0.8-1.4 m/s, before PLV), in the

middle (i.e., limb velocity above 1.4 m/s, before and after PLV), or late (i.e., limb velocity of

1.4-0.8 m/s, after PLV) in the movement. Kennedy et al. (2015) reported that endpoint

consistency was comparable to full vision in the early and middle visual windows. However, the

middle visual feedback condition led to longer limb deceleration phase durations compared to

full vision. As such, the early window alone, which lasted only 43 ms, led to endpoint precision

and limb deceleration duration comparable to full vision. Thus, the spatiotemporal window for

processing online visual feedback could be considerably narrow. As a result, the gathering of

visual feedback can vary significantly during voluntary reaching movements and seems to

predominantly take place quite early in the trajectory.

Sensory information from other modalities (e.g., audition) can also provide valuable spatial

feedback during reaching movements. In the experimental trials of a study conducted by Levy-

Tzedek, Hanassy, Abboud, Maidenbaum, and Amedi (2012) sighted participants performed

upper-limb reaching movements while blindfolded and received auditory feedback via a sensory

substitution device (SSD). When provided with auditory feedback only, participant’s movements

did not differ significantly in terms of movement time, peak velocity, or amplitude as compared

to reaches performed with visual feedback. Although participants were more accurate when

21

visual information was available, reaches performed using the SSD had endpoint errors of less

than 0.5 cm on average (Levy-Tzedek et al. 2012). Overall, auditory feedback can also be used to

complete upper-limb reaches with excellent levels of precision.

Visual and auditory feedback can also be combined and integrated to yield an optimal

percept of spatial position. The Maximum Likelihood Estimation Model (MLE) forwarded by

Ernst and Bülthoff (2004) has provided a theoretical framework for how the central nervous

system accomplishes this task. According to the MLE, the central nervous system combines

incoming sensory information across multiple modalities in a statistically optimal fashion (e.g.,

Alais & Burr, 2004). Stimulus reliability affects the “weight” given to each sensory modality

during the integration process, with lesser weight assigned to unreliable cues (e.g., Ernst &

Banks, 2002). By taking a weighted average during the sensory integration process, sensory

estimates from different modalities are combined such that ambiguity is reduced, and a more

stable percept of a sensory event is formed. A well-cited example of multisensory integration is

spatial ventriloquism. This illusion can be characterized by watching television in which the

spatial location of auditory signals is biased towards the spatial location of the visual signal

(Alais and Burr, 2004). In fact, multisensory processing is so robust at the perceptual,

behavioural, and neuronal level (see Driver & Noesselt, 2008 for a review) that some researchers

hypothesize that the brain is largely organized for multisensory integration (e.g., Ghazanfar, &

Schroeder, 2006).

However, it is important to note here that support for the MLE and other Bayesian models

of sensory utilization have primarily stemmed from tasks requiring little or no movement (see

Witten & Knudsen, 2005 for a review; cf. Körding & Wolpert, 2006). Considering that we

hardly remain stationary in daily life (or at least we should not), it is important to consider how

sensory information from multiple modalities is combined and integrated during goal-directed

22

action. Interestingly, studies that have examined multisensory integration have revealed

considerable differences between multisensory integration at rest and multisensory integration

during action.

It has been reported that audiovisual processing is modulated during fast and accurate

reaching movements. In an experiment conducted by Tremblay and Nguyen (2010), participants

performed a 30 cm movement to a 0.5 cm target. Beneath the terminal point of this movement,

participants were presented with an audiovisual fission/ fusion illusion (see Shams, Kamitani, &

Shimojo, 2000). In this audiovisual illusion, the presence of auditory beeps has been reported to

bias the number of perceived visual flashes. For example, when two flashes and one beep were

presented, these events were fused such that participants often erroneously report seeing only one

flash (i.e., fusion illusion). Tremblay and Nguyen (2010) reported that participants became less

susceptible to the fusion illusion during goal-directed action. Critically, it was found the degree

to which participants fused the two visual stimuli was minimized at the highest limb velocity

stage of the movement. When taken as a whole, limb velocity can be a useful proxy to test

multisensory combination and integration (see Tremblay & de Grosbois, 2015), especially

because engaging in voluntary action significantly alters the integration of multisensory cues (see

also Juravle, Deubel, Tan, & Spence, 2010).

The experiments cited above (e.g., Proteau et al., 2009; Kennedy et al., 2015) employed

tasks where the participant’s goal was to terminate their movement onto a target (i.e., upper-limb

reaches). In such instances, visual feedback gathered early in the trajectory was utilized to bring

the limb to a halt, on its final position. However, the critical position of voluntary movements

can be during the trajectory, which could alter when sensory information is primarily gathered.

For example, to properly execute an overarm throw or golf swing, release or contact with the ball

at peak velocity is absolutely critical (e.g., Jegede, Watts, Stitt, & Hore, 2005). To the best of our

23

knowledge, however, no experiments have been conducted on the use of feedback presented at

peak limb velocity in order to control the limb position at peak limb velocity (i.e., PLV being the

critical portion of the trajectory).

The aim of the current study was to test whether the utilization and integration of

multisensory cues was optimal when auditory, visual, or audiovisual feedback were presented at

peak limb velocity, and that is, when peak limb velocity was the critical part of the trajectory

(i.e., a flinging movement). Although providing a visual cue normally facilitates performance,

this improvement may be modest with a flinging task because visual information about the limb

at peak limb velocity less certain than at rest (re.: blurred image of the limb). In addition, because

of audition’s excellent temporal sensitivity (Celesia, 1976), providing auditory cues could also

yield an improvement in performance, especially at the fastest stage of a rapid limb movement.

Finally, based on the presumption that the central nervous system still attempts to integrate

multisensory cues at peak limb velocity (Ernst & Bülthoff, 2004), visual and auditory cues

presented together should yield better endpoint precision than with visual or auditory feedback

alone. Alternatively, if the integration of the cues is suboptimal at peak limb velocity,

performance in one of the unisensory conditions (i.e., vision or audition alone) should yield

better, or at least as good, performance than the combined condition.

4.1.3 Methods

4.1.4 Participants

The University of Toronto Research Ethics Board approved the experimental protocol

reported herein. Thirteen individuals (6 males) with an average age of 23.7 years (SD = 2.2)

provided informed consent prior to participating in the experiment. Participants were recruited

from the graduate and undergraduate student populations at the University of Toronto. All

24

participants self-reported to be right-handed with normal or corrected-to-normal vision.

Participants were financially compensated $10.

4.1.5 Apparatus

Participant’s heads rested on a chin rest (38 cm tall) fastened to a 61 cm by 75 cm table

(see Figure 3 for a depiction of the experimental apparatus). A custom built wooden frame (48

cm tall) with a reflective surface 12 cm in diameter was positioned 13 cm away from the home

position. Attached to the upper most portion

of the chin rest was a yellow LED light (1

cm in diameter), which served as the target

for the experiment (see Figure 3). The

participant viewed the target via the

reflective surface such that the target

appeared elevated from the surface of the

table (i.e., a virtual target). Each trial began

with the participant’s finger on a 1.5 cm by

1.5 cm piece of Velcro, which served as the

home position for the experiment. The movements of the participant were measured along the Y-

axis (i.e., primary movement axis), as well as the X and Z-axes (i.e., secondary movement axes).

The resultant distance between the home position and observed target location was

approximately 40 cm. Along the X, Y, and Z axes, the target was approximately 15 cm to the

right, 27 cm away and 25 cm above the home position, respectively (see axes in Figure 3).

The position of the participant’s right index finger was monitored throughout the

experiment using an Optotrak Certus (Northern Digital Inc., Waterloo, ON, Canada) motion

Figure 3. Participants completed the experiment while their heads

rested in a chin rest and a virtual target was displayed in front of

them. This schematic is not drawn to scale.

25

capture system. An infrared light emitting diode (IRED) was attached to a banjo pick and placed

on the tip of the participant’s right index finger. The Optotrak real-time sampling rate was set at

500 Hz and controlled using a custom Matlab script (The Mathworks Inc., Natick, MA, USA).

Likewise, the Matlab script controlled the real-time feedback through an analog output board

(PCI-6024E, National Instruments Corp., Austin, TX, USA). The experiment was conducted in a

dark room to prevent the participants from viewing their limb. Also, participants wore a black

arm sleeve and the lights in the room were illuminated every five minutes to minimize the effects

of retinal and pupil adaptation to darkness (Fedorov & Mkrticheva, 1938; Hecht, 1920).

A custom piezo electric buzzer (2350 Hz, 75 dB) with a green LED light affixed to it was

positioned proximal to the IRED on the participant’s right index finger. This device (i.e., piezo-

LED) was used to provide participants with augmented feedback during the movement (i.e., for

20 ms only). Instantaneous velocity was calculated using the position data collected from two

subsequent samples gathered by the Optotrak system. Movement start location was defined when

the limb first exceeded a velocity of 0.03 m/s. Peak velocity was marked as two samples with a

velocity decrease, after the limb reached a minimum velocity of 0.8 m/s. The IRED position at

peak velocity was used to calculate constant and variable error, as well as movement time. The

resultant distance between movement start and peak limb velocity location was also calculated.

4.1.6 Procedure

Prior to beginning the experimental trials, the target and home position were estimated by

the participant and recorded using the Optotrak. The home position was estimated by having the

participant rest their index finger on the home position for a single trial. In order to prevent

participants from receiving augmented tactile feedback that would yield an improvement to

performance, participants performed their movement towards a virtual target. To estimate the

26

position of the virtual target, the participant placed their finger where the virtual target was

observed. At this time, the Optotrak recorded the position of the IRED and saved this location as

the target position. In the experimental trials, participants were instructed to “fling” their index

finger through the target as quickly and accurately as possible (see Figure 4). That is, the

participant’s task was to reach their peak limb velocity (i.e., as measured by the velocity of the

index finger) as they passed through the centre of the virtual target (i.e., no tactile feedback

provided). Following completion of the flinging movement, participants returned their finger to

the home position and awaited the signal for the next trial.

Figure 4. The flinging movement performed by participants and the corresponding velocity profiles.

27

Due to the novel dynamics of the task, participants first completed twenty familiarization

trials, followed by thirty baseline trials, where no feedback was given (i.e., no feedback

condition). The no feedback conditions were designed as a baseline and were thus not

counterbalanced with the experimental feedback conditions. After, participants received auditory

(i.e., piezo beep), visual (i.e., LED flash), or audiovisual feedback for thirty trials each in a

blocked and counter-balanced order. Real-time feedback was provided for 20 ms when and

where peak limb velocity was reached via the custom built piezo-LED apparatus affixed to the

finger. Based on the sampling frequency of 500 Hz, Matlab processing delays, and hardware

transmission delays; the onset of the flash or beep was no more than 10 ms after peak limb

velocity was reached, which was deemed to be in real-time. In the audiovisual condition, the

auditory and visual cues were presented simultaneously. Although previous work has shown that

to be perceived as simultaneous, the visual stimulus must lead the auditory stimulus by roughly

75-100 ms (e.g., Zampini, Shore, & Spence, 2003), the auditory and visual cues were presented

simultaneously in this study. This was done in part because peak limb velocity was detected in

real-time. As well, auditory and visual cues are presented simultaneously in the real world.

4.1.7 Data Analysis

To assess the flinging performance across the feedback conditions, movement time (i.e.

MT), average peak limb velocity, constant error (i.e., CE), variable error (i.e., VE) and resultant

displacement (i.e., DISP) were analyzed. Constant error was defined as the sum of the

differences between peak limb velocity position and the estimated target position divided by the

number of trials. Variable error refers to the square root of the sum of the squared differences

between peak limb velocity occurrence and mean peak limb velocity position divided by the

number of trials. In addition, resultant displacement and movement time were calculated as the

28

distance from the home position traveled when peak limb velocity was reached. To determine the

effect of feedback condition on these performance measures, separate one-way repeated

measures ANOVAs with four levels (i.e., no feedback, auditory, visual, audiovisual) were

performed for constant and variable error, across all movement axes. Alpha was set at .05 and

Huynh-Feldt corrections were applied when the sphericity assumption was violated. In all

instances of sphericity assumption violations, the degrees of freedom were reported to one

decimal place. Effects sizes are reported using partial eta squared. Tukey’s HSD comparisons

were used for post hoc analyses, when required. Additionally, the predicted weights of variability

in the audiovisual condition were computed using the equation:

𝜎!" =!!!  !  !!

!

!!!!  !!

!

(1)

In this equation (e.g., Ernst & Bülthoff, 2004), the product of the squared variability in the

spatial position of PLV occurrence in the auditory (A) and visual (V) condition is divided by the

sum of squared variability of PLV occurrence in the auditory and visual conditions.

4.1.8 Results

Means and standard deviations for all dependent variables can be found in Table 1. On

average, participants completed the movement in 150 ms, with an average peak velocity of 2.92

m/s. The ANOVAs performed for MT and PLV revealed that movement time did not vary

significantly across feedback conditions, F (3, 36) = 1.5, p = .23, 𝜂!! = .03, nor did peak velocity,

F (3, 36) = .18, p = .91, 𝜂!! = .02. When analyzing constant error in the primary movement axis,

no significant effect of feedback condition was found, F (3, 36) = 2.9, p = .11, 𝜂!! = .14. This

result was also found in the X-axis, F (3, 36) = .21, p = .88, 𝜂!! = .03, and Z-axis F (3, 36) = .43,

29

p = .73, 𝜂!! = .04, respectively. Lastly, when analyzing constant error for resultant displacement,

no significant effect of feedback condition was found, F (3, 36) = 1.3, p = .30, 𝜂!! = .11.

Table 1

Means and Between-subject Standard Deviations for Movement Time (MT), Average Peak Limb Velocity (PLV),

Constant error (CE) and Variable error (VE) Across the Three Movement Axes (i.e., Y-X-Z), and CE and VE for

Resultant Displacement (R). Values in Bold Indicate a Statistically Significant Difference Between Feedback

Conditions.

MT (ms)

PLV (m/s)

CE-Y (mm)

CE-X (mm)

CE-Z (mm)

CE-R (mm)

VE-Y (mm)

VE-X (mm)

VE-Z (mm)

VE-R (mm)

Feedback Condition

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

No Feedback

152.4 (24.8)

2.8 (0.7)

-110.3 (73.6)

-3.2 (32.1)

-64.9 (48.5)

54.6 (64.1)

48.4 (20.2)

17.2 (11.3)

34.5 (14.1)

56.2 (24.5)

Auditory

149.9 (19.2)

2.9 (0.6)

-93.5 (49.2)

-3.2 (33.6)

-58.7 (55.5)

66.6 (59.2)

27.7 (11.5)

10.3 (3.8)

20.1 (8.4)

30.8 (14.8)

Visual

159.1 (19.1)

2.9 (0.5)

-90.1 (55.7)

-3.8 (34.3)

-59.9 (40.7)

71.4 (53.2)

37.1 (18.7)

12.1 (8.2)

25.2 (16.3)

43.2 (25.5)

Audiovisual

157.9 (24.2)

2.9 (0.5)

-86.3 (48.3)

-6.7 (30.4)

-57.9 (40.1)

80.6 (45.1)

38.3 (14.4)

14.1 (7.6)

22.1 (7.8)

41.1 (17.2)

In contrast, the analyses of variable error in the primary, secondary, and tertiary axis (i.e.,

Y-axis, X-axis, Z-axis, respectively) yielded main effects of feedback condition: Y-axis, F (3,

36) = 6.5, p = .01, 𝜂!! = .35; X-axis, F (2.4, 29.1) = 3.3, p = .01, 𝜂!! = .21; and Z-axis, F (3, 36) =

4.1, p = .09, 𝜂!! = .27. Tukey’s HSD post hoc contrast thresholds were calculated for the three

axes: X-axis: HSD = 6.83 mm; Y-axis: HSD = 12.89 mm; Z-axis: HSD = 13.05 mm. The post

hoc contrasts revealed that variability of the finger position at peak limb velocity was

significantly lower in the auditory condition than in the no feedback condition across all three

axes: Y-axis: p < .01; X-axis: p = .03; Z-axis: p = .03 (see Table 1). As well, these variable error

30

values in the visual and audiovisual conditions were not significantly lower than in the no

feedback condition while being not significantly worse than in the auditory condition (i.e., all ps

> .09, see Table 1). Accordingly, these main effects and post hoc contrasts were replicated for

the variable error values of the resultant displacement, F (3, 36) = 6.4, p = .01,  𝜂!! = .35, HSD =

17.5 mm (see Figure 5).

To follow up these variable error findings, the predicted weights of the auditory and visual

cues in the audiovisual condition were computed using Equation 1. In this equation, the observed

variability in peak limb velocity occurrence was calculated for the auditory and visual conditions

and used to predict the variability in peak limb velocity occurrence in the audiovisual condition.

This analysis revealed the mean predicted variability in spatial position of peak limb velocity

occurrence along the Y-axis (i.e., primary axis) was 21.1 mm (SD = 9.3). This mean predicted

variability significantly differed from the mean observed variability t (12) = 4.08, p = .002, Mean

Observed = 38.3 mm (SD = 14.4). This relationship was also found for the secondary axes: X-

axis; Mean Predicted = 7.4, SD = 2.8, Mean Observed = 14.1 (SD = 7.6), t (12) = 3.66, p = .003,

as well as the Z-Axis; Mean Predicted = 14.7 (SD = 7.6), Mean Observed = 22.1 mm (SD = 7.8),

t (12), 2.7, p = .02. Finally, as shown in Figure 5, when analyzing resultant displacement, the

predicted value of 23.7 mm (SD = 12.5 mm) significantly differed from the observed variability

in displacement of 41.1 mm, t (12) = 3.44, p = .005.

31

Figure 5. Variability in resultant displacement across all experimental conditions as well as MLE predicted

variability in the audiovisual condition.

4.1.9 Discussion

The processing and integration of multisensory stimuli at peak limb velocity to control

limb position at that kinematic marker was the primary focus of the current experiment. To this

end, participants were tasked with flinging their index finger through a virtual target, and

aligning the moment their right index finger reached its peak velocity with the position of the

virtual target (i.e., participants did not make contact with the target thus avoiding terminal

feedback). At peak limb velocity, participants received auditory, visual, or audiovisual feedback.

In addition, trials where no feedback was given were also completed. Based on the MLE (Ernst

& Bülthoff, 2004), it was hypothesized that performance would be better in the audiovisual

condition relative to at least one of the unisensory conditions. Contrary to this, it was found that

only auditory feedback yielded a significant reduction in performance variability as compared to

when no feedback was provided. The results thus indicate that the most reliable sensory cue to

32

detect the spatial position of the limb at peak velocity can be audition, when peak limb velocity

is the critical part of a fast and accurate voluntary action.

An explanation for the lack of differences between the no feedback and visual feedback

conditions may be related to the utilization of visual feedback. Although many studies have

shown that visual information facilitates the execution of voluntary action (e.g., Zelaznik,

Hawkins, & Kisselburgh, 1983; Keele & Posner, 1968), augmented visual feedback failed to

yield a significant improvement to the flinging task employed in this study. This may be related

to the differences in visual processing between traditional tasks where movement end is the

critical moment for sensory feedback utilization and tasks where peak limb velocity is of greater

importance, as in the current study. As introduced above, feedback utilization before peak limb

velocity for an endpoint control task such as those employed by Proteau et al. (2009) and

Kennedy et al. (2015) may occur very early on in the trajectory. In contrast, for tasks where peak

limb velocity is the critical part of a movement, the use of visual information may be

significantly limited.

An alternate yet not mutually exclusive explanation for the lack of visual feedback

facilitation may be related to the quality of the visual cue. As noted in the Methods section, the

entire experiment was conducted in darkness. Because the limb was traveling at such a high

velocity (i.e., approximately 2.9 m/s) when the visual feedback was presented, the LED created a

streak of light instead of a discrete indication of where peak limb velocity occurred. Although

peak limb velocity was associated with the first point along this streak, the participants may have

been unable to form a stable representation of where peak limb velocity occurred and thus

augmented visual feedback did not reliably improve their performance. Nevertheless, visual

feedback did not yield significant improvements in performance for the flinging task employed.

33

A consideration regarding the audiovisual condition is related to the simultaneity in the

presentation of auditory and visual cues. One might surmise that because auditory information is

processed faster than visual cues (e.g., Celesia, 1976; Zampini et al., 2003), presenting the

auditory and visual information together would have led to asynchronous integration of the two

modalities. This could explain why the auditory information led to the best performance. If this

were the case, it would have been expected that the position of peak limb velocity occurrence in

both the visual and audiovisual conditions would reflect the approximate 100 ms difference

associated with the temporal window of integrating visual and auditory cues (e.g., Colonius &

Diederich, 2004). That is, when traveling at 2.92 m/s when peak limb velocity was reached, it

would be expected that the position of peak limb velocity occurrence would be biased roughly

0.29 m further than observed in the resultant axis. These differences between the actual values,

(Mean Visual = 71.4 mm, Mean Auditory = 66.6 mm), however are negligible when compared to

the predicted values.

When considering the results of the auditory and visual conditions alone, the MLE (Ernst

& Bülthoff, 2004) would also predict that the results of the audiovisual condition would be at

least as good as the auditory condition. However, the results suggest that in the audiovisual

condition, participants have heavily weighted the visual cue instead of the most reliable sensory

cue (i.e., audition) and thus performance was not facilitated. Support for this claim can be found

in the significant differences found for the predicted performance in the audiovisual condition.

Specifically, the predicted values across the movement axes, and well as resultant displacement

were found to be significantly less than the observed values, suggesting that the weights assigned

to the auditory and visual cues at peak limb velocity was suboptimal. This finding has

implications for the optimal integration of multisensory cues during goal-directed action.

Specifically, when peak limb velocity is the critical moment in the trajectory, optimal

34

multisensory integration may not necessarily occur. Finally, the experimental task employed here

(i.e., a flinging task) provides a novel approach to studying sensory processing during an

ongoing movement. This particular focus certainly has practical applications for skills such as

throwing a baseball or hitting a golf ball.

4.1.10 What Does This Article Add?

This study adds to the literature in two ways. The flinging task we employed provides a

useful and novel approach to studying sensory processing throughout an ongoing movement.

Although there is predominant focus in the literature on sensory processing during reaches to

terminal targets, it is important to focus on sensory processing at peak limb velocity because of

practical applications in sports contexts, such as baseball and golf. Also, the main findings add to

the growing literature on how the central nervous system integrates sensory cues across various

modalities during voluntary movements. Overall, this study supports the position that the CNS

combines and integrates sensory information in a flexible and task specific manner, but such

integration during voluntary action is not necessarily optimal.

35

4.1.11 References

Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal

integration. Current Biology, 14(3), 257–262. doi:10.1016/j.cub.2004.01.029

Cavanagh, P. R., & Landa, J. (1976). A biomechanical analysis of the karate chop. Research

Quarterly, 47(4), 610–618. doi:10.1080/10671315.1976.10616718

Celesia, G. G. (1971). Organization of auditory cortical areas in man. Brain, (99), 403–414.

Colonius, H., & Diederich, A. (2004). Multisensory interaction in saccadic reaction time: A

time-window-of-integration model. Journal of Cognitive Neuroscience, 16(6), 1000–9.

doi:10.1162/0898929041502733

Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on

“sensory-specific” brain regions, neural responses, and judgments. Neuron, 57(1), 11–23.

doi:10.1016/j.neuron.2007.12.013

Dulberg, M. S., Amant, R. S., & Zettlemoyer, L. S. (1999). An imprecise mouse gesture for the

fast activation of controls. In A. M. Sasse, & C. Johnson (Eds.), Proceeding of Human-

Computer Interaction – INTERACT 1999 (pp. 375-385) Amsterdam, NL: IOS Press.

Ernst, M. O., & Banks. M. S. (2002). Humans integrate visual and haptic information in a

statistically optimal fashion. Nature, 415, 429–434. doi:10.1038/415429a

Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in

Cognitive Sciences, 8(4), 162–169. doi:10.1016/j.tics.2004.02.002

36

Fedorov, N. T., & Mkrticheva, L. (1938). Mechanism of light flicker fusion during the course of

dark and light adaptation. Nature, 142, 750–751.  doi:10.1038/142750a0

Fleischauer, M. A., & Sherwood, D. E. (2008). Development of motor error detection capability

in an underhand throwing task. In Beaulieu, N. P. (Ed) Physical Activity and Children:

New Research, 195-210e. New York: Nova Science Publishers.

Gepshtein, S., Burge, J., Ernst, M. O., & Banks, M. S. (2005). The combination of vision and

touch depends on spatial proximity. Journal of Vision, 5(11), 1013–23. doi:10.1167/5.11.7

Ghazanfar, A. A, & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in

Cognitive Sciences, 10(6), 278–285. doi:10.1016/j.tics.2006.04.008

Hecht, S. (1920). The dark adaptation of the human eye. The Journal of General Physiology,

2(5), 499–517. doi:10.1085/jgp.2.5.499

Jeannerod, M. (1984). The timing of natural prehension movements. Journal of Motor Behavior,

16(3), 235–254. doi:10.1080/00222895.1984.10735319

Jegede, E., Watts, S., Stitt, L., & Hore, J. (2005). Timing of ball release in overarm throws

affects ball speed in unskilled but not skilled individuals. Journal of Sports Sciences, 23(8),

805–16. doi:10.1080/02640410400021393

Juravle, G., Deubel, H., Tan, H. Z., & Spence, C. (2010). Changes in tactile sensitivity over the

time-course of a goal-directed movement. Behavioural Brain Research, 208(2), 391- 401.

doi:10.1016/j.bbr.2009.12.009

37

Keele, S. W., & Posner, M. I. (1968). Processing of visual feedback in rapid movements. Journal

of Experimental Psychology, 77(1), 155–158. doi:10.1037/h0025754

Kennedy, A., Bhattacharjee, A., Hansen, S., Reid, C., & Tremblay, L. (2015). Online vision as a

function of real-time limb velocity: another case for optimal windows. Journal of Motor

Behavior, 47(6), 465-475. doi:10.1080/00222895.2015.1012579

Körding, K. P., & Wolpert, D. M. (2006). Bayesian decision theory in sensorimotor control.

Trends in Cognitive Sciences, 10(7), 319–326. doi:10.1016/j.tics.2006.05.003

Levy-Tzedek, S., Hanassy, S., Abboud, S., Maidenbaum, S., & Amedi, A. (2012). Fast, accurate

reaching movements with a visual-to-auditory sensory substitution device. Restorative

Neurology and Neuroscience, 30(4), 313–323. doi:10.3233/RNN-2012-110219

Proteau, L., Roujoula, A., & Messier, J. (2009). Evidence for continuous processing of visual

information in a manual video-aiming task. Journal of Motor Behavior, 41(3), 219–231.

doi:10.3200/JMBR.41.3.219-231

Saunders, J. A., & Knill, D. C. (2003). Humans use continuous visual feedback from the hand to

control fast reaching movements. Experimental Brain Research, 152(3), 341–352.

doi:10.1007/s00221-003-1525-2

Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408,

788. doi:10.1038/35048669

Tremblay, L., & de Grosbois, J. (2015). Why encode limb and body displacements in the

velocity domain? Neurophysiological and behavioral evidence. In T. Heinen, (Ed.),

38

Advances in visual perception research (pp. 279-292). Hauppauge, NY, USA: Nova

Science Publishers.

Tremblay, L., & Nguyen, T. (2010). Real-time decreased sensitivity to an audio-visual illusion

during goal-directed reaching. PLoS One, 5(1), e8952. doi:10.1371/journal.pone.0008952

Witten, I. B., & Knudsen, E. I. (2005). Why seeing is believing: merging auditory and visual

worlds. Neuron, 48(3), 489–496. doi:10.1016/j.neuron.2005.10.020

Zelaznik, H. N., Hawkins, B., & Kisselburgh, L. (1983). Rapid visual feedback processing in

single-aiming movements. Journal of Motor Behavior, 15(3), 217–236.

doi:10.1080/00222895.1983.10735298

39

Multisensory Integration at Peak Limb Velocity in a Within-4.2

Modality Temporal Order Judgment Task

40

4.2.1 Abstract

During goal-directed action, sensory processing is critical. When sensory information

from the various modalities is concurrently available, the cues are integrated in a statistically

optimal fashion. However, previous reports have shown that limb velocity may have a

modulatory effect on unimodal and multisensory processing. The current study focused

specifically on multisensory integration at peak limb velocity. Participants were required to fling

their limb through a virtual target and align their peak limb velocity with the intersection of the

target. At peak limb velocity, participants were presented with two auditory, visual, or

audiovisual cues. Following completion of the flinging movement, participants reported which

side of the target the first sensory cue was presented (i.e., a temporal order judgment task [TOJ]).

Participants also made this TOJ when the sensory cues were presented at rest. It was found that

TOJs were more accurate following presentation at rest than at peak limb velocity. At rest,

participants were more accurate in the auditory and audiovisual condition relative to the visual

condition. In addition, when comparing accuracy in the audiovisual condition at rest to peak limb

velocity, it was found that performance significantly decreased only in this condition. Overall,

the results suggest that multisensory estimates obtained at peak limb velocity may be less robust

than those obtained under unimodal conditions.

41

4.2.2 Introduction

Integrating information across the sensory modalities shapes the perception of our

environment. Temporal information from the auditory modality and spatial information from the

visual modality are relied upon not only to perceive our environment, but also to execute goal-

directed actions (Sedda, Monaco, Bottini, & Goodale, 2011). Within the context of goal-directed

action, visual information is heavily relied upon to execute spatially driven tasks (e.g., Keele &

Posner, 1968), while auditory information dominates the temporal domain (e.g., Recanzone,

2003). When auditory and visual information are concurrently available, the central nervous

system typically integrates the information in a statistically optimal way (e.g., Ernst & Bülthoff,

2004). Although optimal audiovisual integration has been reported at rest (e.g., Alais & Burr,

2004), engaging in action has been shown to have a modulatory effect on multisensory

integration (e.g., Tremblay & Nguyen, 2010). In addition, the results of shown in Chapter 4.1

further suggest that optimal integration of audiovisual feedback may not occur when presented

specifically at peak limb velocity (PLV). In the experiment reported herein, we provide further

evidence that the weighting of auditory and visual cues presented at peak limb velocity may be

suboptimal.

A classic approach used to study sensory processing has been the temporal order

judgment task. In this task, participants are exposed to a pair of sensory cues presented at various

stimulus onset asynchronies (SOA), with the participant reporting which cue was presented first

(e.g., Hirsh & Sherrick 1961; Kanabus, Szelag, Rojek, & Pöppel, 2002; Spence, Shore, & Klein,

2001). Due to the processing latencies across the senses (e.g., Jaekl & Harris, 2007), an

audiovisual event is perceived as simultaneous, if the visual stimulus is presented before the

42

auditory stimulus by ~75-100 ms (e.g., Zampini, Shore, & Spence, 2003; Jaekl & Harris, 2007).

Temporal order judgment tasks are an excellent paradigm to assess sensory processing because

the temporal and spatial constraints of the task can be manipulated independently. By orienting

participants to attend to which cue was presented first (i.e., temporal constraints) as opposed to

which side was presented first (i.e., spatial constraints) it is possible to assess how temporal and

spatial information are taken into account when perceiving audiovisual events (e.g., Zampini et

al., 2003).

Despite the long-standing interest in temporal order judgment tasks (e.g., Hamlin, 1895),

this paradigm is not without its shortcomings and confounds. One of the most commonly cited

issues is the lack of spatial compatibility between the presentation of the auditory and visual

cues. Specifically, visual information is sometimes presented directly in front of the participant,

whereas auditory information is presented via headphones worn by the participant (e.g., Bushara,

Grafman, & Hallett, 2001; Jaskowski, Jaroszyk, & Hojan-Jezierska, 1990; cf: Spence et al.,

2001; Zampnini et al., 2003). As such, there exists a lack of precision in understanding how

sensory information is processed within the TOJ task due to this spatial confound (Zampini et al.,

2003). Further, the experiments cited above had participants perform the TOJ task only at rest,

neglecting sensory processing during action. Considering that we spend a large portion of time

executing goal-directed actions towards multisensory stimuli, it is pertinent to understand how

sensory information is processed during action. In the current study, participants were exposed to

spatially congruent auditory and visual information and completed a TOJ following stimulus

presentation at peak limb velocity or at rest.

Outside of the temporal order judgment literature, many experiments have examined how

sensory information is processed and utilized specifically during goal-directed action. During

43

action, vision is deemed to be heavily relied upon throughout an ongoing movement (e.g., Keele

& Posner, 1968; Zelaznik, Hawkins, & Kisselburgh, 1983; Proteau, Roujoula & Messier, 2009).

In fact, some researchers have proposed that visual information is critical specifically during the

latter phases of a movement (e.g., Carlton, 1981; Woodworth, 1899), while others suggest that

early trajectory information is critical (e.g., Kennedy Bhattacharjee, Hansen, Reid, & Tremblay,

2015).

Auditory information can also be utilized to during goal-directed actions. Levy-Tzedek,

Hanassy, Abboud, Maidenbaum, and Amedi (2012) provided sighted participants with a sensory

substitution device (SSD) conveying movement related information via auditory feedback. When

comparing movement time, peak velocity, and amplitude, no differences were found for

movements performed with visual feedback, or with the SSD. In a similar study conducted by

Rosati, Oscari, Spagnol, Avanzini, and Masiero (2012), participants tracked a target on screen

while receiving continuous auditory feedback. Of particular interest are the results from

Experiment 1 and 3 of Rosati et al. (2012). These authors found that auditory feedback

significantly reduced tracking errors relative to when no auditory feedback was available. In

addition, auditory and visual feedback yielded greater performance on a novel visuomotor

transformation task relative to visual feedback alone (Rosati et al., 2012). Overall, this evidence

suggests that task specific auditory information can facilitate sensorimotor control.

When both auditory and visual information are available, the central nervous system

integrates this information in a statistically optimal way (Ernst & Bülthoff, 2004). Redundant

sensory information within a modality is combined, and information across multiple modalities

is integrated (i.e., in a body-centred frame of reference) to yield an optimal percept (Ernst &

Banks, 2002). An example of optimal audiovisual integration is the ventriloquist effect whereby

44

the perceived spatial position of auditory information is biased towards the spatial position of the

simultaneously presented visual information (Alais & Burr, 2004; Howard & Templeton, 1966).

Multisensory integration has been reported at the neuronal level (e.g., Meredith, Nemitz & Stein,

1987) and numerous experiments have shown that the central nervous system is largely

organized for multisensory integration (see Driver & Nosselet, 2003; Ghazanfar & Schoerder,

2006 for reviews).

Despite the evidence that optimal integration occurs whenever auditory and visual events

are simultaneously presented, recent evidence suggests this may not always be the case.

Critically, many of the experiments reporting optimal multisensory integration involve a

participant whom passively observed an audiovisual event (see Witten & Knudsen, 2005 for a

review). Interestingly, studies examining multisensory integration during action have revealed

optimal multisensory integration may not always occur. In a study conducted by Tremblay and

Nguyen (2010), participants were exposed to the audiovisual fission/ fusion illusion (Shams,

Kamitani, & Shimojo, 2000) during a 30 cm reaching movement. The authors reported that the

degree to which participants were fusing the audiovisual event was modulated by limb velocity,

such that the least amount of fusion was reported at the highest limb velocities. Based on these

findings, it appears that multisensory integration may not always be optimal during goal-directed

action (see also Juravle, Deubel, Tan, & Spence, 2010).

It is important to note that the experiments cited in the preceding paragraph examined

motor skills where movement end was the critical moment in the trajectory. However, for other

movements such as overarm throwing and hitting a golf ball, the critical moment in the trajectory

is ideally peak limb velocity. As such, how visual information is utilized specifically when peak

limb velocity is the critical moment in the trajectory has gone relatively understudied. In the

45

current study, sensory processing and integration was assessed specifically at this kinematic

marker.

In line with this approach, the experiment reported in Chapter 4.1 had participants

perform an upper-limb flinging movement where their goal was to align peak limb velocity with

the intersection of a virtual target (i.e., in the absence of terminal feedback). At peak limb

velocity, participants received augmented auditory, visual, or audiovisual feedback in an attempt

to facilitate performance. Relative to when no feedback was provided, it was found that

providing auditory feedback significantly reduced the variability in position of the peak limb

velocity occurrence along the primary movement axis, as well as the displacement traveled when

peak limb velocity was reached. Further, no significant improvements were found when visual or

audiovisual feedback was provided. Critically, the performance in the audiovisual condition was

significantly worse than the predicted MLE values (e.g., Ernst and Bülthoff, 2004). Overall, the

results from this study, in conjunction with those reported by Tremblay and Nguyen (2010),

suggest that when the limb is traveling at its peak velocity, optimal integration of audiovisual

cues may not occur.

In line with the Tremblay and Nguyen (2010) results, the current experiment sought to

further test whether optimal integration occurs when the limb is traveling at peak limb velocity.

To this end, participants performed an upper-limb flinging movement with the goal of aligning

their peak limb velocity with the intersection of a virtual target. At peak limb velocity,

participants were presented with two auditory, visual, or audiovisual cues and reported which

cue was presented first in a within-modality temporal order judgment. Participants also

completed this judgment while remaining stationary. During action, it was expected that

performance across conditions would decrease relative to TOJs performed at rest (e.g., Juravle et

46

al., 2010). Based on the MLE (Ernst & Bülthoff, 2004) it was hypothesized that participants

would be more accurate at rest in the audiovisual condition relative to the auditory and visual

conditions. If audiovisual information is integrated optimally during action, performance in the

TOJ should still be best in the audiovisual condition. Alternatively, if audiovisual cues are not

integrated optimally at peak velocity, performance should significantly decrease following

presentation at peak velocity compared to at rest.

4.2.3 Methods

4.2.4 Participants

The University of Toronto Research Ethics Board approved the experiment reported

herein. Of the sixteen participants recruited, three were excluded because their TOJ accuracy at

rest was below chance (i.e., average of 30 or less correct responses across conditions). As such,

data from thirteen individuals (3 males) with an average age of 22.75 years (SD = 4.5) was

analyzed. All participants provided informed consent prior to participation. Participants were

recruited from the undergraduate student population at the University of Toronto. All

participants were right-handed (i.e., self-reported), and had normal or corrected-to-normal vision

at the time of participation. For completing the experiment, participants were compensated $10.

4.2.5 Apparatus

Participant’s heads rested on a chin rest (38 cm tall) fastened to a 61 cm by 75 cm table

(see Figure 6 for a depiction of the experimental apparatus) for during the experimental trials. A

custom built wooden frame (48 cm tall) with a reflective surface 12 cm in diameter was

positioned 13 cm away from the home position. Attached to the forehead mount portion of the

47

chin rest was a yellow LED light (1 cm in diameter), which served as the target for the

experiment. The target was visible to the participant through the reflection such that the target

appeared suspended in the air above the table. Two piezo-LED devices were placed on the

wooden frame 1.5 cm to the left and right of the virtual target. Both devices consisted of a green

LED light affixed to a piezo-buzzer (2350 Hz, 75 dB).

Participants wore a Life Brand model 225ZZ wrist brace with splint (Shoppers Drug

Mart, Toronto, ON, Canada) to restrict movement of the hand and wrist. Each trial began with

the participant’s finger on the home position (i.e., a 1.5 cm by 1.5 cm piece of Velcro). The

position of the participant’s right limb was

recorded throughout the experiment using an

Optotrak Certus (Northern Digital Inc.,

Waterloo, ON, Canada) motion capture system.

An infrared emitting diode (IRED) was

positioned on the distal epiphysis of the

participant’s ulna. The Optotrak real-time

sampling rate was set at 500 Hz and controlled

using a custom Matlab script (The MathWorks

Inc., Natwick, MA, USA). This Matlab script

also controlled the piezo-LED devices through an analog output board (PCI-6024E, National

Instruments Corp., Austin, TX, USA). Participant’s movements were measured along the Y-axis

(i.e., primary movement axis), as well as the X and Z-axes (i.e., secondary movement axes).

The piezo-LED devices presented sensory cues to participants either at peak limb

velocity or while the participant remained at rest. On every trial, both piezo-LED devices would

Figure 6. A depiction of the experimental apparatus. This figure

is not to scale.

48

present two auditory beeps, two visual flashes, or two audiovisual events. The stimulus onset

asynchrony (SOA) was + or - 20 ms. When the cues were presented at peak limb velocity,

instantaneous velocity was calculated using the position data collected from two subsequent

samples gathered by the Optotrak. Movement start location was defined when the limb first

exceeded a velocity of 0.03 m/s. Peak velocity (i.e., as measured by the IRED position on the

wrist) was marked as two consecutive samples with a decrease in velocity, after the limb had

exceeded a minimum velocity of 0.8 m/s. Because participants were aiming with their index

finger while velocity was measured on the wrist, it was pertinent to ensure the moment of peak

velocity occurrence was consistent between the finger and wrist. To this end, three participants

who did not participate in this experiment completed a brief pilot study. In this study,

participants performed the flinging task for thirty trials where no feedback was given. In this

pilot study, limb velocity was measured via an IRED positioned on the index finger (i.e., Chapter

4.1) and the ulna (i.e., Chapter 4.2). Pearson correlations were used to compare the sample at

which peak limb velocity occurred for both IRED locations. This analysis revealed a .99

correlation between the sample at which peak limb velocity occurred on the finger and ulna.

Thus, although limb velocity was measured on the wrist, the point at which peak limb velocity

occurred was consistent with the moving finger.

4.2.6 Procedure

The target and home position were estimated by the participant and recorded using the

Optotrak prior to beginning the experiment. The home position was estimated by having the

participant rest their index finger on the home position for a trial. To estimate the target position,

the participants then moved their finger out to the observed position of the virtual target. During

the experimental trials, participants were instructed to “fling” their index finger through the

49

virtual target (i.e., participants did not make physical contact with a target) and align their peak

velocity when the finger intersected the target (see Figure 7). A virtual target was used because

physical contact with an object would have yielded tactile feedback, hence obscuring the actual

effects of the investigated visual and auditory information processing. Upon completion of the

flinging movement, participants returned to the home position and awaited the signal for the next

trial.

Figure 7. An illustration of the flinging movement performed by participants. The target appeared roughly 40 cm away from the

starting position.

50

Participants first completed 20 familiarization trials to habituate to the task dynamic (i.e.,

viewing a target via a reflection). During these trials, participants were given audiovisual

feedback when their limb reached peak velocity via the piezo-LED devices positioned on either

side of the virtual target. Both devices were triggered simultaneously, with the participant

instructed to have their finger as close to the target as possible when the audiovisual feedback

was given. After, participants completed 120 experimental trials per sensory condition in a

blocked and counter-balanced order. Within each block (i.e., auditory, visual, or audiovisual),

participants were presented with two auditory beeps, visual flashes, or audiovisual events from

either side of the virtual target as described above. As mentioned above, the onset of the stimuli

was held constant at a SOA of + or - 20 ms. The side of the first cue was pseudo-randomized

during each block. Specifically, participants were exposed to 30 trials where the first cue was

presented to the left of the target, and 30 where the first cue was presented from the right of the

target, at rest and at peak limb velocity.

Participants also completed TOJs following stimuli presentation at rest. During these

trials, the virtual target would become illuminated, followed 100 ms after by the first sensory cue

on either side of the target as described above. Participants were instructed to maintain fixation

on the remembered location of the virtual target. Following completion of all trials, participants

responded “Left first” or “Right first” via a button box held in their left hand. Lastly, the

calculation of our dependent variables (i.e., constant error, variable error, resultant displacement,

and movement time) was based on the IRED position at peak limb velocity.

4.2.7 Results

Means and standard deviations, as well as accuracy results can be found in Table 2.

Participants completed the flinging movement in 203 ms on average. Across conditions, the

51

average peak limb velocity (i.e., as measured by the IRED location) achieved was 1.84 m/s.

Movement time and average peak velocity were analyzed using separate one-way repeated

measures ANOVAs. These tests confirmed that movement time, F (2, 24) = .28, p = .76, 𝜂!! =

.02, and peak velocity, F (2, 24) = 1.1, p = .37, 𝜂!! = .08, did not significantly vary as a function

of sensory cue. In addition, the ANOVA conducted for end position along the primary movement

axis (i.e., Y-axis) yielded no significant effects of sensory cue, F (2, 24) = 1.5, p = .24, 𝜂!! = .21.

This effect was also found when analyzing end position along the secondary axes: X-axis: F (2,

24) = 1.7, p = .85, 𝜂!! = .01; Z-axes: F (2, 24) = .30, p = .97, 𝜂!! = .02. Finally, the ANOVA

conducted for resultant displacement yielded no significant effect of sensory cue, F (2, 24) = .72,

p = .50, 𝜂!! = .01.

Table 2

Means and Between-subject Standard Deviations for Movement Time (MT), Average Peak Limb

Velocity (PLV), Constant Error (CE) and Variable Error (VE) Across the Three Movement Axes

(Y-X-Z), and CE and VE for Resultant Displacement (R).

MT (ms)

PLV (m/s)

CE-Y (mm)

CE-X (mm)

CE-Z (mm)

CE-R (mm)

VE-Y (mm)

VE-X (mm)

VE-Z (mm)

VE-R (mm)

Sensory Condition

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

M (SD)

Auditory

206.1

(64.1)

1.9

(0.1)

-46.79

(31.7)

-43.8

(62.1)

-54.3

(28.7)

99

(32.8)

29.5

(10.7)

16.1

(6.3)

23.4

(10.1)

35.4

(19.1)

Visual

198.3

(33.6)

1.8

(0.1)

-49.31

(40.4)

-46.1

(69.4)

-53.4

(33.6)

99.3

(43.3)

24.3

(7.1)

19.3

(10.3)

25.6

(8.8)

43.2

(19)

Audiovisual

204.5

(44.3)

1.9

(0.1)

-40.1

(30.7)

-44.4

(58.6)

-54.1

(29.5)

104.9

(33)

23.6

(7)

14.2

(4.5)

18.9

(5.3)

25.5

(7.2)

52

In contrast, when analyzing accuracy (i.e., the number of correct responses) when the

participant remained at rest, a significant main effect of sensory cue was found, F (2,24) = 7.7, p

= .01, 𝜂!! = .49. Tukey’s HSD post hoc comparisons revealed that participants were more

accurate in the auditory condition than the visual condition (HSD = 5.3, Mean Auditory = 41.6,

Mean Visual = 33.7, p = .01). Post hoc comparisons also revealed participants were more

accurate in the audiovisual condition compared to the visual condition (HSD = 4.4, Mean

Audiovisual = 39.1, Mean Visual = 33.7, p = .05). The analysis conducted on TOJ accuracy

following stimuli presentation at rest versus at peak velocity yielded a significant main effect, t

(38) = 2.7, p = .01, with participants being more accurate at rest (M = 38.2, SD = 7.9) than at

peak limb velocity (M = 35.1, SD = 6.9). In addition, the interaction between sensory cue and

movement failed to reach significance, F (2, 24) = 2.1, p = .15, 𝜂!! = .13. However, because of

the large effect size associate with the interaction, it was deemed theoretically relevant to carry

out post hoc comparisons (e.g., Field, 2009). When analyzing within the sensory modalities, it

was also found that accuracy only decreased significantly between movement conditions in the

audiovisual condition (HSD = 5.4, Mean Audiovisual Rest = 39.1, Mean PLV = 33.6, p = .05).

These findings are depicted in Figure 8 (see also Table B).

53

Figure 8. Number of correct responses as a function of movement and sensory conditions.

4.2.8 Discussion

The processing of unimodal cues, and the integration of multisensory cues at rest and at

peak limb velocity (i.e., as measured by the velocity of the wrist) was examined in the current

experiment. Participants were required to fling their index finger through a virtual target, while

reaching peak limb velocity when intersecting the virtual target (i.e., without terminal feedback).

At peak limb velocity, participants were presented with two auditory, visual, or audiovisual cues.

After the trial, their task was to discern which side of the virtual target the first sensory cue was

presented. This temporal order judgment task was also completed while the participant remained

stationary. In the resting condition, participants were most accurate in the auditory and

audiovisual conditions relative to the visual condition. However, when comparing within sensory

conditions and between movement conditions, accuracy only significantly decreased in the

audiovisual condition between the rest and flinging conditions. These results from the TOJ task

54

being secondary to the flinging task actually reflect those of Chapter 4.1 where the cues were

relevant for the flinging task. The results thus suggest that engaging in goal-directed action has a

modulatory effect on multisensory integration.

The results of the current experiment suggest that visual information gathered at peak

limb velocity may not be particularly useful when peak limb velocity is the critical moment in

the trajectory, as evidenced by the poor performance in the visual condition. As such, the results

of the current experiment further suggest that visual information collected early on in the

trajectory is critical to control the amplitude of a rapidly moving limb. This finding is also in

agreement with Kennedy et al. (2015), who found that visual information provided early on in

the trajectory yielded comparable end-point variability to when vision was provided throughout

the trajectory.

Participants were most accurate at rest in the auditory and audiovisual conditions. As

such, the results are in agreement with the MLE (Ernst & Bültoff, 2004) whereby audiovisual

stimuli presentation facilitated performance in the TOJ relative to unimodal cues (e.g., vision).

One potential explanation for the advantage of performing TOJs when auditory information was

available comes from the Modality Appropriateness Hypothesis put forward by Welch and

Warren (1980). In this view, the influence of each modality for sensory integration is dependent

on the modality’s appropriateness within the task. Perhaps, participants took advantage of the

temporal precision of the auditory modality (e.g., Celesia, 1976). When visual information was

presented alone, the estimate obtained was less robust than audiovisual estimates due to the

temporal ambiguity of the visual estimate.

When auditory and visual information were presented at peak limb velocity, the results

suggest these cues were not optimally integrated. One potential explanation for this finding may

55

be that participants failed to re-weight bimodal sensory information collected at peak limb

velocity in a statistically optimal fashion. Specifically, the estimate of the limb obtained at peak

limb velocity is particularly noisy due to the speed of the moving limb (e.g., Ghez & Sainburg,

1995). When this information is integrated with the audiovisual information, a more reliable

percept is not yielded due to weight assigned to the visual information. As such, an optimal

strategy would be to lower the weight of the visual information. However, it appears that

participants actually increased the weight on visual information when engaging in the visually

guided flinging movement (e.g., Sober & Sabes, 2005). Thus, performance in the TOJ task may

have decreased only when audiovisual information was presented at peak limb velocity because

of the increased weight on visual information, despite its worse reliability as compared to the

auditory cue during a visuomotor task. As such, it would be of interest to perform this

experiment with an auditory spatial target to see how target modality influences sensory

processing.

Overall, the current study provides further evidence that sensory processing is modulated

during voluntary movement. Specifically, the results suggest that the integration of audiovisual

information yields less robust estimates than if the estimates would be optimally integrated.

Although the inability to optimally re-weight sensory information presented at peak limb

velocity in a statistically optimal fashion may help explain the results, further work is needed to

uncover specifically how the central nervous system processes information during rapid goal-

directed actions. Overall, the results of this study in conjunction with those found in Chapter 4.1,

suggest that when a limb is traveling at high velocities during visuomotor tasks, the integration

of audiovisual information is suboptimal.

56

References

Adhikari, M, B., Goshorn, E. S., Lamichhane, B., & Dhamala, M. (2013). Temporal-order

judgment of audiovisual events involves network activity. Brain Connectivity, 3(5), 536–

545. doi:10.1089/brain.2013.0163

Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal

integration. Current Biology, 14(3), 257–262. doi:10.1016/j.cub.2004.01.029

Bushara, K. O., Grafman, J., & Hallett, M. (2001). Neural correlates of auditory – visual

stimulus onset asynchrony detection. Journal of Neuroscience, 21(1), 300–304.

Carlton, L. G. (1981). Visual information: The control of aiming movements. The Quarterly

Journal of Experimental Psychology, 33(1), 87-93.

Celesia, G. G. (1976). Organization of auditory cortical areas in man. Brain, 99(3), 403–414.

Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on

“sensory-specific” brain regions, neural responses, and judgments. Neuron, 57(1), 11–23.

doi:10.1016/j.neuron.2007.12.013

Ernst, M. O., & Banks. M. S. (2002). Humans integrate visual and haptic information in a

statistically optimal fashion. Nature, 415, 429–434. doi:10.1038/415429a

Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in

Cognitive Sciences, 8(4), 162–169. doi:10.1016/j.tics.2004.02.002

Field, A. P. (2009). Discovering statistics using SPSS. London, England: SAGE Publications

LTD

57

Ghazanfar, A. A, & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in

Cognitive Sciences, 10(6), 278–285. doi:10.1016/j.tics.2006.04.008

Ghez, C., & Sainburg, R. (1995). Proprioceptive control of interjoint coordination. Canadian

Journal of Physiology and Pharmacology, 73(2), 273–284.

Hamlin, A. J. (1895) On the least observable interval between stimuli addressed to disparate

senses and to different organs of the same sense. American Journal of Psychology, 6, 564–

575

Hirsh, I. J., & Sherrick, C. E. J. (1961). Perceived order in different sense modalities. Journal of

Experimental Psychology, 62(5), 423–432. doi:10.1037/h0045283

Howard, I. P., & Templeton, W. B. (1966). Human spatial orientation. Oxford, England: Wiley

and Sons.

Jaekl, P. M., & Harris, L. R. (2007). Auditory-visual temporal integration measured by shifts in

perceived temporal location. Neuroscience Letters, 417(3), 219–24.

doi:10.1016/j.neulet.2007.02.029

Jaśkowski, P., Jaroszyk, F., & Hojan-Jezierska, D. (1990). Temporal-order judgments and

reaction time for stimuli of different modalities. Psychological Research, 52(1), 35–38.

doi:10.1007/BF00867209

Juravle, G., Deubel, H., Tan, H. Z., & Spence, C. (2010). Changes in tactile sensitivity over the

time-course of a goal-directed movement. Behavioural Brain Research, 208(2), 391- 401.

doi:10.1016/j.bbr.2009.12.009

58

Kanabus, M., Szelg, E., & Rojek, E. (2002). Temporal order judgment for auditory and visual

stimuli. Acta Neurobiologiae Experimentalis, 62, 263–270.

Keele, S. W., & Posner, M. I. (1968). Processing of visual feedback in rapid movements. Journal

of Experimental Psychology, 77(1), 155–158. doi:10.1037/h0025754

Kennedy, A., Bhattacharjee, A., Hansen, S., Reid, C., & Tremblay, L. (2015). Online vision as a

function of real-time limb velocity: another case for optimal windows. Journal of Motor

Behavior, 47(6), 465-475. doi:10.1080/00222895.2015.1012579

Levy-Tzedek, S., Hanassy, S., Abboud, S., Maidenbaum, S., & Amedi, A. (2012). Fast, accurate

reaching movements with a visual-to-auditory sensory substitution device. Restorative

Neurology and Neuroscience, 30(4), 313–323. doi:10.3233/RNN-2012-110219

Meredith, M. A., Nemitz, W. J., & Stein, B., E. (1987). Determinants of multisensory integration

neurons. I. temporal factors in superior colliculus. Journal of Neuroscience, 7(10), 3215–

3229.

Proteau, L., Roujoula, A., & Messier, J. (2009). Evidence for continuous processing of visual

information in a manual video-aiming task. Journal of Motor Behavior, 41(3), 219–231.

doi:10.3200/JMBR.41.3.219-231

Recanzone, G. H. (2003). Auditory Influences on Visual Temporal Rate Perception. Journal of

Neurophysiology, 89(2), 1078–1093.

Rosati, G., Oscari, F., Spagnol, S., Avanzini, F., & Masiero, S. (2012). Effect of task-related

continuous auditory feedback during learning of tracking motion exercises. Journal of

Neuroengineering and Rehabilitation, 9(79), 2-13. doi:10.1186/1743-0003-9-79

59

Sedda, A., Monaco, S., Bottini, G., & Goodale, M. A. (2011). Integration of visual and auditory

information for hand actions: Preliminary evidence for the contribution of natural sounds to

grasping. Experimental Brain Research, 209(3), 365–74. doi:10.1007/s00221-011-2559-5

Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408,

788. doi:10.1038/35048669

Spence, Charles, Shore, I., D., Klein, M, R. (2001). Multisensory prior entry. Journal of

Experimental Psychology: General, 130(4), 799–832.

Tremblay, L., & de Grosbois, J. (2015). Why encode limb and body displacements in the

velocity domain? Neurophysiological and behavioral evidence. In T. Heinen (Ed.),

Advances in visual perception research (pp. 279-292). Hauppauge, NY, USA: Nova

Science Publishers.

Tremblay, L., & Nguyen, T. (2010). Real-time decreased sensitivity to an audio-visual illusion

during goal-directed reaching. PLoS One, 5(1), e8952. doi:10.1371/journal.pone.0008952

Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory

discrepancy. Psychological Bulletin, 88(3), 638-667

Witten, I. B., & Knudsen, E. I. (2005). Why seeing is believing: Merging auditory and visual

worlds. Neuron, 48(3), 489–496. doi:10.1016/j.neuron.2005.10.020

Woodworth, R. S. (1899). The accuracy of voluntary movement. Psychological Review, 3, 1-

119.

60

Zampini, M., Shore, D. I., & Spence, C. (2003). Audiovisual temporal order judgments.

Experimental Brain Research, 152(2), 198–210. doi:10.1007/s00221-003-1536-z

Zelaznik, H. N., Hawkins, B., & Kisselburgh, L. (1983). Rapid visual feedback processing in

single-aiming movements. Journal of Motor Behavior, 15(3), 217–236.

doi:10.1080/00222895.1983.10735298

61

Chapter 5 – General Discussion 5

Experiment 1 Summary 5.1

The use of unisensory and multisensory feedback was the focus of Experiment 1 (i.e.,

feedback experiment). In this experiment, participants were tasked with flinging their right limb

through the centre of a virtual target. Their goal was to reach peak limb velocity (i.e., as

measured by index finger velocity) when the finger intersected a virtual target. At peak limb

velocity, participants were presented with auditory, visual, or audiovisual feedback via a piezo-

LED device affixed to the right index finger. It was hypothesized that performance would be best

in the multisensory conditions as evidenced by reductions in constant and variable errors as well

as resultant displacement. Contrary to this hypothesis, it was found that auditory feedback alone

reduced variability in peak velocity occurrence across all movement axes (see Figures 1 & 5),

relative to when no feedback was provided. No significant effects were observed for the visual or

audiovisual feedback conditions. The results from this experiment suggest that when the critical

moment in the trajectory is peak limb velocity (i.e., as opposed to movement end), the use of

visual feedback may be limited. In addition, performance in the audiovisual condition suggested

that participants failed to re-weight unreliable sensory information in a statistically optimal

fashion, as evidenced by the significant differences between MLE predicted and observed

variable errors.

Experiment 2 Summary 5.2

Experiment 2 (i.e., TOJ experiment) further assessed whether audiovisual cues were

optimally integrated, or not, at peak limb velocity when peak limb velocity is the critical portion

of the trajectory. In this experiment, participants were again required to fling their limb through

62

the centre of a virtual target. At peak limb velocity (i.e., as measured by wrist velocity)

participants were presented with two auditory, visual, or audiovisual cues via two piezo-LED

devices affixed on either side of the virtual target. Participants had to report which side of the

virtual target was presented first in a temporal order judgment (TOJ) task. Participants also

completed the TOJ task while remaining stationary. It was found that at rest, participants were

more accurate in the auditory and audiovisual condition relative to the visual condition. When

the sensory cues were presented at peak limb velocity, it was found that participant’s TOJ

accuracy was significantly worse in the audiovisual condition compared to at rest. The results of

this experiment suggested that participants prioritized the processing of the least reliable cue

(i.e., vision) when performing the flinging movement towards a visual target.

General Discussion Overview 5.3

Sensory processing at peak limb velocity was assessed across two experiments in the

current thesis. The results reported here have implications for sensory processing throughout an

ongoing movement and the integration of audiovisual information, specifically at peak limb

velocity when this kinematic marker was the critical moment in the trajectory. The discussion

below pertains to sensory processing during goal-directed action with emphasis on how sensory

processing differs when the critical moment in the trajectory is peak limb velocity, not

movement end. In addition, how sensory information is integrated at peak limb velocity will be

discussed with emphasis on the implications of the current results on the use of sensory

information during goal-directed action.

63

5.3.1 Critical Temporal Window For Visual Processing

The current thesis sought to determine how sensory information is processed and

integrated during movements where the critical moment for feedback utilization is peak limb

velocity, as opposed to movement end. Although the empirical focus on movement end is

certainly warranted to understand the fundamental aspects of common upper-limb movements

(i.e., reach-to-grasp and pointing), greater emphasis could be placed on movements with

alternative critical moments. By investigating how sensory information is utilized specifically at

peak limb velocity, the results of the current thesis have implications for feedback processes

occurring throughout an ongoing movement.

Visual information provides important feedback at the end of a movement. Although

previous studies have shown the utility of visual feedback at movement end (e.g., Zelaznik et al.,

1983; Chua & Elliott, 1993; Ma-Wyatt & McKee, 2007) the current thesis provides evidence that

visual feedback at peak limb velocity does little to facilitate precision when the limb is traveling

at peak velocity. This finding may be due to differing feedback processes between the critical

moments in the trajectories (i.e., pointing versus flinging). In a study conducted by Proteau,

Roujoula and Messier (2009), participants performed a manual-aiming task where the position of

the limb was represented by a cursor on screen. During the trajectory, the position of the cursor

was perturbed (i.e., lateral cursor jump soon after movement start). Proteau et al. (2009) reported

that participants were able to perform fast and accurate corrections to the cursor jump and

complete the movement accurately. This finding led the authors to conclude that when

movement end is the critical moment in the trajectory, visual information is continuously

monitored. However, other work has shown that visual processing is critical during specific

phases of a movement.

64

Visual information gathered at specific limb velocities may be critical when executing

goal-directed actions. In a study conducted by Kennedy et al. (2015), participants performed a 30

cm reaching movement within a 270-350 ms movement time bandwidth. Visual information was

provided to participants at certain limb velocities. That is, visual information was provided early

(i.e., limb velocity of 0.8-1.4 m/s, before peak limb velocity), in the middle (i.e., limb velocity

above 1.4 m/s, up to peak limb velocity), or late (i.e., limb velocity of 1.4-0.8 m/s, after peak

limb velocity) in the trajectory. It was found that endpoint precision and time after peak limb

velocity were comparable to full vision when visual information was provided during the early

phase of the trajectory. In contrast, the middle window did exhibit comparable endpoint

precision to full vision, but at a significant temporal cost. This finding in conjunction with those

reported by Proteau et al. (2009) suggest that when the critical moment in the trajectory is

movement end, visual processing within certain phases of the movement are more salient than

others. The current thesis was able to extend this knowledge by manipulating the accuracy

requirements of the task (i.e., make peak limb velocity the critical moment instead of movement

end). The results from the current thesis highlight the importance of visual feedback early in the

trajectory, when controlling movement amplitude. This is evidenced by the lack of facilitation of

the visual feedback in the feedback experiment (see Figure 5), and the poor performance in the

visual condition of the TOJ experiment (see Figure 8). Thus, it appears that visual information at

peak limb velocity is not very salient during upper-limb movements, but rather visual

information gathered early on in the trajectory is critical for the successful execution of goal-

directed actions.

65

5.3.2 Multisensory Integration At Peak Limb Velocity

The experiments conducted in the current thesis assessed whether audiovisual integration

is optimal, specifically when the limb is travelling at peak velocity. This pursuit is warranted

given previous work reporting that multisensory integration was modulated by limb velocity

(e.g., Tremblay & Nguyen, 2010). In addition, the perception of unisensory stimuli has been

shown to decline during a movement (e.g., Juravle et al., 2010). The results of the current thesis

suggest that during action, audiovisual integration may not be optimal. The discussion below

pertains to why multisensory integration may not be optimal at peak limb velocity. As well,

alternative sensory processing mechanisms the central nervous system may be employing at peak

limb velocity will be discussed.

At peak limb velocity, sensory information may not be weighted in a statistically optimal

fashion. A potential explanation for the lack of audiovisual facilitation found in the experiments

is that an unreliable cue, and its weight in the integrated percept, reduced the overall reliability of

the perceptual estimate. Based on the results, it would appear that the least reliable stimulus was

the visual stimulus. In the feedback experiment for example, when the LED was illuminated in

both the visual and audiovisual conditions, it created a streak of light due to the high velocity of

the moving limb. This streak of light would most certainly reduce the quality of the visual

estimate, given that the participant could associate peak limb velocity with any point along this

streak. According to the spatial nature of the task, it was predicted that participants would

heavily weight the visual cue (e.g., Rock & Victor, 1964). However, the quality of this estimate

is limited at peak limb velocity. As such, an optimal strategy would have been to re-weight the

visual estimate (i.e., reduce its overall weight during integration) in the audiovisual condition.

However, the current results do not suggest that participants were able to do so, as evidenced by

66

the significant difference between observed and MLE predicted performance in the audiovisual

condition.

Previous work on sensory weighting has shown the weights assigned to each modality are

flexible. In a study conducted by Fetsch, Pouget, DeAngelis, and Angelaki (2011), monkeys

performed a heading discrimination task (i.e., judging the direction of linear self-motion)

utilizing visual and vestibular cues. On a trial-by-trial basis, the reliability of the visual and

vestibular cues was manipulated such that one cue would become more reliable than the other.

Of particular interest was neuronal activity in the dorsal medial superior temporal area (MSTd),

an area of the cortex believed to encode heading perception (i.e., integrating visual and vestibular

cues). Fetsch et al. (2011) reported that when the reliability of either cue was manipulated, the

monkeys were able to adapt and perform the task successfully, while corresponding fluctuations

in the firing rate of neurons within the MSTd was observed. This pattern of results was

interpreted as evidence that the monkeys re-weighted the sensory information on a trial-by-trial

basis based on the reliability of the cue.

A potential explanation for the lack of facilitation found for the visual and audiovisual

conditions is that participants fail to re-weight this cue (i.e., reduce its overall weight during

integration), when it is presented specifically at peak limb velocity. It is important to note that in

the study conducted by Fetsch et al. (2011), the animal was a passive observer of the sensory

events and did not actively engage in any sort of action. In the current thesis, participants were

required to integrate auditory and visual cues while the limb is travelling at a high velocity

during a goal-directed action. As would be expected given the results of Fetsch et al. (2011),

participants should have become aware of the visual estimate’s unreliability and adjusted its

weighted within the first few trials of the block. In order to further determine how participant’s

67

performance changed over time in the feedback and TOJ experiment, an additional block

analysis was conducted (see Appendices A & B). In this analysis, trials were divided into thirds

(i.e., first third, middle third, and final third) with the interest here being whether participants

improved their performance over time. However, when analyzing the first third, middle third,

and final third trials of the block for the feedback experiment (see Appendix A) no significant

differences in variable error were found (i.e., all ps > .12). In addition, when conducting the same

analysis for the TOJ experiment, participant’s accuracy in the TOJ task again did not improve

over time (see Appendix B). This analysis suggests that the participant’s strategy, and weights

assigned to the sensory cues, did not change throughout the task. As such, it appears that when

unreliable sensory information is presented at peak limb velocity, the cues are not re-weighted in

a statistically optimal fashion.

An alternative explanation for participant’s performance across experiments may be

related to the reliance of visual information during reaches performed to visual targets. Sober and

Sabes (2005) contend that the sensory modality congruent with the target modality will be relied

upon during action. In both experiments, participants were presented with a visual target. Based

on Sober and Sabes (2005), it would be expected that visual processing would be prioritized

because it is congruent with the target modality. Within both experiments however, visual

information did not provide very salient information in comparison to the auditory cue (i.e., an

incongruent modality). As such, no facilitation in performance was found when visual

information was available across both experiments because participants would have upregulated

the processing of the lesser reliable sensory cue. Conducting the same study as described in

Chapter 4.1 with an auditory target could test this hypothesis. In this follow up study,

participants would perform the flinging movement towards a localizable auditory target. If

sensory information congruent with the target modality were upregulated, it would be expected

68

that performance would be best in the audiovisual conditions, due to the upregulation of the most

reliable modality within the task.

Overall, more work is required to understand how the central nervous system processes

information specifically during instances of rapid goal-directed action. Although the results of

this thesis suggest that the weights assigned to auditory and visual information may be

suboptimal, it does not provide a definitive explanation as to why. As such, future work could

focus on how sensory information is processed within movement phases prior to peak limb

velocity. This approach could yield valuable insight into the acute moments of a movement that

are critical to its outcome. In addition, it is also important to study multisensory integration

during goal directed action in general, as this focus has gone relatively overlooked in the

literature (see Witten & Knudsen, 2005 for a review).

5.3.3 The Advantage of Auditory Feedback

The results of both experiments emphasize that auditory information is particularly useful

when peak limb velocity is the critical moment in the trajectory. Therefore, providing augmented

auditory feedback may be beneficial when learning certain motor skills. One such skill where

peak limb velocity is the critical moment in the trajectory is overarm throwing. Empirical work

on overarm throwing has shown that when a series of throws are performed, ball speed tends to

vary from throw to throw as a result of variability in ball release. Given that limb velocity peaks

at a specific moment in the trajectory, ball release either before or after this point negatively

affects ball speed (Timmann, Citron, Watts, & Hore, 2001). Because the feedback experiment

showed that auditory feedback reduced variability in displacement of the limb when presented at

peak velocity, auditory feedback may be a particularly useful to convey movement related

feedback during overarm throws.

69

Auditory feedback may be particularly useful when learning ball release point in the

novice throwing population. In a study directly comparing release points between skilled and

unskilled throwers, Jegede, Watts, Stitt, and Hore (2005) found strong correlations between

release timing and forearm velocity for skilled, but not unskilled throwers. As a result of

variability in release, the unskilled throwers achieved a lower ball speed when compared to the

skilled throwers, while skilled throwers were timing their release close to peak limb velocity

(Jegede et al., 2005). The results of that study suggest that the optimal release window for

overarm throwing centres on peak limb velocity and that novice throwers fail to release the ball

reliably at this kinematic marker. As shown in Chapter 4.1, auditory feedback reduced the

variability in peak limb velocity occurrence across all movement axes as well as resultant

displacement. Therefore, auditory feedback provided at peak limb velocity may be useful for

novice throwers when learning ball release timing during a throwing task.

5.3.4 Limitations of the Current Thesis

A limitation of the feedback experiment was that participants always completed the no

feedback condition before the counterbalanced auditory, visual, and audiovisual blocks. This

block order certainly warrants the criticism that participants were more accurate in the feedback

trials relative to no feedback, simply due to more practice with the task. As such, in future

experiments involving the same flinging task, the no feedback condition should be

counterbalanced with the feedback conditions. Another limitation in both experiments is the

location of the virtual target. As noted under General Methods, each participant estimated the

location of the virtual target. As such, the observed position of the target may have varied

between participants. To correct for any potential variation, future experiments should calculate

the position of the virtual target by measuring the distance between the participant’s eyes and

70

the reflective surface as well as the LED to the reflective surface. This position should then be

used when calculating dependent variables such as constant and variable error.

5.3.5 Concluding Remarks

Processing sensory information accurately is critical to successfully execute actions.

Although sensory information is typically combined in a statistically optimal way, the current

thesis suggests that weights assigned to auditory and visual information when the limb is

travelling at peak velocity is suboptimal. These findings may be related to the inability of

participants to reweight sensory information presented at peak limb velocity in a statistically

optimal fashion. As such, future work is required to understand how sensory information is

processed specifically when the limb is traveling at high velocities, such as when performing

overarm throws or hitting a golf ball. Pursuing this research could yield valuable applied and

fudamental knowledge. For example, understanding how non-visual sensory information is

processed specifically during action could lead to advances in augmented feedback provided

during motor learning (e.g., Effenberg, 2005). Finally, this focus may also lead to further

knowledge regarding sensory processing mechanisms utilized during movement planning and

execution.

71

Chapter 6 – References 6

Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal

integration. Current Biology, 14(3), 257–262. doi:10.1016/j.cub.2004.01.029

Bremmer, F., Schlack, A., Shah, N. J., Zafiris, O., Kubischik, M., Hoffmann, K.-P., Ziles, K., &

Fink, R., G. (2001). Polymodal motion processing in posterior parietal and premotor

cortex: A human fMRI study strongly implies equivalencies between humans and

monkeys. Neuron, 29, 287–296.

Bushara, K. O., Grafman, J., & Hallett, M. (2001). Neural correlates of auditory – visual

stimulus onset asynchrony detection. Journal of Neuroscience, 21(1), 300–304.

Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S.

(1999). Response amplification in sensory‐specific cortices during crossmodal binding.

Neuroreport, 10(12), 2619-2623.

Carlton, L. G. (1981). Processing visual feedback information for movement control. Journal of

Experimental Psychology: Human Perception and Performance, 7(5), 1019–1030.

Carlton, L. G. (1992). On the specificity of learning and the role of visual information for

movement control. In L. Proteau & D. Elliott (Eds.), Vision and motor control. Advances in

psychology, (pp. 3-31). New York, NY, USA: Elsevier Science & Technology.

Celesia, G., G. (1976). Organization of the auditory cortical areas of man. Brain, 99(3), 403-414.

72

Chapman, C., Bushnell, M., Miron, D., Duncan, G., & Lund, J. (1987). Sensory perception

during movement in man. Experimental Brain Research, 68(3), 516-524.

Chua, R., & Elliott, D. (1993). Visual regulation of manual aiming. Human Movement Science,

12(4), 365–401. doi:10.1016/0167-9457(93)90026-L

Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on

“sensory-specific” brain regions, neural responses, and judgments. Neuron, 57(1), 11–23.

doi:10.1016/j.neuron.2007.12.013

Effenberg, A. O. (2005). Movement sonification: Effects on perception and action. IEEE

Multimedia, 12(2), 53–59. doi:10.1109/MMUL.2005.31

Elliott, D., Carson, R. G., Goodman, D., & Chua, R. (1991). Discrete vs. continuous visual

control of manual aiming. Human Movement Science, 10, 393–418.

Elliott, D., Hansen, S., & Khan, M. A. (2010). The optimization of speed, accuracy, and energy

in goal-directed aiming. In D. Elliott & M. Khan (Eds.), Vision and goal-directed

movement: Neurobehavioral perspectives (pp. 21-38). Champaign, IL, USA: Human

Kinetics

Ernst, M. O., Banks. M. S. (2002). Humans integrate visual and haptic information in a

statistically optimal fashion. Nature, 415, 429–434.

Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in

Cognitive Sciences, 8(4), 162–169. doi:10.1016/j.tics.2004.02.002

73

Fetsch, C. R., Pouget, A., DeAngelis, G. C., & Angelaki, D. E. (2012). Neural correlates of

reliability-based cue weighting during multisensory integration. Nature Neuroscience,

15(1), 146–54. doi:10.1038/nn.2983

Ghazanfar, A. a, & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in

Cognitive Sciences, 10(6), 278–85. doi:10.1016/j.tics.2006.04.008

Hershenson, M. (1962). Reaction times as a measure of intersensory facilitation. Journal of

Experimental Psychology, 63(3), 289–293.

Hirsh, I. J., & Sherrick, C. E. J. (1961). Perceived order in different sense modalities. Journal of

Experimental Psychology, 62(5), 423–432. doi:10.1037/h0045283

Kanabus, M., Szelg, E., & Rojek, E. (2002). Temporal order judgment for auditory and visual

stimuli. Acta Neurobiologiae Experimentalis, 62, 263–270.

Jaekl, P. M., & Harris, L. R. (2007). Auditory-visual temporal integration measured by shifts in

perceived temporal location. Neuroscience Letters, 417(3), 219–24.

doi:10.1016/j.neulet.2007.02.029

Jaśkowski, P., Jaroszyk, F., & Hojan-Jezierska, D. (1990). Temporal-order judgments and

reaction time for stimuli of different modalities. Psychological Research, 52(1), 35–38.

doi:10.1007/BF00867209

Juravle, G., Deubel, H., Tan, H. Z., & Spence, C. (2010). Changes in tactile sensitivity over the

time-course of a goal-directed movement. Behavioural Brain Research, 208 (2), 391- 401.

Keele, S. W., & Posner, M. I. (1968). Processing of visual feedback in rapid movements. Journal

of Experimental Psychology, 77(1), 155–158.

74

Kennedy, A., Bhattacharjee, A., Hansen, S., Reid, C., & Tremblay, L. (2015). Online vision as a

function of real-time limb velocity: another case for optimal windows. Journal of Motor

Behavior, 47(6), 465-475. doi:10.1080/00222895.2015.1012579

Lakatos, P., O’Connell, M. N., Barczak, A., Mills, A., Javitt, D. C., & Schroeder, C. E. (2009).

The leading sense: Supramodal control of neurophysiological context by attention. Neuron,

64(3), 419–30. doi:10.1016/j.neuron.2009.10.014

Lesevre, N. (1982). Chronotopographical analysis of the human evoked potential in relation to

the visual field (data from normal individuals and hemianopic patients). Annals of the New

York Academy Of Sciences, 388, 156–182.

Levy-Tzedek, S., Hanassy, S., Abboud, S., Maidenbaum, S., & Amedi, A. (2012). Fast, accurate

reaching movements with a visual-to-auditory sensory substitution device. Restorative

Neurology and Neuroscience, 30(4), 313–23. doi:10.3233/RNN-2012-110219

Ma-Wyatt, A., & McKee, S. P. (2007). Visual information throughout a reach determines

endpoint precision. Experimental Brain Research, 179(1), 55–64. doi:10.1007/s00221-006-

0767-1

McGurk, H., MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.

Meredith, M. A., Nemitz, W. J., & Stein, B. E. (1987). Determinants of multisensory integration

neurons. I. temporal factors in superior colliculus, 7(10), 3215–3229

Proteau, L., Roujoula, A., & Messier, J. (2009). Evidence for continuous processing of visual

information in a manual video-aiming task. Journal of Motor Behavior, 41(3), 219–31.

doi:10.3200/JMBR.41.3.219-231

75

Rock, I., Victor, J. (1964). Vision and touch  : An experimentally created conflict between the

two senses. Science, 143(3606), 594–596.

Russo, G. S., & Bruce, C. J. (1994). Frontal eye field activity preceding aurally guided saccades.

Journal of Neurophysiology, 71(3), 1250–1253.

Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408,

788.

Shams, L., Kamitani, Y., & Shimojo, S. (2002). Visual illusion induced by sound. Cognitive

Brain Research, 14(1), 147–152.

Sober, S. J., & Sabes, P. N. (2005). Flexible strategies for sensory integration during motor

planning. Nature Neuroscience, 8(4), 490–497. doi:10.1038/nn1427

Spence, Charles, Shore, I., D., & Klein, M, R. (2001). Multisensory prior entry. Journal of

Experimental Psychology: General, 130(4), 799–832.

Stein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade, L. (1989). Behavioral indices of

multisensory integration: Orientation to visual cues is affected by auditory stimuli. Journal

of Cognitive Neuroscience, 1(1), 12–24. doi:10.1162/jocn.1989.1.1.12

Tremblay, L., & Nguyen, T. (2010). Real-time decreased sensitivity to an audio-visual illusion

during goal-directed reaching. PloS One, 5(1), e8952. doi:10.1371/journal.pone.0008952

van Atteveldt, N., Murray, M. M., Thut, G., & Schroeder, C. E. (2014). Multisensory integration:

Flexible use of general operations. Neuron, 81(6), 1240–53.

doi:10.1016/j.neuron.2014.02.044

76

van Beers, R. J., Sittig, A. C., & Gon, J. J. D. Van Der. (1999). Integration of proprioceptive and

visual position-information: An experimentally supported model. Journal of

Neurophysiology, 81, 1355–1364.

van Beers, R. J., Wolpert, D. M., & Haggard, P. (2002). When feeling is more important than

seeing in sensorimotor adaptation. Current Biology, 12(10), 834–837.

Warren, J. E., Wise, R. J. S., & Warren, J. D. (2005). Sounds do-able: Auditory-motor

transformations and the posterior temporal plane. Trends in Neurosciences, 28(12), 636–

643. doi:10.1016/j.tins.2005.09.010

Woodworth, R. S. (1899). The accuracy of voluntary movement. Psychological Review, 3, 1–

119.

Zampini, M., Shore, D. I., & Spence, C. (2003). Audiovisual temporal order judgments.

Experimental Brain Research, 152(2), 198–210. doi:10.1007/s00221-003-1536-z

Zelaznik, H. N., Hawkins, B., & Kisselburgh, L. (1983). Rapid visual feedback processing in

single-aiming movements. Journal of Motor Behavior, 15(3), 217–236.

doi:10.1080/00222895.1983.10735298

77

Appendices 7

78

Appendix A – Block Analysis of Experiment 1 7.1

The additional analysis reported below tested how participants utilized feedback

throughout the experimental block. As such, the first third (i.e. trials 1-10), middle third (i.e.,

trials 11-20), and final third (i.e., trials 21-30) trials of the block were analyzed using repeated

measures ANOVA, across the movement axes. The analysis of variable error revealed that

participants did not significantly improve their performance across the block of trials in any

feedback condition (i.e., all p ≥ .27). Specifically, performance along the primary, F (2, 24) =

1.4, p = .27, 𝜂!! = .16, secondary, F (2, 24) = 1.41, p = .27, 𝜂!! = .11, and tertiary, F (2, 24) = 1.2,

p = .32,  𝜂!! = .09, axes when visual feedback was provided showed participants did not

significantly reduce their variability over time. This pattern of results was mirrored along the

primary, F (2, 24) = .47, p = .63, 𝜂!! = .04, secondary, F (2, 24) = .65, p = .53, 𝜂!! = .05, and

tertiary, F (2, 24) = .65, p = .53, 𝜂!! = .04, axes when audiovisual feedback was provided. Lastly,

no significant reduction in variability of displacement traveled when peak velocity was reached

when visual, F (2, 24) = 1.75, p = .12, 𝜂!! = .01, or audiovisual, F (2, 24) = .38, p = .69,  𝜂!! = .03,

feedback was provided. The means and standard deviations associated with these results are

shown in Table A.

79

Table A

Variable Error (VE) Across Feedback Block for the Three Movement Axes (i.e.,

Y-X-Z) and Resultant Displacement (R).

Proportion of Experimental Block

VE-X (mm)

VE-Z (mm)

VE-R (mm)

Sensory Condition

First Third M (SD)

Middle Third M (SD)

Final Third M (SD)

Total M (SD)

Auditory VE-Y VE-X VE-Z VE-R

28.6 (14) 12.2 (3.6) 20.9 (11.5) 32.4 (17.6)

27.4 (8.1) 9.1 (2.8) 19.3 (4.5) 30.8 (8.8)

25.6 (11.4) 9.4 (4.2) 19.9 (5.6) 29.1 (10.2)

27.7 (11.5) 10.3 (3.8) 20.1 (8.4) 30.8 (14.8)

Visual VE-Y VE-X VE-Z VE-R

34.2 (13.4) 12.2 (4.3) 24.4 (5.4) 42.2 (13.1)

39.2 (23.1) 11.3 (5.5) 26.2 (21.7) 45.7 (31.1)

37.9 (28.8) 12.6 (11) 24.9 (20.4) 41.7 (36.8)

37.1 (18.7) 12.1 (8.2) 25.2 (16.3) 43.2 (25.5)

Audiovisual VE-Y VE-X VE-Z VE-R

36.7 (14.4) 15.7 (9.6) 22.8 (9.7) 43.3 (18.3)

37.3 (26.5) 13.2 (5.9) 22.2 (13.3) 40.8 (29.7)

40.8 (16.1) 13.6 (9) 21.2 (9.7) 39.2 (20.1)

38.3 (14.4) 14.1 (7.6) 22.1 (7.8) 41.1 (17.2)

80

Appendix B – Block Analysis of Experiment 2 7.2

Of particular interest in Experiment 2 was the optimal integration of audiovisual cues

specifically when the limb was travelling at peak limb velocity (i.e., as measured by wrist

velocity). In order to determine whether accuracy in this task changed throughout the

experimental block, a repeated measures ANOVA was conducted contrasting across the block of

trials in all sensory conditions (i.e., auditory, visual, and audiovisual). Specifically, trials were

grouped into the first third (i.e., trials 1-20), middle third (i.e., trials 21-40), and final third (i.e.,

trials 41-60). When analyzing response accuracy across the auditory block, no significant

differences were found when judgments were made at rest, F (2, 24) = 2.6, p = .10, 𝜂!! = .12, or

following presentation at peak limb velocity, F (2, 24) = 1.4, p = .28, 𝜂!! = .10. In the visual

condition, no significant differences were found across the block at rest, F (2, 24) = .58, p = .57,

𝜂!! = .05, or at peak limb velocity, F (2, 24) = .67, p = .52, 𝜂!! = .05. Lastly, in the audiovisual

condition, no significant differences were found when at rest, F (2, 24) = .16, p = .85, 𝜂!! = .01,

or at peak limb velocity, F (2, 24) = 2.7, p = .08, 𝜂!! = .18. Overall, this pattern of results

suggests that performance in this task was not influenced by time or presentation condition.

81

Table B

Means and Between-subject Standard Deviations for Accuracy in the Temporal

Order Judgment Task Performed Following Stimuli Presentation At Rest and At

Peak Limb Velocity Across the First Third, Middle, and Final Third Number of

Trials.

Number of Correct Responses

Condition First Third

M (SD) Middle Third

M (SD) Final Third

M (SD) Total M (SD)

Auditory At Rest At PLV

14.6 (3.2) 12.4 (2.9)

14.4 (2.6) 13.4 (2.7)

12.6 (4.5) 12.9 (3.2)

41.6 (8.7) 38.7 (8.1)

Visual At Rest At PLV

11.6 (2.7) 10.8 (2.6)

11.3 (2.9) 10.8 (2.2)

10.8 (2.6) 11.8 (3.4)

33.7 (5.9) 33.4 (5.4)

Audiovisual At Rest At PLV

13.1 (2.5) 12.2 (2.8)

12.7 (3.7) 11.4 (2.2)

13.3 (3.1) 10 (3.7)

39.1 (7.3) 33.6 (6.1)