Available online at www.sciencedirect.com

www.elsevier.com/locate/displa

Displays 29 (2008) 152–158

Adapting to virtual environments: Visual-motor skillacquisition versus perceptual recalibration q

Robert. B. Welch a,*, Anthony. C. Sampanes b

a NASA Ames Research Center, Mail Stop 262-2, Moffett Field, CA 94035, USAb University of California, Santa Cruz Department of Psychology, Santa Cruz, CA 95064, USA

Available online 1 October 2007

Abstract

Virtual environment (VE) technology exposes users to a variety of intersensory and sensory-motor discordances to which they mustadapt for optimal performance. Our research has distinguished two types of adaptation: Visual-motor skill acquisition and perceptual

recalibration. The first involves learning a new way to coordinate hand and eye, while the second is an automatic, restricted processof perceptual learning. We conclude that an understanding of the controlling conditions and defining characteristics of these two adap-tive mechanisms allows one to predict which is the more likely to occur with a given VE and how best to train its users.� 2007 Elsevier B.V. All rights reserved.

Keywords: Virtual environments; Adaptation; Intersensory conflicts; Sensory-motor conflicts

1. Problems with virtual environments and the response to

users’ complaints

Despite remarkable advances over the last decade, vir-tual environments (VEs) continue to suffer from numerousdefects, including poor lighting, unrealistic graphics, asmall visual field of view (FOV), heavy headgear, and var-ious inter-sensory and sensory-motor conflicts (e.g., [1,2]),all of which produce a diverse array of perceptual, behav-ioral, and physical complaints (see Table 1). Of the latter,perhaps the most problematic is the motion sickness-likesyndrome known as ‘‘cybersickness,’’ whose symptomsinclude pallor, sweating, fatigue, drowsiness, and nausea(e.g., [3]). Clearly, serious complaints such as these arelikely to interfere with or prematurely terminate use ofthe device in question, discourage repeat encounters, and,in general, produce a chilling effect on the diffusion ofVE technology (e.g.,[4]).

0141-9382/$ - see front matter � 2007 Elsevier B.V. All rights reserved.

doi:10.1016/j.displa.2007.09.013

q This research was supported by a Grant from the National Aeronau-tics and Space Administration (Project UPN 131-20-30-00).

* Corresponding author.E-mail addresses: Robert.B.Welch@nasa.gov (Robert. B. Welch),

acsampanes@yahoo.com (Anthony. C. Sampanes).

Cybersickness and many other untoward perceptual andbehavioral effects of VE systems are due, at least in part, toconflicts between sensory modalities (e.g., vision and thevestibular sense [5]) or between sensory and motor systems.There are two ways in which the VE community hasresponded to these problems These are to (1) build betterdevices and (2) provide users with systematic trainingand/or strategies for reducing or circumventing the defi-ciencies of the VE system. Although not mutually exclu-sive, the latter approach has been used relativelyinfrequently, based on the strong faith among engineersthat improved technology will eventually prevail. However,the second strategy has the distinct advantage that it can beimplemented immediately, that is, without having to waitfor the necessary engineering improvements to be designedand implemented. Unfortunately, such problems as dis-crepancies between felt and seen limb position, delays ofvisual feedback, distortions of perceived depth, and areduced FOV are unlikely to be completely conqueredany time soon (e.g., [2]). However, these are precisely thekinds of sensory and sensory-motor rearrangements thatare likely to prove amenable to user-training procedures,in particular, those that promote the process commonlyreferred to as adaptation.

Table 1Sensory/perceptual, behavioral, and physical complaints of VE usersa

Perceptual problems

• Momentary reduction in binocular acuity• Misperception of depth• Changes in dark accommodative focus• ‘‘Delayed flashbacks’’ (e.g., illusory experiences of climbing,

turning, and inversion) several hours after an aircraft simulatortraining session

Disruptive behavioral effects of VEs

• Disrupted perceptual-motor (e.g., hand–eye) coordination duringand after using the device

• Locomotory and postural instability• Degraded task performance

Physical/physiological complaints reported by VE users

• Eye strain, or ‘‘asthenopia,’’which may be symptomatic ofunderlying distress of or conflict between oculomotor subsystems

• Headaches• Cardiovascular, respiratory, or biochemical changes• Cybersickness (e.g., pallor, sweating, fatigue, drowsiness, and

nausea)

a Adapted from Table 31.1 of 3 with the permission of ErlbaumAssociates.

Robert. B. Welch, Anthony. C. Sampanes / Displays 29 (2008) 152–158 153

It has long been known that human beings are verygood at adapting behaviorally and, to a lesser extent, per-ceptually to rearranged sensory environments, usually pro-duced by optical means (see [6] and, more recently, [7] forreviews of this literature). In the typical study, observersare measured on their hand–eye coordination before, dur-ing, and after active exposure to the visual rearrangement.The classic example is ‘‘prism adaptation’’ in which duringexposure to a 10–15 deg prismatic displacement of thevisual field, observers rapidly correct their initial reachingerrors (the ‘‘reduction of effect’’), only to err in the oppositedirection when the prism is removed (the ‘‘negative afteref-fect’’). These, then, represent the two primary measures ofadaptation. Other sensory rearrangements to whichobservers have adapted include displacement and right-leftreversal of auditory space (e.g., [8,9]), optical curvature ofvisual contours [10], optical tilt (e.g., [11]), optical right-leftor up-down reversal (e.g., [12]), and 180-deg optical rota-tion (e.g., [13]). Given the robust adaptability of observersto this wide range of sensory rearrangements, it is reason-able to assume that they will be equally capable of adaptingto the inadvertent rearrangements found in VEs. It also fol-lows that the variables shown to influence adaptation torearrangements in general (e.g., active exposure, error-cor-rective feedback) will apply as well to VE adaptation andtraining.

2. Perceptual calibration versus visual-motor skill acquisition

Clower and Boussaoud [14] have proposed that adapta-tion to intersensory and sensory-motor rearrangementscomes in two qualitatively different flavors known as per-ceptual recalibration and visual-motor skill acquisition.[Redding and Wallace [15] have made a very similar

distinction, although with different terminology]. Percep-tual recalibration exists when one sensory system (typi-cally, proprioception) has been calibrated in terms ofanother sensory system (typically, vision). This is consid-ered to be an automatic, non-volitional process of percep-tual learning. The prototypical example is prismadaptation, in which it has been shown that both the reduc-tion of effect and the negative aftereffect occur because theobserver’s hand has come to feel located where it was seenthrough the prism (e.g., [16]). Little or no inter-manualtransfer is obtained with prism adaptation and re-adapta-tion to normal vision occurs extremely rapidly when visualfeedback is provided (e.g., [17, pp. 86–87]).

It is argued here that perceptual recalibration will be thepredominant adaptive response to sensory rearrangementsthat are relatively small and are limited to a single spatialdimension. An example would be a 10-deg displacementbetween seen and felt limb position in the horizontaldimension. The end product of this type of adaptation isalso understood to be limited. For example, with hand–eye coordination, the proprioceptive recalibration isassumed to be restricted to the exposed hand. Therefore,observers whose interaction with the rearranged visualenvironment is limited to one hand may expect to experi-ence little or no inter-manual transfer when tested withthe other hand. Furthermore, this proprioceptive recalibra-tion is likely to result in a substantial post-exposure nega-tive aftereffect, since, in the absence of visual feedback,observers instructed to align the felt hand with a visual tar-get will respond by placing the hand off to one side.Clearly, all of these are well-documented characteristicsof prism adaptation.

In sharp contrast to perceptual recalibration, visual-motor skill acquisition can be expected when rearrange-ments are large and/or involve two or more spatial dimensions.A hypothetical example from the area of tele-operation iswhen a rightward turn of a joystick by an astronaut in theInternational Space Station causes a robotic arm, viewedon a wall monitor, to move off at a 45-deg angle into space.According to the present distinction, learning to correctfor such an extreme, multi-spatial sensory-motor conflictcan be likened to a cognitive problem-solving task and isthus a higher-order, voluntary process. Therefore, unlikethe restricted, non-cognitive process of perceptual recali-bration, visual-motor skill acquisition can be expected totransfer from trained to untrained hand. Furthermore,because this kind of adaptation is not based on a changein felt limb position, little or no post-exposure negativeaftereffect should be expected.

It follows from the paucity of negative aftereffects forvisual-motor skill acquisition that this form of adaptationshould be very susceptible to ‘‘dual adaptation,’’ in whichobservers who have repeatedly adapted and re-adapted totwo (or more) mutually conflicting sensory environmentsare eventually able to make this transition with minimaldisruption of their perception and/or perceptual-motorcoordination (e.g., [18]). Clearly, the presence of substan-

154 Robert. B. Welch, Anthony. C. Sampanes / Displays 29 (2008) 152–158

tial negative aftereffects will impede the ability to shift fromone sensory environment to the other. Therefore, sincevisual-motor skill acquisition is assumed to be relativelyimmune to negative aftereffects, dual adaptation shouldbe more easily acquired and more complete for this typeof adaptation than for perceptual calibration. A relatedprediction is that an acquired visual-motor skill will bemuch more resistant to forgetting or decay than is percep-tual recalibration. As described in the following sections,our laboratory has supported and elaborated these pro-posed distinctions between perceptual recalibration andvisual-motor skill acquisition.

Fig. 2. Root-mean-square (RMS) error (in mm) by trial blocks overTraining Days 3–5 and the Day 6 retention block for participants exposedto a 108-deg rotated stylus-cursor arrangement. Note: Error bars representstandard error of the mean for each plotted point. Filled circles denoterotated trial blocks; open circles denote normal trial blocks. (Adapted withthe permission of the American Psychological Association from Figure 5of 31).

3. Empirical support for the distinction

3.1. Comparison between multi-spatial and uni-spatial

rearrangements

In a series of studies from our laboratory (Cunningham& Welch, 19), participants controlled a cursor by means ofa hand-held stylus applied to a horizontal digitizing tablet,both of which were shielded from sight by an occludingboard. Their task was to keep the cursor centered continu-ously on a small, randomly moving target presented on anupright desktop computer monitor, 57 cm distant. Two sty-lus-cursor relationships were used: (1) the normal one inwhich forward-backward hand movements cause up-downcursor motion and leftward-rightward movements causeleft-right cursor motion, and (2) a rotated condition inwhich the cursor was programmed to move off at a 108-deg clockwise (CW) or counterclockwise (CCW) anglefrom its normal trajectory. Fig. 1 shows the results (rootmean square tracking error) of this experiment for a repre-sentative participant who engaged in the target-centeringtask under the rotated condition on a series of 33-s trialsthat were preceded and followed by the normal stylus-cur-sor condition. It can be seen that by the last exposure trial

Fig. 1. Root-mean-square (RMS) errors (in mm) by a representative particrepresents the first exposure to the normal mapping, adaptation the first exposuwith the permission of the American Psychological Association from Figure 1

the participant’s large initial rotation-induced errors haddeclined almost to the normal, baseline level. Despite thisnearly complete reduction of effect, however, the post-exposure aftereffect was negligible and quickly abolishedby further exposure.

In the critical part of the experiment, participants wereexposed first to the rotated condition, then without warn-ing to the normal condition, then to the rotated condition,and so forth. As expected, the first few transitions from onestylus-cursor relationship to the other produced very largeinitial errors that declined with continued exposure to thatcondition. More importantly, as alternations between thetwo stylus-cursor arrangements continued over an hour’straining on five consecutive days, participants became soadept at switching between conditions that by Day 5 theycould re-acquire nearly accurate tracking performancewithin a fraction of a second after the onset of a reversaland their tracking performance approached that of the nor-

ipant exposed to a 108-deg rotated stylus-cursor arrangement. Baselinere to the rotation, and aftereffect the return to normal mapping. (Adaptedof 31).

Fig. 3. Average root-mean-square (RMS) error (in mm) for Day 1 pretestand Day 4 posttest for exposed and non-exposed hands for participantsexposed to a 108-deg rotated stylus-cursor arrangement. Error barsrepresent standard error of the mean for each plotted point. (Adapted withthe permission of the American Psychological Association from Figure 12of 31).

Robert. B. Welch, Anthony. C. Sampanes / Displays 29 (2008) 152–158 155

mal condition (Fig. 2). In other words, they had achievedvery substantial dual adaptation, consistent with the claimthat this task was producing visual-motor skill acquisitionand not perceptual recalibration. This conclusion is furthersupported by the marginal and evanescent aftereffects seenin Fig. 1 and the presence of nearly 80% inter-manualtransfer of adaptation, as measured by comparing Day 1versus Day 4 performance for the exposed hand and Day1 versus Day 4 performance for the non-exposed hand(Fig. 3). Finally, a test administered three days after thelast training session revealed that the dual adaptation dem-onstrated at the end of Day 5 had been preserved withoutdecrement (see Fig. 2).1

In marked contrast to the preceding results, an earlierexperiment from our laboratory (20) found only limiteddual adaptation when participants alternated between8.5-deg prismatic displacement of the hand to the rightand the same displacement to the left. Fig. 4 (Panel B)shows that the reduction of effect for participants’ target-reaching errors (in deg) was somewhat more rapid on thetenth reversal between the two displacements than on thefirst, evidence of modest, albeit statistically significant, dualadaptation. It can also be seen in the figure (Panel C) thatthe post-exposure negative aftereffect, although reduced bythe tenth cycle as compared to the first, was quite substan-tial in both cases. As we have argued, such resistance todual adaptation and the presence of large aftereffects canbe considered evidence of perceptual recalibration andnot visual-motor skill acquisition. The obvious differencesbetween Cunningham and Welch [19] and Welch et al.[20] is that the former used a very large rearrangement

1 In an informal test several months after the completion of this researchthe second author was able perform the rotated task nearly perfectly. Thefact that this intervening period entailed countless hours at the computercontrolling a cursor in the usual way attests to the relative permanence ofdual adaptation.

(108-deg rotation) in which the plane of the visual displaywas orthogonal to (and some distance away from) that ofthe hand, whereas the latter presented participants with asmall rearrangement within a single spatial plane.

3.2. Other rearrangements and tasks

In a recent series of experiments we examined anothersensory-motor rearrangement and task as a candidate forvisual-motor skill acquisition. Participants were exposedto a 45-deg CW or CCW rotated stylus-cursor relationshipusing a physical arrangement similar to that of Cunning-ham and Welch (19). Participants moved a hand-held sty-lus on a digitizing tablet (shielded by an occluding board)and viewed the resulting cursor motion on an upright lap-top computer monitor. Whereas the exposure task ofCunningham and Welch [19] involved visual tracking, thetask in this experiment was to move the cursor from a start-ing position in the center of the monitor to a designatedtarget at the top, left, right, or bottom edge of the display.The measure of adaptation during exposure (i.e., the reduc-tion of effect) was the length of the cursor path from start-ing position to target. Thus, as participants learned tocorrect their rotation-induced errors earlier, these pathsshould get straighter and thus shorter.

An important difference between this research and thatof Cunningham and Welch [19] is the way in which poten-tial aftereffects were measured. It will be recalled that inthe latter study, the cursor motion was displayed at alltimes. Therefore, any post-exposure aftereffects thatoccurred were quickly abolished by the visual feedback(see Fig. 1). In order to provide a more stable measureof aftereffects in the present experiment we required thatthe post-exposure (as well as pre-exposure) pointingresponses be made open loop, as in the typical prismadaptation study. Thus, before and after exposure, thecursor was deactivated, forcing participants to make theirresponses in the absence of visual feedback. The depen-dent measure was the linear distance between the desig-nated target and the final position of the (unseen)cursor. The difference between pre- and post-exposureserved as the measure of any negative aftereffects thatmight have occurred.

On the early exposure trials, as participants began mov-ing the cursor away from the center of the monitor towarda target in its upper half of the display, they would startmoving the cursor at a 45-deg angle to one side, whilefor a target in the lower half of the display, they woulderr by 45 deg to the opposite side. Upon seeing this initialerror, they would start correcting the direction of the cur-sor’s motion until it intersected the target, which resultedin markedly curved cursor paths. As expected, these pathshad straightened out (and thus shortened) considerably bythe last exposure trial, as participants anticipated or morequickly corrected for their errors (Fig. 5). This reductionof effect was followed by a very small, although statisticallysignificant, negative aftereffect, as indicated by the open-

Fig. 4. Reduction of effect, aftereffect, and re-adaptation curves (in deg.) for the first and tenth cycle of reversal between 8.5-deg rightward prismaticdisplacement and 8.5-deg leftward prismatic displacement. (Adapted with the permission of Psychonomic Publications from Figure 2 of 32).

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37

EXPOSURE TRIALS (Blocks of 4)

Fig. 5. Reduction of effect for adapting to a 45-deg rotated stylus-cursor arrangement. Note: Cursor translation efficiency was calculated by taking thedifference between the actual path length and the theoretical minimum of 476 and dividing the resultant by 476; the smaller the ratio, the more efficient thecursor translation.

156 Robert. B. Welch, Anthony. C. Sampanes / Displays 29 (2008) 152–158

loop target-pointing responses. Calculated as a percentageof the imposed 45-deg rotation, this aftereffect came to

approximately 8.5%, substantially less than the figure of50% or more commonly observed in prism adaptation

Robert. B. Welch, Anthony. C. Sampanes / Displays 29 (2008) 152–158 157

studies [9], including the one by Welch et al. [20] describedabove (Fig. 4). Thus, using an open-loop measure, we stillfound only minimal negative aftereffects, which supportsthe conclusion that our multi-spatial rearrangement andtargeting task were subject to visual-motor skill acquisi-tion, rather than perceptual recalibration.

Because it is possible that the dearth of aftereffects inthis experiment was simply evidence of the absence of sub-stantial adaptation we performed two follow-up experi-ments designed to maximize adaptation. In the secondexperiment, we increased the number of exposure trials

-50

0

50

100

150

200

250

1 2 3 4

EXPOSURE

POST

-PR

E SH

IFT

IN F

ELT

STR

AIG

HT-

AH

EAD

HA

ND

PO

SITI

ON

(pix

els)

Fig. 6. Post-pre shift in felt straight-ahead limb position (propriocepti

0

20

40

60

80

100

120

140

160

180

1 2 3 4 5

INTE

RM

AN

UA

L TR

AN

SFER

(pix

els)

Fig. 7. Intermanual transfer of adaptatio

and limited targets to the upper half of the display in orderto simplify the adaptive ‘‘rule’’ for counteracting the effectsof the rotation. In the third experiment, we provided par-ticipants with ‘‘terminal,’’ rather than continuous, visualfeedback. All of these variables are known or presumedto augment prism adaptation as measured by the negativeaftereffect (e.g., [6]). Nevertheless, the aftereffects obtainedin these two experiments were virtually identical to those ofthe first. Thus, in all three experiments we replicated Cunn-ingham and Welch [19] by finding only miniscule afteref-fects, strengthening the claim that this form of

5 6 7 8 9

TRIALS (Blocks of 2)

ve recalibration) for adaptation to 23-deg prismatic displacement.

6 7 8 9 10

n to 23-deg prismatic displacement.

158 Robert. B. Welch, Anthony. C. Sampanes / Displays 29 (2008) 152–158

adaptation is best described as visual-motor skillacquisition.

3.3. The relationship between visual-motor skill acquisition

and perceptual recalibration over the course of the

adaptation period

It is important to note that we do not assume that per-ceptual recalibration and visual-motor skill acquisition aremutually exclusive. Rather, it seems likely that in some sit-uations both can occur, although varying in relative pro-portion over the course of the adaptation period.Congruent with this expectation, Welch and Sampanes[21] tested the specific hypothesis that when confrontedwith the relatively small, uni-spatial rearrangement of lat-eral prismatic displacement, observers first acquire thevisual-motor skill of corrected target reaching, which, asexposure to the displacement continues, is replaced (at leastpartially) by perceptual recalibration.

To test this prediction, we carried out a study in which12 participants were measured periodically during exposureto a 23-deg leftward prismatic displacement on one of thetwo types of adaptation and 12 participants were measuredperiodically on the other. During this exposure phase, theyreached with the right hand for a target and received error-corrective feedback at the terminus of each response. Forthe measure of perceptual recalibration, participants wereinstructed at regular intervals to place the prism-exposed,but now unseen, hand straight ahead of their nose. Recal-ibration would be revealed if participants came to place theunseen hand in the direction opposite the prismatic dis-placement, signifying that its felt position had been shiftedto one side. The measure of visual-motor skill acquisitionwas the presence of inter-manual transfer of adaptation –the degree to which the corrected target-pointing accuracyof the prism-exposed (right) hand transferred to the accu-racy of the non-exposed (left) hand.

In brief, our results showed that as exposure continued,perceptual recalibration, as measured by proprioceptiveshift, increased (Fig. 6), whereas visual-motor skill acquisi-tion, as measured by inter-manual transfer, decreased,(Fig. 7). Thus, as we had surmised, even with a relativelysmall, uni-spatial visual rearrangement, the initial adaptiveresponse is characterized primarily by visual-motor skillacquisition, which is then partially replaced by perceptualrecalibration.

4. Summary and conclusions

An understanding of the necessary conditions for per-ceptual recalibration versus visual-motor skill acquisitionallows one to predict the primary type of adaptation a par-ticular VE (or tele-operator device) will produce. If a VE

entails a relatively minor inter-sensory or sensory-motorconflict that is limited to one spatial plane (e.g., a lateralmisalignment of felt limb position and its visual surrogateviewed in a head-mounted display), the end product ofadaptation will be perceptual recalibration, a restrictedform of perceptual learning that is accompanied by sub-stantial negative aftereffects. If, on the other hand, theVE presents the user with a very large and/or multi-spatialrearrangement, adaptation is expected to take the guise of amore cognitive, problem-solving process referred to asvisual-motor skill acquisition, which produces little or nonegative aftereffects. As we have seen, being able to predictwith some confidence which of these two types of adapta-tion a given VE device will produce allows one to anticipatethe usefulness of dual adaptation training, whether signifi-cant inter-manual transfer of training is likely to occur,how persistent the adaptation will be, and whether oneshould be on the lookout for negative aftereffects.

References

[1] E.B. Nash, G.W. Edwards, J.A. Thompson, W. Barfield, Interna-tional Journal of Human–Computer Interaction 12 (2000) 1–41.

[2] R.B. Welch, Adapting to virtual environments, in: K. Stanney (Ed.),The Handbook of Virtual Environments Technology (HVET),Lawrence Erlbaum, Nahwah, NJ, 2002, pp. 619–636.

[3] L.J. Hettinger, G.E. Riccio, Presence: Teleoperators and VirtualEnvironments 1 (1992) 306–310.

[4] F. Biocca, Presence: Teleoperators and Virtual Environments 1 (1992)334–343.

[5] J.T. Reason, J.J. Brand, Motion Sickness, Academic Press, London,1975.

[6] R.B. Welch, Adaptation of Space Perception, in: K.R. Boff, L.Kaufman, J.P. Thomas (Eds.), Handbook of Perception and HumanPerformance, John Wiley and Sons, New York, 1986, pp. 24-1–24-45.

[7] G.M. Redding, B. Wallace, Adaptive Spatial Alignment, Erlbaum,Mahwah, NJ, 1997.

[8] R. Held, American Journal of Psychology 68 (1955) 526–548.[9] P.T. Young, Journal of Experimental Psychology 11 (1928) 399–429.

[10] J.C. Hay, H.L. Pick Jr., Journal of Experimental Psychology 72(1966) 640–648.

[11] S.M. Ebenholtz, D. Mayer, Perceptual and Motor Skills 26 (1968)507–509.

[12] I. Kohler, Psychological Issues 3 (1964) 1–173.[13] G.M. Stratton, Psychological Review 4 (1897) 182–187.[14] D.M. Clower, D. Boussaoud, Journal of Physiology 84 (2000) 2703–

2708.[15] G.M. Redding, B. Wallace, Journal of Motor Behavior 34 (2002)

126–138.[16] C.S. Harris, Psychological Review 72 (1965) 419–444.[17] R.B. Welch, Perceptual Modification: Adapting to Altered Sensory

Environments, Academic Press, New York, 1978.[18] J.P. Flook, B.O. McGonigle, Perception 6 (1977) 15–29.[19] H.A. Cunningham, R.B. Welch, Journal of Experimental Psychology:

Human Perception and Performance 20 (1994) 987–999.[20] R.B. Welch, B. Bridgeman, S. Anand, K. Browman, Perception &

Psychophysics 54 (1993) 195–204.[21] R.B. Welch, A.C. Sampanes, Paper presented at the meeting of the

Psychonomic Society, Meanneapolis (2004).