Animation and interactivity in computer-based physics experiments to support thedocumentation of measured vector quantities in diagrams: An eye tracking study

Christoph Hoyer and Raimund GirwidzLudwig-Maximilians-Universität München, Chair of Physics Education, Theresienstraße 37, 80333

München, Germany

(Received 20 June 2020; accepted 17 September 2020; published 19 October 2020)

Simulations and virtual or remote laboratories are increasingly used in schools. The extent to whichindividual experimental skills can be acquired when experimenting in digital applications is, however,questionable. This paper focuses on finding multimedia features for digital experiments to support thetransfer of measured values from the laboratory system to a diagram. Beside physical considerations,spatial translation processes could be crucial for a successful assignment. Therefore, the influence of thesubjects’ spatial ability is examined. Using a pretest post-test design (N ¼ 119), the effects of training withsupportive animation (animation group) and training with an interactive task and feedback (interactivegroup) were tested. The results of both groups were each compared to those of a reference group. Eyetracking data were recorded during training to investigate the origin of different training effects. Hence,fixations and saccades during training were analyzed. For the investigation of the distribution of thesaccadic movements, polar diagrams were used in combination with estimated probability densityfunctions. The results show that the score in the pretest is correlated to the score achieved in the cardrotation test, which measures the spatial rotation skills of the subjects. Further, the subjects in the interactivegroup benefited from the training more than the subjects in the reference group did. There were nosignificant differences in the effect of the training between the animation group and the reference group.Eye tracking data reveal that the training in the interactive group caused the most comparative eyemovements between the laboratory system and the diagram. The training in the animation group led to thehighest visual attention; however, subjects in this group concentrated on the dynamic elements. Theseresults indicate that especially students with weak spatial skills need additional support when transferringmeasured values from the laboratory system to the diagram. This assignment can be practiced in computer-based experiments, in particular with an interactive training task and feedback. Additionally, the analysisshowed that the training is equally suitable for learners with different spatial abilities. A corresponding taskwas implemented into a virtual laboratory.

DOI: 10.1103/PhysRevPhysEducRes.16.020124

I. INTRODUCTION

Collecting, organizing, and interpreting measured valuesare key competences for processing experimental data. Thecreation and interpretation of graphical representations ofmeasured values plays a particularly important role. Thefollowing examines how the learners’ skill of transferringmeasurement results from the laboratory system to thecorrect points in the diagram can be supported whenworking with computer-based experiments. This wasinvestigated using a virtual laboratory with which the fieldof a permanent magnet can be measured and visualized.

Supporting multimedia tools are derived from theory andtheir effectiveness is tested. However, first the relevance ofpracticing the documentation of measurement results isshown and associated difficulties are described.Science lessons in school require students to develop a

deeper understanding of how science works and to practicerelevant skills (e.g., Ref. [1]). Therefore, in schools,planning and performing experiments should play a centralrole. Simulations and virtual or remote laboratories can bevaluable supplements to real experiments as described,among others, by Finkelstein et al. [2] for electrical circuitsand by Martínez et al. [3] for image formation and opticalaberration. Experimenting on the computer can also beadvantageous if cost and security reasons do not allow allstudents to carry out a real experiment. When decidingwhether to conduct a real or computer-based experiment,the advantages and disadvantages should be taken intoaccount. Also, the expected learning outcomes have to beconsidered [4,5]. Teaching physics includes imparting

Published by the American Physical Society under the terms ofthe Creative Commons Attribution 4.0 International license.Further distribution of this work must maintain attribution tothe author(s) and the published article’s title, journal citation,and DOI.

PHYSICAL REVIEW PHYSICS EDUCATION RESEARCH 16, 020124 (2020)

2469-9896=20=16(2)=020124(18) 020124-1 Published by the American Physical Society

knowledge about physical content and experimental skills.An interesting question is to what extent individualexperimental skills can be practiced by using digitalapplications. To help answer this question, the followingpresents an investigation into what support in computer-based experiments improves the documentation of meas-urement results in diagrams.Both the experimental setup and the diagram are repre-

sentations that the learner must relate to each other tosuccessfully document measurement results. Therefore,research on learning with multiple representations [6] offersthe opportunity to identify potential difficulties in thedocumentation process.The DeFT framework [7] describes factors that make it

difficult to build relations between representations. Theinfluence of three of these factors on the documentation ofmeasurement results in computer-based experiments isbriefly described as an example:

• Level of abstraction: If the axes of the laboratorysystem of the experiment and the axes of the diagramare coded differently, the translation process betweenboth could be difficult for the learners. This, forexample, is the case if a movement is documentedin a time-velocity diagram.

• Dimensionality: If measurements are recorded inthree-dimensional space and measurement resultsare visualized in a two-dimensional diagram, therequired abstraction can be an obstacle for the learn-ers. An example is the representation of the gravita-tional potential on the surface of a mountain usingequipotential lines.

• Whether representations are static or dynamic: If thespatial relationship between the laboratory system andthe coordinate system of the diagram changes duringdata acquisition, the translation process can be chal-

lenging. An example of such an experiment is thedetermination of the directional characteristic of areceiver dipole. The emitter is usually more massiveand therefore more difficult to move than the receiver.Because of this, it is useful to turn the receiver insteadof moving the emitter. Simultaneously, by rotating thereceiver, the laboratory system is rotated. In contrast,the orientation of the coordinate system of the diagramstays the same.

These examples show that the challenges of documentingdata in diagrams are diverse. Therefore, the supportingmultimedia content must be adapted for the specific task.The case investigated concerns documenting measured

values of the field of a permanent magnet in a virtuallaboratory. To construct multimedia tools to support thistask, the exact area of application must be defined first.Measuring the field of a permanent magnet necessitates a

sensitive but unwieldy measurement apparatus. This isbecause the absolute value of the magnetic flux densitydecreases rapidly with increasing distance from the magnet.Minimal deviations in the alignment of the sensor causelarge fluctuations in the measured value. When construct-ing such a laboratory, instead of moving the measurementdevice around the magnet, it is appropriate to restrict itsmovement to the radial direction. Using a linear driveenables highly accurate changes in the distance between themiddle of the magnet and the sensor. This linear movementin combination with a rotation of the magnet allows thedetermination of the magnetic field around the magnet.However, by rotating the magnet, the associated laboratorysystem is rotated (see Fig. 1). As derived from the DeFTframework [7] above, the changing orientation between thelaboratory system and the coordinate system of the diagramcan pose challenges for the learners.

FIG. 1. View of the webpage containing the application. In the middle (within the red dashed line) the experimental setup (left) and thediagram (right) can be seen. The orientation of the laboratory system changes during data acquisition, while the orientation of thecoordinate system of the diagram stays the same. The position of the sensor is marked in red. The red markings (dashed line and circle)were added here for illustrative purposes and were not included in the application.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-2

The focus of the investigation is therefore on findingmultimedia techniques for computer-based experimentsthat support this transfer between the laboratory systemof the experiment and the coordinate system in the diagram.Beside physical considerations, since this process can

also be seen as a spatial transformation between thelaboratory system and the coordinate system in the dia-gram, success in this task may also depend on the learners’spatial thinking abilities (see Sec. I A 2). If so, this shouldbe considered when choosing appropriate supportive multi-media tools. In the next step, possible approaches arederived from theory. Subsequently, their effectiveness istested. Eye tracking data will help to interpret the results.

A. Theory

1. Experimental skills

In recent years, more and more attention has been paid toteaching experimental skills in physics courses (e.g.,Refs. [8–12]). There are various categorizations in theliterature to classify experimental skills. An overview isgiven byEmden [13]. A categorization of experimental skillsin undergraduate labs is provided by the AAPT Committeeon Laboratories [11]. Experimental skills can be roughlyclassified as skills required to prepare experiments, skillsnecessary to carry out experiments, or skills relevant toevaluate experimental data [14]. Recent studies have shownthat lab courses focused on teaching experimentation do notnecessarily convey less content knowledge than those con-centrating on content reinforcement [15].With the rapid development of digital applications,

conducting experiments is no longer tied to working in alaboratory. Digital applications offer new possibilities tocarry out experiments on the computer [8]. When exper-imenting with digital applications, however, it is question-able to what extent experimental skills can be acquired.Many computer-based experiments document measured

data automatically. Such automated documentation can freeup cognitive resources for other learning activities [16].However, by documenting measured values automaticallyit cannot be taken for granted that the students’ ability todocument measured values in diagrams is supported.Nevertheless, having a wide range of application, multi-media can conceivably help training the documentation ofmeasurement results in diagrams.

2. Spatial rotation ability

There are different categorizations of spatial abilities inthe literature [17–19]. The review by Cole et al. [20] givesan overview and summarizes research results on the role ofspatial thinking in the teaching of astronomy. A componentof spatial skills that is included in all of the categorizationsis mental rotation ability. This ability concerns the mentalrotation of two- and three-dimensional objects. A commontest to measure two-dimensional rotation ability is the card

rotation test developed by Ekstrom et al. [21] (see alsoRef. [22]). A suitable test for measuring three-dimensionalrotation ability is the cube comparison test [21] or themental rotation test, originally developed by Vandenbergand Kuse [23] (for an overview, see Ref. [22]). There is alsoa revised version of the latter test [24].Findings suggest that students with higher visual

processing skills find it easier to use visualizations andother multimedia modules (for a summary, see Ref. [22]).Several research papers report that spatial thinkinginfluences problem solving and interpreting graphs inscience [25–29]. Especially the translation between framesof reference, a challenging task for learners [30], isinfluenced by the learners’ visual spatial abilities [29].In the case examined, the learners have to map positions

from the laboratory system to the diagram. Since this task issimilar to the problem of translating between referencesystems, it should be considered whether this task alsodepends on the learners’ spatial thinking ability.

3. Linking experiment and diagram

To support the transfer of measurement results of themagnetic flux density from the laboratory system to thecorrect position in the coordinate system of the diagram, so-called relational cues [31] could help. These can emphasizethe connection between the experiment and the diagram.Multimedia offers various options for providing such cuesin simulations and virtual or remote laboratories. Variousapproaches in the literature are promising.

Animated visualizations. Computer-based experimentsthat automatically document data display a static diagramcontaining the correctly plotted measurement values. Theliterature shows that compared to a static representation, ananimation could be superior in supporting the transferbetween the laboratory system and the diagram.The results of a meta-analysis by Höffler and Leutner

[32] suggest that animations seem to outperform staticpictures if the motion depicted in the animation explicitlyrefers to the learning topic. Then animations can help tobuild mental models of the dynamic [33]. Berney andBetrancourt [34], in their meta-analysis, are only partiallyable to replicate those results. They confirm that animationsseem to be more effective compared to static pictures, butthey cannot find significant differences in this effectbetween different types of knowledge. Instead of a com-parison of static pictures and animations, the authorsrequest an analysis of when, why, and for whom animationsare beneficial for learning [34,35].An important property of animations is that they can

attract attention and help focus on relevant aspects. Theonset of motion or appearance of objects can guide thefocus of the learners [31,36,37]. However, results show thatattention guidance does not necessarily lead to betterlearning outcomes [38–40].

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-3

When designing animations, the characteristics of thelearning content should be taken into account. It waspreviously described that the translation between thelaboratory system and the coordinate system of the diagramcould be influenced by the students’ visual spatial abilities.If the learners indeed use a spatial translation process forthe documentation task according to Salomon’s supplanta-tion framework [41], an animation illustrating the mentaltranslation task may foster the internal mental process.Therefore, animations could be particularly effective forsupporting such tasks. Accordingly, Gilligan et al. [42]show that a continuous animation depicting the trans-formation of an object can have a positive effect on laterspatial rotation tasks.In computer-based experiments, a dynamic animation

could be an appropriate aid for both binding visual attentionand improving the transfer of measured values of themagnetic flux density from the laboratory system to thecorrect point in the coordinate system in the diagram. Apossible implementation is the following: When a learnermeasures the field at a specific position in the laboratorysystem, an animation shows how to transfer this position tothe corresponding point in the diagram. After the animationhas ended, the measured value stays marked in the diagram.

Interactivity and feedback. Chi and Wylie [43] distinguishbetween four groups of learning activities. They point outthat learning increases while “activities move from passiveto active to constructive to interactive.” Animations couldleave the learners in a passive recipient role. Thus, a moreactive task could improve the transfer of measured valuesfrom the laboratory system to the diagram to a greaterextent. The feature of interactivity distinguishes new mediafrom “classic” media, which show the same information toevery learner without taking the learners’ activities intoaccount. Interactivity is a key property for the successfuluse of multimedia [44]. Interactivity in multimedia appli-cations is characterized by mutual actions between thelearner and the learning environment [45].A model for interaction in multimedia applications is

described in Ref. [46]. In addition to the interactionbetween the learning environment and the learner, theauthors also consider learner characteristics, motivationaland emotional aspects, cognitive and metacognitive char-acteristics, and the learner’s mental models. An optimallearning outcome requires finding the right level of inter-activity in the learning environment. However, finding thislevel is difficult [47].For cognitive tutors, interactive elements were identified

that are contained in all such systems [47]. These elementsare implicit yes-no feedback, specific feedback messagesfor commonly occurring errors, and next-step hints.In the context of this application, feedback can be

defined as “information communicated to the learner that

is intended to modify his or her thinking or behavior toimprove learning” [48].Therefore, for the case examined, providing feedback

helping the learners to rethink and reflect on their answerscould induce deeper processing. Immediate feedback attask level, highlighting the correct position in the diagram,could cover these requirements. This is also justified bytheory.Hattie and Timperley [49] give an overview of research

results showing that corrective feedback at task level can beeffective, especially if it draws attention to faulty inter-pretations. Van der Kleij et al. [50] showed that studentspaid more attention to immediate feedback than to delayedfeedback. Suddenly appearing feedback could additionallyenforce this effect as research shows that the appearance ofobjects can capture attention [31,37,51]. In this way, givingimmediate feedback clearly delineated from the rest of theapplication can be used to guide the learners’ attention. It isworthy of note that, like for animations, binding attentiondoes not necessarily result in better learning outcomes [38].In summary, immediate feedback in computer-based

experiments could be helpful for improving the transfer ofmeasured vector quantities from the laboratory system to thecoordinate system of the diagram. Such feedback could beintegrated as follows: After learners have carried out ameasurement in the computer-based experiment, they areasked to mark the point at which they would plot themeasured value in the diagram. Simultaneously with theappearance of this marking, the laboratory visualizes andhighlights themeasured quantity at the correct position in thediagram.

4. Eye tracking

In recent years, there has been an increasing number ofeye tracking studies in teaching and learning (for anoverview, see the metastudies [52,53]). Recent investiga-tions show how eye tracking data can provide insights intostudents’ visual attention in physics education (for exam-ple, Refs. [54–56]).Eye tracking systems allow the recording of learners’ eye

movement during a learning task. The eye-mind assumption[57] states that eye fixations indicate processing of thefixated information. Accordingly, eye tracking data helpchecking whether elements intended for learning wereperceived in the expected way or if other salient elementsattracted the learners’ attention.For the investigation described in this paper, eye tracking

enables the analysis and comparison of eye movementsduring training. In this way, processes can be identified thatare caused by the training and that are decisive for improvingthe task. For the comparability of the eye movementsbetween the different training conditions, it is important thatthe conditions include the same components of the userinterface and differ only in the given training stimulus.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-4

In the past, research concentrated mainly on the analysisof the total number of fixations, the duration of fixations,and the dwell time in AOIs. Also, the total number,direction, length, and duration of the movements betweentwo fixations, the so-called saccades, were investigated (foran overview, see Refs. [52,58]). Figure 2 shows the scanpath of a test person as an example. The circles correspondto fixations, with the size of the circles representing theduration of each fixation.The number of saccadic movements was often examined

to analyze integration processes between representations.While in some studies an increased number of saccades ledto better performance (e.g., Refs. [56,59–61]), this couldnot be confirmed in other studies (e.g., Refs. [62–64]).In the case investigated, the relationship between ele-

ments in the laboratory system and the diagram is ofparticular interest. Appropriate eye movements may benecessary to perceive these relationships. Therefore, acombined view of the directional and length distributionsof saccades could reveal the perception of relations betweenthe depicted elements. A suitable procedure was developedand is presented below.

B. Research questions

This paper examines four research questions (RQs).In general, the laboratory system and the coordinate

system in the diagram have no fixed spatial relationship toeach other.RQ1: If the laboratory system and the coordinate system

in the diagram can be transformed into each other usingrotation and translation and if the task is to transfer ameasurement location of the laboratory system as accu-rately as possible into the coordinate system of the diagram,is there a correlation between spatial rotation ability andsuccess in this assignment task?In most computer-based experiments, as soon as mea-

surements are recorded, they are immediately visualized atthe correct positions in a diagram. In comparison, an

animated visualization or an interactive assignment taskwith feedback could improve the subjects’ accuracy whenperforming the assignment on their own.RQ2: When the laboratory system is rotated relative to

the coordinate system of the diagram, are there significantdifferences in the accuracy with which measurementlocations are transferred from the laboratory system tothe diagram after subjects attend one of the following threetrainings:

(i) Training with animations illustrating the transferbetween the laboratory system and the coordinatesystem of the diagram.

(ii) Training with interactive assignment tasks withfeedback where subjects have to transfer positionsfrom the laboratory system to the diagram.

(iii) Training with depictions of correctly solved assign-ment tasks.

As described above, the three training conditions havedifferent characteristics. Subjects with different spatialabilities could therefore benefit from training in differentforms. To select the most suitable training, it should thus beexamined whether a relationship exists between the spatialabilities of the subjects and the performance gain in theindividual training conditions.RQ3: Is there a relationship between the spatial abilities

of the subjects and the performance gain during the threetraining conditions?Analyzing the fixations and saccadic movements pro-

vides information about the visual processing of the contentof the training. It is crucial for the training that theinformation content is perceived and that it receives thevisual attention of the learners. This can be determined byanalyzing fixations. However, saccadic eye movements canalso indicate cognitive processes. Such eye movements, forexample, are necessary to perceive the position of themeasuring location in relation to the laboratory system or tocheck the position of marked values in a diagram relative to

FIG. 2. Visualization of the scan path of a subject. The red circles represent fixations. The green lines show saccadic movements. Theorder in which the fixations occurred is described by the number in the circles. The duration of each fixation is indicated by the size ofthe corresponding circle and, in addition, is displayed next to the circle.

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-5

the corresponding coordinate system. Other characteristicsaccadic movements should result from establishing arelationship between the laboratory system and the coor-dinate system of the diagram. The various training con-ditions could have different effects on those components ofgaze behavior.RQ4: Is there a difference between the training groups in

the number of fixations and in the distribution of thesaccades during training? Furthermore, can those measuresindicate visual processes important for successful training?

II. METHODS

A. Participants

119 students (81 males, 38 females) from eight different11th grade classes took part in the study. The curriculum ofthe 11th grade in a Bavarian high school includes describ-ing magnetic fields as vector quantities. To guarantee thatall students bring basic knowledge about magnetic fieldswith them, the examination was carried out shortly beforethe end of the school year.

B. Design

The design of the study is depicted in Fig. 3. At thebeginning, the 119 subjects worked on the card rotation test(CRT). This test indicates the spatial rotation ability of thesubjects and is a paper and pencil test. Subsequently, theytook the pretest. After the pretest, the intervention tookplace. For the intervention, the subjects were dividedrandomly into three groups. The animation group consistedof Nanimation ¼ 38, the interactive group of Ninteractive ¼ 41,and the reference group of Nreference ¼ 40 participants.Finally, all of the subjects took the post-test. The exami-nation on the computer took only a short time. To ensurethat the test subjects were not disturbed by the eye trackingand in order not to interrupt the sequence of pretest,training, and post-test, the eye tracking system was startedwhen the subjects began to work with the computerapplication. For this reason, the eye tracker was calibratedonly once at the beginning. Thus, eye movements wererecorded while the subjects worked on the pretest, theintervention, and the post-test.

C. Material

In this section, the technical equipment, the tests used,and the different training conditions are described. Also,the procedures of the investigation and statistical methodsare presented.

1. Computer environment

The pretest, intervention, and post-test were carried outon a Windows 10 computer workstation. The resolution ofthe 24” monitor was 1920 × 1200 pixels. The applicationwas written in HTML and JavaScript. It was opened in theChrome browser. During the pretest and the post-test, clickdata were recorded using JavaScript and PHP. A high-quality mouse and mousepad were used to ensure bestconditions. The mouse resolution was the same for allparticipants. An average setting was chosen based onresults of a pilot test (N ¼ 8).

2. Computer-based experiment

A modified version of a virtual laboratory with which thefield of a permanent magnet can be measured [65] was usedfor the assessment. The application runs in all standardbrowsers. Figure 1 shows a screenshot. The virtual lab isbased on a real experiment. To give a realistic impression,animated pictures of the real experiment are used in thevirtual lab. For the measurement of the field of the magnet,the sensor can be moved radially while the magnet can berotated. Thus, the apparatus allows a highly accurate posi-tioning of the sensor at all positions within 100 mm aroundthe magnet (further explanations can be found in theintroduction).

3. Eye tracker

The eye tracking system used was an Eye-Follower fromLC Technology. The system has an accuracy of less than0.4° of visual angle over the whole range of head move-ment. Four cameras are used for tracking eye movements.Two of them record the motion of the head. In this way, thesystem can accurately track the movements of the eyes,even if the subjects move their head during computeroperation. This makes it possible to work on the computeras naturally as possible, uninfluenced by the eye trackingsystem. The sampling rate of the system is 120 Hz. A nine-point calibration was carried out for every subject. Thecalibration process was successful if the accuracy was lessthan 0.63 cm (0.25 inches). To discriminate betweenfixations and saccades, the dispersion-based algorithmLC Fixation Detector was used (for more informationabout eye tracking algorithms, see Ref. [66]).

4. Tests

Card rotation test. To measure the spatial rotation abilityof the subjects, the card rotation test of the kit of factor-referenced cognitive tests was chosen [21]. This test is a

FIG. 3. Illustration of the design of the study. First the subjectsworked on the card rotation test. Afterward, eye movements wererecorded while the subjects participated in the pretest, theintervention, and the post-test.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-6

paper-and-pencil test and measures the capacity to mentallyrotate two-dimensional geometries. Its internal consistencyis excellent (α ¼ 0.96) as Burton and Fogarty [67] reported.As the introductory material was available only in English,it was translated into German. In the test, one two-dimensional object is depicted on the left of a line andeight objects on the right of this line. The eight objects canbe transformed into the object on the left using rotation andreflection. The task in this test is to compare if the objectson the right of the line either are the same but rotated or aremirrored in comparison to the referential object on the leftof the line. The subjects received one point for eachcorrectly answered subtask, and one point was deductedfor each incorrectly answered subtask. The time on taskwas limited to 3 min.

Pretest. In the pretest, the subjects’ task was to click asaccurately as possible the position in the diagram corre-sponding to the location of the sensor in the laboratorysystem (see Fig. 1). Each subject worked on eight suchassignment tasks. The eight measuring locations wereevenly distributed over the four quadrants of the coordinatesystem. In the background, the system recorded the dis-tance between the clicked location and the correct one. Thesmaller this distance was, the more points the participantsreceived. A maximum of five points could be achieved witheach task. Overall, this resulted in a maximum of 40 pointsfor the pretest. The total pretest score was used for furtheranalysis.

Post-test. In the post-test, the subjects had to fulfil thesame task as in the pretest. The only difference was thateight new measuring locations were chosen. Like in thepretest, these locations were distributed over the coordinatesystem so that there were exactly two in each quadrant ofthe coordinate system. The system again recorded thedistance between the clicked location and the correctone. A maximum of five points could be achieved witheach task. Overall, this resulted in a maximum of 40 pointsfor the post-test. For further analysis, the total post-testscore was used.

5. Training conditions

Between the pretest and the post-test, the training tookplace. To avoid time pressure negatively impacting learn-ing, participants could freely choose when to move on tothe next training task. Each training provided informationonly on the correct assignment of the measurement locationto the diagram. Subjects were distributed randomly to thefollowing three conditions:

• In the animation group, a partially transparent dy-namic animation visualized the transfer of the labo-ratory system including the measurement location tothe diagram. From the time the transformed laboratorysystem matched the coordinate system of the diagram,the correct point in the diagram remained marked.Later the animation showed how to transfer thecoordinate system of the diagram back to the labo-ratory system of the experiment. Figure 4 depictsvideo stills of an animation for a measurementlocation in the fourth quadrant. The training consistedof eight animations showing how to transfer themeasurement locations of the pretest. By clickingthe “Next” button, the subjects could move on to thenext animation.

• In the interactive group, the subjects were asked to dothe pretest tasks again. This time, when they clickedon the diagram, the point they clicked was marked inblack. Simultaneously, the correct position wasmarked in red (see Fig. 5). The markers remainedin their positions until the subjects clicked the “Next”button to proceed to the next training task.

• In the reference group, for each of the eight meas-urement locations of the pretest, the correct solutionswere presented. The correct points were marked in thediagram. By clicking the “Next” button, the subjectscould move through the eight solutions. Figure 6shows the marking for the first of the eight measure-ment locations. The marking of the correct positioncorresponds to the behavior when in computer-basedexperiments, the measurement values are auto-matically documented at the correct position in thediagram. This group provides a reference level. In

FIG. 4. Illustration of the animation as a series of still images captured at different points in time. The animation dynamicallytransforms the laboratory system to the coordinate system of the diagram.

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-7

relation to this level, the effects of the other twotrainings were assessed.

D. Procedure

The investigation was part of class visits at the Chair ofPhysics Education at the LMUMunich. The classes visitedthe university on different days.At the beginning of each visit, all students took the CRT.

An automated slide show led through the manual of the test.After the subjects’ questions were answered, the test wascarried out. Testing time was limited to 3 min. Anacoustical signal indicated the end. After this, the subjectsworked in groups on physics experiments using Arduinos.Those experiments required no documentation of meas-urement results so that the results of the investigation werenot falsified.While the others continued to work on their experiments,

a researcher led the subjects one by one into a separateroom in which a computer workstation and the eye trackingsystem were set up. After calibration of the eye trackingsystem, the students started to work with the computerapplication. From this point onwards, the eye trackerrecorded the eye movements.The computer application was separated into four parts:1. Part one introduced the user interface, which con-

tains a view of the experimental setup including theaxes of the laboratory system and the diagram to

which the measurement locations should be trans-ferred (see Fig. 1). On-screen text described theindividual components. The introduction finishedwith two example tasks where students could prac-tice assigning measurement locations to the diagram.The first part lasted an average of 185.1 sec.

2. Part two of the application consisted of the pretest.The pretest lasted an average of 98.0 sec.

3. In part three, the training was carried out. Participantswere randomly assigned to one of three trainingconditions. The training lasted on average 108.5 sec.

4. In part four, the subjects worked on the post-test.This part took an average of 90.1 sec.

In total, each student worked with the application for anaverage of 481.7 sec.

E. Statistical procedures

1. Pearson’s product-moment correlation coefficient

To assess the relationship between the CRT score and thepretest score as well as between the CRT score and theperformance gain, Pearson’s correlation coefficients werecalculated. By using boxplots, the data was checkedbeforehand for outliers. Also, the assumptions of linearityand homoscedasticity were tested using a scatter plot. Toprove the assumption of bivariate normality, Henze-Zirkler’s test was calculated.

FIG. 5. Screenshot of the first training task in the interactive group. After clicking on the diagram, the clicked position is marked inblack and the correct position appears in red.

FIG. 6. Screenshot of the first part of the training in the reference group. The correct position is marked in red in the diagram.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-8

2. ANCOVA

Following the suggestions of Dimitrov and Rumrill [68],a one-way analysis of covariance (ANCOVA) was con-ducted to evaluate the results of the pretest and the post-test.The pretest score was used as a covariate and the post-testscore as the dependent variable. In this way, the variancethat existed in the pretest is corrected in the post-test score.Before calculating the ANCOVA, the underlying assump-tions were checked. Data were searched for outliers usingboxplots. The Shapiro-Wilk test showed whether data wasnormally distributed for each group. Furthermore, thehomogeneity of variance was tested by calculating Levene’stest. The independence of the covariate from the groupmembership was investigated by using an ANOVA. To testwhether the assumption of homogeneity of regression slopeswas met, a customized ANCOVA model was calculated thatincluded the interaction between the covariate and theindependent variable.Contrasts for pairwise comparisons allowed the analysis

of the results of the ANCOVA in more detail.

3. Kruskal-Wallis test

The time on task and the numbers of eye fixations andsaccades were not normally distributed. To investigatedifferences between the groups in those variables, Kruskal-Wallis tests were calculated. To examine the differences inmore detail, Mann-Whitney U tests were used for pairwisecomparisons of the medians. In accordance with Divineet al. [69], the distributions of the dependent variable werechecked for differences between the groups. Therefore,Kolmogorov-Smirnov tests were calculated. For reportingthe effects of the pairwise comparisons, a Bonferronicorrection was taken into account.

F. Preliminary analysis

To check whether the randomization of the 119 subjectsinto the animation group, interactive group, and referencegroup was successful, the pretest results were examined fordifferences between the training groups.A one-way ANOVA was conducted to assess if there

were differences in the pretest score. There were nooutliers, according to inspection with a boxplot. Datawas normally distributed for each group (Shapiro-Wilktest, p > 0.05) and the assumption of homogeneity ofvariance (Levene’s test, p > 0.05) was met.The mean score in the pretest decreased from the

reference group (M ¼ 11.20, SD ¼ 5.73) via the interac-tive group (M ¼ 10.78, SD ¼ 5.64) to the animation group(M ¼ 10.39, SD ¼ 4.73).There was no significant difference in the pretest score

between training conditions, Fð2; 116Þ ¼ 0.217, p > 0.05,partial η2 ¼ 0.004. This indicates that the randomizationwas successful.

III. RESULTS

A. Spatial ability and task performance

To answer RQ1, Pearson’s correlation coefficient wascalculated to assess the relationship between the CRTscores and the scores in the pretest. It was suspected thatthere is a positive correlation between students’ CRT scoresand the pretest scores.There were no outliers in the data; also, the assumptions

of linearity, homoscedasticity and bivariate normality(Henze-Zirkler’s test, p > 0.05) were fulfilled. A smallpositive correlation between the CRT score and the pretestscore, r ¼ 0.234, N ¼ 119, p (one-tailed)< 0.01, with anR2 ¼ 0.055, was found. This shows that 5.5% of thevariability in the CRT score is shared with the pretest score.

B. Training effect on task performance

To answer RQ2, it was tested if the type of trainingaffected post-test performance. To avoid the influence oftime pressure, the subjects were free to decide when tomove on to the next training task. They had enough time tothink about all the details that seemed relevant to them.The average duration of training differed between thegroups. The animation group spent the most time ontask (M ¼ 149.2), followed by the interactive group(M ¼ 103.5) and the reference group (M ¼ 75.0). Thetime is given in seconds. The time on task was not normallydistributed for each group; therefore, a Kruskal-Wallis testwas calculated to check whether there are significantdifferences in the processing time of the training.Indeed, the processing time was significantly affected by

group membership, Hð2Þ ¼ 78.68, p < 0.001. To examinethe differences between the groups in more detail, Mann-Whitney U tests were used. Considering a Bonferronicorrection, all effects are reported at a 0.0167 level ofsignificance.A Mann-Whitney U test was calculated to determine if

there were differences in the processing time of the trainingbetween the animation group and the reference group. Thedistributions differed between both groups (Kolmogorov-Smirnov, p < 0.05). There was a statistically significantdifference in the processing time of the training between theanimation group (Mrank ¼ 59.5) and the reference group(Mrank ¼ 20.5),U¼ 0.0, z¼−7.60,p < 0.001, r ¼ −0.86.Also, a Mann-Whitney U test was calculated to deter-

mine if there were differences in the processing time of thetraining between the interactive group and the referencegroup. The distributions did not differ between both groups(Kolmogorov-Smirnov, p > 0.05). There was a statisticallysignificant difference in the median of the processingtime of the training between the interactive group(Mdn ¼ 106.1) and the reference group (Mdn ¼ 72.26),U ¼ 324.5, z ¼ −4.68, p < 0.001, r ¼ −0.52.A Mann-Whitney U test was calculated to determine if

there were differences in the processing time of the training

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-9

between the animation group and the interactive group. Thedistributions differed between both groups (Kolmogorov-Smirnov, p < 0.05). There was a statistically significantdifference in the processing time of the training between theanimation group (Mrank ¼ 57.5) and the interactive group(Mrank¼ 23.8),U¼113.0, z¼−6.54, p<0.001, r¼−0.74.The mean score in the post-test decreased from the

interactive group (M ¼ 15.12, SD ¼ 5.97) via the refer-ence group (M ¼ 12.90, SD ¼ 5.07) to the animationgroup (M ¼ 12.84, SD ¼ 4.84). Figure 7 depicts the meansof the pretest and post-test scores for each group and thecorresponding standard errors.A one-way ANCOVAwas calculated to assess the effect

of training condition on the post-test score while control-ling for the pretest score. There were no outliers. Data wasnormally distributed for each group (Shapiro-Wilk test,p > 0.05) and the assumption of homogeneity of variance(Levene’s test, p > 0.05) as well as homogeneity ofregression slopes was met. Also, the preliminary analysisshowed that the covariate did not differ between groups.The covariate (pretest score) was significantly related to

the post-test score, Fð1; 115Þ ¼ 89.306, p < .001, r ¼ .66.There was also a significant effect of the training conditionon the post-test score when controlling for the pretest score,Fð2; 115Þ ¼ 4.42, p < 0.05, partial η2 ¼ 0.071. Accordingto Cohen [70], this corresponds to a medium effect.Planned contrasts revealed that belonging to the interactive

group significantly increased the post-test score in compari-son to belonging to the reference group, tð115Þ ¼ −2; 796,p < 0.01, r ¼ 0.25, d ¼ 0.521. In contrast, belonging to theanimation group did not significantly increase post-testperformance in comparison to belonging to the referencegroup, tð115Þ ¼ 0.51, p > 0.05.

C. Spatial ability and performance gain

For the examination of RQ3, Pearson’s correlationcoefficient was calculated to assess the relationship

between the CRT score and the performance gain in thetraining groups. The three groups were analyzed separately.In the animation group, there were no outliers in the

data; also, the assumptions of linearity, homoscedasticity,and bivariate normality (Henze-Zirkler’s test, p > 0.05)were fulfilled. There was no significant relationship found(p > 0.05).In the interactive group, there were no outliers in the

data; also, the assumptions of linearity, homoscedasticityand bivariate normality (Henze-Zirkler’s test, p > 0.05)were fulfilled. Like in the animation group, no significantrelationship was found (p > 0.05).In the reference group, there were no outliers in the data;

also, the assumptions of linearity, homoscedasticity andbivariate normality (Henze-Zirkler’s test, p > 0.05) werefulfilled. Like for the other two groups, no significantrelationship was found (p > 0.05).

D. Analysis of eye gaze pattern during training

To answer RQ4 for all three training conditions, the samearea of interest (AOI) was defined. This AOI corresponds tothe area within the red dashed line in Fig. 1. It contains theexperimental setting and the diagram. The decision tochoose this AOI was made for two reasons:

• While quasistatic stimuli are shown in the training ofthe interactive group and in the reference group, thetraining stimulus in the animation group movesdynamically between the laboratory system and thecoordinate system. So that the eye movements of thethree training conditions can be compared, the definedAOI must be suitable for all of them. The smallestsuitable option is therefore the area marked in redin Fig. 1.

• The AOI should not be chosen larger as it shouldrecord only eye movements directly related to process-ing the training stimulus. Eye movements that enter orleave the area where the training stimulus is presenteddo not connect any causally related elements of thetraining and should therefore be filtered out by thechoice of the AOI. For this reason, the AOI should bechosen as small as possible and should contain onlythe training stimulus.

As the eye tracking data was incomplete for six subjects,they were removed from the data set for further analysis.Those missing values were due to dropouts in the eyetracking system. The subjects were still almost evenlydistributed across the groups (Nanimation ¼ 38, Ninteractive ¼37, Nreference ¼ 38).

1. Analysis of the total number of fixations in the AOI

First the investigation focused on looking for differencesin the total number of fixations in the AOI between the threegroups. Data were not normally distributed for each group;therefore, a Kruskal-Wallis test was calculated. The totalnumber of fixations was significantly affected by group

FIG. 7. Average scores of the pretest and the post-test for thethree training groups. The error bars represent the standard errorof the mean.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-10

membership, Hð2Þ ¼ 56.51, p < 0.001. As before whenanalyzing differences in the processing time of the training,Mann-Whitney U tests were calculated to investigate thedifferences in the total number of fixations in the AOI inmore detail. Again, a Bonferroni correction was taken intoaccount.A Mann-Whitney U test was calculated to determine if

there were differences in the total number of fixationsbetween the animation group and the reference group.The distributions did not differ between both groups(Kolmogorov-Smirnov, p > 0.05). There was a statisticallysignificant difference in the median of the total number offixations between the animation group (Mdn ¼ 201.5) andthe reference group (Mdn ¼ 88.0), U ¼ 41.0, z ¼ −7.08,p < 0.001, r ¼ −0.81. According to Cohen [70], thiscorresponds to a large effect.Also, a Mann-Whitney U test was calculated to deter-

mine if there were differences in the total number offixations between the interactive group and the referencegroup. The distributions did not differ between both groups(Kolmogorov-Smirnov, p > 0.05). There was a statisticallysignificant difference in the median of the total number offixations between the interactive group (Mdn ¼ 138.0) andthe reference group (Mdn ¼ 88.0), U ¼ 282.5, z ¼ −4.46,p < 0.001, r ¼ −0.51. According to Cohen [70], thiscorresponds to a large effect.A Mann-Whitney U test was calculated to determine if

there were differences in the total number of fixationsbetween the animation group and the interactive group.The distributions did not differ between both groups(Kolmogorov-Smirnov, p > 0.05). There was a statisticallysignificant difference in the median of the total number offixations between the animation group (Mdn ¼ 201.5) andthe interactive group (Mdn¼ 138.0),U¼341.0, z¼−3.84,p < 0.001, r ¼ −0.44. According to Cohen [70], thiscorresponds to a medium effect.

2. Analysis of the saccadic distribution inside the AOI

As shown in the previous section, there are significantdifferences in the total number of fixations between thethree training conditions. The three training conditionsdiffer only in the training stimulus. The rest of the userinterface is identical. Because of this common underlyingvisual structure, eye gaze patterns should show similaritiesin all of the three training conditions. However, the totalnumber of fixations neither showed any of those structuralsimilarities during training nor reflected the differences intest performance between training groups. To analyzesimilarities and differences of the eye pattern in moredetail, the distribution of saccades, which led to thefixations, was investigated. Therefore, polar diagrams werecreated for each training task as follows.The diagrams visualize saccades in dependence on their

direction, length and number. According to their direction,saccades are classified in angular fields of 30° each. The

saccadic length is encoded in the radial distance to theorigin of the diagram. When visualizing saccades this way,at some positions in the diagram, the markers wouldoverlap. To take this into account, colored markers wereused to indicate the frequency of saccades which are ofabout the same length and direction. As shown above, thetotal number of fixations and, accordingly, the number ofsaccades that lead to these fixations differed between thetraining conditions. To improve the comparability of thediagrams, the number of saccades was normalized for eachtraining condition. The color of the marked saccadecorresponds to the value of a probability density functionthat describes the relative frequency of each saccade in agiven training condition. These probability density func-tions of the saccades were estimated by using a kerneldensity estimation. In the following section, the diagramsof the first of the eight training tasks are discussed for eachof the three training groups. All the other seven trainingtasks showed a similar structure.Figure 8 presents the saccadic distributions during

training task 1 for the three training conditions. It can beseen that in all training conditions, most saccades can beclassified into the two angular fields of 345°–15° and 165°–195°. What is remarkable is that the saccadic distribution inthese sections differs between the training conditions.While in the animation group the probability for havingshort saccadic amplitudes is higher than for having longsaccadic amplitudes, in the interactive group and thereference group long saccadic amplitudes are more likely.To analyze those differences in more detail, the saccadic

distributions for the two angular fields (345°–15° and 165°–195°) were investigated. Those fields contained most of thesaccades. Again, kernel density estimations were calculatedfor the saccades in each of the two angular fields for each ofthe three training groups and for all of the eight trainingtasks. In total 48 functions were calculated. For the firsttraining task, the probability density functions in theangular field 345°–15° are shown in blue in Fig. 9. Thefigure additionally contains histograms showing, in grada-tions of one centimeter, the relative frequency of saccadesin the corresponding length interval.The probability density functions (Fig. 9) contain two

local maxima at about the same saccadic lengths. Betweenthose maxima, the probability density reaches a localminimum. This indicates that saccadic movements canbe divided into two categories according to their length. Atotal of 42 of the 48 calculated functions showed thisstructure with a local minimum between the two localmaxima. The remaining six functions, all belonging to theanimation group, formed inflection points instead. Thisdeviation was caused by the large number of short saccadescompared to the number of long saccades in the anima-tion group.A threshold was defined to divide the saccades into two

categories according to their length. For this purpose, the

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-11

mean was calculated from the positions of the 42 localminima (M ¼ 6.80 cm, SD ¼ 1.71 cm).This division indicates further differences between the

training groups. In the animation group, most of thesaccades have a length less than 6.8 cm. In the interactivegroup and the reference group, saccadic lengths above thisthreshold are more likely. As shown before, the totalnumber of fixations in the animation group was signifi-cantly higher than in the two other training groups.Consequently, most of those fixations in the animationgroup are the end points of saccades that have a lengthbelow 6.8 cm. Conversely, in the interactive group and thereference group, the probability is higher that the fixationsare end points of saccades of length larger than 6.8 cm. Asthe interactive group performed best in the post-test,especially those saccades could indicate processes benefi-cial for improving the assignment task. This assumptionwould be reinforced if there were differences between thegroups in the total number of saccades of length larger than6.8 cm. As the diagrams were normalized, they do not give

this information. For this reason, the next section deals withthe question of whether significant differences existbetween the training groups in the total number of saccadeslonger than 6.8 cm.

3. Analysis of the total number of saccades of lengthlarger than 6.8 cm

The following examines the subjects’ total number ofsaccadic movements with a length larger than 6.8 cm thatoccurred during the training tasks in the AOI. This totalnumber was calculated by summing over all training tasks.As the total number of saccades inside the AOI that werelonger than 6.8 cm was not normally distributed for eachgroup, a Kruskal-Wallis test was calculated.The total number of saccades longer than 6.8 cm was sig-

nificantly affected by group membership, Hð2Þ ¼ 47.482,p < 0.001.To examine the differences between the groups more

closely, Mann-Whitney U tests with Bonferroni correctionwere carried out again (see above).

FIG. 9. The subfigures show the probability density function of saccades (blue line) in the angular field 345°–15° according to theirsaccadic length (in cm) during training task 1 in the animation group (a), interactive group (b), and reference group (c). While in theanimation group the probability for shorter saccades is higher than for longer saccades, the interactive group and reference group showthe opposite behavior.

FIG. 8. The diagrams show the lengths and angular distributions of saccades during training task 1 in the animation group (a),interactive group (b), and reference group (c). Saccades were grouped into angular fields depending on their direction. The saccadiclength in centimeters is encoded in the radial distance to the origin. The probability density of the saccades is color coded.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-12

A Mann-Whitney U test was calculated to determine ifthere were differences in the total number of saccadeslonger than 6.8 cm between the animation group and thereference group. The distributions did not differ betweenboth groups (Kolmogorov-Smirnov, p > 0.05). There wasa statistically significant difference in the median of thetotal number of saccades longer than 6.8 cm between theanimation group (Mdn ¼ 27.0) and the reference group(Mdn ¼ 36.0), U ¼ 398.5, z ¼ −3.37, p < 0.01, r ¼−0.39. According to Cohen [70], this corresponds to amedium effect.Also, a Mann-Whitney U test was calculated to deter-

mine if there were differences in the total number ofsaccades longer than 6.8 cm between the interactive groupand the reference group. The distributions did not differbetween both groups (Kolmogorov-Smirnov, p > 0.05).There was a statistically significant difference in the medianof the total number of saccades longer than 6.8 cm betweenthe interactive group (Mdn ¼ 59.0) and the referencegroup (Mdn ¼ 36.0), U ¼ 245.0, z ¼ −4.86, p < 0.001,r ¼ −0.56. According to Cohen [70], this corresponds to alarge effect.A Mann-Whitney U test was calculated to determine if

there were differences in the total number of saccadeslonger than 6.8 cm between the animation group and theinteractive group. The distributions did not differ betweenboth groups (Kolmogorov-Smirnov, p > 0.05). There wasa statistically significant difference in the median of thetotal number of saccades longer than 6.8 cm between theanimation group (Mdn ¼ 27.0) and the interactive group(Mdn ¼ 59.0), U ¼ 124.5, z ¼ −6.13, p < 0.001, r ¼−0.71. According to Cohen [70], this corresponds to alarge effect.These differences therefore only partially reflect the

assumption that the information acquired through longsaccades is decisive for success in the training. The inter-active group,whose trainingwas themost successful, had themost long saccadic movements, as expected. The number oflong saccadic movements in the animation group and thereference group differed, while the training was almostequally successful. A possible explanation for the deviationfrom the expectation is given in the discussion.

IV. DISCUSSION AND CONCLUSIONS

Results regarding RQ1 showed a significant positivecorrelation between the pretest score and the CRT score.This indicates that the pretest task is indeed related to thespatial thinking ability of the subjects. Especially studentswith weak spatial skills need more support when training inthe documentation of measured values. However, the sizeof the correlation was small. A larger correlation coefficientwas not to be expected as the task in the CRTwas of lowercomplexity than the task in the pretest was: The CRT justmeasures the subjects’ ability to transform two-dimensional

figures into each other using rotation and reflection.Similarly, in the pretest, the laboratory system and thecoordinate system must first be related to each other. Inaddition, the information obtained this way must be used totransfer the measuring position in the laboratory system tothe correct position in the diagram. The results foundindicate that spatial rotation processes are involved whensolving the pretest task. According to Salomon [41], anexternal visualization could support the internal spatialtranslation process and contribute to a performance gain inthe post-test. Therefore, as described in Sec. I A, consid-ering animation as a possible supportive tool was justified.The results regarding RQ2 show that the three training

conditions differ in their influence on the post-test perfor-mance. Group membership affected the performance sig-nificantly with a medium effect size. Remarkably, themedium effect size resulted despite the short time on taskduring the intervention (M ¼ 108.5 sec).The animation group spent more time on tasks than the

reference group did. Accordingly, this shows that thesubjects took more time to complete the training. Theprocessing depth, however, was not improved as the post-test performance of the animation group did not differsignificantly from the results of the reference group.Therefore, the animation group did not profit significantlymore from training with animations than the referencegroup did from inspecting the correctly marked positions.The intended better support from the animation could notbe observed. Previous studies (see Sec. I A) reportedsimilar observations. Animation often attracted attentionbut did not necessarily contribute to learning. Also, thetransience of the animation limited the processing depth ofthe information. When discussing the results regardingRQ4, eye tracking data will help to interpret the ineffi-ciency of the training in the animation group.The interactive group spent more time on task than the

reference group did. This indicates that subjects in theinteractive group took more time to complete the trainingthan subjects in the reference group did. In contrast to theanimation group, the interactive group indeed outper-formed the reference group significantly in the post-test.The presented information in the reference group and theinteractive group concerning the correct point in thediagram did not differ in its content. The better performanceof the interactive group therefore shows that the interactivetask and the feedback helped to process the presentedinformation more deeply.The investigation of RQ3 showed that there is no

relationship between the spatial rotation ability of thelearners and the performance gain during the three train-ings. This result suggests that, in the case under inves-tigation, considering the spatial rotation abilities of thelearners is not necessary when choosing the most suitabletraining.

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-13

The analysis of RQ4 showed similarities and differencesin eye tracking measures during the three training con-ditions. Eye movements characteristic of the differenttrainings were identified. For the analysis, first the totalnumber of fixations within the defined AOI was inves-tigated. Results showed significantly more fixations duringtraining in the animation group than in the interactive groupand significantly more during training in the interactivegroup than in the reference group. This result is closelylinked to the differences in processing time describedabove. However, processing time alone does not provideinformation about whether the content of the training wasprocessed or whether the test subjects’ gaze moved outsidethe user interface or even outside the screen. An animationthat runs too slowly can, for example, due to boredom, leadthe test subjects to observe objects that are not part of thetraining. Therefore, compared to the time on task alone, thenumber of fixations within the AOI gives more accurateinsight. The high number of fixations in the animationgroup compared to the other two training conditions showsthat the animation worked best to attract visual attention. Inthe interactive group, there were fewer fixations than in theanimation group, however the number of fixations in thisgroup was larger than in the reference group. Thus,compared to the reference group, also the training in theinteractive group was successful in attracting visualattention.Most of the saccadic movements during training can be

assigned to the two angular fields from 345° to 15° andfrom 165° to 195° and correspond to eye movements in thehorizontal direction. Thus, further analysis concentrated onthe distribution of those saccadic movements. The structureof the calculated probability density functions (see Fig. 9for the probability density functions of the first training taskin the angular field 345° to 15°) indicated similarities andfurther differences in the eye movement between the threetraining conditions.The number of saccadic movements was shown to

consist of two components. For the division of the saccadesinto long saccadic movements and short saccadic move-ments, a saccadic length of 6.8 cm was derived from thedataset as a threshold. This length corresponds to approx-imately a quarter of the width of the AOI (half the distancebetween the origin of the laboratory system and the originof the coordinate system in the diagram) (see Fig. 1).Correspondingly, it is possible to differentiate between

two characteristic saccadic movements:• Short saccadic movements of length below 6.8 cm:These movements are more likely to occur withineither the illustration of the experimental setup or thediagram. This can involve, for example, those eyemovements that detect the position of the sensor inrelation to the laboratory system. Such short saccadicmovements are also required to follow the course ofthe animation. Because of their short length, these

saccades affect matching processes between the ex-periment and diagram less.

• Long saccadic movements of length larger than6.8 cm: These movements especially indicate match-ing processes between the experiment and the dia-gram. Such saccades are necessary, for example, tocompare locations in the laboratory system of theexperiment with positions in the coordinate system ofthe diagram.

The probability of long and short saccadic movementsdiffered between the groups (see Fig. 9). The majority ofthe saccades in the animation group, in contrast to theinteractive group and reference group, belong to thecategory of short saccadic movements. The recorded eyetracking data showed, that the reason for the increasedoccurrence of these movements is that the subjects in theanimation group followed the course of the animation. Incontrast, in the interactive group and the reference group,those saccades belonging to the group of long saccadicmovements occurred more often than the shorter ones (seeFig. 9). Consequently, most of the fixations in the anima-tion group were end points of short saccadic movements,while in the interactive group and the reference group, mostof the fixations were end points of long saccadic move-ments. This shows that the three training conditionsaffected the visual attention of the test subjects differently.Since the training of the interactive group was the mostsuccessful and many of the long saccadic movements wereobserved in this group, it was assumed that especially thedeeper processing of the information perceived throughthe longer saccadic movements is important for improvingthe assignment task during the training.The statistical analysis of the number of saccadic move-

ments of length above 6.8 cm showed that there aresignificantly more of them in the interactive group thanthere are in the reference group and that there aresignificantly more in the reference group than in theanimation group. Thus, the interactive task in the inter-active group led to the most eye movements comparing theexperiment and the diagram. The immediate feedbackencouraged further comparison between the experimentand diagram. The animation in the animation group,however, suppressed these eye movements in favor ofthe shorter saccadic movements with which the animationwas followed. Interestingly, even despite the fact that thetime on task in the animation group was longer than it wasin the interactive group and the reference group, the numberof saccadic movements of length of above 6.8 cm wassmallest in the animation group.It is therefore true for the interactive group that the

highest number of long saccadic movements occurred inthe most effective training. The subjects in this group hadby far the most long saccadic movements (Mdn ¼ 59.0).On the other hand, when comparing the animation group

and the reference group, the results are not so clear. The

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-14

reference group scored slightly better in the post-test thanthe animation group did, but this difference already existedin the pretest. Thus, no significant differences in trainingsuccess were found between these two groups, while therewere significantly more long saccadic movements in thereference group (Mdn ¼ 36.0) than in the animation group(Mdn ¼ 27.0). However, this difference in the number oflong saccadic movements is far smaller than that betweenthe interactive group and the reference group. That nosignificant difference in the effect of the training was foundcan therefore also be attributed to the number of longsaccadic movements in the two groups not being suffi-ciently different.In any case, the better performance of the interactive

group shows that processing information obtained throughthe long saccadic movements seems to be particularlyimportant for improving the assignment task. Long sac-cadic movements are necessary to compare positions in thelaboratory system and positions in the coordinate system ofthe diagram. These results suggest that a higher number ofsaccadic movements leads to better integration of theinformation presented and thus to better performance. Inprevious research, this relationship was not always evident(see Sec. I A). As an explanation, it was assumed that theperceived elements are initially stored in memory.Subsequently, integration takes place without the needfor comparative eye movements [62]. In the study pre-sented in this paper, most long saccadic movementsoccurring in the most successful training could indicatethat the two elements to be related to each other (laboratorysystem and diagram) were too complex to be stored inmemory at the same time. This would necessitate saccadiceye movements to integrate the two elements.However, it should be noted that it is not just about the

number of saccadic movements but rather the activeprocessing of the information obtained through them.This processing was facilitated by the immediate feedback.The subjects were immediately made aware of their errors,which prompted them to investigate the reason for theirmistakes. A large number of saccadic movements withoutthe corresponding cognitive activation of the test subjectswould not necessarily have had the same effect.For example, the passivity of the test subjects could be

responsible for the poorer performance in the referencegroup. The test subjects are immediately shown the correctpositions in the diagram. Passive test subjects may there-fore not even be aware that they would have chosen anincorrect point on their own initiative.During the training in the animation group, no indica-

tions were found that the animation might have helped tosupport the translation process between the laboratorysystem and diagram. However, this could be due to theanimation not being similar enough to the internal cognitivetranslation process it should support (see Ref. [41]). In theanalysis of the gaze data of the animation group, the longer

saccadic movements necessary to link elements in thelaboratory system and the coordinate system were rarelyobserved. Instead, many short saccadic movements tookplace to follow the dynamics of the animation. Attentionwas bound by the dynamics, which obviously did not leadto an increase in performance.However, the results do not imply that animation is a

poor teaching tool in general. Only for the described area ofapplication, involving the most accurate documentationpossible, were there no advantages over the other twotraining type. If only a visualization of the relationshipbetween the laboratory system and the coordinate system inthe diagram is intended, or if the gaze of the learners is to beguided, an animation can be the means of choice.Overall, the training in the interactive group was best

suited to support the documentation of measured vectorquantities in diagrams when experimenting with computer-based experiments. An example showing a possible imple-mentation in a virtual laboratory can be found in Ref. [71].

V. LIMITATIONS

For the investigation, the eye tracking data of thetrainings in the animation group, the interactive group,and the reference group were compared. The trainings inthe interactive group and the reference group show quasi-static training stimuli, while the training in the animationgroup contains a dynamic one. In general, it is difficult tocompare eye tracking data from static and dynamic stimuli.In the present study, however, care was taken to ensure thatthe evaluated parameters are comparable. On the one hand,this applies to the total number of fixations as this quantitywas used only as a measure of visual attention. On the otherhand, when interpreting the saccadic movements in theanimation group, the following was considered.For saccadic movements in the animation group, three

processes are responsible: saccadic movements within theanimated element, saccadic movements between thedynamic and the static element of the display, and saccadicmovements within the static element of the display.When interpreting the results found, it was assumed that

these three categories have the following influence onsaccadic movements:

• Saccadic movements within the animated element:The animation dynamically transforms an entire partof the display, namely, the laboratory system of theexperimental setup, into the part of the user interfacewhich contains the diagram. Thus, in terms of sac-cadic length, eye movements that occur within theanimated element are comparable to those that takeplace within the laboratory system. Eye movementsthat follow the animation should therefore belong tothe category of short saccadic movements.

• Saccadic movements between the dynamic and thestatic elements of the display: The animation moveswith uniform speed between the laboratory system and

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-15

the diagram. Therefore, saccadic movements that takeplace between the animated element and the labora-tory system as well as between the animated elementand the diagram were assumed to be distributed evenlyover all lengths. Consequently, this effect increases thetotal number of observed saccadic movements butdoes not impair the distinction between the categoriesof long and short saccadic movements.

• Saccadic movements within the static element of thedisplay: Because the animation is transparent, theunderlying structure of the application can be seen atall times. Since this structure is the same as in thequasi-static trainings, such eye movements are com-parable between all training groups.

The results show that the number of long saccadicmovements was highest in the most successful training.The procedure that led to this statement was purelyexploratory.This is only a first hint that there is a relationship

between the number of long saccadic movements and theincrease in performance in the task under consideration. Toverify this, further investigations are necessary. Theseshould also be extended to other similar measurementdocumentation tasks (e.g., the documentation of measured

values of the directional characteristic of a receiver dipole,of the directional characteristic of a speaker box, or of thespatial distribution of luminance for different LED lamps).The results described were examined in a situation where

the laboratory system and coordinate system could beconverted into each other by using the geometric operationsof rotation and translation. A generalization to morecomplex situations is conceivable but not a matter ofcourse. Further research has to show whether the resultsfound also apply if the axes are coded differently (forexample, if a movement is visualized in a time-velocitydiagram).

VI. FUTURE PROSPECTS

In the future, further research has to show if multimediafeatures in computer-based experiments can also supportmore general documentation processes. Additionally, thetransferability of the results found to real experimentsshould be investigated. It is thinkable that augmentedreality features in three-dimensional space could supportthe transfer between the laboratory system and the coor-dinate system of a diagram in a similar way as shown herefor two-dimensional computer-based experiments.

[1] Committee on a Conceptual Framework for New K-12Science Education Standards; National Research Council,A Framework for K-12 Science Education: Practices,Crosscutting Concepts, and Core Ideas (The NationalAcademies Press, Washington, DC, 2012).

[2] N. D. Finkelstein, W. K. Adams, C. J. Keller, P. B. Kohl,K. K. Perkins, N. S. Podolefsky, S. Reid, and R. LeMaster,When learning about the real world is better done virtually:A study of substituting computer simulations for laboratoryequipment, Phys. Rev. ST Phys. Educ. Res. 1, 010103(2005).

[3] G. Martínez, F. L. Naranjo, Á. L. Perez, M. I. Suero, andP. J. Pardo, Comparative study of the effectiveness ofthree learning environments: Hyper-realistic virtual simu-lations, traditional schematic simulations and traditionallaboratory, Phys. Rev. ST Phys. Educ. Res. 7, 020111(2011).

[4] V. Potkonjak, M. Gardner, V. Callaghan, P. Mattila, C.Guetl, V. M. Petrović, and K. Jovanović,Virtual laborato-ries for education in science, technology, and engineering:A review, Comput. Educ. 95, 309 (2016).

[5] A. Hofstein and V. N. Lunetta, The laboratory in scienceeducation: Foundations for the twenty-first century, Sci.Educ. 88, 28 (2004).

[6] S. Ainsworth, The functions of multiple representations,Comput. Educ. 33, 131 (1999).

[7] S. Ainsworth, DeFT: A conceptual framework for consid-ering learning with multiple representations, Learn. Instr.16, 183 (2006).

[8] R. Trumper, The Physics laboratory—a historical overviewand future perspectives, Sci. Educ. 12, 645 (2003).

[9] S. R. Singer, M. L. Hilton, and H. A. Schweingruber,America’s Lab Report (National Academies Press,Washington, DC, 2006).

[10] S. Olson and D. G. Riordan, Engage to Excel: ProducingOne Million Additional College Graduates with Degrees inScience, Technology, Engineering, and Mathematics, Re-port to the President (Executive Office of the President,Washington, DC, 2012).

[11] J. Kozminski, H. J. Lewandowski, N. Beverly, S. Lindaas,D. Deardorff, A. Reagan, R. Dietz, R. Tagg, M. Eblen-Zayas, J. Williams, R. Hobbs, and B. M. Zwickl, AAPTRecommendations for the Undergraduate Physics Labo-ratory Curriculum (AAPT, College Park, MD, 2014).

[12] N. G. Holmes and C. E. Wieman, Introductory physicslabs: We can do better, Phys. Today 71, No. 1, 38 (2018).

[13] M. Emden and E. Dumfleth, Assessing students’ exper-imentation processes in guided inquiry, Int. J. Sci. Math.Educ. 14, 29 (2016).

[14] H. Theyßen, M. Dickmann, K. Neumann, H. Schecker, andB. Eickhorst, Measuring experimental skills in largescale assessment: A simulation-based test instrument, in

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-16

Electronic Proceedings of ESERA 2015 Conference. Sci-ence Education Research: Engaging Learners for Sustain-able Future, edited by J. Lavonen, K. Juuti, J. Lampiselkä,A. Uitto, and K. Hahl (co-ed. Dolin, Jens; Kind, Per)(University of Helsinki, Helsinki, 2016), pp. 598–606.

[15] E. M. Smith, M.M. Stein, C. Walsh, and N. G. Holmes,Direct Measurement of the Impact of Teaching Experimen-tation in Physics Labs, Phys. Rev. X 10, 011029 (2020).

[16] S. Becker, P. Klein, A. Gößling, and J. Kuhn, Using mobiledevices to enhance inquiry-based learning processes,Learn. Instr. 69, 101350 (2020).

[17] M. C. Linn and A. C. Petersen, Emergence and characteri-zation of sex differences in spatial ability: A meta-analysis,Child Development 56, 1479 (1985).

[18] J. B. Carroll, Human Cognitive Abilities (CambridgeUniversity Press, New York, NY, 1993).

[19] N. S. Newcombe and T. F. Shipley, Thinking about spatialthinking: New typology, new assessments, in StudyingVisual and Spatial Reasoning for Design Creativity, editedby J. S. Gero (Springer Netherlands, Dordrecht, 2015),pp. 179–192.

[20] M. Cole, C. Cohen, J. Wilhelm, and R. Lindell, Spatialthinking in astronomy education research, Phys. Rev. Phys.Educ. Res. 14, 010139 (2018).

[21] R. B. Ekstrom, J. W. French, H. H. Harmann, and D.Dermen, Manual for Kit of Factor-Referenced CognitiveTests (Educational Testing Service, Princeton, NJ, 1976).

[22] J. C. Castro-Alonso, P. Ayres, M. Wong, and F. Paas,Visuospatial tests and multimedia learning, in Advances inCognitive Load Theory, edited by S. Tindall-Ford, S.Agostinho, and J. Sweller (Routledge, New York, 2019),pp. 89–100.

[23] S. G. Vandenberg and A. R. Kuse, Mental rotation, a grouptest of three-dimensional spatial visualization, Percept.Mot. Skills 47, 599 (1978).

[24] M. Peters, B. Laeng, K. Latham, M. Jackson, R. Zaiyouna,and C. Richardson, A redraw Vandenberg and Kuse mentalrotations test: Different versions and factors that affectperformance, Brain Cognit. 28, 39 (1995).

[25] M. Hegarty and V. K. Sims, Individual differences inmental animation during mechanical reasoning, Mem.Cogn. 22, 411 (1994).

[26] M. Kozhevnikov, M. Hegarty, and R. E. Mayer, Revisingthe visualizer-verbalizer dimension: Evidence for twotypes of visualizers, Cognit. Instr. 20, 47 (2002).

[27] M. Kozhevnikov and R. Thornton, Real-time data display,spatial visualization ability, and learning force and motionconcepts, J. Sci. Educ. Technol. 15, 111 (2006).

[28] M. Stieff, Mental rotation and diagrammatic reasoning inscience, Learn. Instr. 17, 219 (2007).

[29] M. Kozhevnikov, M. A. Motes, and M. Hegarty, Spatialvisualization in physics problem solving, Cogn. Sci. 31,549 (2007).

[30] S. Küchemann, P. Klein, H. Fouckhardt, S. Gröber, and J.Kuhn, Students’ understanding of non-inertial frames ofreference, Phys. Rev. Phys. Educ. Res. 16, 010112 (2020).

[31] B. B. de Koning, H. K. Tabbers, R. M. J. P. Rikers, and F.Paas, Towards a framework for attention cueing in instruc-tional animations: guidelines for research and design, Educ.Psychol. Rev. 21, 113 (2009).

[32] T. N. Höffler and D. Leutner, Instructional animationversus static pictures: A meta-analysis, Learn. Instr. 17,722 (2007).

[33] M. Betrancourt and B. Tversky, Effect of computeranimation on users’ performance: A review, Trav. Hum.63, 311 (2000).

[34] S. Berney and M. Betrancourt, Does animation enhancelearning? A meta-analysis, Comput. Educ. 101, 150 (2016).

[35] R. K. Lowe and W. Schnotz, Animation Principles inMultimedia Learning, in The Cambridge Handbook ofMultimedia Learning, 2nd ed., edited by R. Mayer (Cam-bridge University Press, New York, 2014), pp. 513–546.

[36] R. A. Abrams and S. E. Christ, Motion onset capturesattention, Psychol. Sci. 14, 427 (2003).

[37] A. P. Hillstrom and Y.-C. Chai, Factors that guide or disruptattentive visual processing, Comput. Hum. Behav. 22, 648(2006).

[38] S. Kriz and M. Hegarty, Top-down and bottom-upinfluences on learning from animations, Int. J. Hum-Comput. St. 65, 911 (2007).

[39] J. R. Kirby, Mental representations, cognitive strategies,and individual differences in learning with animations:Commentaries on sections one, and two, in Learning withAnimation, edited by R. Lowe and W. Schnotz (CambridgeUniversity Press, New York, 2008), pp. 165–182.

[40] B. B. de Koning, H. K. Tabbers, R. M. J. P. Rikers, and F.Paas, Attention guidance in learning from a complexanimation: Seeing is understanding?, Learn. Instr. 20,111 (2010).

[41] G. Salomon, Interaction of Media Cognition and Learning(Erlbaum, Hillsdale, NJ, 1979).

[42] K. A. Gilligan, M. S. C. Thomas, and E. K. Farran, Firstdemonstration of effective spatial training for near transferto spatial performance and far transfer to a range ofmathematics skills at 8 years, Dev. Sci. 23, e12909 (2019).

[43] M. T. H. Chi and R. Wylie, The ICAP framework: Linkingcognitive engagement to active learning outcomes, Educ.Psychol. 49, 219 (2014).

[44] T. Feldmann, Multimedia (Chapman & Hall, London,1994).

[45] R. E. Clark and D. F. Feldon, Ten common but questionableprinciples of multimedia learning, in The Cambridge Hand-book of Multimedia Learning, edited by R. Mayer (Cam-bridge University Press, Cambridge, 2014), pp. 151–173.

[46] S. Domagk, R. N. Schwartz, and J. L. Plass, Interactivity inmultimedia learning: An integrated model, Comput. Hum.Behav. 26, 1024 (2010).

[47] K. R. Koedinger and V. Aleven, Exploring the assistancedilemma in experiments with cognitive tutors, Educ.Psychol. Rev. 19, 239 (2007).

[48] V. J. Shute, Focus on formative feedback, Rev. Educ. Res.78, 153 (2008).

[49] J. Hattie and H. Timperley, The power of feedback, Rev.Educ. Res. 77, 81 (2007).

[50] F. M. van der Kleij, T. J. H. M. Eggen, C. F. Timmers, andB. P. Veldkamp, Effects of feedback in a computer-basedassessment for learning, Comput. Educ. 58, 263 (2012).

[51] H. M. Oonk and R. A. Abrams, New perceptual objectsthat capture attention produce inhibition of return, Psy-chon. Bull. Rev. 5, 510 (1998).

ANIMATION AND INTERACTIVITY IN … PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-17

[52] M.-L. Lai, M.-J. Tsai, F.-Y. Yang, C.-Y. Hsu, T.-C. Liu,S. W.-Y. Lee, M.-H. Lee, G.-L. Chiou, J.-C. Liang, andC.-C. Tsai, A review of using eye-tracking technology inexploring learning from 2000 to 2012, Educ. Res. Rev. 10,90 (2013).

[53] E. Alemdag and K. Cagiltay, A systematic review of eyetracking research on multimedia learning, Comput. Educ.125, 413 (2018).

[54] P. Klein, J. Viiri, S. Mozaffari, A. Dengel, and J. Kuhn,Instruction-based clinical eye-tracking study on the visualinterpretation of divergence: How do students look atvector field plots?, Phys. Rev. Phys. Educ. Res. 14,010116 (2018).

[55] P. Klein, S. Küchemann, S. Brückner, O. Zlatkin-Troitschanskaia, and J. Kuhn, Student understanding ofgraph slope and area under a curve: A replication studycomparing first-year physics and economics students,Phys. Rev. Phys. Educ. Res. 15, 020116 (2019).

[56] P. Klein, J. Viiri, and J. Kuhn, Visual cues improvestudents’ understanding of divergence and curl: Evidencefrom eye movements during reading and problem solving,Phys. Rev. Phys. Educ. Res. 15, 010126 (2019).

[57] M. A. Just and P. Carpenter, A theory of reading: from eyefixations to comprehension, Psychol. Rev. 87, 329 (1980).

[58] K. Holmqvist and R. Andersson, Eye Tracking: A Com-prehensive Guide to Methods, Paradigms, and Measures(Lund Eye-Tracking Research Institute, Lund, 2017).

[59] C. I. Johnson and R. E. Mayer, An eye movement analysisof the spatial contiguity effect in multimedia learning, J.Exp. Psychol. Appl. 18, 178 (2012).

[60] L. Mason, M. C. Tornatora, and P. Pluchino, Do fourthgraders integrate text and pictures in processing andlearning from an illustrated science text? Evidence fromeye-movement patterns, Comput. Educ. 60, 95 (2013).

[61] P. A. O’Keefe, S. M. Letourneau, B. D. Homer, R. N.Schwartz, and J. L. Plass, Learning from multiple repre-sentations: An examination of fixation patterns in a sciencesimulation, Comput. Hum. Behav. 35, 234 (2014).

[62] K. Scheiter and A. Eitel, Signals foster multimedia learningby supporting integration of highlighted text and diagramelements, Learn. Instr. 36, 11 (2015).

[63] A. Schüler, Investigating gaze behavior during processingof inconsistent text-picture information: Evidence for text-picture integration, Learn. Instr. 49, 218 (2017).

[64] N. Ott, R. Brünken, M. Vogel, and S. Malone, Multiplesymbolic representations: The combination of formula andtext supports problem solving in the mathematical field ofpropositional logic, Learn. Instr. 58, 88 (2018).

[65] C. Hoyer and R. Girwidz, A remote lab for measuring,visualizing and analysing the field of a cylindrical perma-nent magnet, Eur. J. Phys. 39, 065808 (2018).

[66] D. D. Salvucci and J. H. Goldberg, Identifying fixationsand saccades in eye-tracking protocols, in Proceedings ofthe 2000 Symposium on Eye Tracking Research & Appli-cations (Association for Computing Machinery, New York,2000), pp. 71–78.

[67] L. J. Burton and G. J. Fogarty, The factor structure of visualimagery and spatial abilities, Science 31, 289 (2003).

[68] D.M. Dimitrov and P. D. Rumrill, Jr., Pretest-post-test designs and measurement of change, Work 20, 159(2003), https://content.iospress.com/articles/work/wor00285.

[69] G. W. Divine, H. J. Norton, A. E. Barón, and E. Juarez-Colunga, TheWilcoxon–Mann–Whitney procedure fails asa test of medians, Am. Stat. 72, 278 (2018).

[70] J. Cohen, Statistical Power Analysis for the BehavioralSciences, 2nd ed. (Erlbaum, Hillsdale, NJ, 1988).

[71] https://www.didaktik.physik.uni-muenchen.de/sims/documentation/.

HOYER and GIRWIDZ PHYS. REV. PHYS. EDUC. RES. 16, 020124 (2020)

020124-18