00pp 2012 - dynamics of mental model construction from text and graphics

Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/257488564

DynamicsofmentalmodelconstructionfromtextandgraphicsARTICLEinEUROPEANJOURNALOFPSYCHOLOGYOFEDUCATIONDECEMBER2013ImpactFactor:0.8DOI:10.1007/s10212-012-0156-z

DOWNLOADS30

VIEWS159

7AUTHORS,INCLUDING:

WolfgangSchnotzUniversittKoblenz-Landau77PUBLICATIONS1,168CITATIONS

SEEPROFILE

MarkUllrichGoethe-UniversittFrankfurtamMain11PUBLICATIONS31CITATIONS

SEEPROFILE

HolgerHorzGoethe-UniversittFrankfurtamMain52PUBLICATIONS164CITATIONS

SEEPROFILE

JrgenBaumertMaxPlanckInstituteforHumanDevelopment201PUBLICATIONS3,150CITATIONS

SEEPROFILE

Availablefrom:HolgerHorzRetrievedon:11September2015

1 23

European Journal of Psychology ofEducationA Journal of Education andDevelopment ISSN 0256-2928 Eur J Psychol EducDOI 10.1007/s10212-012-0156-z

Dynamics of mental model constructionfrom text and graphics

Ulrike Hochpchler, Wolfgang Schnotz,Thorsten Rasch, Mark Ullrich, HolgerHorz, Nele McElvany & Jrgen Baumert

1 23

Your article is protected by copyright andall rights are held exclusively by InstitutoSuperior de Psicologia Aplicada, Lisboa,Portugal and Springer Science+BusinessMedia BV. This e-offprint is for personaluse only and shall not be self-archived inelectronic repositories. If you wish to self-archive your work, please use the acceptedauthors version for posting to your ownwebsite or your institutions repository. Youmay further deposit the accepted authorsversion on a funders repository at a fundersrequest, provided it is not made publiclyavailable until 12 months after publication.

Dynamics of mental model construction from text and graphics

Ulrike Hochpchler & Wolfgang Schnotz &Thorsten Rasch & Mark Ullrich & Holger Horz &Nele McElvany & Jrgen Baumert

Received: 22 March 2012 /Revised: 7 August 2012 /Accepted: 16 August 2012# Instituto Superior de Psicologia Aplicada, Lisboa, Portugal and Springer Science+Business Media BV 2012

Abstract When students read for learning, they frequently are required to integrate text andgraphics information into coherent knowledge structures. The following study aimed atanalyzing how students deal with texts and how they deal with graphics when they try tointegrate the two sources of information. Furthermore, the study investigated differencesbetween students from different school types and grades. Forty students from grades 5 and8 from higher track and lower track of the German school system were asked to process and

Eur J Psychol EducDOI 10.1007/s10212-012-0156-z

U. Hochpchler :W. Schnotz : T. RaschFaculty of Psychology, University of Koblenz-Landau, Fortstrae 7, 76829 Landau, Germany

U. Hochpchlere-mail: [email protected]

T. Rasche-mail: [email protected]

M. Ullrich :H. HorzInstitute of Psychology, University of Frankfurt, Mertonstrasse 17 (Jgel-Haus), 60325 Frankfurt, Germany

U. Ullriche-mail: [email protected]

H. Horze-mail: [email protected]

N. McElvanyInstitut fr Schulentwicklungsforschung (IFS), Technical University of Dortmund, Vogelpothsweg 78,44227 Dortmund, Germanye-mail: [email protected]

J. BaumertMax-Planck-Institute of Human Development, Lentzeallee 94, 14195 Berlin, Germanye-mail: [email protected]

W. Schnotz (*)General and Educational Psychology, University of Koblenz-Landau, Fortstr. 7, 76829 Landau, Germanye-mail: [email protected]

Author's personal copy

integrate texts and graphics in order to answer items from different levels of a textpictureintegration taxonomy. Students eye movements were recorded and analyzed. Resultssuggest fundamentally different functions of text and graphics, which are associated withdifferent processing strategies. Texts are more likely to be used according to a coherence-formation strategy, whereas graphics are more likely to be used on demand as visualcognitive tools according to an information-selection strategy. Students from different tracksof schooling revealed different adaptivity with regard to the requirements of combining textand graphic information.

Keywords Textbooks . Reading . Cognitive processing

After having learned to read in primary school, students begin using their reading skillsincreasingly for the acquisition of new knowledge, which allows them to answer questionsor to solve tasks. Learning material in secondary school usually includes not only writtentext, but also instructional graphics such as schematic diagrams, maps, and various kinds ofgraphs that visualize and explain complex subject matter. Students are expected to use textand graphics as complementary sources and integrate verbal and pictorial information for theconstruction of mental representations of the learning content (Ainsworth 1999; Mayer1997, 2005a, 2009; Schnotz 2005). Abundant research has demonstrated since severaldecades that students learn generally better from text and graphics than from text alone(Levie and Lentz 1982; Levin et al. 1987). As Mayer (1997, 2001, 2009) has pointed out,this applies not only to retention, but also to deep comprehension, which finally turns intobetter problem solving skills. Mayer has introduced the term multi-media effect to sum-marize these findings.

However, many students underestimate the informational value of graphics and processvisual displays only superficially assuming that a passing view of a graphic would besufficient to grasp the essential information (Mokros and Tinker 1987; Weidenmann1989). Hannus and Hyn (1999) found for 10-year-old children, that learning was heavilydriven by the text and that children inspected illustrations only superficially. In a recentstudy with students from grades 5 to 8 from the different tracks of the German schoolsystem, who were tested for their textgraphic integration skills, Schnotz et al. (2010) foundthat nearly half of the total variance of textgraphic integration performance could beexplained by school type, whereas one fourth of the total variance could be explained bygrade. According to the well-known Matthews effect (Rigney 2010; Stanovich 1986, 2000),one would expect that students from higher tracks of schooling would profit more fromsystematic schooling and, thus, show higher increase of textpicture integration skills fromgrade to grade than students from lower tracks. However, no such interaction between schooltype and grade was found: Students from higher tracks seemed not to benefit more fromschooling than students from lower tracks. Instead, the increase from grade to grade wasapproximately the same for all school types (cf. Schnotz et al. submitted). A relativelysimple explanation of this finding would be that there is at present no or only little systematicschooling of textpicture integration skills. Accordingly, the competency of integrating textsand graphics seems to be a by-product of schooling rather than the result of systematicteaching. We will come back to this later in the discussion.

The just-mentioned previous research was primarily quantitative in nature and providedonly a general survey of the development of textpicture integration skills. It did not allowderiving instructional measures for the improvement of the corresponding skills of students.Enhancing these skills requires a more differentiated view of the cognitive requirements of

U. Hochpchler et al.


textpicture integrations, that is, a deeper understanding of how students deal with texts andhow they deal with graphics when they try to integrate the two sources of information. Morespecifically, one needs to know in which way and to what extent their dealing with textsdiffers from their dealing with graphics. On top of that, one should know which kinds ofdifferences exists between students from different school types and grades.

In the following paper, we aim at answering these questions through an in-depth analysisof textpicture integration processes with the help of an experiment that builds on thepreviously mentioned quantitative research, but uses a research strategy that combinesquantitative and qualitative methods. First, we will provide some theoretical background:we will describe theoretical models of textgraphic integration, suggest a taxonomy of textgraphic integration tasks, and describe idealtypical strategies of using texts and graphics inorder to integrate the corresponding information for answering different kinds of questions.Second, we will specify our research questions and formulate hypotheses with regard to theuse of strategies for the combined processing of texts and graphics. Third, we will describean experiment which is aimed at testing these hypotheses. Fourth, we will report the resultsof this experiment and (fifth) discuss these results in terms of our research questions andtheir theoretical and practical consequences.

Theoretical background

Models of textpicture integration

The superiority of learning from text and pictures rather than from text alone was tradition-ally explained by the dual coding theory of Paivio (1986), which hypothesizes that thehuman cognitive system consists of a verbal subsystem and an imagery subsystem. Whereasverbal information was assumed to be encoded only in the verbal subsystem, pictorialinformation was assumed to be encoded in the imagery and in the verbal subsystem.Kulhavy et al. (1994) have developed this theory further and created their conjoint-processing theory, which assumes that pictorial unitscalled intact imagesallow espe-cially efficient usage of limited working memory capacity (Kulhavy, Stock & Kealy, 1993).

In his cognitive theory of multimedia learning, Mayer (2005b, 2009) adopts Paivios dualcoding concept, assuming that the human cognitive system includes a verbal and pictorial(image) subsystem. Accordingly, individuals can use different representational formats tointernally encode and store knowledge. Based on the working memory model of Baddeley(1992), Mayer also accepts that two sensory subsystems exist in working memory: anauditory system and a visual system. His first basic assumption on multimedia learningmerges these two concepts. Humans are supposed to process information in workingmemory through two channels: an auditoryverbal channel and a visualpictorial channel.The second basic assumption reflecting both the work of Baddeley (1992) and of Swellerand Chandler (1994) is that these two channels have a limited capacity to convey andprocess information. The third basic assumption is that humans are active sense-makers:they engage in active cognitive processing to construct coherent knowledge structures fromboth the available external information and their prior knowledge. According to the cogni-tive theory of multimedia learning, active learning from multimedia instructional messagesincludes a set of five co-ordinated processes: (1) selecting relevant words, (2) selectingrelevant images, (3) organizing the selected words into a verbal mental model, (4) organizingthe selected images into a pictorial mental model, and (5) integrating the verbal model andthe pictorial model with prior knowledge into a coherent mental representation.

Dynamics of mental model construction


Because texts and pictures are based on different sign systems and use quite differentprinciples of representation, Schnotz and Bannert (2003) have proposed another model oftext and picture comprehension, which gives more emphasis to these representationalprinciples (cf. Schnotz 2002). Their integrated model of text and picture comprehensionconsists of a descriptive and a depictive branch of representations. The descriptive branchcomprises the (external) text, the (internal) mental representation of the text surface structureand the propositional representation of the texts semantic content. The interaction betweenthese descriptive representations is based on symbol processing. The depictive branchcomprises the (external) picture, the (internal) visual perception or image of the pictureand the (also internal) mental model of the subject matter presented in the picture. Theinteraction between these depictive representations is based on processes of structuremapping due to the structural correspondences (i.e., analogy relations) between the repre-sentations (Gentner 1989).

In text comprehension, the reader constructs a mental representation of the text surfacestructure, generates a propositional representation of the semantic content and constructsfrom this text basis a mental model of the described subject matter (Graesser et al. 1997; vanDijk and Kintsch 1983; Weaver et al. 1995). Text information is processed with regard tomorphological and syntactic aspects by verbal organization processes, which lead to amental representation of the text surface structure. This text surface structure in turn triggersconceptual organization processes, which result in a structured propositional representationand a mental model.

In picture comprehension, the reader first creates a visual mental representation of thepictures graphic display through perceptual processing. Task-relevant information is select-ed and visually organized through automated visual routines (Ullman 1984). Perceptualprocessing includes identification and discrimination of graphic entities as well as the visualorganization of these entities according to the Gestalt laws (Wertheimer 1938; Winn 1990).The resulting visual perception is assumed to be kept in the imagery part of workingmemory, the visual sketchpad (Baddeley 1992; Kruley et al. 1994; Sims and Hegarty1997). Then, based on his/her perceptual processing, the reader constructs a mental modeland a propositional representation of the subject matter shown in the picture throughsemantic processing. Mental model construction is assumed to take place through aschema-driven mapping process, in which graphic entities are mapped onto mental model entitiesand in which spatial relations are mapped onto semantic relations. In other words, picturecomprehension is considered to be a process of analogical structure mapping between a systemof visuo-spatial relations and a system of semantic relations (Falkenhainer et al. 1989/90; Schnotz1993). While understanding pictorial illustrations or maps, the individual can use cognitiveschemata of everyday perception. Perception-based construction of mental models can, forexample, occur for mechanical devices such as pulleys or gears, which allow to internally simulatemechanical movements (Hegarty 2004; Hegarty and Just 1993; Keehner et al. 2008). Whileunderstanding graphs, the individual requires specific cognitive schemata (graphic schemata) inorder to be able to read off information from the visuo-spatial configuration (Pinker 1990).

Taxonomy of textpicture integration

Integrating verbal and pictorial information requires mapping between corresponding elementsin the text and the picture. This mapping can occur on the level of surface structures and on thelevel of semantic deep structures. Surface structure mapping includes connecting verbalelements (words) and graphical elements (lines and shapes) based on cohesive devices suchas common color coding, common numbers, common symbols or common labels. Semantic



deep structure means connecting conceptual structures and structural characteristics of themental model. As simple structures can be embedded into more complex structures, hierarchiesof structure mapping can emerge with complex structures at the higher levels and simplestructures at the lower levels. The example of a textpicture combination shown in Fig. 1should demonstrate the basic idea of structuremapping hierarchies. The text and the pictures aretaken (slightly modified) from a text book on biology that explains the structure of the legs ofinsects. The text states that the legs of insects presented in four pictures have all the samestructure consisting of hip (orange), leg ring (brown), thigh (green), bar (pink), and foot (blue),and that the leg of an ant is specialized for running, the leg of a water bug is specialized forswimming, the leg of a grasshopper is specialized for jumping and the leg of a honey-bee isspecialized for cleaning.1

With reference to Wainer (1992), we can distinguish between three different levels ofstructure mapping with increasing complexity. At the lowest level (A), the student has toextract only a specific piece of information based on the mapping of elements. An exampleof a level A item would be What is the name of the end part of an insects leg?. Answeringthe item requires only mapping of the blue color of the legs end part on the blue color in thetext legend, which leads to foot as the right answer. At the medium level (B), the studenthas to extract and map simple relations between elements in the text or in the picture. Anexample of a level B item would be Does the leg for swimming have a shorter thigh than theleg for jumping?. Answering this item requires identifying the corresponding pictures(swimming and jumping), identifying the corresponding parts of the leg and to make acomparison between them. At the highest level (C), the student has to extract and mapcomplex relations between elements in the text or in the picture. An example of a level Citem would be Does the leg for running have a longer bar than the leg for swimming, but ashorter bar than the leg for jumping?. Answering this item requires identifying thecorresponding pictures, identifying the corresponding parts, comparing the parts and inte-grating both comparisons into one semantic structure.

To summarize: Regarding structure mapping between text and pictures, a distinction canbe made in terms of the complexity of the structures to be mapped. Accordingly, integratingtext and pictures can take place at different levels: Level A (extraction and mapping of singleinformation), level B (extraction and mapping of simple relations), and level C (extractionand mapping of complex relations). As complex structures include more simple structures,and the latter are embedded into the former, a hierarchy of structure mapping emerges,wherein the embedded structures at lower levels are prerequisites for the embeddingstructures at higher levels. The hierarchy can serve as a taxonomy of textpicture integrationtasks, wherein the levels of the taxonomy represent structure mappings of increasingcomplexity forming a sequence of logical preconditions within each unit.

Strategies for integrative processing of text and graphics

When students process texts and graphics in order to answer specific questions or to solvespecific tasks, they will try, within their capabilities, to apply appropriate strategiesmethods to bring about the desired results. Former research on learning from text has shownthat pre-posed questions about the text content can result in highly selective processing at theexpense of a global understanding of the text, which indicates an inherent conflict betweentask-oriented information selection and global coherence formation during goal-oriented text

1 In the original picture, the different parts of the legs are not labeled by color names, but by the correspondingcolors. Within the original text, colors are used instead of color names as reference to the picture.



processing (Rickards and Denner 1978). The same kind of conflict can also be assumed forintegrative processing of text and graphics. We assume that a student learning from text andgraphics in order to answer questions can choose between two opposing strategies, namely(1) a task-specific information-selection strategy versus (2) a global coherence-formationstrategy. Whereas the first strategy puts the emphasis from the beginning on goal-orientedselection of task-relevant information, the second strategy puts the emphasis first on a globalunderstanding of the content, and deals only afterwards with specific questions (Kintsch1998; van Dijk and Kintsch 1983).

Task-specific information-selection strategy When students use this strategy, their processingis task-specific from the beginning. That is, they focus primarily on the task to be solved andselect the required verbal and pictorial information from the text and the graphic. Text andgraphic processing are guided to a large extent by top-down activation of cognitive schemataaccording to the task at hand. As mentioned before, items at level A of the taxonomy describedabove require less information than items at level B, which in turn require less information thanitems at level C. Thus, when students choose a task-specific information-selection strategy, theirtext-reading times and graphic observation times for answering an item at taxonomy level Ashould on the average be shorter than for an item at level B, which should in turn be shorter thanfor an item at level C.

Global coherence-formation strategy When students use this strategy, their processing is nottask-specific from the beginning. Instead, they first read the whole text and observe theaccompanying graphic in order to understand the subject matter before dealing with specificquestions. In other words, they engage in a non-task-specific initial construction of acoherent mental model with a stronger emphasis on bottom-up processing before turningto specific items. When students choose a global coherence-formation strategy, their text-reading times and graphic observation times for answering the first item should on theaverage be higher than for the following items with the same text and graphic, regardless ofthe items taxonomy level. Thus, even if the first item is at a lower level in the taxonomy,students would invest more time into reading the text and observing the graphic than forhigher level items that might follow and which then would require only a few mental modelupdates (if any).

Fig. 1 Example of a textpicture integration task from a text book on biology



Research questions and hypotheses

The aim of the present investigation was to find out how students deal with texts andgraphics when they try to integrate the two sources of information for answering questions.In view of the previous considerations on processing strategies, we analyze whether and towhat extent students adopt a task-specific information-selection strategy or a globalcoherence-formation strategy when they try to integrate text and graphics for answeringquestions. More specifically, we addressedin the context of goal-oriented text graphicintegrationthe following research questions:

1. Do students use a task-specific information-selection strategy or a global coherence-formation strategy for the processing of texts?

2. Do students use a task-specific information-selection strategy or a global coherence-formation strategy for the processing of graphics?

3. Is there a difference between processing texts and processing graphics in terms ofstrategy usage?

4. Do students from different school types differ with regard to their strategy usage?5. Do students from different grades differ with regard to their strategy usage?

Following the integrated model of textpicture comprehension, we hypothesized an asym-metry between text processing and picture processing. On the one hand, due to the represen-tational relation of an analogy between the picture and the mental model, pictures provide amore direct way to mental model construction than texts do. Pictures can therefore serve aseasily accessible external scaffolds for mental model construction. Texts, on the other hand,guide the readers conceptual analysis by a description of the subject matter leading to acoherent semantic network, which in turn contributes to further elaborating the mental model.Accordingly, we assumed on the one hand that texts contribute more to coherence formationand, thus, are more likely to be used for a coherence-formation strategy than pictures. On theother hand, we assumed that pictures are more likely to be used as easily accessible externalrepresentations for a task-specific information-selection strategy than texts.

According to the integrated model, text comprehension and picture comprehension providedifferent ways to mental model construction. Both ways can support each other, but can alsoreplace each other to some extent. When pictures are combined with texts, in particular lessskilled learners can replace text reading to some extent by picture observation, when they readthe text more superficially with pictures than without (Schnotz and Bannert 1999). Becausestudents from lower track schools and from lower grades are usually less skilled than studentsfrom higher track schools and from higher grades, we assumed that students from lower trackschools would use the pictures more intensively being more selective with regard to textprocessing than students from higher track schools. By the same token, we assumed thatstudents from lower grades would use the pictures more intensively being more selective withregard to text processing than students from higher grades.

Method

In order to test the previous hypotheses, we presented sets of textgraphic combinations tostudents from different grades and from different tracks of schooling and asked them to answersequentially presented items from different taxonomy levels. Measurement of cognitive skillsstarts usually with simple, easy items and then proceeds to more complex and difficult items.



Accordingly, we presented each textgraphic combination with a sequence of three items: Thefirst item was from taxonomy level A, the second item was from taxonomy level B, and the thirditem was from taxonomy level C. Under this condition, one can derive different predictions froma task-specific information-selection strategy and a global coherence-formation strategy.

As mentioned before, items from level A of the taxonomy require less information thanitems from level B, which in turn require less information than items from level C. Thus,when students choose a task-specific information-selection strategy, the time they invest intoan information source (text or graphic) for answering an item from taxonomy level A shouldon the average be shorter than for an item from level B, which should in turn be shorter thanfor an item from level C (provided that the items are stochastically independent, whichimplies that there is no semantic overlap between the A-, the B-, and the C item2). Thus,predictions based on a task-specific information-selection strategy are:

Time on Source (Item A)Time on Source (Item C)

Eye-tracking In order to measure the time invested into the texts and the graphics foranswering the presented items, we used eye-tracking methodology with an EyeLink IIsystem from SR Research, which allows participants to move their head relatively freely.Eye-tracking collects behavioral data about the sequence and duration of time an individuallooks at a specific area of his/her environment. A cognitive interpretation of these datarequires assumption about the relation between eye movements and cognitive processingsuch as the eyemind assumption and the immediacy assumption (Just and Carpenter 1980).They eyemind assumption suggests that the locus of eye fixations corresponds generally to theinformation being processed in the cognitive system, whereas the immediacy assumptionsuggests that the received information is immediately cognitively processed. The underlyingconcept derives from the idea of cognitive economy and suggests that there should be as littleunanalyzed information as possible to be temporarily stored during processing. Insofar, the twoassumptions can be considered as two sides of the same coin. Despite of occasional criticism ofthe general validity of the eyemind assumption, there is evidence that eye movements canprovide useful information about cognitive processes (Rozenblit et al. 1998).

Test development The combinations of texts and graphics used in the following experimentwere taken from a test for measuring textgraphic integration skills of students in a largescale assessment study (Schnotz et al. 2010). For test development, 60 textbooks on biology

2 It should be noted that although the hierarchy between the levels A, B and C in the taxonomy is based on theprinciple of lower level structures being embedded into higher level structures, it is possible to construct A-items, B-items and C-items from different hierarchies within the same textpicture unit. In this case, the itemsare located at different taxonomy levels, but are nevertheless stochastically independent.



and geography from grade 5 to grade 8 currently used in Germany were searched for textgraphic units. We decided for biology and geography because the textbooks include a well-balanced diversity of visualizations, whereas textbooks in physics include primarily realistic(although schematic) drawings and textbooks in mathematics include primarily graphs. Thetext of a unit was required to have a maximum length of 200 words, and the graphic had tobe either a photograph, a schematic drawing, a map, or a graph. Twelve units per grade wererandomly selected, which resulted in a total of 48 units. Text length per unit varied from 38to 160 words, and number of graphics per unit varied from 1 to 3. For each unit, six multiplechoice test items were created, which required to integrate verbal and pictorial information:Two items were on taxonomy level A, two items on level B, and two items on level C. Theresulting 288 test items were presented via a multiple matrix design to 1,060 students fromgrade 5 to grade 8. The items were analyzed based on item-response theory with a one-parametric logistic model (Rasch-model) including DIF analyses for gender, grade, andschool track. Furthermore, the items underwent a rational task analysis (Schnotz et al. 2011).

Selection of textgraphic units Due to the high demands of eye-tracking studies, we selectedonly four units for the experiment according to the following criteria. Two units should be onbiology and the other two on geography. The units should include a diversity of visual-izations, including realistic schematic drawings, maps, and graphs, and they should vary indifficulty. For each unit, we selected three out of the six items, one item from taxonomy levelA, one item from level B, and one item from level C. The items had to be Rasch-homogenous (Eggen 2004) and stochastically independent (Zenisky et al. 2002). Items fromlevel A had to be easier than items from level B which in turn had to be easier than itemsfrom level C. Furthermore, items from level A had to require fewer textgraphic mappingsaccording to the rational task analysis than items from level B which in turn had to requirefewer mappings than items from level C. According to these criteria, the following textgraphic units on biology were selected:

1 Social behavior of apes. This unit was relatively easy (beta01.10). Text length was 87words. The unit consisted of two pie graphs about the time budget of apes with andwithout a leader of the troop.

2 Human respiratory system. This unit was relatively difficult (beta0+1.3). Text lengthwas 116 words. The unit contained a realistic schematic drawing of the human respira-tion organs including nasal cavity, throat, trachea, bronchia, and lung.

The selected units on geography were:3 States of Australia. This unit had medium difficulty (beta00.09). Text length was 36

words. The unit contained a map showing the states of Australia and the location of somecities of the country.

4 Production of chocolate. This unit had high difficulty (beta0+1.80). Text length was 82words. The unit contained a schematic drawing of a chocolate manufacturing plant includingcleaning, roasting, cooling down and grinding of cocoa beans, squeezing the cocoa mass aswell as kneading, stirring, and pouring the mass with ingredients into chocolate bars.

The average beta value on the Rasch scale was 0.13 for items from taxonomy level A, +0.61for items from level B, and +0.84 for items from level C. The average number of textgraphicmappings according the rational task analysis was 1.25 for items from taxonomy level A, 1.50 foritems from level B, and 3.00 for items from level C.

Participants Forty students, 20 of grade 5 and 20 of grade 8 participated in the study. Ten ofthe fifth graders and ten of the eighth graders were students from the higher track of the



German school system (Gymnasium). The other half of the fifth graders and the eighthgraders were students from the lower track of the German school system (Hauptschule).Fifth graders had an average age of 10.7 years (SD00.58), eighth graders an average age of13.9 years (SD00.66). Twenty-four students were male, 16 were female. The students wereinvited to our university lab, where they were asked to read the four selected textgraphicunits and to answer the corresponding total set of 12 textgraphic integration items: 4 itemsof level 1, 4 items of level 2 and 4 items of level 3. Students were paid 15 Euros for theirparticipation.

Procedure The study was performed by individual sessions with computer-based presenta-tion of the material. Screen size was 370 mm horizontal274 mm vertical. Text areas sizewas 139 mm horizontal274 mm vertical. Graphic areas size was 231 horizontal189 mmvertical. Item areas size was 231 mm horizontal85 mm vertical. The areas were constantacross textgraphic units. Students had been informed before the session that their eyemovements would be recorded with a head-mounted system. At the beginning of the session,they were instructed how to operate the system (how to answer the items) with a game pad.They were also informed that time per item was limited to 3 min and that the next itemwould be presented automatically afterwards. A previous pilot study had shown that 3 min issufficient even for most of the fifth graders or for poorer students. Units were presented in afixed order according to difficulty as mentioned above. Within each unit, the text and thegraphic(s) were presented simultaneously on the screen, first combined with an item fromtaxonomy level A, then combined with an item from level B, and finally combined with anitem from level C. So, the students engagement with each unit consisted of a first phasewhen dealing with the item from level A (A-phase), a second phase when dealing the itemfrom level B (B-phase), and a third phase when dealing with the item from level C(C-phase).

Students eye movements were registered with an EyeLink II system. After suc-cessful calibration, students worked self-paced (except for the 3 min limit per item)and answered the items with the game pad. After entering the answer of an item, thenext item (where required with the next textgraphic unit) appeared automatically.Turning pages backwards was not possible. With four students, successful calibrationwas not possible. This reduced the number of participants for further analysis to 36students (10 fifth graders and 9 eighth graders from Gymnasium, 9 fifth graders and8 eighth graders from Hauptschule).

Scoring The display of each textgraphic unit with each item was subdivided into threeareas of interest: the text area, the graphic area, and the item area. For each item and for eachtextgraphic unit, the total fixation time was determined separately for the text, for thegraphic(s) and for the corresponding item. Based on these aggregated times of eachparticipants and for each item, we computed for further analysis the total fixation times ofthe text, the total fixation times of the graphic(s) and the total fixation times of thecorresponding item across all textgraphic units. This averaging procedure was performedseparately for the A-phase (when a level A item was presented), for the B-phase (when alevel B item was presented), and for the C-phase (when a level C item was presented).Furthermore, we determined the average fixation time for the text, for the graphic(s) and forthe items. The system registered the students item responses as well as the time required foranswering the corresponding item automatically. Based on these data, we determined thenumber of correct item answers per minute working time for each student as an index of his/her efficiency of textgraphic integration.



Results

For a task-specific information-selection strategy, we had expected the times invested in aninformation source (text or graphic) to be shorter for phase A (when the student works on anitem from taxonomy level A) than for phase B (when he/she works on an item from level B).By the same token, we had expected shorter times for phase B (when the student works on alevel B item) than for phase C (when the student works on a level C item). For a globalcoherence formation strategy, we had expected the opposite: Times for phase A would belonger than times for phase B, which would in turn be longer than times for phase C. In orderto test these assumptions, we performed 22(3) ANOVAS with the between-factors schooltype (higher track/lower track) and grade (5th/8th) and the within-factor phase (A/B/C) forthe averaged total fixation times of the text areas and for the averaged total fixation times ofthe graphic areas, and we computed a priori contrasts for the comparison between the A-phase and the B-phase and for the comparison between the B-phase and the C-phase.

Reading texts

Total fixation times invested by students from higher track schools (Gymnasium) andstudents from lower track schools (Hauptschule) into texts during the phases A, B, and Caveraged across all units are shown graphically in Fig. 2. Accordingly, reading timesdecreased dramatically from answering item A to answering item B and then furtherdecreased from answering item B to answering item C. For the contrast between phases Aand B, we found a highly significant effect of phase (F(1, 32)078.48; p

students from the lower track (Hauptschule) needed more time for answering the items thanstudents from the higher track (Gymnasium). No significant main or interaction effect wasfound for grade.

Observing graphics

Total fixation times invested by students from higher track schools (Gymnasium) andstudents from lower track schools (Hauptschule) into graphics during the phases A, B,and C averaged across all units are shown graphically in Fig. 3. Accordingly, observationtimes decreased from answering item A to answering item B both for students from thehigher track and students from the lower track. Afterwards, observation times for graphicsincreased again from answering items B to answering items C, but only with students fromthe higher track, whereas for students from the lower track, observation times remained at alow level both for answering items B and answering items C. For the contrast betweenphases A and B, we found a highly significant effect of phase (F(1, 32)018.66; p

coherence-formation strategy. As for the text-reading times, the results clearly support thecoherence formation strategy, as all predictions derived from this strategy were confirmed byhighly significant differences. No indication of a task-specific information-selection strategywas found with regard to text processing.

As for the graphic observation times, the results are mixed. On the one hand, thedifferences between answering the first (level A) item and the second (level B) item supportthe global coherence formation strategy, which corresponds to the assumption that graphicscan serve as external scaffolds for mental model construction. On the other hand, thedifferences between answering the second (level B) item and the third (level C) item provideoverall support to the task-specific information-selection strategy, but with a highly significantmoderator effect of school type: Whereas the data of the higher track students clearly supportthe task-specific information-selection strategy, there is no such support with the lower trackstudents. In fact, the data of the lower track students support neither an information-selectionstrategy nor a coherence formation strategy, as despite of the increase of difficulty from level Bitems to level C items lower track students show in both cases the same low usage of graphics. Itseems that higher track students adapt their usage of graphics to the higher difficulty of level Citems, whereas the lower track students do not.

Average times per fixation

Regarding the average time per fixation during text reading, the corresponding ANOVArevealed a significant main effect of school type (F(1,32)05.46; p0 .026; Eta20 .15) and ahighly significant main effect of phase (F(1.62, 51.83)014.60; p

Regarding the average time per fixation during graphic observation, the correspondingANOVA did not reveal any remarkable difference between school types. The average timeper fixation during graphic observation was 230 ms (SD042 ms) for students from the lowertrack (Hauptschule) and 245 ms (SD032 ms) for students from the higher track(Gymnasium). Neither were there essential differences between phases A, B, and C. Theaverage time per fixation on graphics was 240 ms (SD036 ms) for answering items A,236 ms (SD037 ms) for answering items B and 239 ms (SD040 ms) for answering items C.However, there was a significant interaction itemsgrade (F(2,64)03.55; p0 .035;Eta20 .10): Whereas fifth graders showed a gradual decrease of the average time per fixationduring graphic observation from 242 ms (SD046 ms) for item A to 229 ms (SD041 ms) foritem C, the eighth graders showed a gradual increase from 237 ms (SD022 ms) with item Ato 249 ms (SD036 ms) for item C.

Regarding the average time per fixation on the item areas, the corresponding ANOVArevealed only a significant main effect for the factor items (F(1.59, 50.72)04.07; p0 .03;Eta20 .11). Other effects were not significant. The average time per fixation increased withthe taxonomy level of the item: it was 238 ms (SD047) for the A items, 246 ms (SD057) forthe B items and 250 ms (SD061) for the C items.

Efficiency of textgraphic integration

The number of correctly answered items per minute working time on items from levels A, B,and C was determined for each student as an index of his/her efficiency of textgraphicintegration. The corresponding ANOVA revealed a highly significant main effect of items(F(2,64)010.27; p

panel) from the higher track (Gymnasium) and the lower track (Hauptschule) graphically.Regarding the school type effect, students from higher track (Gymnasium) solved on theaverage 1.31 items per minute (SD0 .56) successfully, whereas students from lower track(Hauptschule) solved only 0.68 items per minute (SD0 .51) successfully. Regarding the gradeeffect, eighth graders solved on the average 1.22 items per minute (SD0 .64) successfully,whereas fifth graders solved only 0.83 items per minute (SD0 .54) successfully. Regarding, theitems effect, students solved on the average 0.768 items from level A per minute (SD0 .36)successfully. Afterwards, their average performance increased to 1.08 per minute (SD0 .76) forlevel B items and to 1.19 per minute (SD0 .73) for level C items.

Concerning the interaction itemgrade, both fifth graders and eighth graders showed anincrease of efficiency from item A to item B. This can be explained by the long text-readingtimes and graphic observation times for the first item. Such long times are no longer neededfor the second item, which increases efficiency for the second item in terms of correctanswers per second. The interaction itemgrade derives from the difference between fifthgraders and eighth graders regarding item C: Whereas efficiency decreased from item B toitem C for the fifth graders, it increased for the eighth graders.

Discussion

The fundamentally different patterns of usage of texts and graphics during integrative compre-hension suggest fundamentally different functions of both information sources. For the usage oftexts, the findings clearly support a global coherence formation strategy. Students seem toengage first in a process of intensive coherence formation which amounts to an initial mentalmodel construction. Accordingly, they invest a high amount of time into the text with higheraverage fixation times, even when the item to be answered is relatively easy. After this initialmodel construction, they seem to use the text only for mental model updates if needed foranswering the item at hand and, thus, invest less time for the following items, even if these itemsare considerably more demanding. For the usage of graphics, on the other hand the findingssupport a global coherence formation strategy first and a task-specific information-selectionstrategy later. Students seem to use graphics as scaffolds (i.e., aids for coherence formation) fortheir initial mental model construction, although to a much lower extent than texts. After theinitial mental model construction, students with higher learning prerequisites (i.e., students fromhigher track schools) seem to adapt their processing to the difficulty of the task at hand, as theyinvest more time into graphic processing when items become more difficult. On the contrary,students with lower learning prerequisites (i.e., students from lower track schools) do not usegraphics more intensively when items become more difficult.

As for students from the higher track, text and graphics seem to serve different functionsand are therefore used according to different strategies. Texts are more likely to be usedaccording to a coherence-formation strategy than graphics: Texts guide the readers concep-tual analysis by a description of the subject matter, which results in a coherent semanticnetwork and mental model regardless of the difficulty of the item at hand. Graphics are morelikely to be used according to a task-specific information-selection strategy than texts.Graphics serve as scaffolds for initial mental model construction, but are afterwards morelikely to be used on demand as easily accessible external representations for item-specificmental model updates.

As for students from the lower track, the situation is somewhat different. Texts are also morelikely to be used by these learners according to a coherence-formation strategy. They obviouslyhave more difficulties with word recognition and lexical access, which is indicated by their



average fixation times. However, they nevertheless invest a high amount of time into the textduring the first phase of initial mental model construction, although the item to be answered isrelatively easy. Afterwards, they invest much less time into the text, even when the followingitems are more demanding. Graphics seem also to serve as scaffolds for initial mental modelconstruction for lower track students . Contrary to higher track students, however, the lowertrack students do not use graphics more intensively afterwards when items become moredifficult. Contrary to our expectations, lower track students (as well as younger students) seemnot to use graphics more intensively and text less intensively than higher track students (andolder students). This is in line with results of Hannus and Hyn (1999), who found that 10-year-old childrens learning was heavily driven by the text and that children inspected illustrations onlysuperficially.

We can only speculate at this point about the reasons why lower track students do notadapt their processing of graphics to the demands of the items. One possible explanationwould be that they are less meta-cognitively sensitive for item difficulty. Students withhigher meta-cognitive skills are possibly better in recognizing the demands of an item andthe usefulness of an accompanying graphic (Flavell and Wellman 1977; Hartman 2001).Another reason for the lack of adaptation would be that they do not know how to deal withgraphics, either because they do not possess the required graphic processing strategies orbecause they do not know when to apply which strategy (Hasselhorn 1996). These issuesneed further research.

Our findings could perhaps contribute to a more differentiated view of both text process-ing and graphic processing from a general, developmental and instructional perspective, asthey suggest that text and graphics play fundamentally different roles in mental modelconstruction (Goldman and Rakestraw 2000; Graesser et al. 1997; McNamara 2007;Rouet 2006; Rouet et al. 1996). We assume that mental model construction from text andgraphics is a dynamic process that consists of an initial phase, which is primarily text drivenalthough graphics may have some scaffolding functions. Afterwards, the text is only used forsome item-specific mental model updates, whereas the graphics are used on demand aseasily accessible visual tools. This implies that processing of text is more systematic, lessitem-dependent and insofar more a bottom-up process, whereas processing of graphics ismore ad hoc, more item-dependent and insofar more a top-down process. In other words,processing of graphics seems to be more task-dependent than processing of texts. Simplyspeaking, texts provide conceptual guidance, whereas graphics serve on demand as externalcognitive tools.

Numerous studies have demonstrated that students learn generally better from text andgraphics than from text alone (Fletcher and Tobias 2005; Levie and Lentz 1982; Levin et al.1987; Mayer 1997, 2001, 2009). These findings might apply when learners possess therequired processing strategies and if they are also able to use them. However, theseconditions are not always met. Sometimes, learners seem to lack the relevant strategies forintegrating text and graphic information and then have little use especially for the pictorialpart. Whereas many studies on multimedia learning found that especially poorer studentsbenefit from adding graphics to texts, we found that the poorer students had less use forgraphics, especially in the case of answering more difficult items.

Whereas the development of text-reading skills receives justifiably much attention inschooling, reading graphs and other types of visuals is only briefly addressed in most schoolcurricula. Integrative processing of text and graphics puts specific demands on the learner,because specific strategies of how to employ the mutual constraints between texts andgraphics are required. Although integrative processing of texts and graphics is becoming anincreasingly fundamental requirement in everyday life and further education, the development



of these skills seems to be a by-product rather than a result of systematic teaching and learningin the present form of schooling. Evidence for this claim comes, on the one hand, from closeranalyses of school books: Even within the higher track of schooling, the different forms ofvisualizationssuch as pie charts, bar charts and (Cartesian) line graphsare explained in theschool books of geography used between grades 5 and 8 only once, and this explanation is limitedto one or two book pages. Evidence for the claim comes, on the other hand, from empirical dataabout students skill improvement: according to our findings, students from higher tracks ofschooling do not benefit more from schooling than students from lower track (Schnotz et al. 2010;Schnotz et al. submitted), as should be expected according to the well-knownMatthews effect incase of systematic teaching and learning (Rigney 2010; Stanovich 1986, 2000).

In order to identify specific strategies of pictorial processing, further research should putmore emphasis on specific graphics formats (Pinker 1990; Zelazny 2006). For these kinds ofanalysis, it might also be fruitful to combine eye-movement methodology with thinking-aloud procedures, although the latter might slow down cognitive processing. However, thebenefits of synchronized behavioral eye-movement data and verbal data might outweigh themethodological difficulties (cf. Kaakinen and Hyn 2005; Hyn et al. 2003).

Acknowledgments This research has been performed as part of the BITE-Project supported by the GermanScience Foundation (Grant Number GZ: SCHN 665/3-1, SCHN 665/6-1) within the Special ResearchProgram Competence models for assessing individual learning results and for balancing of educationalprocesses. We are grateful to Dipl.-Psych. Wienke Wannagat for her help in running the empirical studiesdescribed in this article.

References

Ainsworth, S. (1999). The functions of multiple representations. Computers in Education, 33, 131152.Baddeley, A. D. (1992). Working memory. Science, 255, 556559.Eggen, T. J. H. M. (2004). Contributions to the theory of practice of computerized adaptive testing. Enschede:

University of Twente.Falkenheiner, B., Forbus, K. D., & Gentner, D. (1989/1990). The structure-mapping engine: algorithm and

examples. Artificial Intelligence, 41, 163.Flavell, J. H., & Wellman, H. M. (1977). Metamemory. In R. V. Kail & J. W. Hagen (Eds.), Perspectives on

the development of memory and cognition. Hillsdale: Erlbaum.Fletcher, J. D., & Tobias, S. (2005). The multimedia principle. In R. E. Mayer (Ed.), Cambridge handbook of

multimedia learning (pp. 117133). Cambridge: Cambridge University Press.Gentner, D. (1989). The mechanisms of analogical learning. In S. Vosniadou & A. Ortony (Eds.), Similarity

and analogical reasoning (pp. 199241). London: Cambridge University Press.Goldman, S. R., & Rakestraw, J. A. (2000). Structural aspects of constructing meaning from text. In M. L.

Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.),Handbook of reading research (Vol. III, pp. 311335).Hillsdale: Erlbaum.

Graesser, A. C., Millis, K. K., & Zwaan, R. A. (1997). Discourse comprehension. Annual Review ofPsychology, 48, 163189.

Hannus, M., & Hyn, J. (1999). Utilization of illustrations during learning of science textbook passagesamong low- and high-ability children. Contemporary Educational Psychology, 24, 95123.

Hartman, H. J. (2001). Metacognition in learning and instruction: Theory, research and practice. Dordrecht:Kluwer.

Hasselhorn, M. (1996). Kategoriales Organisieren bei Kindern: Zur Entwicklung einer Gedchtnisstrategie.Gttingen: Hogrefe.

Hegarty, M. (2004). Mechanical reasoning as mental simulation. Trends in Cognitive Sciences, 8, 280285.Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal

of Memory and Language, 32, 717742.



Hyn, J., Radach, R., & Deubel, H. (Eds.). (2003). The minds eye: cognitive and applied aspects of eyemovement research. Amsterdam: North Holland.

Just, M. A., & Carpenter, P. A. (1980). A theory of reading: from eye fixations to comprehension. Psycho-logical Review, 87, 329354.

Kaakinen, J. K., & Hyn, J. (2005). Perspective effects on expository text comprehension: evidence fromthink-aloud protocols, eyetracking and recall. Discourse Processes, 40, 239257.

Keehner, M., Hegarty, M., Cohen, C. A., Khooshabeh, P., & Montello, D. R. (2008). Spatial reasoning withexternal visualizations: what matters is what you see, not whether you interact. Cognitive Science, 32,10991132.

Kintsch, W. (1998). Comprehension: a paradigm for cognition. New York: Cambridge University Press.Kruley, P., Sciama, S. C., & Glenberg, A. M. (1994). On-line processing of textual illustrations in the

visuospatial sketchpad: evidence from dual-task studies. Memory & Cognition, 22, 261272.Kulhavy, R. W., Stock, W. A., & Kealy, W. A. (1993). How geographic maps increase recall of instructional

text. Educational Technology Research and Development, 41, 4762.Kulhavy, R. W., Stock, W. A., & Caterino, L. C. (1994). Reference maps as a framework for remembering

text. In W. Schnotz & R. Kulhavy (Eds.), Comprehension of graphics (pp. 153162). Amsterdam: NorthHolland.

Levie, H. W., & Lentz, R. (1982). Effects of text illustration: a review of research. Educational Communi-cation and Technology Journal, 30, 195232.

Levin, J. R., Anglin, G. J., & Carney, R. N. (1987). On empirically validating functions of pictures inprose. In D. M. Willows & H. A. Houghton (Eds.), The psychology of illustration (pp. 5185).New York: Springer.

Mayer, R. E. (1997). Multimedia learning: are we asking the right questions? Educational Psychologist, 32,119.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.Mayer, R. E. (2005a). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge

handbook of multimedia learning (pp. 3148). New York: Cambridge University Press.Mayer, R. E. (2005b). The Cambridge handbook of multimedia learning. Cambridge: Cambridge University

Press.Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.McNamara, D. S. (Ed.). (2007). Reading comprehension strategies: theories, interventions, and technologies.

Mahwah: Erlbaum.Mokros, J. R., & Tinker, R. F. (1987). The impact of microcomputer based labs on childrens ability to

interpret graphs. Journal of Research in Science Teaching, 24(4), 369383.Paivio, A. (1986). Mental representations: a dual coding approach. New York: Oxford University Press.Pinker, S. (1990). A theory of graph comprehension. In R. Freedle (Ed.), Artificial intelligence and the future

of testing (pp. 73126). Hillsdale: Erlbaum.Rickards, J. P., & Denner, P. R. (1978). Inserted questions as aids to reading text. Instructional Science, 7,

313346.Rigney, D. (2010). The Matthew effect: how advantage begets further advantage. New York: Columbia

University Press.Rouet, J. F. (2006). The skills of document use: from text comprehension to Web-based learning. Mahwah:

Erlbaum.Rouet, J. F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason

about history. Journal of Educational Psychology, 88, 478493.Rozenblit, L., Spivey, M., & Wojslawowicz, J. (1998). Mechanical reasoning about gear-and-belt diagrams:

Do eye movements predict performance? In Proceedings of Mind III: The Annual Conference of theCognitive Science Society of Ireland, 158165.

Schnotz, W. (1993). On the relation between dual coding and mental models in graphics comprehension.Learning and Instruction, 3, 247249.

Schnotz, W. (2002). Towards an integrated view of learning from text and visual displays. EducationalPsychology Review, 14(2), 101120.

Schnotz, W. (2005). An integrated model of text and picture comprehension. In R. E. Mayer (Ed.), Cambridgehandbook of multimedia learning (pp. 4969). Cambridge: Cambridge University Press.

Schnotz, W., & Bannert, M. (1999). Einfluesse der Visualisierungsform auf die Konstruktion mentalerModelle beim Text- und Bildverstehen [Influence of the type of visualization on the construction ofmental models during picture and text comprehension]. Zeitschrift fuer Experimentelle Psychologie, 46,217236.

Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representations.Learning and Instruction, 13, 141156.



Schnotz, W., Horz, H., McElvany, N., Schroeder, S., Ullrich, M., Baumert, J., Hachfeld, A., & Richter, T.(2010). Das BITE-Projekt: integrative Verarbeitung von Bildern und Texten in der Sekundarstufe I.Zeitschrift fr Pdagogik, 56, 143153.

Schnotz, W., Ullrich, M., Hochpchler, U., Horz, H., McElvany, N., Schroeder, S., & Baumert, J. (2011).What makes textpicture integration difficult? A structural and procedural analysis of textbook require-ments. Ricerche di Psicologia, 1, 103135.

Sims, V. K., & Hegarty, M. (1997). Mental animation in the visuospatial sketchpad: evidence from dual-tasksstudies. Memory & Cognition, 25, 321332.

Stanovich, K. E. (1986). Matthew effects in reading: some consequences of individual differences in theacquisition of literacy. Reading Research Quarterly, 21(4), 360407.

Stanovich, K. E. (2000). Progress in understanding reading: Scientific Foundations and New frontiers. NewYork: Guilford Press.

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction,12, 185233.

Ullman, S. (1984). Visual routines. Cognition, 18, 97159.van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.Wainer, H. (1992). Understanding graphs and tables. Educational Researcher, 21(1), 1423.Weaver, C. A., III, Mannes, S., & Fletcher, C. R. (Eds.). (1995). Discourse comprehension. Hillsdale: Erlbaum.Weidenmann, B. (1989). When good pictures fail: An information-processing approach to the effects of

illustrations. In H. Mandl & J. R. Levin (Eds.), Knowledge acquisition from text and pictures (pp. 157170).Amsterdam: North-Holland.

Wertheimer, M. (1938). Laws of organization in perceptual forms in a source book for Gestalt Psychology.London: Routledge & Kegan Paul.

Winn, W. D. (1990). A theoretical framework for research on learning from graphics. International Journal ofEducational Research, 14, 553564.

Zelazny, G. (2006). Say it with charts: the executives guide to visual communication. New York:McGraw-Hill.

Zenisky, A. L., Hambleton, R. K., & Sireci, S. G. (2002). Identification of local item dependencies in theMedical College Admissions Test. Journal of Educational Measurement, 39(4), 291301.

Ulrike Hochpchler. Faculty of Psychology, University of Koblenz-Landau, Fortstrae 7, 76829 Landau,Germany. E-mail: [email protected]

Current themes of research:

Textpicture integration.

Wolfgang Schnotz. Faculty of Psychology, University of Koblenz-Landau, Fortstrae 7, 76829 Landau,Germany. E-mail: [email protected]


TextPicture Integration, Knowledge Acquisition from Multiple Representations, Knowledge Acquisi-tion with Conflicting Information, Multimedia Learning, Learning with Hypermedia, Learning withAnimation.

Most relevant publications in the field of Psychology of Education:

Schnotz, W. (2011). Colorful bouquets in multimedia research: A closer look at the modality effect. Zeitschriftfr Pdagogische Psychologie, 25, 269276.



Schnotz, W., Baadte, C., Mller, A., & Rasch, R. (2010). Creative thinking and problem solving withdepictive and descriptive representations. In L. Verschaffel, E. De Corte, T. de Jong & J. Elen (Eds.)Use of representations in reasoning and problem solving (pp. 1135). London: Routledge.

Schnotz, W., & Hei, A. (2009). Semantic scaffolds in hypermedia learning environments. Computers inHuman Behavior, 25(2), 371380.

Schnotz, W., & Krschner, C. (2007). A reconsideration of cognitive load theory. Educational PsychologyReview, 19(4), 469508.

Schnotz, W., & Lowe, R.K. (2008). A unified view of learning from animated and static graphics. In R.K.Lowe & W. Schnotz (Eds.), Learning with animation. Research implications for design (pp. 304356).New York: Cambridge University Press.

Thorsten Rasch. Faculty of Psychology, University of Koblenz-Landau, Fortstrae 7, 76829 Landau,Germany. E-mail: [email protected]


Textpicture integration, learning from animation, executive functions of working memory.


Lowe, R.K., Rasch, T., & Schnotz W. (2010). Aligning affordances of graphics with learning task requirements.Applied Cognitive Psychology. doi:10.1002/acp.1712.

Meyer, K., Rasch, T., & Schnotz, W. (2010). Effects of animations speed of presentation on perceptualprocessing and learning. Learning and Instruction, 20(2), 136145.

Rasch, T. (2006). Verstehen abstrakter Sachverhalte. Semantische Gestalten in der Konstruktion mentalerModelle. Berlin: wvb.

Rasch, T., & Schnotz, W. (2009). Interactive and non-interactive pictures in multimedia learning environ-ments: Effects on learning outcomes and learning efficiency. Learning and Instruction, 19, 411422.

Schnotz, W., & Rasch, T. (2005). Enabling, facilitating, and inhibiting effects of animations in multimedialearning: why reduction of cognitive load can have negative results on learning. Educational Technology:Research and Development, 53(3), 4758.

Mark Ullrich. Institute of Psychology, University of Frankfurt, Mertonstrasse 17 (Jgel-Haus), 60325Frankfurt, Germany. E-mail: [email protected]


Learning from text and pictures, adult education.


Lintorf, K., McElvany, N., Rjosk, C., Schroeder, S., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M. (2011).Zuverlssigkeit von diagnostischen LehrerurteilenRealiabilitt verschiedener Urteilsmae bei derEinschtzung von Aufgabenschwierigkeiten. Unterrichtswissenschaft, 39, 102120.

McElvany, N., Schroeder, S., Hachfeld, A., Baumert, J., Richter, T., Schnotz, W., Horz, H., & Ullrich, M., &(2010). Diagnostische Fhigkeiten von Lehrkrften bei der Einschtzung von Schlerleistungen undAufgabenschwierigkeiten bei Lernmedien mit instruktionalen Bildern. Zeitschrift fr PdagogischePsychologie, 23(34), 223235.

McElvany, N., Schroeder, S., Richter, T., Hachfeld, A., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M.,(2010). Texte mit instruktionalen Bildern als Unterrichtsmaterial - Kompetenzen der Lehrkrfte. Unter-richtswissenschaft, 38, 98116.

Schnotz, W., Horz, H., McElvany, N., Schroeder, S., Ullrich, M., Baumert, J., Hachfeld, A., & Richter, T. (2010).Das BITE-Projekt: Integrative Verarbeitung von Texten und Bildern in der Sekundarstufe I. Zeitschrift frPdagogik, 56, 143153. In Klieme, Eckhard/Leutner, Detlev/Kenk, Martina: Kompetenzmodellierung.Zwischenbilanz des DFG-Schwerpunktprogramms und Perspektiven des Forschungsansatzes. 56. Beiheftder Zeitschrift fr Pdagogik, Weinheim u.a.: Beltz.



Holger Horz. Institute of Psychology, University of Frankfurt, Mertonstrasse 17 (Jgel-Haus), 60325Frankfurt, Germany. E-mail: [email protected]


Learning from text and pictures, instructional media, adult education, cognitive load, instructional prompts.


Horz, H. (2010). Lernen mit Medien. In Reinders, H., Ditton, H., Grsel, C., & Gniewosz, B. (Hrsg.),Lehrbuch Empirische Bildungsforschung (S. 2132). Wiesbaden: VS-Verlag fr Sozialwissenschaft.

Horz, H. (2012). Situated prompts in authentic learning environments. In N. M. Seel (Ed.), Encyclopedia ofthe sciences of learning (pp. 30863087). Heidelberg: Springer.

Horz, H., & Schnotz, W. (2010). Cognitive load in learning with multiple representations. In J. L. Plass, R.Moreno, & R. Bruenken (Eds.), Cognitive load: Theory & application (pp. 229252). New York:Cambridge University Press.

Horz, H., Winter C., & Fries, S. (2009). Differential benefits of instructional prompts. Computers in HumanBehavior, 25, 818828.

Ullrich, M., Schnotz, W., Horz, H., McElvany, N., Schroeder, S., & Baumert, J. (2012). Kognitionspsycho-logische Aspekte der Bild-Text-Integration. Psychologische Rundschau, 63, 1117.

Nele McElvany. Institut fr Schulentwicklungsforschung (IFS), Technical University ofDortmund, Vogelpothsweg78, 44227 Dortmund, Germany. E-mail: [email protected]


Lehr-Lernforschung im schulischen Kontext und im Elternhaus, Lesekompetenz, Kompetenzen von Lehrk-rften, Bildung und Migration, bergang von der Grundschule auf die weiterfhrende Schule.


Lintorf, K., McElvany, N., Rjosk, C., Schroeder, S., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M.(2011). Zuverlssigkeit von diagnostischen Lehrerurteilen - Reliabilitt verschiedener Urteilsmaebei der Einschtzung von Schlerleistungen und Aufgabenschwierigkeiten. Unterrichtswissen-schaft, 39(2), 102120.

McElvany, N. (2011). Familire Bedingungsfaktoren von Lesekompetenz und Effektivitt systematischerFrderung. In: Bundesverband Alphabetisierung und Grundbildung e.V./Bothe, Joachim (Hrsg.).Funktionaler Analphabetismus im Kontext von Familie und Partnerschaft. Alphabetisierung undGrundbildung Band 8, (S. 6271). Mnster/NewYork/Mnchen/Berlin: Waxmann.

Rjosk, C., McElvany, N., Anders, Y., & Becker, M. (2011). Diagnostische Fhigkeiten von Lehrkrften bei derEinschtzung der basalen Lesefhigkeit ihrer Schlerinnen und Schler. Psychologie in Erziehung undUnterricht, 58(2), 92105.

Schroeder, S., Richter, T., McElvany, N., Hachfeld, A., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M. 547(2011). Teachers beliefs, instructional behaviours, and students engagement in learning from texts with548 instructional pictures. Learning and Instruction, 21(3), 403415.

van Steensel, R.C., McElvany, N., Kurvers, J.J., & Herppich, S. (2011). How effective are family literacyprograms?: results of a meta-analysis. Review of Educational Research, 81, 6996. March 2011

Jrgen Baumert. Max-Planck-Institute of Human Development, Lentzeallee 94, 14195 Berlin, Germany.E-mail: [email protected]


Learning and instruction, Teacher expertise, Large-scale assessments, Development of educational systems,Cognitive and motivational development in childhood and adolescence.




Baumert, J., Nagy, G., & Lehmann, R. (2012). Cumulative Advantages and the Emergence of Social andEthnic Inequality: Matthew Effects in Reading and Mathematics Development Within ElementarySchools? Child Development, 83(4), 1347-1367. doi:10.1111/j.1467-8624.2012.01779.x.

Stanat, P., Becker, M., Baumert, J., Ldtke, O., & Eckhardt, A. G. (2012). Improving second language skills ofimmigrant students: a field trial study evaluating the effects of a summer learning program. Learning andInstruction, 22(3), 159-170.

Becker, M., Ldtke, O., Trautwein, U., Kller, O., & Baumert, J. (2012). The differential effects of schooltracking on psychometric intelligence: Do academic-track schools make students smarter? Journal ofEducational Psychology, 104(3), 682-699.

Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., Klusmann, U., Krauss, S., Neubrand,M., & Tsai, Y. (2010). Teachersmathematical knowledge, cognitive activation in the classroom, and studentprogress. American Educational Research Journal, 47(1), 133-180. doi:10.3102/0002831209345157.

Baumert, J., Becker, M., Neumann, M., & Nikolova, R. (2010). Besondere Frderung von Kern- kompetenzenan Spezialgymnasien? Der Frhbergang in grundstndige Gymnasien in Berlin. Zeitschrift fr Pda-gogische Psychologie, 24(1), 5-22. doi:10.1024/1010-0652.a000001.

Baumert, J., Ldtke, O., Trautwein, U., & Brunner, M. (2009). Large-scale student assessment studiesmeasure the results of processes of knowledge acquisition: evidence in support of the distinction betweenintelligence and student achievement. Educational Research Review, 4(3), 165-176.



Dynamics of mental model construction from text and graphicsAbstractTheoretical backgroundModels of textpicture integrationTaxonomy of textpicture integrationStrategies for integrative processing of text and graphics

Research questions and hypothesesMethodResultsReading textsObserving graphicsAverage times per fixationEfficiency of textgraphic integration

DiscussionReferences

00pp 2012 - dynamics of mental model construction from text and graphics

Documents