An evaluation report of multimedia environments as cognitive learning tools

Download An evaluation report of multimedia environments as cognitive learning tools

Post on 15-Sep-2016




2 download


An evaluation report of multimedia environments as cognitivelearning toolsNorbert M. Seel*, Katharina SchenkFreiburg Institute of Educational Science, Albert-Ludwigs University, Freiburg, GermanyReceived in revised form 1 December 2001AbstractThis article deals with the evaluation of a multimedia learning environment which has been developed and evaluated within the broadercontext of a research project on the learning-dependent progression of mental models in economics. To carry out formative evaluations, wehave adapted a particular evaluation approach which allows and requires the implementation of specific evaluation instruments. The crucialquestions of our evaluation studies were the efficacy of a multimedia-based realization of the cognitive apprenticeship (CA) approach, thediagnosis of mental model progression through the CA based instruction, and the effects of implemented metacognitive training. For theassessment of the learning-dependent progression of the mental models, we developed and used a special diagnostic instrument for causaldiagrams, which are understood as reproductions of students mental models. In order to be able to meet statements about the practicability ofa multimedia based realization of CA, we measured the results of the tasks of learning during each different learning phase. Additionally,several motivational variables and persistent learning strategies were measured. In this article, we will specify the adapted evaluationinstruments. Furthermore, we will report on the results of five replication studies and discuss the consequences for instructional design inconnection with the design of constructive learning environments.q 2003 Elsevier Science Ltd. All rights reserved.Keywords: Causal diagrams; Cognitive apprenticeship; Instructional design; Mental models; Metacognitive training1. IntroductionThere is considerable concern that students thinkingskills, motivational dispositions, and domain-specificknowledge might be inadequate for them to lead fulfillinglives in a global, information-rich, technology-orientedworld. Informed by recent theory and research on learningand teaching, efforts to reform classroom instruction andcreate learning environments that promote these ends areunderway. Hannafin (1992) argues that the improvement ofproblem-solving abilities and other key skills requiresemergent technologies in order to design effective learningenvironments that provide opportunities for reflectivethinking. However, this argument can be seen from differentpoints of view. One approach focuses on the improvement ofstudents technological literacy and advocates a new type ofunderstanding of information and communication technol-ogy in educational settings. Another approach focuses on theeffective instructional design (ID) of multimedia environ-ments as opposed to the technology itself. Obviously, theseperspectives are not mutually exclusive as each may beconsidered within the context of ID.ID is a theoretically sound educational technology for thedevelopment, implementation, and evaluation of learningenvironments that are adapted to learners, tasks, resources,and contexts (Tennyson, Schott, Seel, & Dijkstra, 1997). Ananalysis of the literature indicates that there is broadconsensus with regard to ID requirements and how toevaluate various designs in different contexts (Weston,McAlpine, & Bordonaro, 1995). Evaluation has beenconsidered a central and necessary part of instructionalplanning from the very beginning of ID (Andrews &Goodson, 1980; Cronbach, 1963), wherein a differentiationis made between two kinds of evaluation (Scriven, 1967). formative evaluation aimed at the improvement ofinstruction by means of feedback of informationconcerning the effective use and outcomes; and,0149-7189/03/$ - see front matter q 2003 Elsevier Science Ltd. All rights reserved.doi:10.1016/S0149-7189(03)00003-XEvaluation and Program Planning 26 (2003)* Corresponding author.E-mail addresses: (N.M. Seel), (K. Schenk). summative evaluation aimed at the measurement of thedegree to which intended results are achieved.A further distinction is often drawn with regard to theinstruments of evaluation, insofar as quantitative orqualitative data are to be assessed.This article is concerned with the evaluation of aparticular multimedia learning environment which wasdeveloped and evaluated as part of a comprehensiveresearch project focusing on the learning-dependentprogression of mental models. Insofar as the constructionof mental models presupposes constructive learning,which occurs when learners actively construct meaningfulmental representations during instruction (Mayer, Moreno,Boire, & Vagge, 1999), multimedia is considered to bethe most effective for the promotion of such constructivelearning. In the case of the given multimedia-learningenvironment, the meaningful mental representation is acoherent mental model of a dynamic model of macro-economics and financial politics, respectively. In ourresearch, learning outcomes were evaluated with multiplemeasuring instruments. On the one hand, students had toproduce causal diagrams of the problem situation; on theother hand, student solutions to learning tasks and transferproblems were tested.In the following sections we will illuminate theformative evaluation of the learning environmentDynamic Systems of Economics (DSE) as applied inseveral replication studies. An evaluation model will bedescribed with the focus on the following components:methodology, data analysis, and interpretation. We willconclude with a discussion of the main results of severalevaluation studies.2. Model-based learning and instructionOur research group has been involved in the develop-ment and investigation of instructional intervention pro-grams aimed at the improvement of model-based learningand thinking for several years. The epistemological andpsychological foundations of this research rest on Seels(1991) theory of mental models.Mental models emerged in the 1980s as a theoreticalconstruct to encompass both situated cognition as well asqualitative reasoning. Greeno (1989) argues that compre-hension of and reasoning in specific situations necessarilyinvolves the use of mental models of different qualities.Mental models are a central construct of symbolic models ofhuman cognition that presuppose the use and manipulationof symbols. Following Wartofsky (1979), cognition takesplace while using mental representations in which individ-uals organize symbols of experience or thought in such away that they effect a systematic representation of thisexperience or thought as a means of understanding orexplaining it to others.Mental models play a central and unifying role inrepresenting objects, states of affairs, sequences ofevents, and the social and psychological actions ofdaily life. They enable individuals to make inferencesand predictions, to understand phenomena, to decidewhat action to take and to control its execution,and, above all, to experience events by proxy (Johnson-Laird, 1983, p. 397)Craik (1943) introduced the idea of internal models topsychology with the notion of a working model. Accordingto Craik, most cognitive theorists agree on the point thatmental models serve primarily to create situation-specificplausibility. Due to an idealized reduction to relevantcharacteristics of its original, a model is a concrete,comprehensible, and feasible representation of non-obviousor abstract objects. The representation of the objectsattributes and components comes second to the represen-tation of structural relationships. Mental models are not aspecific representational format such as images andpropositions, but rather higher-order cognitive constructions(artifacts) which refer primarily to the content of mentalrepresentations (Seel, 1991).The functions of mental models, including structuralfeatures, are defined on the basis of the objectives of themodel-constructing person. In physics and other sciences,the term model is always used in a functional sense.Appearance models may serve to simplify a complexphenomenon or represent structural relationships visually.On the other hand, derivative (thought) models (e.g.Rutherfords model of the atom) serve primarily to aidanalogical reasoning in exploring phenomena (e.g. quantummechanisms). Mental simulations occur when cognitiveoperations simulate (in the sense of thought experiments)specific transformations of objects that may occur in real-life situations. In sum, mental models run in the mindseye to produce qualitative inferences with respect to thesituation to be cognitively mastered.Although mental models may differ markedly in theircontent, there is no evidence to suggest that they differ inrepresentational format or in the processes that constructand manipulate them. What is at issue is how suchmodels develop as an individual progresses from noviceto expert, and whether there is any pedagogicaladvantage in providing people with models of tasksthey are trying to learn (Johnson-Laird, 1989, p. 485)In accordance with Snow (1990) we have identified thelearning-dependent progression of mental models as aspecific kind of transition mediating between studentpreconceptions, which describe the initial states of thelearning process, and causal explanations, which describethe desired end states of the learning process (Seel,Al-Diban, & Blumschein, 2000). From the perspective ofinstructional psychology, the guiding principle forN.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224216influencing the construction of mental models has beenexpressed by Mayer (1989) as follows: Students givenmodel-instruction may be more likely to build mentalmodels of the systems they are studying and to use thesemodels to generate creative solutions to transfer problems(p. 47). This presupposes that the learner is sensitive to themodel-relevant characteristics of the learning environment,such as the availability of certain information at a giventime, the way this information is structured and mediated,and the ease with which it can be found in the environment.An analysis of the relevant literature indicates that thesuggested instructional strategy to provide learners with adesigned conceptual model actually constitutes the maintrend of instructional research on mental models. Accordingto Carlson (1991), instruction can be designed to involve thelearner in an inquiry process in which facts are gatheredfrom data sources, similarities and differences among factsnoted, and concepts developed. In this process, theinstructional program serves as a facilitator of learning forstudents who are working to develop their own answers toquestions. In this case, mental models are more proactiveand direct the learning experiences so that the result oflearning is dependent on the initial model, defined as thelearners a priori understanding of the material to belearned. On the other hand, instructional programs canpresent concepts with clear definitions followed by clearexamples. A conceptual model may be presented before thelearning tasks in order to direct the learners comprehensionof the learning material. Over the past decades muchresearch applying this strategy has been done to providestudents with model-based instruction but several authors(Royer, Cisero, & Carlo, 1993; Snow, 1990) have objectedthat this kind of research has typically been done piece-meal, in small-scale, specialized contexts. In order toovercome these shortcomings we need a more comprehen-sive instructional approach. Cognitive apprenticeship (CA)(Collins, Brown, & Newman, 1989) provides a fundamentalbasis for initiating and directing model-based learning.Our research group started in 1994 with the developmentof a multimedia environment aimed at an externally guided,goal-oriented, and systematic influence upon the learnersprogression of mental models. CA (Collins et al., 1989) wasthe only promising instructional strategy correspondingwith the idea of providing the students with model-instruction in the aforementioned sense. There are sixinstructional methods in CA: modeling, coaching, scaffold-ing, articulation, reflection, and exploration. The instruc-tional intervention of apprenticeship starts with thepresentation of an experts conceptual model of the tasksto be accomplished, and then the students are coached andscaffolded to adapt this model for their own solutions(exploration) to the learning tasks designed. CA is based onresults of cognitive psychology and applies these results in aprescriptive way in order to identify ideal features oflearning environments. This approach prescribes in detailwhat the learner has to do and in which sequence in order toachieve particular objectives. However, the question as towhether the CA approach may be appropriate for the designof multimedia environments could not be answered at thistime. There are several studies (Casey, 1996; Chee, 1996;Jarvela, 1995) that have investigated this. However, thesestudies run parallel with our investigations and finalconclusions are not available. Therefore, we focused onthe issue of whether the preferred use and applicationof multimedia technology allows a strict adaptation ofinstructional regularities to individual regularitiesof learning.The research we have done in the past 6 years hascentered around two main topics:1. the investigation of the learning-dependent progressionof mental models, more specifically of analogy models ofDSE; and,2. how this progression can be guided or influenced througha particular instructional intervention program designedas a multimedia environment in accordance withprinciples of CA. We focus on the second line ofresearch in this paper.3. The evaluation modelWe have adopted the evaluation approach of Ross andMorrison (1997) with these main components: (1) needsassessment, (2) methodology, (3) data analysis andinterpretation, and (4) dissemination results. The evaluationof DSE focused on methodology and data analysis andinterpretation. Accordingly, we have realized: a program analysis in order to determine the content andmethods of their mediation within the multimediaprogram; a participants analysis in order to determine the (groupsof) learners as well as the scope of the instructionalprogram; a specification of the evaluation design; the development of instruments of measurement; and, the implementation and control of the evaluationdesign.Following these methodological steps, the analysis andinterpretation of data was to be done in order to modify orrevise the instructional program or parts of it. Accordingly,the data analysis centers around the formative evaluation ofinstruction during the development phase for the purpose ofimprovement.3.1. Program analysisThe multimedia program is designed to explain thedynamics of economic systems and to introduce themonetary policy of the European Central Bank intoN.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224 217students discussion. In order to facilitate student construc-tion of adequate mental models, several conceptual models,especially a circuit model of economic systems, arepresented. According to the CA approach, effective learningenvironments can be characterized by 18 features in fourbroad dimensions: content, methods, sequencing, and thesociology of teaching. We focused especially on methods,which encompass the features of modeling, coaching,scaffolding, articulation, reflection, and exploration.Altogether, we integrated the relevant aspects of the CAapproach into a comprehensive model, as described in Fig. 1.We realized CA in the following way (Seel et al., 2000).In modeling, an expert demonstrates a solution to a problem,and the students acquire an initial model of this process byobserving the experts approach. In coaching, the studentsare supervised and given guidance as they try to findsolutions to a given task in an adaptive manner. Inscaffolding a special problem-solving heuristics is taught.The realization of articulation and reflection proved to beproblematic within the multimedia program. Articulation isthe process of thinking aloud while working on a task, andreflection is the comparison of the problem solvingprocedures applied by the learner and the expert respect-ively. We realized both methods in the form of a teach-back procedure (Jih & Reeves, 1992; Sasse, 1991) in asocial learning situation. This procedure is based on theconstructive interaction between two communicationpartners who share similar domain-specific knowledge.In exploration, the final part of the apprenticeshipinstruction, the learners have to solve transfer tasksoneof them requires a near transfer (i.e. the task remains in thesame subject matter domain of economics) the other onerequires a far transfer from economics to ecology. On thewhole, the multimedia program realized the methods of CAin the sequence illustrated in Fig. 2.Additionally, two different instructional strategies aimedat the improvement of analogical problem solving wererealized in modeling as well as in scaffolding: subsumption of analogous learning tasks under a schemaof a problem-solving structure, followed by its instantia-tion through a detailed, worked-out example; and, induction of a more general problem-solving schemafrom analogous learning tasks by the comparison ofdifferent examples in order to extract structuralsimilarities.3.2. SubjectsTaking curriculum constraints into account, the multi-media program DSE is directed at 12th grade students ofGerman secondary schools (on average 18 years old). Twopilot studies with university students indicated that theprogram could also be used effectively with college studentsin non-economic disciplines. In sum, more than 400students have worked with DSE in various instructionalsettings.3.3. The evaluation designIn order to ensure that goals were being achieved and toimprove the multimedia program, the evaluation designfocused on formative evaluation as an iterative process oftrying out and revising of instruction during development.However, formative evaluation can be conceived of in avariety of ways (Weston et al., 1995). We implementedformative evaluation as an iterative process in a series ofreplication studies aimed at the gradual enhancement ofconfidence. This process included empirically gatheredperformance data and their interpretation.Altogether, we realized five replication studies aimed at(a) the formative evaluation of the multimedia program, and(b) the diagnosis of mental models. The strategy offormative evaluation applied related to the methods of theFig. 1. A view of the CA model (Seel et al., 2000).N.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224218CA, and aimed at the gradual improvement of theinstruction (Table 1). This strategy can be sketched asshown in Table 1.This evaluation plan is based on the assumption that theinstruction can be considered effective with regard to theachievement of goals if theoretically prognosed changes incriterion variables can be observed. However, the efficacy ofthe instruction may also depend on effects other than thepostulated ones, for example, on characteristics of theinstruction that are independent of the multimedia programeffects. This means, that measurable effects of an instruc-tional intervention are not sufficient to justify the drawing ofconclusions about the model of efficacy applied.3.4. Instruments of evaluationA central goal of instruction is to improve studentperformance, defined in terms of domain-specific knowl-edge, skills, strategies, attitudes, and behavioral disposi-tions. We assessed student domain-specific knowledgeusing a test developed by Beck (1993). This test isconstructed in accordance with Blooms taxonomy ofcognitive objectives and permits statements about thequality of knowledge in the field of economics. We assessedthe quantity and quality of learning and transfer tasksaccomplished within the learning environment. In coaching,we measured the frequency and self-corrections of errors aswell as the time required to accomplish learning tasks,whereas in scaffolding and exploration the frequency andtype of correct solutions were assessed. In order to assess thelearning-dependent progression of mental models, ourresearch group developed a special test procedure of causaldiagrams, which can considered to be a combination ofcognitive modeling and a particular structure-spreadingtechnique similar to concept mapping. Additionally, in twostudies we applied receptive interviews to assess the qualityof mental models, but this procedure proved to beineffective (Al-Diban, 2001). Finally, we also used protocolanalyses of the teach-back phase of instruction.Additionally, several motivational variables and persist-ent learning strategies were measured, including: learnerachievement motivation, by means of a questionnaireTable 1Overview of the evaluation studiesStudy Methods of CA implementedPilot study Modeling Coaching Study 1 Modeling Coaching Scaffolding ExplorationStudy 2 Modeling Coaching Scaffolding Computer-based, and individual articulation and reflection ExplorationStudy 3 Modeling Coaching Scaffolding Articulation and reflection as teach-back procedure ExplorationStudy 4 Modeling Coaching Scaffolding Articulation and reflection as teach-back procedure ExplorationStudy 5 Modeling Coaching Scaffolding Articulation and reflection as training of metacognitive strategies ExplorationFig. 2. The methods of CA as implemented in the multimedia program.N.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224 219developed and validated by Rollett and Bartram (1977);learner interests and attitudes, measured with a self-constructed questionnaire; student perceptions of thelearning situation, also measured with a self-constructedquestionnaire to assess the climate of multimedia-learning(Seel, 1980); and, student persistent learning strategies, witha questionnaire by Wild and Schiefele (1994).In addition to these external criteria we realized anobjective task analysis and a subjective task analysis todetermine whether the instruction was appropriated withregard to curriculum and content criteria. Task analysisprocedures (Jonassen, Tessmer, & Hannum, 1999) pre-suppose different criteria to achieve a sufficient differen-tiation of the requirements and a classification of learningtasks: (a) sub-task differences must be unambiguouslyobservable; (b) the task analysis should be derivedindependently from a specific topic and should have generalvalidity; and (c) the analysis should be independent fromknowledge and dispositions of the application. In accord-ance with these requirements, we applied a procedure ofproblem and task analysis which Hacker, Sachse, andSchroda (1998) developed especially for complex andauthentic tasks. With this procedure it was possible toseparate several dimensions, such as the complexity oflearning tasks (e.g. number of partial functions of a systemand of their relationships), the consistency of objectives(number of goals, number of contradictory goals, degree ofconcurrence of goals), the transparency of possiblesolutions, degrees of freedom (variants of solutions),dynamics (change of constraints), and the prior knowledgenecessary. By following this procedure, we were able tocharacterize each learning task by means of a multi-dimensional and objective profile of demands.3.5. Implementation of the evaluation planThe evaluation plan contains one pilot study withuniversity students and five replication studies with 12thgrade German students (Table 1). All studies were carriedout outside the school at the Laboratory for MultimediaResearch of the Technical University of Dresden (19941998) and the University of Freiburg (1998 2001),enabling a strict control of external factors. The recruitmentof subjects took place on the basis of announcements inschools, and the volunteers received a nominal fee. Subjectswere assigned to the different treatments randomly.4. ResultsThis section starts with the report on findings of the tasksanalyses. In the next section we describe the main results ofdifferent evaluation studies on the effectiveness of themultimedia program in accordance with CA principles andthe development of mental models.4.1. Results of task analysesWe realized a task analysis as part of the secondreplication study in order to get information concerning thedegree of difficulty and the content-related quality of thelearning tasks in coaching, scaffolding, and exploration. Weasked three experts in the field of economics to analyze thelearning tasks, applying the aforementioned procedure ofHacker et al. (1998). The experts judgments concerning thedimensions of the task analysis obtained an averageagreement of at least 65% (Table 2).The results indicate that the experts estimated thedimension complexity, dynamics, and transparency tobe most important for the task difficulty whereas thedimensions inconsistent goals and degrees of freedomwere rated as less important. The task analysis alsodemonstrated that the instruction satisfied the apprentice-ship features of increasing complexity and variety as well asof increasing abstraction of the learning tasks in coaching,scaffolding, and exploration. As a result of the experts taskanalysis, three learning tasks of scaffolding and two of thefive exploration tasks were crossed from the list due toevident weaknesses in content and overly strong demandson knowledge.In addition to the experts task analysis, we also askedstudents to estimate the difficulty of the learning taskswithin the various apprenticeship methods. In the secondstudy, 84 subjects rated the difficulty of the coaching-tasksas not difficult, whereas the learning tasks of scaffoldingand exploration were judged to be more difficult anddifficult to solve. Within each method there were nosignificant differences concerning the difficulty of thelearning tasks. However, there were significant differencesTable 2Average agreements of experts and judgments on the relevance of the dimensionsDimensions Inconsistencyof goalsLevel ofcomplexityLevel oftransparencyDegree offreedomLevel ofdynamicsNecessity ofprior knowledgeAgreement ofexperts on average (%)69.8 73.0 72.0 73.0 65.0 78.9Average relevanceof the dimensions1.7 4.3 3.3 2.3 4.0 2.6N.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224220between these methods, coaching being the least difficult,scaffolding more difficult, and exploration the most difficultof the three. Interestingly, this increase of cognitive effortinvolved in the solution of the learning tasks depending onthe method of apprenticeship instruction correlated with thecognitive performance as measured with the knowledge testof Beck (1993). There was a correlation of r 0:469 withthe measures in coaching but only a correlation of r 20:74 with the scaffolding tasks and of r 20:379 withthe exploration tasks. This result corresponds with theobjective results of the various evaluation studies, whichindicated that the learners could not perform the learningtasks in scaffolding and exploration as well as the coachedtasks.4.2. Results of the evaluation studiesOn the whole, the results of the five evaluations withmore than 400 subjects allow the statement that the CAapproach can be considered to be a sound framework for theID of environments aiming at constructivist learning.Results of the second evaluation study indicated that theattempts to install articulation and reflection in the multi-media program failed insofar as the written statements andcomments required of the learners were revealed to be noteffective with regard to the reflective thinking aboutproblem-solving procedures which was intended. Theseresults correspond with Caseys (1996) observation thatinteraction with the computer can provide a great deal ofinformation on why learners make the choices they make,but computer-based intelligence seems pale in comparisonto the open peer dialogue we observed learners havingduring the testing of CI (p. 83).As a result of these findings we implemented articulationand reflection in the third study and later in the form of aparticular teach-back-procedure in accordance with Sasse(1991). Moreover, in the fifth replication study we alsoapplied the method of generative teaching by Kourilskyand Wittrock (1992), but only with moderate effects. Bothof these alternatives for the realization of articulation andreflection in a cooperative manner should be investigated inmore detail in further studies.All replication studies agree on the point that the learnerscome out on top in accomplishing the learning tasks ofcoaching. Successful learners are characterized both byfewer mistakes in task solutions of coaching and by longerlearning times for these tasks, indicating a well-plannedmethod in accomplishing the learning tasks. This appren-ticeship method evidently aims at controlled content-oriented learning and narrows down the learners to imitatethe expert model they are provided in modeling.In all of the studies we observed a significant decrease ofperformance in scaffolding, which is characterized by a fadingof external guidance maintained. The learning outcomes of allreplication studies show that learners had difficulty indeveloping and applying their own problem-solvingstrategies. The significant decrease of performance fromcoaching to scaffolding, which was observable in all of thereplication studies, indicates that the learners obviouslycould not progress effectively from content- to process-oriented learning in the sense of an increasingly self-regulated accomplishment of analogous tasks.In designing scaffolding we followed the statements ofCollins et al. (1989) and the standard procedures ofcognitive task analysis (Jonassen et al., 1999). Accordingly,we organized the tasks in a structure that we considered tobe best for the solution of learning tasks by analogy. That is,we provided the learners with a learning task which theycould solve easily and then we increased the difficulty oftasks until the learners were no longer able to solve them ontheir own. As an alternative, Hmelo and Guzdial (1996)considered the redesign of learning tasks in such a way thatthey support task performance with the help of asupplantation (Salomon, 1979) of those cognitive oper-ations that are involved in the task solutions. Moreover, taskperformance can be supported with cognitive tools whichgive advice to learners by representing a problem and thenmanipulating the representation in the process of finding asolution. These forms of scaffolding are taken by Hmelo andGuzdial (1996) as examples of glass-box scaffoldingwhich aims at giving help to learners in situations whichthey cannot master on their own. Moreover, the results ofour studies concerning the efficacy of scaffolding alsocorrespond with other investigations aimed at constructivistlearning.Comparable with the learning results in scaffolding, thesubjects did not perform the two transfer tasks administeredin exploration well. One of the tasks remained within thesame subject matter domain of fiscal policy and thus aimedat a near transfer whereas the second task required thetransfer of freshly acquired knowledge into the differentsubject matter area of ecology (i.e. far transfer). As inscaffolding, the subjects achieved only average results onthese transfer tasks in the various replication studies. Thisresult generally corresponds with the literature on learningtransfer (Seel, 2000) insofar as the learners strong fixationon content-oriented learning was consistently to thedetriment of learning effectiveness. Actually, with regardto the accomplishment of transfer tasks, student perform-ance in coaching was revealed to be a significant predictor.From a didactic point of view, these results are notsatisfactory, especially with regard to the results ofregression analyses, according to which scaffolding didnot influence the requested transfer in exploration.Jonassen, Marra, & Palmer (2003) point out with regardto the efficacy of so-called constructivist learning environ-ments that they often do not meet the expectations ofstudents (and teachers) with regard to their effects onlearning. This in turn may lead to motivational problems instudents (Jonassen & Tessmer, 1996/1997). With regard tothe cognitive effects of the instruction on learning andtransfer, the data of our replication studies correspond withN.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224 221the argument of Jonassen and Marra. However, with regardto motivational effects, we found consistently positivelearning emotions as well as explicitly good and persistentmotivation to learn. Actually, the findings of the variousreplication studies show that there is a strong tendencytowards a positive motivation to learn and related interestsfor these students in the domain of economics. Therefore, itwas not surprising that in the fifth study the learnerscontent-oriented interests contributed to the variance ofperformance in accomplishing the learning tasks ofscaffolding. Taking into account this finding as well as theobservation that enduring learning strategies evidently didachieve greater effects than metacognitive training, we canconclude that instructional interventions of a short durationare strongly limited with regard to their properties.In the pilot study and in two replications, we investigatedthe efficacy of subsuming worked examples under a generalschema of solving problems vs. inducing such a schemafrom worked examples. The data of these studies proved tobe inconsistent insofar as the strategy of subsumption wassuperior to the strategy of induction in the pilot study withregard to both the amount of mistakes and the time neededfor accomplishment of the learning tasks in coaching,whereas in the second study a superiority of the inductivestrategy could be assessed with regard to the performancesin scaffolding. However, the fourth replication indicatedthat there are no significant differences between the twoteaching strategies. Only in one case, namely in themeasured procedural knowledge in performing the secondtransfer task in exploration, was the inductive instructionrevealed to be more effective.With regard to the measurement of the learning-dependent progression of mental models, in all of thevarious replication studies we could observe significantchanges in the accomplishment in producing causaldiagramsunderstood here as external representations ofmental models. Much clearer were the results of regressionanalyses, which indicated that the quality of the accom-plishment of the learning tasks of increasing complexitywere influenced both by the learning experiences in thecourse of instruction and the enduring learning strategies ofthe subjects. The empirical findings of the fifth study showsignificant learning progress in all experimental groupsdepending on the complexity of their causal diagrams. Weinterpret the substantial changes of causal diagramsobserved as a proof for the general effectiveness of theinstructional intervention. Our findings also support thetheoretical assumption that subjective causal modelsasmeasured with the help of causal diagramsare situation-dependent constructions. Remaining within the samecontext of contents, the learners did construct causaldiagrams as cognitive artifacts correlating only minimallywith each other. Instructionally relevant are the results ofour investigations concerning the adoption of the concep-tual models the students were provided with in theinstruction. Our data contradict the assumption that studentsadapt externally provided models and apply them to solvetasks. In our replication studies the learners causaldiagrams showed only minor similarities with the concep-tual models provided during instruction. Although con-tingency coefficients indicate that the causal diagrams arenot fully independent of the conceptual models, thecorrelation was not significant. Basically, we can agreewith Mayers (1989) verdict that students given model-instruction may be more likely to build mental models of thesystems they are studying and to use these models togenerate creative solutions to transfer problems (p. 47), butat the same time it is clear that the students do not adapt theconceptual model provided through instruction one-to-one.The results of the fifth evaluation study (Schenk, 2003)did not indicate that metacognitive training had any effecton the performance in the scaffolding and exploration partof the learning program, but it did show significantdifferences in different qualities of metacognitive knowl-edge. The decisive predictors for the performances inscaffolding were the students content-oriented interests.When metacogitive training was implemented, severallearning strategies had a significant effect on the accom-plishment of the transfer tasks in exploration. Thiscorresponds with the results of previous investigations(Seel & Dinter, 1995). However, the amount of metacog-nitive statements correlated positively with performances,measured by the complexity of causal diagrams anddeclarative knowledge in scaffolding.5. ConclusionsOver the projects life it was fruitful to realize and toevaluate the methods of the CA approach with the help of astep-by-step formative evaluation strategy. Parallel withthis, we investigated the learning-dependent progression ofmental models with the help of a specific diagnosis thatconfirmed central assumptions of the theory of mentalmodels.In the course of five replication studies, the suitabilityof CA for the design of particular multimedia learningenvironments could be proved in generalexceptionallythe methods of articulation and reflection, for which therealization of a teach-back procedure proved to be moreeffective with regard to the required metacognitiveactivities. Independent of two instructional strategies(subsumption vs. induction) that were applied to designthe methods of modeling and scaffolding, the subjectsachieved excellent performance in the accomplishment ofthe learning tasks administered in coaching. This consist-ently positive result in all of our evaluation studiesindicates that the multimedia instruction was very effectivefor the improvement of content-oriented learning. Ingeneral, scaffolding aimed at fading instructional advicecan be considered the weak spot of the multimediainstruction. Although the subjects would have liked moreN.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224222influencing control and alternatives during instruction,they were not able to construct satisfactory problemsolutions or transfer their knowledge to other fields ofcontent. So far our results correspond with observationsand empirical results of other studies, such as Casey(1996), Chee (1996) and Lajoie and Lesgold (1989),which suggest that CA principles are suitable for the ID oflearning environments. Reservations include the obser-vation that it was very difficult to implement articulationand reflection in multimedia environments, and thatscaffolding and exploration, both of which aim to improveself-regulated learning, were less effective than expected.Taking into consideration the experiences we made indesigning the CA environment, we do not suggest therealization of the methods articulation and reflection formetacognitive improvement in a computer-based formatsince it has not yet been possible to implement an intelligenttutoring system in the program that might be able to guidethe learners self-verbalization and reflective thinkingadequately. Rather, we suggest the application of coopera-tive procedures such as the teach-back procedure in orderto realize articulation and reflection.The weak learning results our subjects achieved inscaffolding in the several replication studies actuallycontradict its popularity in the literature (Hmelo &Guzdial, 1996; Jonassen et al., 2003). This may be dueto the fact that the subjects of our studies wereconstrained by the instructional program and did notreceive additional advice through a teacher as advocatedby Palincsar (1986), who considers such a dialogue to bea solid basis for effective scaffolding. The multimediainstruction was not capable of adapting the learning tasksto the individual learner. For example, it cannot adaptthe difficulty of a learning task to the learners abilitiesin such a way that a learners missing knowledge couldbe compensated. Furthermore, the multimedia instructiondid not make appropriate cognitive tools available tosupport the learners in accomplishing the learning tasks.As in a study by Niegemann (1995), the availableprogram resources (e.g. icons, glossary, cribs) wereevidently not sufficient and they were not usedeffectively by the subjects of the replication studies.The lesson learned from the students average perform-ance in scaffolding is that this method should beprimarily oriented towards the idea of glass-boxscaffolding in the future, with the aim of expandingthe opportunities of multimedia learning with appropriatecognitive tools. We believe that the consistently averageachievement in exploration as well as in scaffolding canbe explained by enduring socialization effects of school-ing. Schooling is targeted primarily at content-orientedlearning along with the acquisition of a lot of contentknowledge. For the development and practice ofcognitive skills, less resources are used and thereforeprocess-oriented learning in the sense of the developmentof problem-solving skills can be seen as a negligiblequantity. This observation corresponds with the critics ofKrohne (1977) as well as with the major results ofTIMSS (Baumert, Bos, & Watermann, 1999).The results of our studies demonstrate that causaldiagrams can be considered to be a suitable method ofassessing mental models as knowledge constructions of ahigher order that develop depending on learning experi-ences. The pedagogical lesson learned from empiricalresults with regard to significant effects on the complexityof causal diagrams is that enduring learning strategies andprevious learning experiences may have greater influenceon the accomplishment of transfer tasks than temporaryinstructional strategies, which perhaps need more time tobecome effective than we could offer in our studies.Konrad (1998) argues that successful self-regulatedlearningwhich was our goal in implementing metacogni-tive training in the fifth studypresupposes that the learnerspossess not only strategic knowledge but also broad worldknowledge. These learners can also fall back on specificprevious experiences in various learning situations. There-fore, in future we should investigate in more detail whetherit would be effective to teach novice learners the applicationof metacognitive strategies in the course of the instruction ofnew subject matter.In examining these evaluation results, we conclude thatlearners constructed situation-bound problem solutionsindependently of teaching strategies provided duringinstruction, and that an effective design of successfullearning environments presupposes the provision of cogni-tive tools which facilitate and support individual model-building and revision aimed at problem solving. In thissense, it is doubtful whether multimedia environmentswhich guide students on a predetermined learning way areactually suitable to overcome the problems with problem-based learning (Hoffman & Ritchie, 1997).AcknowledgementsWe gratefully acknowledge financial support for thisresearch from a generous grant provided by the GermanResearch Association (Deutsche Forschungsgemeinschaft)with Grant-No. Se 399/4-1-3.ReferencesAl-Diban, S. (2001). Padagogische Diagnose mentaler Modelle. Freiburg:Albert-Ludwigs-Universitat.Andrews, D. H., & Goodson, L. A. (1980). A comparative analysis ofmodels of instructional design. Journal of Instructional Development,3(4), 216.Baumert, J., Bos, W., & Watermann, R. (1999) (2. Aufl.). TIMSS/III.Schulerleistungen in Mathematik und den Naturwissenschaften amEnde der Sekundarstufe II im internationalen Vergleich. Zusammen-fassung deskriptiver Ergebnisse, Berlin: Max-Planck-Institut furBildungsforschung.N.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224 223Beck, K. (1993). Dimensionen der okonomischen Bildung. Messinstru-mente und Befunde. Nurnberg: Universitat (Abschlussbericht zumDFG-Projekt Wirtschaftskundlicher Bildungs-Test [WBT]. Normier-ung und internationaler Vergleich).Carlson, H. L. (1991). Learning style and program design in interactivemultimedia. Educational Technology Research and Development,39(3), 4148.Casey, C. (1996). Incorporating cognitive apprenticeship in multi-media.Educational Technology Research and Development, 44(1), 7184.Chee, Y. S. (1996). Mind bridges: A distributed, multimedia learningenvironment for collaborative knowledge building. InternationalJournal of Educational Telecommunications, 2(2/3), 137153.Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprentice-ship: Teaching the crafts of reading, writing, and mathematics. In L. B.Resnick (Ed.), Knowing, learning, and instruction (pp. 453494).Hillsdale, NJ: Erlbaum.Craik, K. J. W. (1943). The nature of explanation. Cambridge: CambridgeUniversity Press.Cronbach, L. J. (1963). Course improvement through evaluation. TeacherCollege Record, 64, 672683.Greeno, J. G. (1989). Situations, mental models, and generative knowledge.In M. Klahr, & K. Kotovsky (Eds.), Complex Information Processing(pp. 285318). Hillsdale, NJ: Erlbaum.Hacker, W., Sachse, P., & Schroda, F. (1998). Design thinkingPossibleways to successful solutions in product development. In P. Badke-Schaub, H. Birkhofer, & E. Frankenberger (Eds.), DesignersThe keyto successful product development. London: Springer.Hannafin, M. J. (1992). Emerging technologies, ISD, and learningenvironments: critical perspectives. Educational Technology Researchand Development, 40(1), 4963.Hmelo, C. E., & Guzdial, M. (1996). Of black and glass boxes: Scaffoldingfor doing and learning. Proceedings of the Second InternationalConference on the Learning Sciences, Charlottesville, VA: Associationfor the Advancement of Computers in Education, pp. 128133.Hoffman, B., & Ritchie, D. (1997). Using multimedia to overcome theproblems with problem based learning. Instructional Science, 25(2),97115.Jarvela, S. (1995). The cognitive apprenticeship model in a technologicallyrich learning environment: interpreting the learning interaction.Learning and Instruction, 5(3), 237259.Jih, H. J., & Reeves, T. C. (1992). Mental models: a research focus forinteractive learning systems. Educational Technology Research andDevelopment, 40(3), 3953.Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive scienceof language, inference, and consciousness. Cambridge: CambridgeUniversity Press.Johnson-Laird, P. N. (1989). Mental models. In M. I. Posner (Ed.),Foundations of cognitive science (pp. 469499). Cambridge, MA: MITPress.Jonassen, D. H., Marra, R., & Palmer, B. (2003). Epistemologicaldevelopment: An implicit entailment of constructivist learningenvironments. In N. M. Seel, & S. Dijkstra (Eds.), Curriculum, plansand processes of instructional design: International perspectives.Mahwah, NJ: Erlbaum, in press.Jonassen, D. H., & Tessmer, M. (1996/1997). An outcomes-basedtaxonomy for instructional systems design, evaluation, and research.Training Research Journal, 2, 1146.Jonassen, D. H., Tessmer, M., & Hannum, W. H. (1999). Task analysismethods for instructional design. Mahwah, NJ: Erlbaum.Konrad, K. (1998). Kooperatives lernen bei studierenden: forderungmetakognitiver selbstauerungen und (meta)kognitive profile. Unter-richtswissenschaft, 26(1), 6787.Kourilsky, M., & Wittrock, M. C. (1992). Generative teaching:an enhancement strategy for the learning of economics incooperative groups. American Educational Research Journal,29(4), 861876.Krohne, H. W. (1977). Kognitive Strukturiertheit als Bedingung und Zielschulischen Lernens. Zeitschrift fur Entwicklungspsychologie undPadagogische Psychologie, 9(1), 5475.Lajoie, S. P., & Lesgold, A. (1989). Apprenticeship training in theworkplace: computer-coached practice environment as a new form ofapprenticeship. Machine-Mediated Learning, 3, 728.Mayer, R. E. (1989). Models for understanding. Review of EducationalResearch, 59(1), 4364.Mayer, R. E., Moreno, R., Boire, M., & Vagge, S. (1999). Maximizingconstructivist learning from multimedia communication by minimiz-ing cognitive load. Journal of Educational Psychology, 91(4),638643.Niegemann, H. M. (1995). Zum einflu von modeling in einercomputergestutzten lernumgebung: quasi-experimentelle untersuchungzur instruktionsdesign-theorie. Unterrichtswissenschaft, 23(1), 7587.Palincsar, A. S. (1986). The role of dialogue in providing scaffoldedinstruction. Educational Psychologist, 2(1/2), 7398.Rollett, B., & Bartram, M. (1977). Anstrengungsvermeidungstest (AVT).Braunschweig: Westermann.Ross, S. M., & Morrison, G. R. (1997). Measurement and evaluationapproaches in instructional design: Historical roots and currentperspectives. In R. D. Tennyson, F. Schott, N. M. Seel, & S. Dijkstra(Eds.), Instructional design: International perspectives (Vol. 1) (pp.327351). Theory, research, and models, Mahwah, NJ: LawrenceErlbaum.Royer, J. M., Cisero, C. A., & Carlo, M. S. (1993). Techniques andprocedures for assessing cognitive skills. Review of EducationalResearch, 63(2), 201243.Salomon, G. (1979). Interaction of media, cognition and learning. SanFrancisco: Jossey Bass.Sasse, M. (1991). How to t(r)ap users mental models. In M. J. Tauber, & D.Ackermann (Eds.), (Vol. 2) (pp. 5979). Mental models and humancomputer interaction, Amsterdam: North-Holland.Schenk, K. (2003). Effekte metakognitives Trainings auf Lernen undProblemlosen. Dresden: Saxoprint.Scriven, M. (1967). In R. W. Tyler, R. M. Gagne, & M. Scriven (Eds.),Perspectives of curriculum evaluation (Vol. 1) (pp. 3983). AERAmonograph series on curriculum evaluation, Chicago: RandMcNally.Seel, N. M. (1980). Lernerleben im Geschichtsunterricht der SekundarstufeI. Eine experimentelle Analyse. Munchen: Minerva.Seel, N. M. (1991). Weltwissen und mentale Modelle. Gottingen: Hogrefe.Seel, N. M. (2000). Psychologie des Lernens. Munchen: Reinhardt.Seel, N. M., Al-Diban, S., & Blumschein, P. (2000). Mental models andinstructional planning. In J. M. Spector, & T. M. Anderson (Eds.),Integrated and holistic perspectives on learning, instruction andtechnology: Understanding complexity (pp. 129158). Dordrecht,NL: Kluwer.Seel, N. M., & Dinter, F. R. (1995). Instruction and mental modelprogression: learner-dependent effects of teaching strategies on knowl-edge acquisition and analogical transfer. Educational Research andEvaluation, (1), 435.Snow, R. E. (1990). New approaches to cognitive and conative assessmentin education. International Journal of Educational Research, 14(5),455473.Tennyson, R. D., Schott, F., Seel, N. M., & Dijkstra, S. (Eds.), (1997).Instructional design: International perspectives (Vol. 1). Theory,research, and models, Mahwah, NJ: Erlbaum.Wartofsky, M. W. (1979). Models: Representation and the scientificunderstanding. Dordrecht: Reidel.Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model forunderstanding formative evaluation in instructional design. EducationalTechnology Research and Development, 43(3), 2948.Wild, K. P., & Schiefele, U. (1994). Lernstrategien im studium. Ergebnissezur faktorenstruktur und reliabilitat eines neuen fragebogens. Zeitschriftfur Differentielle und Diagnostische Psychologie, 15, 185200.N.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224224An evaluation report of multimedia environments as cognitive learning toolsIntroductionModel-based learning and instructionThe evaluation modelProgram analysisSubjectsThe evaluation designInstruments of evaluationImplementation of the evaluation planResultsResults of task analysesResults of the evaluation studiesConclusionsAcknowledgementsReferences


View more >