An evaluation report of multimedia environments as cognitive learning tools

Download An evaluation report of multimedia environments as cognitive learning tools

Post on 15-Sep-2016




2 download

Embed Size (px)


  • An evaluation report of multimedia environments as cognitive

    learning tools

    Norbert M. Seel*, Katharina Schenk

    Freiburg Institute of Educational Science, Albert-Ludwigs University, Freiburg, Germany

    Received in revised form 1 December 2001


    This article deals with the evaluation of a multimedia learning environment which has been developed and evaluated within the broader

    context of a research project on the learning-dependent progression of mental models in economics. To carry out formative evaluations, we

    have adapted a particular evaluation approach which allows and requires the implementation of specific evaluation instruments. The crucial

    questions of our evaluation studies were the efficacy of a multimedia-based realization of the cognitive apprenticeship (CA) approach, the

    diagnosis of mental model progression through the CA based instruction, and the effects of implemented metacognitive training. For the

    assessment of the learning-dependent progression of the mental models, we developed and used a special diagnostic instrument for causal

    diagrams, which are understood as reproductions of students mental models. In order to be able to meet statements about the practicability of

    a multimedia based realization of CA, we measured the results of the tasks of learning during each different learning phase. Additionally,

    several motivational variables and persistent learning strategies were measured. In this article, we will specify the adapted evaluation

    instruments. Furthermore, we will report on the results of five replication studies and discuss the consequences for instructional design in

    connection with the design of constructive learning environments.

    q 2003 Elsevier Science Ltd. All rights reserved.

    Keywords: Causal diagrams; Cognitive apprenticeship; Instructional design; Mental models; Metacognitive training

    1. Introduction

    There is considerable concern that students thinking

    skills, motivational dispositions, and domain-specific

    knowledge might be inadequate for them to lead fulfilling

    lives in a global, information-rich, technology-oriented

    world. Informed by recent theory and research on learning

    and teaching, efforts to reform classroom instruction and

    create learning environments that promote these ends are

    underway. Hannafin (1992) argues that the improvement of

    problem-solving abilities and other key skills requires

    emergent technologies in order to design effective learning

    environments that provide opportunities for reflective

    thinking. However, this argument can be seen from different

    points of view. One approach focuses on the improvement of

    students technological literacy and advocates a new type of

    understanding of information and communication technol-

    ogy in educational settings. Another approach focuses on the

    effective instructional design (ID) of multimedia environ-

    ments as opposed to the technology itself. Obviously, these

    perspectives are not mutually exclusive as each may be

    considered within the context of ID.

    ID is a theoretically sound educational technology for the

    development, implementation, and evaluation of learning

    environments that are adapted to learners, tasks, resources,

    and contexts (Tennyson, Schott, Seel, & Dijkstra, 1997). An

    analysis of the literature indicates that there is broad

    consensus with regard to ID requirements and how to

    evaluate various designs in different contexts (Weston,

    McAlpine, & Bordonaro, 1995). Evaluation has been

    considered a central and necessary part of instructional

    planning from the very beginning of ID (Andrews &

    Goodson, 1980; Cronbach, 1963), wherein a differentiation

    is made between two kinds of evaluation (Scriven, 1967).

    formative evaluation aimed at the improvement ofinstruction by means of feedback of information

    concerning the effective use and outcomes; and,

    0149-7189/03/$ - see front matter q 2003 Elsevier Science Ltd. All rights reserved.


    Evaluation and Program Planning 26 (2003) 215224

    * Corresponding author.

    E-mail addresses: (N.M. Seel), schenk@ (K. Schenk).

  • summative evaluation aimed at the measurement of thedegree to which intended results are achieved.

    A further distinction is often drawn with regard to the

    instruments of evaluation, insofar as quantitative or

    qualitative data are to be assessed.

    This article is concerned with the evaluation of a

    particular multimedia learning environment which was

    developed and evaluated as part of a comprehensive

    research project focusing on the learning-dependent

    progression of mental models. Insofar as the construction

    of mental models presupposes constructive learning,

    which occurs when learners actively construct meaningful

    mental representations during instruction (Mayer, Moreno,

    Boire, & Vagge, 1999), multimedia is considered to be

    the most effective for the promotion of such constructive

    learning. In the case of the given multimedia-learning

    environment, the meaningful mental representation is a

    coherent mental model of a dynamic model of macro-

    economics and financial politics, respectively. In our

    research, learning outcomes were evaluated with multiple

    measuring instruments. On the one hand, students had to

    produce causal diagrams of the problem situation; on the

    other hand, student solutions to learning tasks and transfer

    problems were tested.

    In the following sections we will illuminate the

    formative evaluation of the learning environment

    Dynamic Systems of Economics (DSE) as applied in

    several replication studies. An evaluation model will be

    described with the focus on the following components:

    methodology, data analysis, and interpretation. We will

    conclude with a discussion of the main results of several

    evaluation studies.

    2. Model-based learning and instruction

    Our research group has been involved in the develop-

    ment and investigation of instructional intervention pro-

    grams aimed at the improvement of model-based learning

    and thinking for several years. The epistemological and

    psychological foundations of this research rest on Seels

    (1991) theory of mental models.

    Mental models emerged in the 1980s as a theoretical

    construct to encompass both situated cognition as well as

    qualitative reasoning. Greeno (1989) argues that compre-

    hension of and reasoning in specific situations necessarily

    involves the use of mental models of different qualities.

    Mental models are a central construct of symbolic models of

    human cognition that presuppose the use and manipulation

    of symbols. Following Wartofsky (1979), cognition takes

    place while using mental representations in which individ-

    uals organize symbols of experience or thought in such a

    way that they effect a systematic representation of this

    experience or thought as a means of understanding or

    explaining it to others.

    Mental models play a central and unifying role in

    representing objects, states of affairs, sequences of

    events, and the social and psychological actions of

    daily life. They enable individuals to make inferences

    and predictions, to understand phenomena, to decide

    what action to take and to control its execution,

    and, above all, to experience events by proxy (Johnson-

    Laird, 1983, p. 397)

    Craik (1943) introduced the idea of internal models to

    psychology with the notion of a working model. According

    to Craik, most cognitive theorists agree on the point that

    mental models serve primarily to create situation-specific

    plausibility. Due to an idealized reduction to relevant

    characteristics of its original, a model is a concrete,

    comprehensible, and feasible representation of non-obvious

    or abstract objects. The representation of the objects

    attributes and components comes second to the represen-

    tation of structural relationships. Mental models are not a

    specific representational format such as images and

    propositions, but rather higher-order cognitive constructions

    (artifacts) which refer primarily to the content of mental

    representations (Seel, 1991).

    The functions of mental models, including structural

    features, are defined on the basis of the objectives of the

    model-constructing person. In physics and other sciences,

    the term model is always used in a functional sense.

    Appearance models may serve to simplify a complex

    phenomenon or represent structural relationships visually.

    On the other hand, derivative (thought) models (e.g.

    Rutherfords model of the atom) serve primarily to aid

    analogical reasoning in exploring phenomena (e.g. quantum

    mechanisms). Mental simulations occur when cognitive

    operations simulate (in the sense of thought experiments)

    specific transformations of objects that may occur in real-

    life situations. In sum, mental models run in the minds

    eye to produce qualitative inferences with respect to the

    situation to be cognitively mastered.

    Although mental models may differ markedly in their

    content, there is no evidence to suggest that they differ in

    representational format or in the processes that construct

    and manipulate them. What is at issue is how such

    models develop as an individual progresses from novice

    to expert, and whether there is any pedagogical

    advantage in providing people with models of tasks

    they are trying to learn (Johnson-Laird, 1989, p. 485)

    In accordance with Snow (1990) we have identified the

    learning-dependent progression of mental models as a

    specific kind of transition mediating between student

    preconceptions, which describe the initial states of the

    learning process, and causal explanations, which describe

    the desired end states of the learning process (Seel,

    Al-Diban, & Blumschein, 2000). From the perspective of

    instructional psychology, the guiding principle for

    N.M. Seel, K. Schenk / Evaluation and Program Planning 26 (2003) 215224216

  • influencing the construction of mental models has been

    expressed by Mayer (1989) as follows: Students given

    model-instruction may be more likely to build mental

    models of the systems they are studying and to use these

    models to generate creative solutions to transfer problems

    (p. 47). This presupposes that the learner is sensitive to the

    model-relevant characteristics of the learning environment,

    such as the availability of certain information at a given

    time, the way this information is structured and mediated,

    and the ease with which it can be found in the environment.

    An analysis of the relevant literature indicates that the

    suggested instructional strategy to provide learners with a

    designed conceptual model actually constitutes the main

    trend of instructional research on mental models. According

    to Carlson (1991), instruction can be designed to involve the

    learner in an inquiry process in which facts are gathered

    from data sources, similarities and differences among facts

    noted, and concepts developed. In this process, the

    instructional program serves as a facilitator of learning for

    students who are working to develop their own answers to

    questions. In this case, mental models are more proactive

    and direct the learning experiences so that the result of

    learning is dependent on the initial model, defined as the

    learners a priori understanding of the material to be

    learned. On the other hand, instructional programs can

    present concepts with clear definitions followed by clear

    examples. A conceptual model may be presented before the

    learning tasks in order to direct the learners comprehension

    of the learning material. Over the past decades much

    research applying this strategy has been done to provide

    students with model-based instruction but several authors

    (Royer, Cisero, & Carlo, 1993; Snow, 1990) have objected

    that this kind of research has typically been done piece-

    meal, in small-scale, specialized contexts. In order to

    overcome these shortcomings we need a more comprehen-

    sive instructional approach. Cognitive apprenticeship (CA)

    (Collins, Brown, & Newman, 1989) provides a fundamental

    basis for initiating and directing model-based learning.

    Our research group started in 1994 with the development

    of a multimedia environment aimed at an externally guided,

    goal-oriented, and systematic influence upon the learners

    progression of mental models. CA (Collins et al., 1989) was

    the only promising instructional strategy corresponding

    with the idea of providing the students with model-

    instruction in the aforementioned sense. There are six

    instructional methods in CA: modeling, coaching, scaffold-

    ing, articulation, reflection, and exploration. The instruc-

    tional intervention of apprenticeship starts with the

    presentation of an experts conceptual model of the tasks

    to be accomplished, and then the students are coached and

    scaffolded to adapt this model for their own solutions

    (exploration) to the learning tasks designed. CA is based on

    results of cognitive psychology and applies these results in a

    prescriptive way in order to identify ideal features of

    learning environments. This approach prescribes in detail

    what the learner has to do and in which sequence in order to

    achieve particular objectives. However, the question as to

    whether the CA approach may be appropriate for the design

    of multimedia environments could not be answered at this

    time. There are several studies (Casey, 1996; Chee, 1996;

    Jarvela, 1995) that have investigated this. However, these

    studies run parallel with our investigations and final

    conclusions are not available. Therefore, we focused on

    the issue of whether the preferred use and application

    of multimedia technology allows a strict adaptation of

    instructional regularities to individual regularities

    of learning.

    The research we have done in the past 6 years has

    centered around two main topics:

    1. the investigation of the learning-dependent progression

    of mental models, more specifically of analogy models of

    DSE; and,

    2. how this progression can be guided or influenced through

    a particular instructional intervention program designed

    as a multimedia environment in accordance with

    principles of CA. We focus on the second line of

    research in this paper.

    3. The evaluation model

    We have adopted the evaluation approach of Ross and

    Morrison (1997) with these main components: (1) needs

    assessment, (2) methodology, (3) data analysis and

    interpretation, and (4) dissemination results. The evaluation

    of DSE focused on methodology and data analysis and

    interpretation. Accordingly, we have realized:

    a program analysis in order to determine the content andmethods of their mediation within the multimedia


    a participants analysis in order to determine the (groupsof) learners as well as the scope of the instructional


    a specification of the evaluation design; the development of instruments of measurement; and, the implementation and control of the evaluation


    Following these methodological steps, the analysis and

    interpretation of data was to be done in order to modify or

    revise the instructional program or parts of it. Accordingly,

    the data analysis centers around the formative eval...