teachers' thinking elicited from interactive multimedia professional development courseware

Teachers' Thinking Elicited from Interactive Multimedia Professional Development Courseware

[] lan Putt Lyn Henderson William Patching

This paper reports data obtained on practicing teachers" thinking as they interacted with a professional development interactive multimedia (IMM) courseware package. The data were obtained from 11 participating teachers via stimulated-recall interviews. Two types of thinking are detailed and discussed. The first type is those mediating processes engaged in by teachers during study sessions that related to the academic and professional content of the IMM package. The second type is those evaluative thoughts reported by teachers that related to instructional design aspects of the IMM courseware. The latter data were used to develop a classification system that provides a conceptualization of the major components, as perceived by the participating teachers, that relate to instructional design.

D Information gained about the effectiveness of interactive multimedia (IMM) computer- based courseware for learning is directly dependent on the research paradigm used for obtaining that information. Because research paradigms used to investigate teaching effectiveness have undergone a fundamental recon- ceptualization over the last two decades, methods adopted to investigate what and how students learn when using IMM courseware should reflect these developments. One such development has been the change from employing a process-product paradigm, based on the assumptions that teaching processes give rise directly to learning outcomes and that students are passive recipients of instructional processes, to a mediating process paradigm (Shulman, 1986). This change directs attention to a study of student thought processes that mediate or come between instructional stimuli and learning outcomes.

One area of research based on this paradigm has involved the study of mediating or thinking processes used by distance learners studying hard copy text (Marland, Patching, & Putt, 1992; Marland, Patching, Putt, & Store, 1984). The present investigation extends this work and is concerned with identifying those thinking processes activated by learners as they study IMM courseware materials. It is readily recognized that learning does not auto- matically occur as a direct result of the ever- increasing sophistication of computers and other educational technologies used for trans- mitting information. Rather, learning is related

ETR~), Vol. 44, No, 4, 1996, pp, 7-22 ISSN 1042-1629 7

8 ErR&D, Vol 44, No. 4

to the quality and quantity of thinking undertaken by leamers (Litchfield, 1992) as they interact with computers as cognitive tools (Jonassen & Reeves, in press). That is, learning outcomes are a function of the mediating processes activated by instructional tasks and other learning activities. Given this assump- tion, we maintain that there is a need for greater focus and emphasis on learner thinking that results from interacting with IMM materials in studies designed to evaluate the effects of instructional software.

This paper reports a study that mapped the thinking in which practicing teachers engaged while studying a professional development IMM package--Social Justice Explored. Two types of thinking were identified from the transcripts of stimulated recall interviews conducted with the teachers after their study sessions on the computer. The first type was cognitive mediating processes, which were the teachers' thinking processes that related to the academic and professional content of the social justice package. These mental processes are those that mediate or come between perusal of instructional materials and learning outcomes, and can be viewed as the fine-grained elements of cognition through which and by

which learning outcomes are realized. Such mental processes were a subset of those found in the Marland et al. (1992) study referred to previously. The mediating processes identified in the present study are shown in Table 1 and will be discussed later in the paper.

The second type consisted of evaluative thoughts about instructional design of the IMM courseware that the teachers reported as occurring, either during study sessions or as reflec- tions immediately after the sessions. From the reported evaluative thoughts, the authors developed a classification system for thoughts relating to the design elements of the IMM courseware, some of which were reflected in the literature. This classification system will be discussed later in the paper (see Table 4). Figure 1 is the authors' attempt to show, via a concept map, the definitions and characteristics of, and interrelationships between, these two types of thinking.

CONTEXTUALIZING THE STUDY

Most of the existing research data regarding the efficacy of computer-based learning systems (CBL) is anchored in the process-product

Figure I [ ] Two types of thinking elicited by the IMM materials,

academic

] pr%dessional content

define~as

fine-gained elements of cognition that mediate between instructional materials and learning outcome

I M Stimulus Materials [ eli~ted

[ types of thinking ]

K N~ ..... [instructional mediating ~ 1 evaluativelrelate~Jdesign processes thoughts I to v I dimensions

" _1 ' [ of the IMM define~as ~ materials

thoughts that resulted in a detailing of the IMM instructional design elements that have potential salience for teacher-learner interaction

~r-ACHERS' THINY-JNG ELICITED FROM IMM 9

paradigm, with particular emphasis on the link between instructional courseware and student learning outcomes in terms of the implicit or explicit goals of the instructional package. For example, Salomon, Globerson, and Guterman (1989) examined the learning outcomes of rhe- torical metacognitive prompts in CBL; Thorn- berg and Pea (1991) analyzed two software programs in terms of the problem-solving and reasoning strategies displayed by the users in tests taken after use of the programs; and Krendl and Lieberman (1988) looked at learning outcomes and strategies used by different- sized groups, comprised of one to four persons, in CBL. There is a dearth of docu- mentation on the cognitive processing undertaken by students as they study IMM courseware. In fact, too little is known about the thought processes of students while they are engaged in CBL.

Of necessity, the present study is largely informed, not by the research literature associated with IMM, but by findings in the related area of student thinking while studying from hard copy text. This information about student thinking is of vital importance to those involved in instructional design. For example, in the literature on distance learners' processing of text, there is evidence showing discrepancies between the processing strategies that students use and those they should use or are expected to use in line with instructional designer anticipations (Martand, Patching, Putt, & Putt, 1990).

Many reasons have been suggested to account for these discrepancies. For instance, Reeve, Palincsar, and Brown (1987) claimed that poor metacognitive skills contributed to student failure to " . . . distinguish between those skills needed for everyday thinking and those skills needed for academic thinking" (p. 123). They referred also to pathologies in human thinking such as biases, erroneous intuitions and unsound generalizing. Gibbs, Lockwood, Morgan, and Taylor (1982) considered that surface level thinking skills may be induced by in-text questions that fixate student attention on specific facts rather than broad understandings, while Dahlgren (1978) identified the demands and complexity of the overall

curriculum as a reason why students adopt a survival mentality and a surface approach to study. More recently, Lockwood (1990) found that another possible reason for the discrepancies between processing strategies used by students and those expected by the instructional designer was the mismatch between designer rhetoric or espoused theory about how to write hard copy text and that actually pro- duced for student usage. In summary, very lit- fie research is reported on the qualitative aspects of thinking in which students engage while studying from hard copy text.

The above information regarding hard copy text has implications with respect to students' thinking when studying IMM courseware materials. Courseware content in IMM is presented through text, color, sound, animation, graphics, moving and still pictures, question- answer-feedback activities, time (when these components are brought onto the screen through programmed or free learner control) and space (placement on the screen), and combined in a manner that should provide users with multisensory pathways for creating knowledge. In a review and critique of instructional software by software evaluation organi- zations, Reiser and Kegelmann (1994) found that most evaluations concentrated on instructional design components of the software, while relatively few attempted to ascertain learning outcomes. Although Harel (1991) con- duded that children who used Logo to create software products were constantly engaged in metacognitive acts, she did not include any direct measures of thinking processes in her study. Rather, she based her conclusions about what students were thinking on analysis of data from observations and what the students wrote about their design processes in a daily journal. Neither of the above studies reported any examination of the thinking processes of the user when working through the software.

While there is a need to focus on the gener- ative thinking processes that give rise to learning, that is, the processes that mediate between instructional stimuli and learning outcomes, it is also important to take into account Reeves' (1995) comments about the inappro-

10 ETR&D, Vol 44, No, 4

priateness of using short-term, controlled, and laboratory-like experiments to collect data on user perceptions about instructional design features of IMM courseware. He made the plea for researchers in the area of instructional technology to eschew what happens "in most pseudo-science studies [whereby] undergradu- ates earn 'extra credit' for less than an hour of their time spent using some form of mediated 'treatment' that has little or no relevance for them" (p.8). Reeves argued that the ethics of conducting these types of reductionist experiments in education should be more closely examined, and that researchers should con- struct and/or adopt research contexts and methodologies that "capitalize on authenticity" (Salomon, 1991, p.16).

The warning was heeded in the present study by ensuring that the students' perceptions about the courseware design were obtained in realistic, ecologically-valid study situations. Specifically, the Social Justice Explored (White, 1993) courseware employed is a professional development package designed to raise teachers' awareness of social justice and equity issues in school contexts. The package is comprised of five IMM units that examine the concept of social justice and its implications for classroom practice. Partici- pants interact with the computer IMM package and complete a reading journal and a learning journal. They are required to conduct practical tasks in their classrooms and schools. In the present study the focus was the interactions of the participants with the computer IMM package. The research focus involved the identification and classification of the teachers' thoughts as they interacted with the IMM courseware in this professional development package.

METHODOLOGY

The current study adopted a research paradigm from a previous investigation of mediating processes used by university students who were distance learners studying individ- ually from hard copy text (Marland et al., 1992). The major difference between the pre-

vious study and the current one is that present participants studied IMM materials rather than hard copy text.

Design

Design parameters, relating to the mediating process paradigm employed in the current research, reflect those associated with qualitative research previously used to study university student learning and thinking (e.g., Gibbs et al., 1982; Mafland et al., 1992; Parer, 1988). Specifically, we adopted a process-tracing approach to investigate the thinking of a small group of subjects studying from IMM courseware. We also used specialized post-study stimulated recall interviews with participants to provide data for the development of theory about learning from IMM courseware. Video recordings of study behaviors were used to stimulate recall of participants' thinking while they interacted with the IMM courseware.

Participants

The participants were a volunteer sample of 11 practicing teachers who came from four small primary schools in a rural district in North Queensland, Australia. They were from similar backgrounds to the nine volunteer teachers in the Marland et al. (1992) study, all of whom were teaching and studying concurrently for a BEd degree. The content studied by these teachers covered factors influencing the curriculum and included readings from a set textbook, lecturer-prepared notes, activities, and self-assessment questions. Indeed there were some similarities between the tasks covered in both studies as teachers were being asked to look critically at the curriculum and its implementation in their own classrooms.

There were eight female teachers (two of whom were principals) and three male teachers (two of whom were principals). All had undertaken major professional development programs sponsored by their employer, the State Department of Education, either in face-

TEACHERS' THINKING ELICITED FROM IMM 11

to-face or in a distance mode, but none through IMM. The teachers agreed to undertake the professional development course, Social Justice Explored, over an eight-week period. During this time they covered five units that required them to interact with the IMM courseware materials on a continuing basis.

Data Collection Procedures

Data were collected on two categories of participants' thinking:

1. Mediating processes generated as a result of interacting with the content of the IMM courseware; and

2. Evaluative thoughts about the instructional design features of the IMM courseware.

Stimulated recall interviews that were used to gain this information on participants' mediating processes followed strict guidelines developed over long experience with the technique (see guidelines in Marland, 1984; Marland et al., 1992). The interviewer and each group of participants together viewed the videotape of the study session. The video records of the IMM study groups contained cues that stimulated participants' recall of mediating processes. These were aural cues (mostly intra-group discussion) and two types of visual cues that related to either the actual content, characteristics, and instructional design features of the IMM courseware or the participants' on-screen non-verbal cues, for example, shrugging of shoulders. Based on what occurred in the video, it was the interviewer's role to facilitate the participants' recall and verbalization of their thinking while studying the content of the social justice IMM package.

Evaluative thoughts on the instructional design features of the IMM courseware were obtained from both the stimulated recall interviews and via direct questioning by the interviewer at the end of the stimulated recall interview session. Data were gathered in school settings at two points during the eight weeks of the professional development period. The participants had scheduled all sessions after school, one day each week. The first data-

gathering episode occurred during their study of the first unit of the IMM package; the second was approximately six weeks later when they were studying the final unit. For the first episode, the teacl~ers had formed four study groups by virtue of their geographical location; group sizes were three, two, two and four. During the professional development program, one participant left the first group and the first two groups then combined to make a single group of four teachers. Thus, for the second episode, there were three groups of four, two and four participants.

Each of the four groups was videotaped for a V2 hour session working at the computer on the first unit. Group stimulated-recall interviews of about 11/2 hours were conducted immediately following each video session and were audiotaped for later transcription and analysis. Because of a misunderstanding between the researchers and some participants it was possible to obtain a videotaped study session and subsequent stimulated-recall interview of only one group while they were actually working on Unit 5 on the computer. The other two groups had completed the study session ahead of time so the participants could only be interviewed to gain their evaluative thoughts about the instructional design of the IMM courseware.

In summary, data for analysis of mediating processes were obtained from five small group stimulated-recall interviews (four associated with Unit 1 and one associated with Unit 5 of the professional development package) while all seven small group interviews provided the data base for evaluative thoughts

DATA ANALYSIS AND RESULTS

To identify the participants' mediating processes (see Table 1) and evaluative thoughts, the interview transcripts were analyzed by two of the researchers and the third then verified their findings. Where disagreement occurred, consensus was achieved through group discussion. Data analysis embraced the following five steps:

12 ETR~D, Vol 44, No. 4

1. Transcr ip t da ta that w e r e cons ide red inval id

(for example , as a resul t of a l ead ing ques-

t ion f rom the in te rv iewer) w e r e ident i f ied

a n d d i s ca rded so that t hey w o u l d receive

no fu r ther cons idera t ion .

2. The r e m a i n i n g t ranscr ipt da ta w e r e ana-

lyzed to classify r epor t ed ins tances of stu-

d e n t th ink ing as e i ther in teract ive (that is,

t h ink ing tha t occu r red whi le s t u d y i n g o n

the compute r ) or non- in te rac t ive ( that is,

Table 1 [ ] Mediating Processes Identified in the Present Study

Type Description Example Mental activity in which a student:

Affective

Analysis

Anticipation

Comparing

Confirming

Deliberation

Evaluation

Generating

Imaging

Linking

Metacognitive

Recalling

Strategy Planning

Reports feelings aroused by content during study

Reduces, breaks down whole (e.g. problem task) into parts

Predicts or states expectations that problem, question, or textual feature will be encountered; looks forward to new unit, material; wonders about: the possibility of an event, occurrence in text, relevance of material, content

Identifies similarities, differences between two statements, concepts, models, situations, ideas, theories, points-of-view, etc.

Judges that ideas in text support one's own beliefs, practices, tactics

Engages in "thinking" about a topic, prose segment, etc. (type of thinking not disclosed)

Makes judgments about the value, worthwhileness of textual materials, activities, in-text questions, own position or point-of-view

Formulates one's own questions, examples, ideas, or problems; interpolating; going beyond the data

Creates a mental image of an idea in text in order to gain a fuller understanding

Associates or brings together two or more ideas, topics, contexts, headings, personal experiences, materials, tasks

Thinks about, reflects on, evaluates or directs own thinking

Brings back into working memory an idea, opinion, fact previously stored in long-term memory

Plans ways of processing or handling content material during study sessions

"Guilty! It did. I felt terrible--that this is the effect that I could have on a child."

"Would I choose that child with the eye problem, no I really wouldn't . . . . Same with the other fellow who looks dark skinned. And girls--because girls are easier to handle."

"I thought 'this is OK, this is interesting and I want to do more of this'."

"So we felt that others in a different school with different children, they certainly would have felt closer and thought about other aspects of the program."

"Oh yes, that matches up with that . . . . that is what we're really talking about."

"It made you think about it a bit like, I should choose that one."

"Look, we were asked to make a judgement. It really wasn't a true judgement because we don't do t h a t . . . The examples were very negative and we were expected to be negative."

"I was really interested in the effect of weight, and the correlation of weight and learning."

"I know, like, you do form a concept of that child as it comes."

"This is interesting, because it says we picked four girls and two boys. So it says we have a preference for girls. But when we picked four light skinned and two dark skinned, it didn't say that there was a preference for that."

"Well I thought at the time what are my agreed definitions for myself?"

"Because I was thinking of, OK, my mother all through those years. Also my experiences through u n i . . , my attitudes are shaped by my experiences."

" . . . I said 'it 's better we write it down here and forget what's up there.'"

Note: Descriptions are adapted from Mafland et al. (1992, pp. 17-18)

TEACHERS" THINKING ELICITED FROM IMM 13

thinking that occurred during the interview but not during the study session).

3. Interactive data were further analyzed to code individual mediating processes reported by participants according to the definitions used by Marland et al. (1992). Table 1 contains mediating processes found in the current study together with definitions from the Marland et al. study and examples taken from the present study.

4. Interactive and non-interactive data were analyzed to identify those thoughts that related to the instructional design of the IMM courseware. As shown in Figure 1, these are referred to as evaluative thoughts. Thoughts about the instructional design were revealed through the interactive data and via the non-interactive data elicited by the interviewer asking direct questions regarding the learner's thoughts and feelings about the instructional design.

5. The evaluative thoughts identified in (4) were used to develop a data-grounded classification system (see Table 4). In generating this system, attention was paid to findings from the literature on instructional design and user perceptions of IMM (e.g., Kozma, 1994; Reeves, 1992).

Data on Mediating Processes while Studying the IMM Courseware

As mentioned previously, data on mediating processes resulted from five small group stimulated recall interviews (four after studying Unit 1 and one after studying Unit 5). This translated to gaining data from 15 individuals. The interactive data were coded according to the procedure outlined above and frequencies for each type of mediating process reported by the participants were tallied. In all, a total of 167 mediating processes were identified from the interactive data in the transcripts and these were classified into 13 of the 19 categories identified in the Marland et al. (1992) study. The 13 mediating processes and their frequency of occurrence are presented in Table 2.

The data in Table 2 reveal that there is a

Table 2 [ ] Frequency of Mediating Processes Relating to Academic and Professional Content

Mediating Process a (0 (~)

Metacognition 33 (19.8) Evaluation 30 (18.0) Affective 24 (14.4) Linking 19 (11.4) Genera0ng 14 (8.4) Anticipation 13 (7.8) Recalling 11 (6.6) Comparing 8 (4.8) Analysis 5 (2.9) Confirming 3 (1.8) Deliberation 3 (1.8) Strategy Planning 3 (1.8) Imaging 1 (0.6) Total 167 (100.0)

a: Based on responses from 15 participants.

wide variation in the frequency of occurrence of the 13 identified mediating processes. These ranged from 0.6% for imaging to 19.8% for metacognition. This variation in occurrence of the 13 mediating processes is displayed in Table 3, that categorizes the frequency of occurrence into four levels, namely, very high, high, low, very low. To enable comparisons between the present study and that of Mar- land et al. (1992), the same frequency bands were used. In the Marland et al. study, the 3% and 10% cut-off figures were arbitrarily cho- sen, whereas 5.3% (100 +19) was the average percentage frequency across all 19 categories found in that study.

The data in Table 3 reveal that four mediating processes--metacognition, evaluation, affective, and linking--have the highest frequencies in the IMM study. Three of these-- metacognition, evaluation, and linking--also rated as very high in the Marland et al. (1992) study. While affective rated as very high in the current study, it rated as high in the Marland et al. (1992) study. Two additional comparisons are worthy of note. Whereas generating

14 ~o, vo~ 44, No, 4

Table 3 [] Mediating Processes Grouped According to Frequency Range

Mediating Processes Mediating Processes Social Justice Explored BEd (In-service)

Frequency (f) % through IMM (Marland et al., 1992)

Very high (f > 10) metacognition a ( 1 9 . 8 ) metacognition evaluation (18.0) evaluation affective (14.4) linking (11.4) linking (10.4)

strategy planning (16.8) High (5.3 < f < 10) generating (8.4)

anticipation (7.8) recall (6.6)

Low (3 < f < 5.3) comparing (4.8)

Very low (f < 3) analysis (2.9) strategy planning (1.8) deliberation (1.8) confirming (1.8) imaging (0.6)

b(12.4) (18.6)

anticipation (5.9)

deliberation (8.1) affective (7.8)

generating (3.6) recall (4.1) analysis (1.1)

confirming (2.8) imaging (0.0) companng (1.5)

a: The number in parentheses is the percentage frequency. b: The percentage frequency is based on the 19 mediating processes identified in the study.

rated high in the present s tudy it rated low in the Marland et al. (1992) study; strategy planning which rated very low in the present study, had a very high rating in the Marland et al. (1992) study.

Data on Evaluative Thoughts Relating to the Instructional Design of the IMM Courseware

As indicated under data collection procedures, t ranscr ipt data obta ined f rom all interviews, both s t imula ted recall and uns t ruc tu red non- s t imula ted recall, y ielded interactive and non- in te rac t ive though t processes that re- la ted to the ins t ruct ional design features of the IMM courseware . From these data, we deve loped a classification sys tem compris ing eight major categories, some of which were based on the l i terature; the others were a r r ived at by consensus be tween the researchers. A brief descr ipt ion of these major categories together wi th their sub-categories

and the frequency of occurrence of each is presented in Table 4. Each sub-category is g rounded in the evaluat ive though t s of the teachers, examples of which are also included in Table 4. The classification provides a frame- work for the fu ture documenta t ion of s tudents ' thoughts about a range of facets of instructional design in IMM.

Table 4 shows that 11 teachers reported 114 evaluative thoughts related to instructional design of the IMM materials after Unit 1 (10.3 thoughts per teacher) and 10 teachers reported 71 evaluative thoughts after Unit 5 (7.1 thoughts per teacher). This represents a decrease of three fewer thoughts from each teacher as they continued to interact with the IMM material throughout the duration of the professional development course. Despite hav- ing one fewer teacher's evaluative thoughts after Unit 5, the frequency for learner control and access and pedagogy increased. In every other category and sub-category, frequency of evaluative thoughts decreased from Unit I to Unit 5.

TEACHERS' THINKING ELICITED FROM IMM 1 5

Table 4 [] A Classification of the Major Categories and Sub-categories of the Teachers" Thoughts Relating to IMM Courseware

aSub-Categories of Evaluative bFreq CFreq Examples of Evaluative Thoughts from Categories of Thoughts relating to each after after the data that give rise to the classification category Unit I Unit 5 sub-categories

Interaction (a) clarity of the instructional 7 0 (a) "I couldn' t figure out what it formats task m e a n t . . , the instructiosn Refers to the weren ' t very clear at that part ." instructional (b) suitability of the 7 5 (b) "It's good in that it makes design of instructional task you a w a r e . . , that 's the point ." various (c) feedback 2 2 (c) s tudents anticipated consistent question- feedback answer- (d) design features 6 1 (d) The answer box size was some- activities, times too limited for the answers

needed.

Category total 22 8

Global Impressions Refers to the students ' overall comments on the IMM courseware.

12 7

Category total 12 7

"We spent more and more time talking and taking notes and thinking about it"; "You really interact."

"I like the idea of working in a unit. This is my unit, and this is what I 'm going to do today. In sixty minutes time I 'm going to finish that unit ."

Learner control and access Refers to students" control over their navigation through the IMM courseware. This takes into account ease of access and the degree of user friendliness.

(a) clarity of requirements 2

(b) constraints of the software 1

(c) constraints of the IMM 8 courseware

(d) program pacing 8

(e) screen interface design 1

(f) awareness of the 1 12 navigation system

Category total 21 25

2 (a)

1 (b)

9 (c)

(d)

(e)

(f)

lack of information on what to type in to get the program started inability to change answers after exiting the interaction with- out going back into the interaction "we would have liked to get out of the introduction e a s i e r . . . maybe that 's a bad thing. I don ' t know." "I thought it was a little bit too slow"; " . . . at one p a g e . . . the box actually faded too fast before I 'd finished reading." " • . • all the commands are on the bottom usually with the Mac it is at the top and you get used to it." "You could page ahead by putt ing in some gobbledygook."

Pedagogy Refers to aspects of the designer 's approach to the learning task•

(a) use of reinforcement 1 6

(b) constructivist 3 1

(a) need for positive reinforcement in the program

(b) "I liked the way it's not just telling me that this is social justice, this is what you do. I like the way it is trying to get me to say what it is that I think and how I feel and how I treat people."

16 a T ~ , VoI 44, No. 4

Table 4 [ ] Continued

aSub-Categories of Evaluative bFreq CFreq Examples of Evaluative Thoughts from Categories of Thoughts relating to each after after the data that give rise to the classification category Unit 1 Unit 5 sub-categories

Pedagogy (c) encourages learner self- 3 4 (c) "It's raising our awareness. (continued) awareness We are being forced to con-

sider buried biases or prejudices, thoughts, or feelings about the children that we teach."

1 0 (d) "The content that we did cover was very in-sync. It followed on well and flowed well."

(e) amount of text 0 1 (e) "Had we had the readings in the book rather than on here[the c o m p u t e r ] . . . That took us a long time, probably two and a half hours . . . . "

(f) length of unit 0 5 (f) " . . . some of the areas there we wondered whether anyone would bother do ing - - not for any other reason than sometimes there was a rather large time f a c t o r . . . "; "We just sort of thought 'we 've got to get going--we ' re only on page such and such' ."

0 2 (g) "It had a bit of jargon in it too, and long sentences. We had to read them three or four t imes and Gary had to translate them for us. Page 14 was a mouthful ."

(d) sequencing of content

(g) structure and language of text

Category total 8 19

Characteris- (a) pictures 5 3 (a) suitability and clarity. tics of IMM (b) color 4 1 (b) "Color made the screen Refers to the attractive." available (c) sound 6 2 (c) suitability of background music programming and voice overs. attributes in (d) animation 1 0 (d) "I like the way the words came IMM software, across the screen."

(e) space/time 1 0 (e) appropriateness of the linear and temporal appearance of text.

Category total 17 6

Cooperative learning Refers to instructional methods and usage in which learners work together in pairs or small groups.

(a) individual versus group 13 1 learning

(b) size of the group 6 1

Category total 19 2

(a) "If I was by myself I would have read more slowly"; "I like the collegial situation very much."

(b) "We would have got through a lot quicker with a smaller group"; "Four are too many--i ts too slow. I think pairs is about it."


Table 4 D Continued

aSub-Categories of Evaluative bFreq CFreq Examples of Evaluative Thoughts from Categories of Thoughts relating to each after after the data that give rise to the classification category Unit I Unit 5 sub-categories

Learner (a) computer as an authority 6 3 (a) "It 's only because the perception of figure machine wants you to say that ." the computer (Personifies the computer and Refers to the assigns it the instructional instructional power of the teacher). role students (b) acknowledges the role of 3 1 (b) "We were trying to guess assign to the the instructional developer what was in the organizer 's computer, mind." (Focusing on what the

lecturer as opposed to what the computer requires).

(c) computer as collaborator 1 0 (c) "We are really interacting with the computer h e r e . . , it 's almost a triangular situation." (Interaction between student, computer courseware and lecturer).

Category total 10 4

Miscellaneous comments Refers to general comments concerning computer literacy and confidence in using the computer.

5 0

Category total 5 0

"I was very intent on proving that I could use the mouse"; "I quite enjoyed doing the typing part."

Total number of thoughts 114 71

Average number of thoughts 10.4 7.1 per teacher

a: The sub-categories were constructed on the basis of these data. b: Frequency refers to the number of evaluative thoughts reported by 11 teachers c: Frequency refers to the number of evaluative thoughts reported by 10 teachers.

18 ETR&D, Vol 44, No, 4

DISCUSSION

This section involves an interpretation of the data on the two types of thinking that were elicited in this study, namely, mediating processes and evaluative thoughts.

Mediating Processes

This discussion of mediating processes will focus on two things:

1. Comparing the results of the study with the Mafland et al. (1992) study using hard copy text; and

2. Discussing mediating processes that had a relatively high frequency of occurrence in the present study.

Comparison of studies. As shown in Table 3, 19 mediating processes were found in the Mar- land et al. (1992) study of which 13 were again evident in this study. Seven of the mediating processes, namely, metacognition, evaluation, linking, anticipation, analysis, confirming, and imaging remained in the same frequency band in both studies. Four went up one frequency band from the Marland et al. study (1992) to the IMM study: affective, generating, recall, and comparing. Two went down one frequency band from the Mafland et al. (1992) study to the IMM study: strategy planning and deliberation. Except for the latter, which went down by more than one frequency band, the frequency bands across the two studies tended to be relatively stable. Similar types of mediating processes were activated whether people were studying content from hard copy text or via IMM courseware. The similarities in the content areas being studied, the interactive tasks (either on computer or in the self-assessment questions), and the required reflection on classroom practices, support Clark's (1994) contention that the medium was not the cru- cial variable in this area of cognitive mediating processes.

Strategy planning warrants some discussion since it moved from the very high category in the Marland et al. (1992) study to the very low category in the present study. Strat-

egy planning refers to thought processes in which students plan ways of processing or handling instructional material or activities during study sessions. Possibly one of the major reasons for the discrepancy across the two studies is that the instructional designer's suggested linear sequence for studying the five units in the Social Justice Explored package allowed less scope for the students to exercise individual ways of negotiating the IMM materials.

Mediating processes with high frequency of occurrence. In the current study, four mediating processes are targeted for discussion because of their relatively high frequency of occurrence. These are metacognition, evaluation, affective, and linking.

Mediating processes classified as metacognitions are those in which students reported awareness of, reflecting on, evaluating, or directing their own thinking. This defi- nition reflects a widely accepted view of metacognition as referring to students' knowledge about, and control over, their cognitive processes (Weinstein & Mayer, 1986). In the current study, metacognitions reported by teachers were at both the awareness level and the control level. Evidence of a relatively high level of metacognitive activity by participants in this study is encouraging. Nevertheless, the vast majority of instances were at the awareness rather than the control level in which participants would actually direct themselves to undertake such things as: (a) checking on their learning and understanding, (b) thinking deeply about or focusing on specific topics, and (c) analyzing what they do or do not know.

Participants' responses that indicated the awareness aspects of their metacognition related to acknowledging their own reasons for making choices; their awareness of peda- gogic intent; reflecting their self-awareness about such things as prejudices, attitudes, and values; monitoring their progress; and realiz- ing their need of assistance. On the other hand, metacognitive control was evidenced only in terms of participants redirecting the focus of their thinking based on feedback

1EACHE~" THINKING ELICffED FROM tMM 19

about their failure to carry out the task as orig- inally intended in the instructional design. That is, metacognitive control was evidenced in this study more as a reactive thinking process (e.g., "I thought: 'I didn' t understand what was required in this activity'") rather than a proactive one whereby participants actually plan and exercise some control or direction over thinking (e.g., "I said to myself: 'I need to concentrate on thinking about classroom examples to discuss in this activity'" or "I need to think of alternative explanations to gain an understanding of this section").

Evaluation was the second most frequently occurring thought process. Evaluation is defined as the mental process in which a judgement is made about the value or worthwhileness of some aspect of the content of the instructional material. Instances of participants ' use of evaluation as a mediating process in this s tudy clustered around three foci: the difficulty of the task (e.g., "I was t h i n k i n g . . . it is a very hard thing for us to do"); interest (e.g., " . . . that aspect was very interesting"); and responding to the requirements of the instructional tasks/activities (e.g., "Look, we were asked to make a judgement. It really wasn ' t a true judgement because we don ' t do t h a t . . . The examples were very negative and we were expected to be negative"). Because the content of the professional development course focused on the teachers' own beliefs about social justice and how these were opera- tionalized in their classrooms, a relatively high frequency of evaluation was not surprising.

Interestingly, Table 3 shows that the level of affective mediating processes is almost dou- ble that revealed in Marland et al. (1992). Affective mediating processes are those in which participants reported an awareness of their affective states during IMM interaction sessions: for instance, enjoyment ("I really liked that testing y o u r s e l f . . . I enjoyed doing that"); frustration ("Frustrated with the activity and a little bit concerned about it"); and disagreement ("We were expected to be negative. I just didn' t like that"). Given that the participants in both studies were practicing elemen- ta D , school teachers, and that the content in both cases related to curriculum issues, it

would appear that the main cause of the difference in frequency was the presentation of the material via the various multisensory attributes of the IMM mode.

Linking is defined as the process of associat- ing, or bringing together in the mind, two or more ideas, topics, contexts, personal experiences, words, and so forth. Linking usually occurred when an item on the screen (e.g., text, explanatory voice overs, and graphics that illustrated a concept) prompted recall of an associated item in a participant's memory. Instances of linking reported by the teachers related to such things as their current classroom experiences, events from their past teaching career, personal experiences from the past, attributes of students known and/or taught, and previous s tudy experiences. It is not surprising that linking fell into the very high frequency range given that the IMM package is a professional development activity related to the classroom and school.

Categories of Evaluative Thoughts

The classification of teachers' evaluative thoughts (Table 4) about aspects of the IMM courseware into major categories and sub-categories was grounded in the data provided by participants, not established a priori by the researchers. An examination of Table 4 shows that the total number of evaluative thoughts reported by teachers decreased by 43 (or approximately 38%) across the two sessions, while the average number of thoughts per teacher decreased from 10.4 to 7.1 (or approximately 32%). This decrease could most likely be attributed to the fact that, as teachers became more comfortable with learning through the IMM mode, they spent fewer thoughts on the characteristics of the instructional design of the IMM materials.

In spite of the decrease, frequencies in two categories, learner control and access and pedagogy, actually increased. Two of the sub-categories of learner control and access show considerable variability between the two data collection sessions. The marked decrease in thoughts concerning program pacing was more

20 ERRS, Vol 44, No. 4

than offset by the large increase in thoughts on awareness of the navigation system. There may be a relationship between these two changes since, as teachers became more knowledgeable users of the navigation system, they devised strategies that allowed shortcuts that gave them control over the pace at which they negotiated the IMM courseware. An implica- tion for IMM courseware design that accom- modates students with different levels of ability, interest, and metacognitive skills is to incorporate a default linear navigation path- way as well as flexible hypermedia access to any material in the IMM courseware.

As teachers became more proficient in working with the IMM materials, some of their thoughts were directed to evaluating the instructional designer's pedagogy (see Table 4). In particular, the participants indicated that they expected the instructional designer to provide reinforcement and/or feedback in text or aural format to their considered responses to the interactions. Two sub-categories in Table 4 are worthy of comment because of their relatively high total across the two data sessions.

First, constraints of the courseware were sig- nificant to the teachers. It appears that ini- tially, the teachers sought to use the IMM courseware like they would a printed text. They wanted to be able to skip over or exit from the introduction in order to proceed with the rest of the program. They also wanted their typed-in answers to text-based interactions to remain so that when they revised any section they could see what they had previously written and edit it rather than rethink and type in a new answer. The teachers' con- cerns could have been accommodated in two ways:

1. Answers can remain and be edited by the user, or

2. Answers can be erased in the actual courseware but stored in an easily accessed electronic notebook that can be edited, printed and saved to a disk. The notebook can be positioned anywhere on the screen so that question, answer, and notebook can be seen simultaneously.

Second, with respect to individual versus

group learning, there was general consensus in favor of small group learning over individual use of the computer. Comments by the teachers suggest that, from their perspective, two was the optimum group size for interacting with the computer and discussing the content. Their perceptions are supported by data reported by Patching, Henderson and Putt (1994) that indicated, for groups of two, the number of reported mediating processes per person was two and three times that for groups of three and four respectively.

Overall, teachers reported positive perceptions about studying the Social Justice Explored package. Specifically, they felt that both the content and the instructional design of how that content was presented through IMM generated discussion among group members, clar- ified and promoted awareness of their own ideas, and enabled them to reflect on their own thinking.

CONCLUSION

The authors agree with Jonassen and Reeves (in press) and argue that IMM should be used as a cognitive tool to enhance the cognitive powers of people during thinking, problem solving, and learning. To attain this goal, the study highlights three major implications for the instructional design of IMM materials.

First, this study helps make instructional designers aware of the existence, types, and relative frequencies of mediating processes that students do engage in while interacting with instructional IMM courseware. It appears, given that the content and interactions had many similarities and the teachers had similar classroom experiences, the types of mediating processes elicited did not differ markedly between IMM courseware and print materials. However, in the present study, there was a higher percentage of relatively very high frequency mediating processes than in the Marland et al. (1992) study that perhaps can be attributed to the presentation of the materials through the IMM mode. Furthermore, some students engage heavily in evaluative thoughts about the features and design of IMM


courseware that may mitigate against the a m o u n t of cognit ive energy directed at courseware content and processes and may ul t imately lessen the efficiency of their learning and quali ty of thinking.

Second, there is a need for instructional designers to p lan educat ional materials, in line with their pedagogy , phi losophy, and goals, that will activate desired mediat ing processes as par t of s tudent thinking. This is related to initial evidence that suggests instructional des igners can actually engineer presentat ions to increase or decrease rates of occurrence of selected media t ing processes. (See previous comment s about the different rates of occurrence of strategy planning between the current s tudy and the Mar land et al. (1992) study.) One approach that could be incorporated in IMM design is to include not only in-text s tudy activities but also to make explicit the thinking processes the s tudents could use to complete the activities satisfactorily. For example, Hen- derson, Patching, and Putt (1994) have had encouraging results with embedded prompts and interactions in other universi ty IMM materials to activate metacognit ion. When a new theory or idea is introduced, s tudents could be chal lenged to engage in such processes as appl icat ion (e.g., convert ing the idea into a practical form for use in the work place), gen- eration (e.g., l isting suitable criteria for assess- ing outcomes), and evaluat ion (e.g., assigning value to outcomes and judging whether the outcomes satisfy the criteria). In essence, the r ecommenda t ion here is that IMM instructional des igners incorporate activities that clearly specify the media t ing processes required for their completion.

Third, in order to increase the quality and quant i ty of s tudent thinking about content and processes, there is a need for the instructional des igner to minimize the time spent on thinking about features of the IMM courseware. One way to do this would be to take the categories and sub-categories developed from data collected in the present ecologically-valid s tudy si tuat ion ( shown in Table 4), and use them as guidel ines to inform instructional designs that result in increased thinking related to the learning outcomes envisaged by

the instructional des igner and decreased thinking related to instructional des ign facets. [ ]

Ian Putt is a senior lecturer in mathematics education in the School of Education at James Cook University, Townsville, Australia. He has been involved in teaching mathematics education courses at a distance via printed text to inservice teachers and through interactive multimedia (IMM) courseware to Aboriginal and Tortes Strait Islander teacher education students in their home communities. His current research interests focus on tertiary students' thinking while studying from IMM materials and their usage patterns of IMM courseware. He also supervises post graduate students in the area of teaching problem solving in upper primary and lower secondary school.

Lyn Henderson is senior lecturer at James Cook University of North Queensland, Australia. Her fields of teaching and research interests concern the cultural contextualization of instructional design of interactive multimedia (IMM) and the World Wide Web (WWW). Other research interests focus on the mental models, thinking processes, and teaching-learning strategies used by teachers and learners when interacting with electronic databases, IMM, and the WWW.

William Patching is senior lecturer at James Cook University of North Queensland, Australia. His areas of expertise are special education and distance education. His current research interests with respect to interactive multimedia (IMM) focus on tertiary students' thinking while studying with IMM materials and their usage patterns of IMM courseware. He is also involved in a number of school curriculum and systemic evaluations.

REFERENCES

Clark, R.E. (1994). Media will never influence learning. Educational Technology Research and Develop- ment, 42(2), 21-29.

Dahlgren, L.O. (1978). Qualitative differences in concep- tions of basic principles in economics: A contribution to the discussion of the validity of examination results. Paper presented to the Fourth International Con- ference on Higher Education, University of Lan- caster.

Gibbs, G., Lockwood, F., Morgan, A., & Taylor, E. (1982). Student learning and course design 1: In-text teaching devices in Open University texts (Study Methods Group Report No. 12). Milton Keynes: Open University.

Harel, I. (1991). Children designers: Interdisciplinary constructions for learning and knowing mathematics in a computer-rich school. Norwood, NJ: Ablex Pub- lishing.

22 ErRs. Vol 44, No, 4

Henderson, L., Patching, W., & Putt, I. (1994). The impact of metacognitive interactive strategies and prompts embedded in interactive multimedia in a cross-cultural context: An exploratory investigation. In OPEN LEARNING "94, Proceedings of the 1st International Conference on Open Learning, (pp. 181- 186). Brisbane, Australia: Queensland Open Learning Network & The University of Queens- land.

Jonassen, D., & Reeves, T. (in press). Learning with technology: Using computers as cognitive tools. In D.H. Jonassen (Ed.), Handbook of research on educational communications and technology (Chapter 24). New York: Scholastic Press in collaboration with the Association for Educational Communications and Technology.

Kozma, R.B. (1994) Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42(2), 7-19.

Krendl, K.A., & Lieberman, D.A. (1988). Computers and learning: A review of recent research. Journal of Educational Computing Research, 4(4), 367-389.

Litchfield, B.C.(1992). Evaluation of inquiry-based science software and interactive multimedia programs. Computing Teacher, 19(6), 41-43.

Lockwood, F. (1990). Activities in distance learning texts. Unpublished PhD thesis, Open University.

Marland, P.W. (1984). Stimulated recall from video: Its use in research on the thought processes of classroom participants. In O. Zuber-Skerrit (Ed.), Video in higher education (pp. 156-165). London: Kogan Page.

Marland, P.W., Patching. W.G., & Putt, I.J. (1992). Learning from text: Glimpses inside the minds of distance learners. Townsville: James Cook University of North Queensland.

Marland, P., Patching, W., Putt, I., & Putt, R. (1990). Distance learners' interactions with text while studying. Distance Education, 1I(1), 71-91.

Marland, P.W., Patching, W.G., Putt, I.J., & Store, R.E. (1984). Learning from distance-teaching materials: A study of students' mediating responses. Distance Education, 5(2), 71-91.

Parer, M.S. (1988). Textual design and student learning. Churchill, Vic.: Gippsland Institute of Advanced Education.

Patching, W., Henderson, L., & Putt, I. (1994). Inter- active multimedia and thinking: A report on research in progress. In C. McBeath & R. Atkinson (Eds.), Proceedings of the 2nd International Interactive

Multimedia Symposium (pp. 389-397). Perth, West- ern Australia: Promaco Conventions Pry. Ltd.

Reeve, R.A., Palincsar, A.S., & Brown, A.L. (1987). Everyday and academic thinking: Implications for learning and problem solving. Journal of Curricu- lum Studies, I9(2), 123-133.

Reeves, T.C. (1992). Effective dimensions of interactive learning systems. Information Technology for Training and Education Conference (ITTE '92) (pp. 99-113). St. Lucia, Brisbane, Australia: University of Queensland.

Reeves, T.C. (1995). Questioning the questions of instructional technology research. Proceedings of the Annual Conference of the Association for Educational Communications and Technology, Research and Theory Division (pp. 459-470). Anaheim, CA: Association for Educational Communications and Technology, Research and Theory Division.

Reiser, R., & Kegelmann, H. (1994). Evaluating instructional software: A review and critique of current methods. Educational Technology Research and Development, 42(3), 63-69.

Salomon, G. (1991). Transcending the qualitative- quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10-18.

Salomon, G., Globerson, T., & Guterman, E. (1989). The computer as a zone of proximal development: Internalizing reading-related metacognitions from a reading partner. Educational Psychology, 80(4),620-627.

Shulman, L.S. (1986). Paradigms and research programs in the study of teaching: A contemporary perspective. In M. C. Wittrock (Ed 0, Handbook of research on teaching (3rd ed.). (pp. 3-36). New York: Macmillan.

Thornberg, D.G., & Pea, R.D. (1991). Synthesizing instructional technologies and educational culture: Exploring cognition and metacognition in the social studies. Journal of Educational Computing Research, 7(2),121-164.

Weinstein, C.E., & Mayer, R.E. 1986). The teaching of learning strategies. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed.). (pp. 315- 327). New York: Macmillan.

White, B. (1993). Social justice explored [CD-ROM]. Townsville: Queensland State Education Depart- ment.

teachers' thinking elicited from interactive multimedia professional development courseware

Documents