Interdisciplinary Journal of E-Learning and Learning Objects Volume 4, 2008 Formerl y the Inter disciplinar y Journal of Knowl edge and Learning Objects

Editor: Susan Nash

Investigating the Use of Learning Objects for Secondary School Mathematics

Robin Kay and Liesel Knaack University of Ontario Institute of Technology, Canada

robin.kay@uoit.ca; liesel.knaack@uoit.ca

Abstract Research on the impact of learning objects in secondary school mathematics classes is limited. To date, only two investigations have been reported in this area. The current study presents a comprehensive analysis of the use and effect of learning objects in secondary school mathematics classrooms. Teacher and student attitudes, as well as student performance, were used to assess the impact of learning objects for 298 students and 11 teachers in 20 different classrooms. Teacher attitudes were positive with respect to quality, engagement, and learning value of learn-ing objects. Student attitudes, on the other hand, varied markedly. Overall, student performance increased significantly after using learning objects, although gains observed were small and high-ly variable. Student performance was significantly related to the intended purpose of a lesson and teaching strategy selected. Planning time, using learning objects for review or to introduce new concepts, and supplying guided handouts improved student performance. It is reasonable to con-clude that mathematics-based learning objects are viable teaching tools when used with the ap-propriate goals and strategies. Keywords: mathematics, use, evaluate, secondary school, learning object

Introduction Learning objects are operationally defined in this study as interactive web-based tools that sup-port the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners. While the design, development, reuse, and accessibility of learning objects has been examined in some detail for almost 10 years (Kay & Knaack, 2007a, 2007b), research on the effectiveness and usefulness of learning objects is limited, particularly in the area of sec-ondary school mathematics (Kay & Knaack, 2007c; Lopez-Morteo & Lopez, 2007). It is argued that learning objects help address a number of obstacles secondary school teachers face with respect to using technology including not having enough time, difficulties in learning new software, creating effective integration strategies, and accessibility (e.g. Agostinho, Bennett, Lockyer, & Harper, 2004; Duval, Hodgins, Rehak, & Robson, 2004; Gadanidis, Gadanidis, & Schindler, 2003; Kay& Knaack, 2007c; Rehak & Mason, 2003). In addition, comprehensive,

theoretically-based, reliable, and valid evaluation tools for assessing learning objects are scarce (Kay& Knaack, in press). The purpose of this study was to examine the impact of learning objects in secondary school mathematics class-rooms by using a comprehensive set of measures to assess teaching strategies, teacher attitudes, student attitudes, and learning performance.

Material published as part of this publication, either on-line or in print, is copyrighted by the Informing Science Institute. Permission to make digital or paper copy of part or all of these works for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage AND that copies 1) bear this notice in full and 2) give the full citation on the first page. It is per-missible to abstract these works so long as credit is given. To copy in all other cases or to republish or to post on a server or to redistribute to lists requires specific permission and payment of a fee. Contact Publisher@InformingScience.org to request redistribution permission.

Inv estigating the Use of Learning Objects

270

Literature Review

Overview Until recently, learning objects were solely used in higher education; consequently the majority of research has taken place in this domain (Haughey & Muirhead, 2005; Kay & Knaack, in press). Increased use of learning objects in the K-12 domain (e.g., Brush & Saye, 2001; Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2005; Liu & Bera, 2005; Lopez-Morteo & Lopez, 2007; Nurmi & Jaakkola, 2006) demands that the focus of investigation shift , at least in part, to the needs of mid-dle and secondary school students. The current literature review will focus on three key areas: the potential of learning objects as an effective educational tool in secondary school classrooms, the proposed pedagogical benefits of learning objects, and research on the use of learning objects in K-12 mathematics classrooms to date.

Potential for Using Learning Objects in Secondary School Without doubt, efforts to increase the use of technology in K-12 classrooms have been substantial in the past 10 years (Compton & Harwood, 2003; McRobbie, Ginns, & Stein, 2000; Plante & Beattie, 2004; US Department of Education, National Center for Education Statistics, 2002). In spite of this push, a number of researchers have argued that technology has had a minor and even negative impact on student learning (e.g., Cuban, 2001; Roberston, 2003; Russell, Bebell, O'D-wyer, & O'Connor, 2003; Waxman, Connell, & Gray, 2002). Part of the problem stems from a considerable list of obstacles that have prevented successful implementation.

Educators face a number of challenges when attempting to use technology in their classrooms including the amount of time required (Eifler, Greene, & Carroll, 2001; Wepner, Ziomek, & Tao, 2003), having to work with limited technological skills (Eifler et al., 2001; Strudler, Archambault, Bendixen, Anderson, & Weiss, 2003; Thompson, Schmidt, & Davis, 2003), fear of technology (Bullock, 2004; Doering, Hughes, & Huffman., 2003), difficulty in understanding how to inte-grate technology into teaching (Cuban, 2001), and insufficient access (Bartlett , 2002; Brush et al., 2003; Russell et al., 2003).

Learning objects offer several promising solutions to the challenges that everyday teachers face with respect to using technology. First and foremost, learning objects are easy to use. Teachers, even those who have limited computer-based skills, do not need to devote considerable blocks of time toward understanding how to use these straightforward tools (Gadanidis, et al., 2003; Kay & Knaack, 2007c). Second, good learning objects have well defined objectives and a clear narrow focus making it easier to develop effective lesson plans and integration strategies (Kay & Knaack, 2007c). Third, learning objects are readily accessible over the Internet. Given that over 90% of all public schools in North America and Europe now have access to the Internet (and therefore learning objects) with most having high-speed broadband connections (Compton & Harwood, 2003; McRobbie et al., 2000; Plante & Beattie, 2004; US Department of Education, National Center for Education Statistics, 2002), teachers need not worry about software accessibility. Fi-nally, reusability permits learning objects to be useful for a large audience, particularly when the objects are placed in well organized, searchable databases (e.g., Agostinho et al., 2004; Duval, Hodgins, Rehak & Mason, 2004).

In summary, the accessibility and straightforward design of reusable, concept-focussed learning objects help address a number of obstacles teacher face with respect to using technology includ-ing time, difficulties in learning new software, integration strategies, and accessibility.

Kay & Knaack

271

Pedagogical Benefits of Learning Objects A number of researchers have argued that learning objects, if carefully selected, have a consider-able potential to aid student learning (Akpinar & Bal, 2006; Liu & Bera, 2005; Nurmi & Jaak-kola, 2006; Reimer & Moyer, 2005). It is hypothesized that effective learning objects (a) require students to construct and manipulate information (Akpinar & Bal, 2006; Baser, 2005; Nurmi & Jaakkola, 2006), (b) provide rich feedback and interactive illustrations (Akpinar & Bal, 2006), (c) help students understand abstract ideas with concrete representations (Akpinar & Bal, 2006; Rei-mer & Moyer, 2005) and (d) support key student weaknesses like limited working memory, diffi-culty in retrieving long term memory, and ineffective cognitive strategies (Liu & Bera, 2005). In addition, it is emphasized that instructional strategies supporting the use of learning objects are critical for success, regardless of the quality of the learning object selected (Akpinar & Bal, 2006; Clarke & Bowe, 2006a; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). A judicious mix of instruction, exploration, practice, and reflections is ideal (Nurmi & Jaakkola, 2006). To date, the above list of hypotheses and speculations about the effective use of learning objects remain large-ly untested, particularly in the secondary school classroom.

Impact of Learning Objects in Mathematics An extensive review of learning objects articles in the past 10 years revealed 10 studies looking at the use of mathematics-based learning objects (Bower, 2005; Clarke & Bowe, 2006a, 2006b; Gadanidis et al., 2003; Kay & Knaack, 2007c; Kong & Kwok, 2005; Lim, Lee, & Richards, 2006; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005. It is worth noting that, aside from Gadanidis et al. (2003), these studies go back no further than 2005, indicating that research in this domain is are relatively new.

Context While the 10 studies reviewed for this paper focussed on mathematics, the context of use varied widely with respect to grade level, t ime spent using learning objects, number of objects evaluated, and implementation strategies. In terms of grade level, four studies looked at elementary school students (Bower, 2005; Clarke & Bowe, 2006a, 2006b; Reimer & Moyer, 2005), three studies investigated middle school students (Bower, 2005; Kong & Kwok, 2005; Nurmi & Jaakkola, 2006), two studies examined high school students (Kay & Knaack, 2007c; Lopez-Morteo & Lo-pez, 2007), and two studies focussed on higher education students (Gadanidis et al., 2003; Lim et al., 2006). T ime spent using learning objects, varied from 40 to 60 minutes (Akinpar & Bal, 2006; Kay & Knaack, 2007c; Nurmi & Jaakkola, 2006) to several weeks (Clarke & Bowe, 2006a, 2006b; Liu & Bera, 2005; Reimer & Moyer, 2005). Regarding the number of learning objects evaluated, half of the papers focussed on a single learning object, and half looked at multiple learning objects (Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2007c; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). Finally, with implementation strategies, some teachers played a facilitating role, allowing students to explore the learning object on their own (Kay & Knnack, 2007c; Kong & Kwok, 2005; Nurmi & Jaakola, 2006), while other teachers reported the use of multiple teaching techniques such as large group discussion, guiding worksheets, collaborative learning, and writing reflective comments (Clarke & Bowe, 2006a; 2006b; Lim et al., 2006).

Teacher perspective Only four studies looked at teacher attitudes toward the use of learning objects in the mathematics classroom (Clarke & Bowe, 2006a; 2006b; Gadanidis et al., 2003; Kay & Knaack, 2007c). Based largely on qualitative data collection methods, teachers valued several characteristics of learning objects: the immediate feedback provided, the ability to replay and redo tasks for both enjoyment

Inv estigating the Use of Learning Objects

272

and mastery, and the motivational impact. Kay and Knaack (2007c) added quantitative evidence suggesting that pre-service teachers felt learning objects benefited student learning.

Student perspective Seven studies looked at student attitudes toward learning objects. Mathematics students reported liking learning objects because they (a) were fun and enjoyable, (b) were easy to control with re-spect to the pace of learning, (c) provided timely feedback, (d) were easy to use, (e) consisted of a number multimedia tools, and (f) helped with learning (Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2007c; Lim et al., 2006; Lopez-Morteo & Lopez, 2007; Nurmi & Jaakkola, 2006; Rei-mer & Moyer, 2005). Lim et al. (2006) and Nurmi and Jaakkola, (2006) added that the accep-tance of learning objects was somewhat dependent on the kind of learning object used. Students in these studies favoured interactive, constructive learning objects over the “electronic” textbook prototype. Kay and Knack (2007c) offered quantitative evidence that students were moderately, but not exceedingly, positive about using mathematics-based learning objects.

Student performance Four studies looked at learning performance after students used math-based learning objects, but none were done at the secondary school level (Bower, 2005; Kong & Kwok, 2005; Nurmi & Ja-akkola, 2006; Reimer & Moyer, 2005). In all four studies, elementary and middle school students who used learning objects showed significant improvement on various performance measures. Several contextual details, though, are worth noting.

Nurmi and Jaakkola (2006) noted that learning performance was dependent on the type of learn-ing object and how it was used. Students working with drill and practice learning objects were more focussed on competing with their peers than on learning. Students involved in a mixed learning object / lab-based lesson, performed significantly better than in other learning scenarios. In addition, Bower (2005) observed that students’ performance improved when they received feedback comparing their personal performance to their peers.

Purpose The purpose of this study was to examine the use of learning objects in the secondary school ma-thematics classroom from the perspective of both teachers and students.

Method

Overview This paper reviewed 10 articles looking at the use of learning objects in mathematics classrooms, although just two papers focussed on secondary schools. While half of the papers used mixed data collection methods, a number of issues need to be addressed to improve the investigation of learning objects. First, eight out of ten studies used a qualitative approach to data collection that relied heavily on descriptive data and anecdotal reports to assess the merits of learning objects. The reliability and validity of these informal qualitative observations are questionable, as is relying on only one data source.

Second, limited information is provided on teacher attitudes toward learning objects. Only four studies examined teacher perspectives’ on the actual use of learning objects (Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2007c), however three of those studies reported “general impres-sions” of relatively small samples.

Kay & Knaack

273

Third, sample populations tested in K-12 studies are small and weakly described making it chal-lenging to extend any conclusions to a larger population. Only two studies (Bower, 2005; Kay & Knaack, 2007c) examined more than 100 students and average sample size was 70 students.

Fourth, triangulation of data collection is somewhat limited with only one study using more than two data collection procedures (Reimer & Moyer, 2005). In addition, no researchers looked at student attitude, teacher attitude, and student performance simultaneously.

Finally, while most evaluation studies reported that students benefited from using learning ob-jects, the evidence is based on assessment tools with no validity or reliability. Only two studies offered reliability estimates (Kay & Knaack, 2007c; Kong & Kwok, 2005) and one study, validity data (Kay & Knaack, in press, 2007c). The lack of reliability and validity of evaluation tools compromises the accuracy of results presented to date.

In summary, previous methods used to evaluate learning objects are limited with respect to col-lecting multiple data sources, evaluating teacher perspectives, acquiring an adequate sample size, and providing estimates of reliability, and validity.

In order to address the key methodological challenges, the following steps were taken:

1. a large, diverse, sample was used; 2. reliability and valid surveys were used ; 3. formal statistics were used where applicable; 4. both qualitative and quantitative data were collected; 5. both teacher and student perspectives were assessed; 6. a measure of student performance was included; and 7. a number of learning objects were tested.

Sample

Teachers The teacher sample consisted of 11 teachers (5 males, 6 females) and 20 classrooms (a number of teachers used learning objects more than once). Teaching experience ranged from 2 to 27 years with a mean of 9.3 (SD = 9.8). A majority of the teachers rated their ability to use computers as strong or very strong (n=9) and their attitude toward using computers as positive or very positive (n=10). In spite of the high computer ability and positive attitudes, only two of the teachers used computers in their classrooms more than once a month.

Students The student sample consisted of 298 secondary school students (161 males, 137 females), 11 to 22 years of age (M = 16.3, SD = 1.2). The population base spanned three separate boards of edu-cation, 9 secondary schools, and 20 different classrooms. The students were selected through convenience sampling and had to obtain signed parental permission to participate.

Learning objects Teachers were permitted to select any learning object they deemed appropriate for their curricu-lum. As a starting point, they were introduced to a wide range of learning objects located at the LORDEC website (http://www.education.uoit.ca/lordec/collections.html). Seventy percent of the teachers selected learning objects from the LORDEC repository - the remaining teachers reported that they used Google. A total of 13 unique learning objects were selected (see http://faculty.uoit.ca/kay/papers/Math_LO.html for links to of all the learning objects used). Key topics covered included algebra, coordinate planes, congruent triangles, equation of a line, integer

Inv estigating the Use of Learning Objects

274

addition and multiplication, slope of a line, proportional relationships, parabolic transformations, investigating triangle centres, exponential functions, and the binomial distribution.

Procedure Teachers from three boards of education were emailed by an educational coordinator and in-formed of the learning object study. Participation was voluntary and a subject could withdraw from the study at any time. Each teacher received a half day of training in November on how to choose, use, and assess learning objects (see http://www.education.uoit.ca/lordec/lo_use.html for more details on the training provided). They were then asked to use at least one learning object in their classrooms by April of the following year. Email support was available throughout the du-ration of the study. All students in a given teacher’s class used the learning object that the teacher selected. However, only those students with signed parental permission forms were permitted to fill in an anonymous, online survey about their use of the learning object. In addition, students completed a pre- and post-test based on the content of the learning object.

All teachers in this study used learning objects in a lab setting. In order to simulate real secon-dary school environment as much as possible, teachers were given full control over the learning object they selected, the purpose for using the learning object, how long the learning object was used, teaching strategies for using the learning objects, and the design of the pre- and post-tests.

Data Sources

Teacher use Teachers were asked (a) how much preparation was involved when using learning objects (e.g., how long it took them to find and integrate learning objects into their classroom), (b) their pur-pose for using the learning object (e.g., motivate students, teach a new concept, review, supple-menting a lesson), (c) strategies they used to integrate learning objects (e.g., demonstration, pro-viding a set of guiding questions, let student explore, discussion after learning object) and (d) how long the learning object was used in their classroom.

Teacher survey After using a learning object, each teacher completed the Learning Object Evaluation Scale for Teachers (LOES-T) to determine their perception of (a) how much their students learned (learn-ing construct), (b) the quality of the learning object (quality construct), and (c) how much their students were engaged with the learning object (engagement construct). Data from the LOES-T showed low to moderate reliability (0.63 for learning construct, 0.69 for learning object quality construct, 0.84 for engagement construct), and good construct validity using a principal compo-nents factor analysis. See Kay and Knaack (2007d) for a detailed of the teacher-based learning object scale.

Teacher comments Finally, teachers were asked to comment on the overall impact that the learning object had on learning (see Q9, Appendix A).

Student survey After using a learning object, students completed the Learning Object Evaluation Scale for Stu-dents (LOES-S) in Appendix B to determine their perception of (a) how much they learned (learning construct), the quality of the learning object (quality construct), how much they were engaged with the learning object (engagement construct). The constructs selected were based on

Kay & Knaack

275

a thorough review of the literature (Kay & Knaack, 2007a, 2007b). The scale showed good reli-ability (0.78 to 0.89), face validity, construct validity, convergent validity, and predictive validity (see Kay & Knaack, in press).

Student comments Students were asked to comment on what they liked and disliked about the learning object (Ap-pendix B – questions 13 and 14). These qualitative items were analysed according to the coding scheme provided in Table 1. This coding scheme (Kay & Knaack, 2007) was used to categorize 447 student comments. Each comment was then rated on a five-point Likert scale (-2 = very neg-

Table 1. Coding Scheme to Categorize Student Comments about Learning O bjects

Learning Categories Criteria Learn Student comments about a specific or general learning/teaching

issue involved in using the LO

Visual The student mention as visual feature of the LO that helped/inhibited their learning

Engagement Categories Challenge Refers to the ease/difficulty of the concepts being covered. Basi-

cally whether the content level of the LO matched the student’s cognitive level/understanding.

Code “it was easy” in here, but not “ it was easy to use”

Compare Student compares LO to another method of learning

Engage Student refers to program as being OR not being fun/enjoyable/engaging/interesting

Interactive Student refers to some interactive part feature of the LO

Technology The student mention a technological issue with respect to using the LO

Quality Categories Animate Refers to quality of animations /moving pictures

Audio Refers to some audio/sound aspect of the learning object

Easy Refers to clarity of instructions or how easy/hard the LO was to use. It does not refer to how easy/hard the concept was to learn.

Graphics Refers to static picture or look of the program (e.g., colours)

Help Refers specifically to help/hints/instructions/feedback provided by the LO

Control Refers to student control of choice/pace in using the LO

Organization/Design Refers to quality of organization/design or the LO

Text Refers to quality/amount of text in LO

Theme Refers to overall/general theme or CONTENT of LO

Inv estigating the Use of Learning Objects

276

ative, -1 = negative, 0 = neutral, 1 = positive, 2 = very positive). Two raters assessed all com-ments made by students based on category and rating value. Comments where categories or rat-ings were not exactly the same were shared and reviewed a second time by each rater. Using this approach, an inter-rater reliability of 99% was attained for categories and 100% for the rating val-ues. Note that the total impact of any one category was determined by multiplying the mean rating by the total number of students who made a comment. For example, from Table 4, the impact of visual supports on learning was calculated by multiplying the mean which was 1.00 by the num-ber of student who commented about visual supports (24) for a total of 24.0.

Student performance Students completed a pre-test and post-test created by each teacher based on the content of the learning object used in class. Questions for pre- and post-test were identical in form, but differed in the raw numbers used. The type of questions asked varied according to the goal of the specific learning objects. Some tests focussed primarily on factual knowledge while others assess higher order thinking focussing on “what-if” scenarios. The measure was used to determine student per-formance. Because of the wide range of learning objects used, it was not possible to assess the validity of this test data.

Key Questions & Data Analysis In order to examine the impact of learning objects on secondary school students, the following questions were addressed in the data analysis:

1. How do teachers use learning objects in their classrooms? (descriptive analysis of teacher use questions);

2. How do teachers rate learning, quality, and engagement of learning objects? (descriptive analysis of teacher survey – LOES-T);

3. What was the overall impact of learning objects according to teachers? (analysis of quali-tative teacher comments);

4. How do students rate learning, quality, and engagement of learning objects? (descriptive analysis of student survey – LOES-S);

5. What do students like and dislike most about learning objects? (qualitative analysis of student comments);

6. How do teacher ratings of learning objects compare with student ratings? (correlation among learning, quality, and engagement constructs) and

7. How do learning objects affect student performance? (t-test comparing pre and post scores); and

8. How does teacher use of learning objects affect performance? (correlations among prepa-ration, purpose, teaching strategies, t ime used, and student performance scores)

Kay & Knaack

277

Results

Use of Learning Objects

Finding a learning object Fifty percent (n=10) of the teachers reported that finding a suitable learning object took them less than 30 minutes. Thirty percent (n=6) took 30 to 60 minutes to find an appropriate learning ob-ject. The remaining 20% (n=4) took over an hour to finding the learning object they wanted to use in their class.

Preparing a learning object lesson With respect to preparation for using the learning object in class, 5% of the teachers (n=1) spent litt le or no time, 40% (n=8) spent less than 30 minutes, 25% (n=5) spent 30 to 60 minutes, and the remaining 30% (n=6) spent over an hour.

Using a learning object On average, teachers used learning objects for 37.1 minutes (S.D. 17.0); however there was con-siderable variability (15 to 70 minutes). Students worked on their own on computers in a major-ity of classrooms (85%, n=17), with cooperative learning chosen only 15% of the time.

Reason for using a learning object The top four reasons cited by teachers for using learning objects were to provide another way of looking at a concept (75%, n=15), review a previous concept (45%, n=9), explore a new concept before a lesson (40%, n=8), and motivate students about a topic (35%, n=7). Teachers rarely or never used learning objects to teach a new concept, explore a new concept after a formal lesson, or for homework (4%, n=2).

Strategies for using learning objects Teachers in this study typically provided a brief introduction to a learning object (55%, n=11) or let the students start exploring on their own (45%, n=9). Only 20% (n=4) offered a formal dem-onstration of the learning object before the class used it . Forty-five percent of teachers (n=9) pre-pared a formal handout or guiding questions (30%, n=6). Forty percent of teachers (n=8) chose to have a class discussion about the learning object after it was used by students.

Teacher Rating of Learning Objects

Learning The mean rating for impact learning was 10.8 (SD = 1.3) or 5.4 on a 7-point scale. This suggests that most teachers agreed somewhat that learning objects had a positive impact on student learn-ing. Note that the range of learning construct scores was relatively narrow (8 to 13) (Table 2).

Quality of Learning Object The mean rating of learning object quality was 16.0 (SD = 2.8) or 5.3 on a 7-point scale. Most teachers somewhat agreed or agreed that learning objects were of good quality. The range of learning object quality scores was broader than that of the learning construct (11 to 20), but never dipped into negative rating (Table 2).

Inv estigating the Use of Learning Objects

278

Engagement Teachers rated engagement of learning objects the highest of all constructs with a mean score of 16.9 (SD = 2.8) or 5.6 on a 7-point scale. A majority of teachers, then, felt students were engaged while using learning objects. The range of learning object engagement scores was relatively large compared to the learning and quality constructs (9 to 21) (Table 2).

Table 2. Teacher Rating of Learning, Quality, and Engagement for Learning O bjects

Scale No. Items

Possible Range

Actual Range O bserved

Mean (S.D)

Learn 2 2 to 14 8 to 13 10.8 (1.3) Quality 3 3 to 21 11 to 20 16.0 (2.8)

Engagement 3 2 to 21 9 to 21 16.9 (2.9)

Teacher comments about learning objects Four themes emerged from the 24 comments that teachers made about the overall impact of the learning objects: overall learning, visual supports for learning, engagement, and time.

With respect to overall learning, three teachers reported that the learning object was not as suc-cessful has they had hoped:

“Although the learning object did not improve as many students' initial understanding of equation solving as I had hoped, it was useful.” “[After the using the learning object] they still had great difficulty distinguishing between vertical and horizontal stretches and compressions.

However, six teachers commented that visual supports in the learning objects helped their stu-dents learn better:

“I felt that this particular learning object helped them gain a solid visual of what happens to a parabola as you change the numbers of the vertex form of a parabola.” “For many students, using the balance scale was an excellent visual representation of solving algebra problems.”

Regarding engagement, five teachers noted that their students seemed to enjoy using the learning objects:

“Students enjoyed using the learning object and spending time on the computer in math class.” “I think students seemed to enjoy using it , especially the quiz.”

Finally, four teachers mentioned that t ime was an issue, either in creating a good lesson plan with a learning object, saving time, booking the right time to use learning objects, or not having enough time.

“The learning object allowed me to take a much shorter t ime to teach to teach the concept after the students had been introduced to it in the learning object.

“I had only had one class prior to using the learning object to introduce the concept to the class because of problems at the school in terms of booking computer time.”

Kay & Knaack

279

Student Rating of Learning Objects

Learning Students rated learning objects lower than teachers with respect to learning (M=15.65, SD = 4.6) with a mean item rating of 3.1 out of 5 (or 4.4 out of 7). Students were relatively neutral with respect to how much they felt the learning objects contributed to their learning. The range of scores was extensive (5 to 25) indicating that there was considerable variability with respect to whether students felt learning objects helped them learn (Table 3).

Quality of learning objects Students rated the quality of learning objects higher than their learning value, although the mean item rating was still lower than that of the teachers. The mean item rating was 3.4 out of 5 (4.8 or out 7) indicated that most students somewhat agreed that the learning objects they used were of good quality. The range of learning object quality scores (4 to 20) was highly variable (Table 3).

Engagement Ratings of learning object engagement were moderate (M=9.34, S.D. = 2.8) with a mean item rat-ing of 3.1 out or 5 (or 4.4 out of 7). In other words, as was the case with the learning construct, students were neutral about engagement value of the learning objects they used. High variability among student engagement ratings is supported by the wide range of scores reported (3 to 15).

Table 3. Description of Student Learning O bject Evaluation Scales (LO ES-S)

Scale No. Items

Possible Range Actual Range Observed

Mean (S.D)

Learn 5 5 to 25 5 to 25 15.4 (4.3) Quality 4 4 to 20 4 to 20 13.5 (3.4)

Engagement 3 3 to 15 3 to 15 9.1 (2.6)

Student comments about learning objects Student comments are summarized in Table 4. With respect to learning, the visual support that a learning object offered toward learning was rated the highest, whereas overall learning and the pedagogical challenge of the learning object were rated quite low. In other words, many students liked the visual affordances of learning objects, however quite a few felt the learning object did not support learning or was not challenging enough.

With respect to rating the quality of learning objects, ease of use was the highest rated feature. On the other hand, the quality of help and having excessive amounts of text were rated the lowest. Finally, regarding engagement, interactivity, and comparison with other methods of teaching were rated the highest. A number of the students liked the interactive qualities of learning objects and felt they were an improvement over other teaching strategies.

Inv estigating the Use of Learning Objects

280

Table 4. Summary of Student Comments about Learning O bjects

Category Mean S.D n Total Effect Mean * n

Learning

Visual Supports 1.00 0.00 24 24.0 Challenge -0.46 1.02 52 -24.0

Overall Learning -0.37 1.11 75 -28.0

Quality Easy 0.88 0.80 24 21.0

Control 0.71 0.76 7 5.0 Animation 1.00 0.00 2 2.0

Graphics -0.14 1.25 29 -4.0

Theme -0.43 1.22 14 -6.0 Organization -0.81 0.98 16 -13.0

Help -0.86 0.75 28 -24.0

Text -1.19 0.40 21 -25.0 Engagement

Interactivity 0.90 0.44 21 19.0

Compare with other method 0.67 0.76 24 16.0 Liking Technology 0.50 0.91 12 6.0

Engagement / Motivation -0.04 1.18 50 -2.0

Student Performance Overall, student performance scores increased by an average of 7.6% from 56.4% to 64.0%. This change was significant (t = -3.85, df = 123, p < .001). The effect size (based on Cohen’s d) of 0.29 is considered a small effect according to Thalheimer & Cook (2002). Note that the average student performance score by class varied considerably ranging from a 4.2% decrease to a 27.1% increase.

Teacher Use and Student Performance Four areas of teacher use were examined with respect to student performance: preparation, pur-pose, strategies for teaching, and the amount of time a learning object was used. With respect to preparation, t ime spent finding a learning object was inversely correlated with student perform-ance (r = -0.34, p < .001). In other words, the more time a teacher spent looking for a learning object, the less successful the learning object was in terms of student performance. Planning time was strongly correlated with student performance (r = 0.69, p < .001). Regarding the main purpose or objective for using a learning object, introducing a new concept, then following up with a formal lesson (r = 0.68, p < .001) and reviewing previously learned con-cepts (r = 0.36, p < .001) were significantly correlated with student performance. However, us-

Kay & Knaack

281

ing a learning object to motivate students and give them another way of looking at a concept was negatively correlated with student performance (r = -0.31, p < .001). In terms of teaching strategies used with learning objects, positive correlations with student per-formance were seen when a worksheet and guiding questions were provided (r = 0.57, p < .001) and when students were encouraged to explore on their own (r = 0.29, p < .005). On the other hand, negative correlations were observed when a brief demonstration of the learning object was given (r = -0.41, p < .001) or when the learning object was discussed after the student had used it (r = -0.43, p < .001). Finally, the amount of time that students used a learning object in class was negatively correlated with student performance (r = -0.28, p < .005).

Discussion The purpose of this study was to investigate the impact of learning objects in secondary school mathematics classrooms. Evidence gathered to evaluate effectiveness included a description of teacher use, teacher attitudes, teacher comments, student attitudes, student comments, and student performance. Each of these sources of evidence will be discussed.

Teacher Use

Preparation Previous research has not been done looking at how teachers find learning objects and prepare to use them in the science classroom. This study provides new information in this area. Searching for and planning to use a learning object does not take an inordinate amount of time – on average, a half hour for each activity. It is interesting to note that planning time was strongly correlated with increases in student performance. While a few teachers commented that t ime was a concern, it is clear that this t ime investment is worthwhile. Paradoxically, t ime spent searching for learning objects was inversely related to student perform-ance. It is possible that teachers who took longer to search for learning objects were simply un-able to find one that fit their needs. They may have settled for a lower quality learning object simply because they ran out of time. Conversely, teachers who found learning objects quickly may have been impressed by a high quality learning object early on in the search process.

Purpose for using learning object Three quarters of the teachers used learning objects to provide another way of looking at a con-cept; however this approach resulted in significantly lower student performance. It is unclear why students faltered with this approach. Perhaps the “other” way of learning a concept was viewed as unnecessary and maybe confusing. Almost half the teachers used learning objects for review, a goal that correlated significantly with gains in student performance. Students appear to have reacted more positively when a deliberate review was planned, as opposed to a less focussed lesson that simply looked at a current concept in an alternative way. Students may also have been more motivated and focussed during a review because a formal test was imminent.

Four out of ten teachers chose to use learning objects to introduce a new concept before a formal lesson, an objective that was strongly correlated with improved student performance. This result is partially confounded by the fact that a formal lesson was used in conjunction with a learning object. It is impossible to determine the relative contribution of the learning object to final per-formance. That said, using this tool to lead into more formal instruction works very well.

Inv estigating the Use of Learning Objects

282

Teaching strategies The number one strategy for using learning objects, namely providing a brief introduction, was negatively correlated with student performance. At first glance, this finding seems counter intui-tive. It would be expected that students, even in the worst case scenario, would react in a neutral manner to a brief introduction, simply because it is “brief”. Then again, learning objects that tru-ly require and introduction may be of lower quality and harder to use. Another strategy that was used relatively often, but that resulted in lower student performance, was the practice of having a class discussion after students used the learning object. Again, this finding is opposite to what one would expect. One explanation might be that class discussion was used when the use of learning objects did not go smoothly, when there were problems, and per-haps when confusion was experienced by students. A more detailed description of the discussion is required to fully understand this result .

A strategy that was significantly correlated with learning was the inclusion of a worksheet with guiding questions. This kind of support may have focussed students to concentrate on the key concepts at hand, thereby resulting in increased understanding of tested items.

Finally, students clearly prefer to explore learning objects on their own, a strategy that resulted in increased student performance. Learning objects that permit students to simply “jump in” may be better designed in terms of usability, particularly if they are supplemented with guiding work-sheets.

Teacher Ratings (Learning, Quality, and Engagement) There is no question that the vast majority of teachers believed that the learning objects they se-lected were good quality, engaging tools that supported learning. Ratings of these constructs were fairly high. On the one hand, it may be obvious that teachers would give high ratings, since they selected the learning objects in the first place. On the other hand, teachers rated these learn-ing objects after they watched them being used by students in their classroom. In addition, they had no personal stake in approving these tools, and three teachers felt open enough to acknowl-edge that learning objects had not fully met their expectations. Positive reaction from teachers in this study is consistent with results reported in previous research (Clarke & Bowe, 2006a; 2006b; Gadanidis et al., 2003; Kay & Knaack, 2007d).

Teacher Comments With respect to learning, teacher comments supported both survey and learning performance re-sults, however a few teachers were disappointed in the final impact. A number of teachers felt that the learning objects offered helpful visual supports and that the tool was engaging for stu-dents. In addition, learning objects were thought to be engaging. These comments are consistent with observations made by previous researchers (Clarke & Bowe, 2006a; 2006b; Gadanidis et al., 2003; Kay & Knaack, 2007d).

Student Ratings (Learning, Quality, and Engagement) On average, students were neutral with respect to the learning, quality, and engagement impact of learning objects. These results are somewhat consistent with the modest enthusiasm expressed in the one other study that formally evaluated the use of math-based learning objects by secondary school students (Kay & Knaack, 2007d). It is critical to note, though, that the range of scores was very broad for all three constructs. For any given learning object, even when it was used by the same class, some students liked it a lot and others disliked it intensely. Learning objects clearly do not suit every student’s learning style. The challenge remaining is to identify the source and perhaps cause of individual differences.

Kay & Knaack

283

Finally, teachers rated learning, quality, and engagement much higher than students. What ma-thematics teachers think is an effective, high quality, motivating teaching tool, may be perceived as neutral or barely acceptable by students. Teachers might be wise to test learning objects with a few representative students, before exposure to the whole class.

Student Comments about Learning Objects Student comments offer insight into what students like and do not like about learning objects. Not surprisingly, students liked easy to use, interactive learning objects that provided good visual supports. They reacted negatively, though, to poor quality help and having to read excessive amounts of text. These reactions are similar to those made by students in previous studies (Clarke & Bowe, 2006a; Clarke & Bowe, 2006b; Kay & Knaack, 2007d). Finally, it is interesting to note that even though students were relatively neutral about the learning objects they used, a noticeable number of students reported that using learning objects was an improvement over oth-er teaching methods.

Student Performance This is a first study on the impact of learning objects in secondary school mathematics classes with respect to learning performance. The fact that learning performance increased is not surpris-ing given the wealth of previous research that has reported similar results at other grade levels (Akpinar & Bal, 2006; Baser, 2005; Liu & Bera, 2005; Nurmi & Jaakkola, 2006; Rieber, Tzeng, & Tribble, 2004; Schoner, Buzza, Harrigan, & Strampel, 2005; Windschitl & Andre, 1998). However, the magnitude of the overall effect (7.6 %) was fairly small. It is important, though, to be cognisant of two other results. First, the variability in student performance among classrooms was considerable – from a drop of 4% to and increase of 27%. So the more critical question is, “Why are learning objects successful in some classes, but ineffective in others?” The results of this study suggest that the answer, at least in part, is increased planning time, using learning ob-jects for review or to introduce a new concept, and supplying guided handouts to focus students

Implications for Education This study offers several practical observations and suggestions for using learning objects in sec-ondary school mathematics classrooms. First , it is important to take sufficient t ime to prepare for lessons involving learning objects. Second, lessons should have a clear focus like introducing a new concept or reviewing previous concepts. Simply using a learning object as another way of looking at a concept may confuse students and have a detrimental effect. Third, a supporting handout with guided questions should improve student performance. Fourth, the effect of learn-ing objects may vary greatly within the same classroom. Accommodations could have to be made for students with different ability and interest levels. Finally, it might be wise to pre-test learning objects on a few students. Students and teachers differ considerably on what they see as high quality, engaging learning objects that promote learning. On average, students appear to have a more accurate sense of these qualities, at least when it comes to predicting learning per-formance.

Caveats and Future Research In this study, careful attention was directed toward collecting good quality data by sampling a large, relatively diverse population, establishing the reliability and validity of measures used, and using multiple data sources to establish triangulation. Nonetheless, several limitations exist which provide opportunities for future researchers. First, the overall results saw inconsistencies between teacher attitudes, student attitudes, and student performance. More extensive qualitative

Inv estigating the Use of Learning Objects

284

data in the form of focus groups and/or interviews might shed further light one why these dis-crepancies exist and what role learning objects have to play. Second, more qualitative data is needed to understand how teacher goals and strategies influence student performance. A number of strong, significant correlations were observed, but it was chal-lenging to interpret what teachers were doing. For, example brief introductions to learning ob-jects were negatively correlated with student performance. It is difficult to comprehend this un-expected finding without knowing the intended message of these introductory segments.

Finally, the impact of specific kinds of learning objects was not looked at. It is possible that cer-tain categories of learning objects may have decidedly different impacts on learning. The wide variability in student performance among classes may be partially explained by teachers’ goals and strategies, but the type of learning object used may also have an impact. Developing and evaluating a classification system for learning objects is an important next step in learning object research.

References Agostinho, S., Bennett, S., Lockyear, L., & Harper, B. (2004). Developing a learning object metadata ap-

plication profile based on LOM suitable for the Australian higher education market. Australasian Journal of Educational Technology, 20(2), 191-208.

Akpinar, Y. & Bal, V. (2006). Student tools supported by collaboratively authored tasks: The case of work learning unit. Journal of Interactive Learning Research, 17(2), 101-119.

Bartlett, A. (2002). Preparing preservice teachers to implement performance assessment and technology through electronic port folios. Action in Teacher Education, 24(1), 90-97.

Baser, M. (2005). Promoting conceptual change through active learning using open source software for physics simulations. Australasian Journal of Educational Technology, 22(3), 336-354.

Bower, M. (2005). Online assessment feedback: Competitive, individualistic, or…preferred form! Journal of Computers in Mathematics and Science Teaching, 24(2), 121-147.

Brush, T., Glazewski, K., Rutowski, K., Berg, K., Stromfors, C., Van-Nest, M., Stock, L., & Sutton, J. (2003). Integrating technology in a field-based teacher training program: The PT3@ASU project. Edu-cational Technology, Research and Development, 51(1), 57-73.

Brush, T. & Saye, J. (2001). The use of embedded scaffolds with hypermedia-supported student-centered learning. Journal of Educational Multimedia and Hypermedia, 10(4), 333-356.

Bullock, D. (2004). Moving from theory to practice: An examination of the factors that preservice teachers encounter as the attempt to gain experience teaching with technology during field placement experi-ences. Journal of Technology and Teacher Education. 12 (2), 211-237.

Clarke, O., & Bowe, L. (2006a). The learning federation and the Victorian department of education and training trial of online curriculum content with Indigenous students. 1-14. Retrieved from http://www.thelearningfederation.edu.au/verve/_resources/tl f_detvic_indig_trial_mar06.pdf

Clarke, O., & Bowe, L. (2006b). The learning federation and the Victorian department of education and training trial of online curriculum content with ESL students. 1-16. Retrieved from http://www.thelearningfederation.edu.au/verve/_resources/report_esl_final.pdf

Compton, V., & Harwood, C. (2003). Enhancing technological practice: An assessment framework for technology education in New Zealand. International Journal of Technology and Design Education, 13(1), 1-26.

Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard Uni-versity Press.

Kay & Knaack

285

Doering, A., Hughes, J., & Huffman, D. (2003). Preservice teachers: Are we thinking with technology? Journal of Research on Technology in Education, 35(3), 342-361.

Duval, E., Hodgins, W., Rehak, D., & Robson, R. (2004). Learning objects symposium special issue guest editorial. Journal of Educational Multimedia and Hypermedia, 13(4), 331-342.

Eifler, K. Greene, T., & Carroll, J. (2001). Walking the talk is tough: From a single technology course to infusion. The Educational Forum. 65(4), 366-375.

Gadanidis, G., Gadanidis, J., & Schindler, K. (2003). Factors mediating the use of online applets in the les-son planning of pre-servi ce mathematics teachers. Journal of Computers in Mathematics and Science Teaching, 22(4), 323-344.

Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of Instructional Sciences and Technology, 8(1). Retrieved June 1, 2007 from http://www.usq.edu.au/electpub/e-jist/docs/vol8_no1/fullpapers/eval_learnobjects_school.htm

Kay, R. H., & Knaack, L. (in press). Assessing learning, quality and engagement in learning objects: The learning object evaluation scale for students (LOES-S). ETR&D.

Kay, R. H., & Knaack, L. (2005). Developing learning objects for secondary school students: A multi-component model. Interdisciplinary Journal of Knowledge and Learning Objects, 1, 229-254. Re-trieved from http://ijello.org/Volume1/v1p229-254Kay_Knaack.pdf

Kay, R. H., & Knaack, L. (2007a). A systematic evaluation of learning objects for secondary school stu-dents. Journal of Educational Technology Systems, 35(4), 411-448.

Kay, R. H., & Knaack, L. (2007b). Evaluating the learning in learning objects. Open Learning, 22(1), 5-28.

Kay, R. H., & Knaack, L. (2007c). Evaluating the use of learning objects for secondary school science. Journal of Computers in Mathematics and Science Teaching, 26(4), 261-289.

Kay, R. H., & Knaack, L. (2007d). Teacher evaluation of learning objects in middle and secondary school classrooms. Manuscript submitted for publication. Retrieved Feb, 2008 from http://faculty.uoit.ca/kay/papers/LOES_Teacher_2007.pdf.

Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common frac-tions: A model of affordances. Computers and Education, 45(2), 245-265.

Lim, C. P., Lee, S. L. & Richards, C. (2006). Developing interactive learning objects for a computing ma-thematics models. International Journal on E-Learning, 5(2), 221-244.

Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environ-ment. Educational Technology, Research and Development, 53(1), 5-21.

Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning environ-ment based on recreational learning objects. Computers and Education, 48(4), 618-641.

McRobbie, C. J., Ginns, I. S., & Stein, S. J. (2000). Preservice primary teachers' thinking about technology and technology education. International Journal of Technology and Design Education, 10, 81-101.

Nurmi, S., & Jaakkola, T. (2006). Effectiveness of learning objects in various instructional settings. Learn-ing, Media, and Technology, 31(3), 233-247.

Plante, J., & Beattie, D. (2004). Education, skills and learning – Research papers Connectivity and ICT integration in Canadian elementary and secondary schools: First results from the Information and Communications Technologies in Schools Survey, 2003-2004. Statistics Canada. Retrieved Aug 29, 2004 from http://www.schoolnet.ca/home/documents/Report_EN.pdf

Rehak, D., & Mason, R. (2003). Chapter 3: Keeping the learning in learning objects. Journal of Interactive Media in Education, 1. Retrieved July 1, 2005 from http://www-jime.open.ac.uk/2003/1/reuse-05.html

Reimer, K., & Moyer, P. S. (2005). Third-graders learning about fractions using virtual manipulatives: A classroom study. Journal of Computers in Mathematics and Science Teaching, 24(1), 5-25.

Inv estigating the Use of Learning Objects

286

Rieber, L. P., Tzeng, S., & Tribble, K. (2004). Discovery learning, representation, and explanation within a computer-based simulation: Finding the right mix. Learning and Instruction, 14(3), 307-323.

Robertson, H. (2003). Toward a theory of negativity: Teacher education and information and communica-tions technology. Journal of Teacher Education, 54(4), 280-296.

Russell, M., Bebell, D., O’Dwyer, L., & O’Connor, K. (2003). Examining teacher technology use: Implica-tions for preservice and inservice teacher preparation. Journal of Teacher Education, 54(4), 297-310.

Schoner, V., Buzza, D., Harrigan, K. & Strampel, K.. (2005). Learning objects in use: ‘Lite’ assessment for fi eld studies. Journal of Online Learning and Teaching, 1(1), 1-18.

Strudler, N., Archambault, L., Bendixen, L., Anderson, D., & Weiss, R. (2003). Project THREAD: Tech-nology helping restructure educational access and delivery. Educational Technology Research and De-velopment, 51(1), 39-54.

Thalheimer, W., & Cook, S. (2002). How to calculate effect sizes from published research articles: A sim-plified methodology. Retrieved July 14, 2007 from http://work-learning.com/effect_sizes.htm

Thompson, A. D., Schmidt, D. A., & Davis, N. E. (2003). Technology collaboratives for simultaneous re-newal in teacher education. Educational Technology Research and Development, 51(1), 73-89.

U.S. Department of Education, National Center for Education Statistics (2002). Internet access in U.S. public schools and classrooms: 1994-2002. Retrieved August 30, 2004 from http://nces.ed.gov/programs/digest/d02/tables/dt419.asp

Waxman, H. C., Connell, M. L., & Gray, J. (2002). A quantitative synthesis of recent research on the ef-fects of teaching and learning with technology on student outcomes. Naperville, IL: North Central Re-gional Laboratory.

Wepner, S. B., Ziomek, N., & Tao L. (2003). Three teacher educators’ perspectives about the shifting re-sponsibilities of infusing technology into the curriculum. Action in Teacher Education, 24(4), 53-63.

Windschitl, M., & Andre, T. (1998). Using computer simulations to enhance conceptual change: The roles of constructivist instruction and student epistemological beliefs. Journal of Research in Science Teach-ing, 35(2), 145-160.

Kay & Knaack

287

Appendix A - Learning Object Survey – Teachers Strongly

Disagree 1

Disagree

2

Slightly Disagree

3

Neutral

4

Slightly Agree

5

Agree

6

Strongly Agree

7

Learning

1. The graphics and an-imations from the learning object helped students learn.

1 2 3 4 5 6 7

2. The students were able to learn from the learning object.

1 2 3 4 5 6 7

Quality

3. The learning object was easy for students to use.

1 2 3 4 5 6 7

4. The learning object was easy to learn.

1 2 3 4 5 6 7

5. The students found the learning object in-structions clear

1 2 3 4 5 6 7

Engagement

6. The students liked interacting with the learning object.

1 2 3 4 5 6 7

7. The students were on task while using the learning object.

1 2 3 4 5 6 7

8. Students were moti-vated while using the learning object.

1 2 3 4 5 6 7

Overall Impact on Learning

9. What was the overall impact of the learning object on your lesson?

Inv estigating the Use of Learning Objects

288

Appendix B - Learning Object Survey – Students Strongly

Disagree 1

Disagree

2

Neutral

3

Agree

4

Strongly Agree

5

Learning

1. Working with the learning object helped me learn.

1 2 3 4 5

2. The feedback from the learning object helped me learn.

1 2 3 4 5

3. The graphics and animations from the learn-ing object helped me learn.

1 2 3 4 5

4. The learning object helped teach me a new concept.

1 2 3 4 5

5. Overall, the learning object helped me learn.

1 2 3 4 5

Quality

6. The help features in the learning object were useful.

1 2 3 4 5

7. The instructions in the learning object were easy to follow.

1 2 3 4 5

8. The learning object was easy to use. 1 2 3 4 5

9. The learning object was well organized. 1 2 3 4 5

Engagement 1 2 3 4 5

10. I liked the overall theme of the learning ob-ject.

1 2 3 4 5

11. I found the learning object motivating. 1 2 3 4 5

12. I would like to use the learning object again.

1 2 3 4 5

13. What, if anything, did you LIKE about the learning object?

14. What, if anything, did you NOT LIKE about the learning object?

Kay & Knaack

289

Biographies Robin Kay, Ph.D. is an Associate Professor in the Faculty of Educa-tion at the University of Ontario Institute of Technology. He has pub-lished over 40 articles in the area of computers in education, presented numerous papers at 10 international conferences, refereed three promi-nent computer education journals, and taught computers, mathematics, and technology for over 18 years. Current projects include research on laptop use in teacher education, discussion board use, electronic evaluation of teacher education programs, and factors that influence how students learn with technology.

Liesel Knaack , Ph.D. is an Associate Professor in the Faculty of Edu-cation at the University of Ontario Institute of Technology. Her re-search interests are in the areas of design, development and evaluation of learning objects, effective integration of computers in the curricula, instructional design of digital learning environments and the process of change in implementing technology use at institutions of higher educa-tion. Current projects include laptop use in preservice education pro-grams, online discussions and digital performance appraisals.