assignment 4.1 formative evaluation of an internet searching tutorial submitted by: sun hong hwang...

Assignment 4.1

Formative Evaluation of an Internet Searching Tutorial

Submitted By:

Sun Hong Hwang

Paul Kelly

Meredith Rogers

Q410 – Formative and Summative EvaluationMarch 15, 2003

Front-End Analysis Problem

The Park Hill School District (Kansas City, Missouri) identified a gap between teacher performance utilizing the Internet (actual) and an optimal expectation that teachers be able to effectively and efficiently find information to support instructional processes.

Proposed Solution Based upon the facts obtained via a front-end analysis (performance analysis,

environmental analysis, learner analysis, and needs assessment), it was determined that K-12 teachers in the Park Hill School District needed training in effectively searching the Internet to close the gap between the “optimals” and “actuals” found in during the front-end analysis.


Introduction

Instructional Materials/Product Materials

“Take the Garbage out of Internet Searching” was developed specifically as a training tool to address the performance problem outlined on the previous slide.

Learner Objectives Learners will. . .

. . .explore a variety of search engines and online directories and note differences of each.

. . . effectively utilize a search engine and a directory for online searches of information.

. . . improve the reliability of simple searches with advanced operators. . . . explore natural language searches. . . . effectively choose the proper search engine for a given online search

task.


Introduction

Instructional Materials/Product Status of Materials

“Take the Garbage out of Internet Searching” is currently being developed, awaiting the results of this formative evaluation prior to distribution.

Media “Take the Garbage out of Internet Searching” is currently available for preview

in an online version and a CD-ROM format. The product/materials were developed using Macromedia Authorware 6.0, which can be exported in a web format or as an executable runtime program from CD-ROM.

The CD-ROM version was utilized for the purposes of this report. The URL to the online version is:

http://content.parkhill.k12.mo.us/users/pvkelly/Web/internetsearching-beta2.htm


Introduction

Goals of Evaluation The goal of this formative evaluation is to improve the effectiveness of

the instructional materials prior to usage by a large population of K-12 teachers. The goals of this formative evaluation process include . . .

1. . . . identifying the deficiencies in learning effectiveness of the instructional materials;

2. . . . locating ease of use problems with the instructional materials; and

3. . . . evaluating the efficiency of the instruction within the materials.

Evaluators An Expert Reviewer Two One-on-One Reviewers


Formative Evaluation Framework

Reviewer Mr. Robert Borst, Middle School Mathematics Teacher at Gentry Middle

School, Columbia, Missouri Expertise

Teacher of courses for the SISLT at the within the track of technology in schools Web page designer & creator of multi-media presentations for the purpose of

conducting teacher in-services for the Columbia Public School District.

Evaluation Goals Evaluate the strengths and weaknesses of the program’s efficiency of

operation Evaluate the usefulness of the included content.

Evaluation Tool A single three-page questionnaire was to Mr. Borst to complete on his own

as he worked through the various screens of the program. (Appendix 01)


Expert Review

Recommendations of Expert Review The information is too wordy. There is too much to read through with

some of the topics outlined in the program. He said that the amount of information is far more in-depth that most K-12 teachers will ever need to know or will ever use.

The information is “presented” for the most part rather than given in an interactive mode for the user to demonstrate his or her skills.

The transitions between screens are inconsistent and very aggravating at times. A single type of quick transition needs to be considered,

rather than a different type of screen change with every new topic.


Expert Review

Reviewer A Amy Carso, Middle School Language Arts Teacher at Gentry

Middle School, Columbia Public School District Expertise

A novice when it comes to understanding how the computer works and all its capabilities.

She describes herself as being “pretty comfortable [with technology now]”

Reviewer B Keri McConnell, 5th Grade Teacher, Park Hill School District,

Kansas City, Missouri Expertise

She serves as a trainer for many technology-based staff development initiatives.

She is well respected by her peers for her use of technology in the classroom, as well as her ability and willingness to show others.


One-on-One Reviews

Evaluation Goals The reviewers evaluated . . .

. . . the directions and commands of this instruction program for clarity and operability from a learner’s perspective.

. . . the aesthetics design of the program for engaging the learner. . . . the level of interaction for the learner. . . .the content with regards to meeting the needs that K-12 teachers

generally desire in order to improve the efficiency of their Internet searching abilities.

Evaluation Tool A single three-page questionnaire was reviewed each user during their

sessions and given to them to complete on their own as they worked through the various screens of the program (Appendix 02).


One-on-One Reviews

Recommendations of One-on-One Reviews Font size is too small making the user having to strain to read

some of the material. There needs to be more interaction for the learner as they are

reading through the information. The reviewers suggest the inclusion of sample exercises, pre-test questions or actual connections to the Internet to practice some of their skills.

There was no real challenge to this program for the user. Both reviewers summarized the program, as just basically being a reading exercise, which many educators know will not keep a

learner engaged in any learning process for very long.


One-on-One Review

Recommendations of Formative Evaluation Based on the data collected by the expert reviewer and the two one-on-one

reviewers, the following items should be considered for revision in order to improve the program with the hope of addressing the objectives of this program more efficiently.

Font Needs to be larger or more bold; change style to a more squared shape.

Directionality Either enlarge the arrows on the quiz screens or remove them and just

make the words the link; be consistent with what are links within the program, add a skip; create ways to restart the program more easily; add a skip button on opening screens to bypass; fix or remove the backwards button on the first two screens.

Aesthetics Use only one kind of transition between screens; use a lighter background

(e.g. white or ecru) and/or a deeper font color (e.g. navy or black).


Recommended Revisions

Recommendations of Formative Evaluation (continued) Based on the data collected by the expert reviewer and the two one-on-one

reviewers, the following items should be considered for revision in order to improve the program with the hope of addressing the objectives of this program more efficiently.

Content Content assume the user has background knowledge on the topic;

simplify key terms or provide examples to explain the terms directly following the term in the write-up; put in correct answers to incorrectly answered questions to give clearer feedback.

Operation A claim statement needs to be made explaining that Authorware software

is needed in order to run the program and that it can be easily downloaded to run the program form the web version, or that it is included on the CD-ROM already.

Interaction Build in more interactive features such as practice examples where the

user could be practicing on the Internet with the instructional materials open; cut down the content and focus on the key elements used in everyday searches; add more quiz questions, provide a pre-test on some skill; add a help menu.


Recommended Revisions


Self-Evaluation

Strengths of our Evaluation The organization of the evaluation tools The use of the scripts during both types of reviews Communication within our evaluation team and the

analysis of our results.

Areas of Improvement The development of clear and specific goals and criteria for

our reviewers to ensure they meet our needs for evaluation The clarity in the objectives of the instructional materials so

that the tools developed for the evaluation are better aligned.


References

Fitzgerald, G., & Koury, K. (2001-2002). The KidTools Support System. U.S. Department of Education. Project #K033271. http://www.coe.missouri.edu/~vrcbd

Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. London, England: Kogan Page Limited.

assignment 4.1 formative evaluation of an internet searching tutorial submitted by: sun hong hwang...

Documents