Evaluating CAI Evaluating CAI LessonsLessons

Dr. Ennis-ColeDr. Ennis-Cole

CECS 5020CECS 5020

Lesson Lesson Objectives Objectives

Have the expected Have the expected outcomes been outcomes been

realized ?realized ?

Were the desired Were the desired outcomes a result of the outcomes a result of the

lesson ? lesson ?

Purpose of Purpose of Evaluation Evaluation

1. Confirmation of Decisions1. Confirmation of Decisions

2. Lesson Improvement2. Lesson Improvement

““An ongoing process used to determine An ongoing process used to determine whether lesson objectives have been met, whether lesson objectives have been met, identify reasons for the observed identify reasons for the observed performance, and identify areas of performance, and identify areas of modification.” p. 299 Hanafin & Peck modification.” p. 299 Hanafin & Peck

Evaluation Evaluation Techniques Techniques

Informal - advice from experts, ideas from Informal - advice from experts, ideas from users, observations, interviews (simplistic- users, observations, interviews (simplistic- anecdotal data)anecdotal data)

Formal - planned evaluation models, formal Formal - planned evaluation models, formal analysis, attitudinal questionnaires, data analysis, attitudinal questionnaires, data collection & collation (formal tabulation - collection & collation (formal tabulation - empirical evidence) empirical evidence)

Levels of Evaluation Levels of Evaluation

FormativeFormative

Summative Summative

Levels of Evaluation Levels of Evaluation

Formative - identification Formative - identification of lesson features that of lesson features that need modification - need modification -

ongoing processongoing process

lesson improvement is lesson improvement is the goalthe goal

Levels of Evaluation Levels of Evaluation

Summative - signature authority

not used for modification, used for purchase or adoption decisions

The Formative The Formative Evaluation Evaluation

- - - - - - - - - -- - - - - - - - - -

Issues addressed: Issues addressed:

Design LogicDesign Logic

Selection of VocabularySelection of Vocabulary

Clarity of Graphics & Branching IssuesClarity of Graphics & Branching Issues

Interrogation of Student/User ResponsesInterrogation of Student/User Responses

Stages of Formative Stages of Formative EvaluationEvaluation

1. One-to-One:1. One-to-One:• Initial lesson design & developmentInitial lesson design & development• Informal procedures Informal procedures • Identify potential problemsIdentify potential problems

1a. Assets of One-to-One Evaluations:1a. Assets of One-to-One Evaluations:• Insight gained before coding or implementation Insight gained before coding or implementation • Identification of problemsIdentification of problems• Expansion of lesson features or redirectionExpansion of lesson features or redirection

Stages of Formative Stages of Formative EvaluationsEvaluations

2. Small-Group Evaluation:2. Small-Group Evaluation:• Conducted at the lesson’s completion Conducted at the lesson’s completion • Determines effectivenessDetermines effectiveness• Collection & Interpretation of Student Performance Collection & Interpretation of Student Performance

Data Data 2a. Assets of Small-Group Evaluations:2a. Assets of Small-Group Evaluations:

• Use Informal techniques & dataUse Informal techniques & data• Identify areas of improvementIdentify areas of improvement• Generalized results from small-group interactionGeneralized results from small-group interaction

Stages of Formative Stages of Formative EvaluationsEvaluations

3. Field Testing:3. Field Testing:• Conducted in semi or actual settings Conducted in semi or actual settings • Occurs when lessons are “final quality”Occurs when lessons are “final quality”• Conducted to confirm lesson effectivenessConducted to confirm lesson effectiveness

3a. Assets of Field Testing:3a. Assets of Field Testing:• ““Real Life” Simulations of Lesson UseReal Life” Simulations of Lesson Use

• Rely heavily on both formal and informal Rely heavily on both formal and informal techniquestechniques

• Minimal Changes can be made Minimal Changes can be made

Evaluating CAI Evaluating CAI

Instructional Adequacy Instructional Adequacy Cosmetic AppealCosmetic Appeal

Program AdequacyProgram Adequacy

Curriculum AdequacyCurriculum Adequacy

Instructional Instructional AdequacyAdequacy

DirectionsDirections ContentContent NavigationNavigation Text PresentationText Presentation Logical Logical

OrganizationOrganization

Instructional Instructional AdequacyAdequacy

Does the lesson: Does the lesson: • have good procedures and activities ? have good procedures and activities ? • amplify important terms & concepts ? amplify important terms & concepts ? • emphasize important sections & content ? emphasize important sections & content ? • promote meaningful user interaction ? promote meaningful user interaction ? • match the learner with the appropriate learning match the learner with the appropriate learning

task ? task ?

Instructional Instructional AdequacyAdequacy

Does the lesson: Does the lesson: • stress retention & transfer ? stress retention & transfer ? • pace base on the learner’s characteristics ? pace base on the learner’s characteristics ? • motivate the learner ? motivate the learner ? • correctly track and record learner performance ? correctly track and record learner performance ? • provide learner control ? provide learner control ?

Cosmetic AdequacyCosmetic Adequacy

Is the lesson:Is the lesson:• appealing in a visual sense? appealing in a visual sense? • using technology effectively ? using technology effectively ? • presenting a consistent and cohesive picture ? presenting a consistent and cohesive picture ? • supporting learning via “special effects ?” supporting learning via “special effects ?” • free from grammatical and mechanical errors ? free from grammatical and mechanical errors ? • effectively communicating objectives, effectively communicating objectives,

outcomes, and procedural information ? outcomes, and procedural information ?

Cosmetic AdequacyCosmetic Adequacy

Is the lesson:Is the lesson:• well structured ? well structured ? • presenting a presenting a

psychologically safe psychologically safe learning learning environment ? environment ?

• supporting the supporting the student’s learning ? student’s learning ?

Program AdequacyProgram Adequacy

Is the lesson:Is the lesson:• free from conceptual loops ? free from conceptual loops ? • free from technical details ? free from technical details ? • performing as intended ?performing as intended ?• processing efficiently ?processing efficiently ?• secure ? secure ? • accurate ? accurate ? • managing resources wisely ? managing resources wisely ?

Curriculum Curriculum AdequacyAdequacy

Does the lesson:Does the lesson:• relate to other lessons & activities ? relate to other lessons & activities ? • use consistent procedures ? use consistent procedures ? • incorporate teacher & learner preferences ?incorporate teacher & learner preferences ?• facilitate the development of other lessons ? facilitate the development of other lessons ?

• become obsolete quickly ?become obsolete quickly ?

Curriculum Curriculum AdequacyAdequacy

Does the lesson:Does the lesson:• match user expectations ? match user expectations ? • augment or supplement the curriculum ? augment or supplement the curriculum ? • permit completion within the allotted time ?permit completion within the allotted time ?• require “special” hardware or additional software ? require “special” hardware or additional software ?

• provide flexibility ? provide flexibility ?

“ “Complete” Software Complete” Software EvaluationsEvaluations

Courseware Evaluation Courseware Evaluation Form:Form:• Background of the UserBackground of the User• Lesson DescriptionLesson Description• Hardware RequirementsHardware Requirements• Evaluator’s Name & Date Evaluator’s Name & Date • Criteria Above-mentionedCriteria Above-mentioned• Summary RecommendationSummary Recommendation

“ “Web” Web” Activity: Activity:

Search for:Search for:• Software ReportsSoftware Reports• PC Clearinghouse Software PC Clearinghouse Software

Directory Directory • Educational Products Educational Products

Information Exchange (EPIE)Information Exchange (EPIE)• Swift’s Educational Software Swift’s Educational Software

Directory Directory

“ “Web” Web” Activity: Activity:

What are the advantages of What are the advantages of commercial software reviews ?commercial software reviews ?

How can these be used ? How can these be used ? What did you find on the What did you find on the

WWW ? WWW ?

The End The End

““That’s all folks.”That’s all folks.”