Online Course Design

EVALUATION Jennifer Freeman

ACADEMIC ■ IMPRESSIONS

• Understand the difference between assessment and evaluation

• Define formative, summative and confirmative evaluation and understand the importance of each

• Explore theories and methods of evaluation• Design a basic revision plan

Evaluation: Session Goals

• Evaluation: measuring the quality and effectiveness of learning materials and activities

• Assessment: measuring students’ learning and achievement of goals and objectives

Evaluation vs. Assessment

• Fix things that are broken• Ensure learning outcomes are being achieved• Discover causes for failures…proactively fix other problems• Discover potential usability/accessibility issues• What works in theory doesn’t always work in practice• Constant maintenance and improvements to content and

strategies• Dynamic nature of online learning

Course Evaluation: Why?

• Instructional materials’ alignment with objectives and the effectiveness of testing instruments

• Quality of instructional materials • Quality of external resources• Effectiveness of instructional strategies• Usability of tools and technology• Effectiveness of teaching skills• Evaluation chart

What Do We Evaluate?

• Why?– Uncover problems early on; fix broken stuff (hopefully

before students find it)– Identify potential usability/accessibility issues– Examine effectiveness and improve functionality– Dynamic nature of online learning

• What? When?– An ongoing process, usually done both during

development and while being taught– Asks the question, “How are we doing?”

Formative Evaluation of Instructional Materials

• Who will use this evaluation information?– Course development team– Instructor

• How? What should be evaluated?– Instructional materials– Instructional strategies– Use of tools and technology

Formative Evaluation of Instructional Materials

• Do learning activities and assessments align with the learning objectives?

• Do learning materials meet quality standards?– Are learning materials error-free?– Are learning materials accessible?– Are learning materials usable?

• Are the technology tools appropriate and working properly?

Formative Evaluation: Questions to Ask

• Course development rubrics• Checklists• Focus group feedback• For consistency and repeatability

– To reduce bias, evaluation should not be done by members of the course development team…you need a “fresh pair of eyes”

– Develop a testing process/protocol/list of questions/evaluation forms– Determine a standardized, set time for testing so it occurs regularly

as a normal part of the process• Help desk error logs• Student FAQ discussion threads• “Extra credit for errors found” idea• Faculty notes jotted down during semester

Formative Evaluation: Gathering data

course isreleased

• Checkpoint #1 – syllabus, outline and first lesson• Checkpoint #2 – half of the course, viewed on multiple

platforms• Checkpoint #3 – Entire course proofread/edited• Checkpoint #4 – Entire course Q/A checked• Checkpoint #5 – Final check (previous errors)

• Student survey after first three lessons• Instructor survey after first three lessons• Examination of help desk error logs

Sample Formative Evaluation Methods

Du

rin

g d

eve

lop

men

tC

ou

rse

is L

ive

• Why?– Examine effectiveness– Improve functionality– Discover causes for failures…fix existing problems– What works in theory doesn’t always work in practice– Constant maintenance and improvements to content and

strategies…the dynamic nature of online learning

• What? When?– Usually done after the completion of each semester– Asks the question, “How did we do?”

Summative Evaluation

• Who will use this evaluation information?– Instructor– Course development team– Administration

• How? What should be evaluated?– effectiveness of instructional materials and strategies– the learning environment – the instructor’s teaching skills– availability and ease of use of tools and technology– instructor satisfaction with the online teaching experience– student satisfaction with the online learning experience

Summative Evaluation

• Did the students succeed? (grades)– Did the learning activities and assessments align with the learning

objectives? – Were assignments and assessment appropriate to the content?– Was time adequate to convey material and complete tasks?

• Level of instructor and student satisfaction (participation and opinion)– Were learning materials easy to use and accessible? What content did

students frequently have problems with? What areas of the course are error-prone?

– Were there any concerns about motivation?– What tools did the instructor or students frequently have problems with?

Should we continue to use chosen tools?• Are program/department needs being met

– accreditations, prerequisites for other courses, competencies– Is the course scaleable?

Summative Evaluation: Questions to Ask

• Student grades• Student surveys• Instructor satisfaction surveys• Learner self-assessments• Pretest/posttest comparisons• Assessment item analysis• Focus group feedback • Help desk error logs• Discussion forum and chat archives

Summative Evaluation: Gathering data

• Why?– Discover long-term effectiveness of the course – Address large-scale changes necessary to the curriculum– Constant maintenance and improvements to technology,

content and strategies…the dynamic nature of online learning

• What? When?– Usually done some time after the completion of each

semester– Asks the question, “How are we doing now?”

Confirmative Evaluation

• Who will use the evaluation information?– Instructor– Administration

• How? What is being evaluated?– Students’ long-term retention of learning,

usefulness to their long-term goals– Long term effectiveness of the course within the

program– LMS and other technology/tools

Confirmative Evaluation

• Are program/department needs being met (accreditations, prerequisites for other courses, competencies)

• Trends in level of student satisfaction– Course valuable/meaningful to long-term goals?

(program/career)

• Is the course scaleable?• Is the course sustainable?• Learning environment, technology, tools still

meeting our needs?

Confirmative Evaluation: Questions to Ask

• Program student surveys

• Departmental administrative opinions

• Faculty peer review of learning materials

• Employer surveys

• Retention data

• Help desk logs

• LMS effectiveness study / survey

Confirmative Evaluation: Gathering data

1. Analysis of problems found– How urgent is it?– How long will it take to fix?

2. Assign each issue a priority score3. Establish a threshold below which a course will be

postponed4. Prioritized list of change requests…when’s the best

time to revise?5. Assign corrections and establish a deadline for

each6. Make note of unaddressed issues

Revision Resources: Sample Formative

Evaluation and Revision Plan

1. Analyze student and faculty surveys; identify themes or trends2. Analyze assessment3. Analyze help desk logs4. Examine course archives5. Compile list of issues (including issues noted during formative

phase that have yet to be addressed)6. Research solutions7. Determine time needed to fix8. Assign priority ratings9. Assign tasks and establish deadlines

Sample Summative Evaluation and Revision Plan

• Ten modules, each includes lecture notes, one or more Powerpoint presentations with audio, a graded listening quiz and a synchronous chat requirement

• Project – “My Favorite Composer” biography assignment submitted as Word doc to instructor

• Midterm – multiple choice exam• Final – multiple choice exam

Revision ActivityMusic Appreciation

• Costs rising at a reasonable rate?• Are server space and maintenance needs being met? Is the

vendor and software in compliance with required standards?• How reliable has the system been? • Have there been any security concerns?• Level of customization possible within the system?• Satisfied with the structure and presentation of courses?• Satisfied with the authoring tools provided?• Satisfied with the tracking capabilities of the system?• Satisfied with the testing engine and/or assessment tools

available in the system?

Common LMS Evaluation Criteria

• Satisfied with the collaboration tools (discussion areas, journaling, help desk, whiteboard) provided through the system?

• Satisfied with the productivity tools (calendar, help files, search engine) provided through the system?

• Is student / faculty / staff documentation or training sufficient?

• How usable do students, faculty and staff find the tools?• What is the vendor’s reputation in the industry?• What is the vendor’s position in the industry?

Common LMS Evaluation Criteria

• The difference between assessment and evaluation

• Definitions of formative, summative and confirmative evaluation and the importance of each

• Methods of evaluation

• Sample evaluation and revision plans

What We’ve Learned

Jennifer Freemanjfreeman@utsystem.edu