Designing and Implementing Course Evaluations –Survival Training for Subject Matter Experts

The importance of course evaluationsLinking course evaluations to the learning and performance objectivesThe most appropriate level of evaluation based on time, resources and how the data will be utilized.

Why do an evaluation?Will the class be taught again?Will this class be taught by others?Will more classes be taught on this subject?

Who will see and use the information? (Who cares?)

Who is going to compile the data and how will it be used?

Level 3: Behavior – Are the participants using the Information taught in the class?

Level 4: Results – What is the impact on the organization?

Level 1: Reaction –Did the participants like the training?

Level 2: Learning – Can the participants remember what was taught in the training?

Each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

How well was the training perceivedGives immediate feedbackEasy and economicalAdministered at the end of the classSometimes known as smile sheets Paper or electronic

If they don’t like it – they probably are not learning

A.D.D.I.E.

Analysis Learning needs and gaps

Design Course outline

Develop Course material

Implement Class or CBT made available to learners

Evaluate Feedback

Analyze

Implement Design

Develop

Evaluate

Evaluation SectionBasic course information…

Course title, instructor’s name, date, locationInstructor’s ability to answer questions Depth and complexityDifficulty of exercisesInstructor’s knowledgePace of the trainingUsefulness of visual aidsLength of training session

Comments Section:Provides descriptive data such as…

Topics to spend additional time onWere assessed needs accomplished?Are the participants more confident in their skills?Logistics – pre, during, post Changes for a more effective course?

Level 1 – The Questionnaire

Creating test items might seem easy at first, but a lot of thought must go into test creation to make it effective. Writing effective questions takes time and practice. Whether your goal is to measure knowledge and skills, survey opinions and attitudes, or enhance a learning experience, poorly worded questions can adversely affect the quality of the results. Here are a few tips to keep in mind when you write and review questions:

Goal of Test ItemsTest items should: Measure the objectives taught at the required skill or information level

(match verbs)Focus on important, not trivial, subject area content Contain accurate information (including correct spelling) Be written at an appropriate level of difficulty Be clear and concise and as short as possibleAvoid vague pronouns (e.g. it, they)Avoid information that helps answer other itemsBe bias free

Responses should:Be plausible Be grammatically consistent with the stem Be of approximately the same length Be structurally parallel Avoid conflicting alternativesProvide only one correct answer Avoid “All of the above” and use of “None of the above” with caution.Avoid syllogistic reasoning choices (e.g. “both a and b are correct”) unless absolutely necessary.Incorrect answers called distractors should not be too obvious Distractors should point out errors in thinking along the levels of Bloom's Taxonomy

Stems should: Provide sufficient information Avoid extra information Be grammatically correct Avoid clues that help give away answers Avoid negatives, especially double negatives Avoid unnecessary modifiers, especially absolutes (e.g. always, never, etc.) Avoid second personUse questions whenever possible (What, Who, When, Where, Why and How)

Level 2: Learning – Can the participants remember what was taught in the training?

A test of knowledge or skillDid they retain the information?Used to assess the amount of learning that has occurred. Often uses tests conducted before training (pretest) and after training (post test).Administered at the end of the class or several days or weeks following the class

Paper or electronic Multiple choice, short answer, true-false Simulation, demonstration or observation Case studyRole playsTable top exercisesOr a game!

SurveyMonkey.combecause knowledge is everything

Measures the changes that have occurred in learners' behavior due to the training program. Evaluating at this level attempts to answer the question –Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner?

May require up-front assessment for benchmark

Level 3: Behavior – Are the participants using the Information taught in the class?

Occur several weeks or months after the trainingObservationSurvey of managersAnalysis of systemsMay require up-front assessment for benchmark

Attempts to assess training in terms of business results.• Decrease in number of

complaints• Decrease in queue time• Increase in salesOccur several weeks or months after the trainingDetermines if the time and $$ for the training was a good investment

Level 4: Results – What is the impact on the organization?

ObservationSurvey of managersAnalysis of systemsWill require up-front assessment for benchmark

Here’s just a partial list of several excellent resources to continue your learning journey!

Books and MagazinesEvaluating Training Programs: The Four Levels – Donald L. Kirkpatick and James D. KirkpatrickEvaluating the Impace of Training – Scott B. ParryHandbook of Training Evaluation and Measurement Methods – Jack B. PhillipsEvaluation Basics – Donald McCainEvaluating E- Learning – William HortonThe Accidental Trainer – Nanette MillerThe Instant Trainer – C. Leslie Charles and Chris Clarke-Epstein The Ten Minute Trainer – Sharon L. BowmanTraining for Dummies – Elaine BiechTraining SMEs to Train – ASTD InfoLine

Teaching Others What You Know –Survival Training for Subject Matter Experts

Next session:

Classroom Delivery Skills – by invitation only to those who have participated in Modules 1- 4 and successfully completed the assessments.Friday, June 20, 2008 1:00 p.m. – 5:00 p.m.

Fidelity Bank’s Training Center, I-85 and Clairmont Road, Atlanta

www.astdatlanta.org

CREATE exams in a Web browserDEPLOY your exam with one mouse clickAUTOMATE registration and exam retakesANALYZE detailed reports and statistics

www.exambuilder.com

Special thanks to ExamBuilder for providing the on-line assessment.

About ASTDASTD (American Society for Training & Development) is the world’s largest association dedicated to workplace learning and performance professionals. ASTD’s members come from more than 100 countries and connect locally in nearly 140 U.S. chapters and 25 Global Networks. Members work in thousands of organizations of all sizes, in government, as independent consultants, and suppliers.

The Greater Atlanta Chapter of ASTD is one of the five largest chapters within the US with nearly 550 members. We are a regional association of workplace performance professionals, providing our members with a forum for the exchange of professional development opportunities and resources as they strive to improve workplace learning and performance.

www.astdatlanta.org

About CiscoCisco (Nasdaq: CSCO) is the worldwide leader in networking that transforms how people connect, communicate and collaborate. Information about Cisco can be found at www.cisco.com. For ongoing news, please go to newsroom.cisco.com.

With 2.3 million registered users, Cisco WebEx is the global leader in on-demand applications for collaborative business on the web. WebEx offers a full suite of web collaboration applications for specific business processes including marketing, sales, training, and support. Companies of every size gain strategic advantage by using WebEx to replace high-cost/high-touch interactions with efficient web-touch interactions. WebEx applications work across firewalls and across platforms, making them ideal for both internal and external collaboration. Cisco WebEx provides its services over the WebEx MediaTone Network, a secure global network specifically designed for on-demand applications. Please call toll free 877-509-3239 or visit www.webex.com for more information

Special thanks to WebEx for partnering with the ASTD Atlanta Chapter to make this event possible.

WebEX team members: Brad KingNisha.AhluwaliaLauren FollettJennifer Smith

SillyMonkey LLC is a boutique training company specializing in innovative games and activities. We design custom board games, card games, electronic games, video games and other interactive multimedia training components for instructor-led sessions, as well as, eLearning, mobile learning, social networks and virtual worlds. We offer high quality training services to a an impressive list of global clients that includes manufacturing, health care, retail, software, pharmaceuticals, and financial services. Our team has over 25 years of experience in traditional training and various training methodologies including ADDIE, constructivist, cognitivism and behaviorism. We use mind mapping, memory techniques, improve and storytelling.www.sillymonkeyinternational.com

Brad King, WebEx Bridget Evans, Randstad Christie Cole, RBS LynkDarcie Drymon, GH& I

Latarsha Horne, OrkinDeborah Lucas, Turner BroadcastingDeborah Thomas - SillyMonkeyGarland Watkins, UPS Contractor Harvey Menden, Kimberly-ClarkJeff Collins - Arby's Laurie Gilmore - Orkin

Jon Wakeman, Atlanta Journal Constitution Katherine Brune, SunTrust Bank Kim Beckett, XinnixMarcia Breese, CNNMary Duncan, Deloitte and ToucheMatt Fazio, ConsultantPatsy Newton, Cabot Corp.Stacy Woods, UPS ContractorStephanie Britz, CarmaxTony Foley, Children’s Healthcare of Atlanta

There is a short assessment to check your knowledge. If you took the assessment for Modules One, Two or Three go to Go to: http://testing.exambuilder.com, type in your e-mail address andpassword: astdatlanta

First Time Users: Go to:http://testing.exambuilder.com/main/exams/index.cfm?loginaction=5Follow the directions below to register. You willreceive an e-mail from ExamBuilder within one hour with log-in information.Type in:Your First Name:Your Last Name:Your Email:Instructor’s name: jcollins@arbys.comInstructor’s creator ID: DADQClick on Register.