thalheimer -- evaluation workshop -- chick-fil-a workbook

Will Thalheimer, PhD

Measuring Learning to

Drive Improved Learning Results

Schedule

Fundamentals of Learning EvaluationMorning

Building Learner‐Feedback Questions Afterward

How to Get Helpful Data from Learner SurveysAfternoon

Bridging Gap between Research and Practice

Research Translation

Research

The Decisive Dozenfor Learning Design and Learning Measurement

http://is.gd/ddResearch

1. Content

2. ExposureBaseline

3. Guiding Attention

4. Creating Correct Conceptions

5. Repetition

6. Feedback

7. Variation

Engagement & Understanding

8. Retrieval Practice

9. Context Alignment

10. SpacingRemembering

11. Persuasion

12. PerseveranceApplication

https://WorkLearning.com/catalog/

Quite simply, the BEST book on smile sheet creation and utilization, Period!

Karl M. KappProfessor of Instructional Technology

Bloomsburg University

Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.

Robert O. BrinkerhoffProfessor Emeritus, Western Michigan University

& Director, Brinkerhoff Evaluation Institute

Evidence‐based practice at the master level.

Julie DirksenAuthor of Design For How People Learn

Learning‐Transfer Evaluation Model

Question 1You’ve got a 1‐hour learning program just about developed. Then, senior management says they want a 90‐minute program instead, with 30

minutes added after the first hour. For the last 30 minutes, what’s will be the best use of the time?

A. Add additional relevant content you couldn’t cover before.

B. Review key concepts from the first hour.

C. Ask questions about key concepts from the first hour.

Allen, G. A., Mahler, W. A., & Estes, W. K. (1969). Effects of recall tests on long-term retention of paired associates. Journal of Verbal Learning and Verbal Behavior, 8, 463-470.Anderson, R. C., & Biddle, W. B. (1975). On asking people questions about what they are reading. In G. H. Bower (Ed.) The psychology of learning and motivation: Advances in research and theory (Vol. 9). New York: Academic Press.Bahrick, H. P. (1979). Maintenance of knowledge: Questions about memory we forgot to ask. Journal of Experimental Psychology: General, 108, 296-308.Bahrick, H. P., & Hall, L. K. (1991). Preventive and corrective maintenance of access to knowledge. Applied Cognitive Psychology, 5, 1-18.Bahrick, H. P., & Phelps, E. (1987). Retention of Spanish vocabulary over 8 years. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 344-349.Bahrick, H. P., & Phelps, E. (1988). The maintenance of marginal knowledge. In U. Neisser & E. Winograd (Eds.), Remembering reconsidered: Ecological and traditional approaches to the study of memory (pp. 178-192). Cambridge, UK: Cambridge University Press.Bahrick, H. P., Bahrick, L. E., Bahrick, A. S., & Bahrick, P. E. (1993). Maintenance of foreign language vocabulary and the spacing effect. Psychological Science, 4, 316-321.Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C. C. (1991). Effects of frequent classroom testing. Journal of Educational Research, 85, 89-99.Berger, S. A., Hall, L. K., & Bahrick, H. P. (1999). Stabilizing access to marginal and submarginal knowledge. Journal of Experimental Psychology: Applied, 5, 438-447.Bjork, R. A. (1988). Retrieval practice and the maintenance of knowledge. In M. M. Gruneberg, P. E. Morris, R. N. Sykes (Eds.), Practical Aspects of Memory: Current Research and Issues, Vol. 1., Memory in Everyday Life (pp. 396-401). NY: Wiley.Carrier, M., & Pashler, H. (1992). The influence of retrieval on retention. Memory & Cognition, 20, 633-642.Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671-684.Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58, 438-481.Crowder, R. G. (1976). Principles of learning and memory. Hillsdale, NJ: Erlbaum.Cuddy, L. J., & Jacoby, L. L. (1982). When forgetting helps memory: An analysis of repetition effects. Journal of Verbal Learning and Verbal Behavior, 21, 451-467.Cull, W. L. (2000). Untangling the benefits of multiple study opportunities and repeated testing for cued recall. Applied Cognitive Psychology, 14, 215-235.Cull, W. L., Shaughnessy, J. J., & Zechmeister, E. B. (1996). Expanding understanding of the expanding-pattern-of-retrieval mnemonic toward confidence in applicability. Journal of Experimental Psychology: Applied, 2, 365-378.d’Ydewalle, G. D., Swerts, A., & De Corte, E. (1983). Study time and test performance as a function of test expectations. Contemporary Educational Psychology, 8, 55-67.Dempster, F. N. (1987). Effects of variable encoding and spaced presentations on vocabulary learning. Journal of Educational Psychology, 79, 162-170.Ebbinghaus, H. (1913). Memory: A contribution to experimental psychology, (Translated by H. A. Ruger and C. E. Bussenius). New York: Teachers College, Columbia University. (Also available 1964 and 1987, New York: Dover Publications. Original published in 1885).Foos, P. W., & Clark, M. C. (1983). Learning from text: Effects of input order and expected test. Human Learning, 2, 177-185.Foos, P. W., & Fisher, R. P. (1988). Using tests as learning opportunities. Journal of Educational Psychology, 80, 179-183.Gall, M. D., Ward, B. A., Berliner, D. C., Cahen, L. S., Winne, P. H., Elashoff, J. D., & Stanton, G. C. (1978). Effects of questioning techniques and recitation on student learning. American Educational Research Journal, 15, 175-199.Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, 26, 1-104.Glover, J. A. (1989). The “testing” phenomenon: Not gone but nearly forgotten. Journal of Educational Psychology, 81, 392-399.Hamaker, C. (1986). The effects of adjunct questions on prose learning. Review of Educational Research, 56, 212-242.Hogan, R. M., & Kintsch, W. (1971). Differential effects of study and test trials on long-term recognition and recall. Journal of Verbal Learning and Verbal Behavior, 10, 562-567.Izawa, C. (1992). Test trials contributions to optimization of learning processes: Study/test trials interactions. In A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (Eds.) From Learning Processes to Cognitive Processes: Essays in Honor of William K. Estes (Volume 2, pp. 1-33). Hillsdale, NJ: Erlbaum.Jacoby, L. L. (1978). On interpreting the effects of repetition: Solving a problem versus remembering a solution. Journal of Verbal Learning and Verbal Behavior, 17, 649-667.Jones, H. E. (1923-1924). Experimental studies of college teaching: The effect of examination on permanence of learning. Archives of Psychology, 10, 1-70.Kuo, T. M., & Hirshman, E. (1996). Investigations of the testing effect. American Journal of Psychology, 109, 451-464.Lachman, R., & Laughery, K. R. (1968). Is a test trial a training trial in free recall learning? Journal of Experimental Psychology, 76, 40-50.Landauer, T. K., & Ainslie, K. I. (1975). Exams and use as preservatives of course-acquired knowledge. Journal of Educational Research, 69, 99-104.Landauer, T. K., & Bjork, R. A. (1978). Optimum rehearsal patterns and name learning. In M. M. Gruneberg, P. E. Morris, & R. N. Sykes, (Eds.), Practical Aspects of Memory (pp. 625-632). London: Academic Press.Landauer, T. K., & Eldridge, L. (1967). Effect of tests without feedback and presentation-test interval in paired-associate learning. Journal of Experimental Psychology, 75, 290-298.Meyer, G. (1934). An experimental study of the old and new types of examination: I. The effect of the examination set on memory. Journal of Educational Psychology, 25, 641-661.Meyer, G. (1935). An experimental study of the old and new types of examination: II. Methods of study. Journal of Educational Psychology, 26, 30-40.Modigliani, V. (1976). Effects on a later recall by delaying initial recall. Journal of Experimental Psychology: Human Learning and Memory, 2, 609-622.Modigliani, V. (1978). Effects of initial testing on later retention as a function of the initial retention interval. In M. M. Gruneberg, P. E. Morris, & R. N. Sykes, (Eds.), Practical Aspects of Memory (pp. 652-659). London: Academic Press.Modigliani, V., & Hedges, D. G. (1987). Distributed rehearsals and the primacy effect in single-trial free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 426-436.Modigliani, V., Rea, C. P., & Hedges, D. G. (1988). Within-story recalls and the retention of ideas in prose processing. Canadian Journal of Behavioural Science, 20, 73-84.Morris, P. E., & Fritz, C. O. (2000). The name game: Using retrieval practice to improve the learning of names. Journal of Experimental Psychology, 6, 124-129.Neely, J. H., & Balota, D. A. (1981). Test-expectancy and semantic-organization effects in recall and recognition. Memory & Cognition, 9, 283-300.Nungester, R. J., & Duchastel, P. C. (1982). Testing versus review: Effects on retention. Journal of Educational Psychology, 74, 18-22.Parkin, A. J., Wood, A., & Aldrich, F. K. (1988). Repetition and active listening: The effects of spacing self-assessment questions. British Journal of Psychology, 79, 77-86.Rea, C. P., & Modigliani, V. (1985). The effect of expanded versus massed practice on the retention of multiplication facts and spelling lists. Human Learning, 4, 11-18.

0

10

20

30

40

50

60

The Power of Retrieval Practice

Add Questions

‐‐‐with

Feedback

Butler, A. C., & Roediger III, H. L. (2007). Testing improves long‐term retention in a simulated classroom setting. European Journal of Cognitive Psychology, 19, 514‐527.

Add Questions

‐‐‐NO

FeedbackExtra Review

Question 2Which of the following statements is true about

forgetting?

A. People forget about 90% of what they learn one week after learning it.

B. People forget at a rate discovered by Hermann Ebbinghaus.

C. People forget about HALF of what they learn within 24 hours.

D. All of the above.

E. None of the above.

The Truth:

Forgetting depends on many things!

www.work‐learning.com/catalog.html

Question 3How much are learner‐

feedback questions correlated with learning outcomes?

A. High marks indicate that the training was likely to be VERY SUCCESSFUL in creating learning.

B. High marks indicate that the training was likely to be at least SOMEWHAT SUCCESSFUL in creating learning.

C. High marks on smile sheets tell us VERY LITTLE about the success of our training programs in creating learning.

Data & Analysis

Why is this farmer gathering and analyzing growing rates,

crop yields, infestation rates, soil quality, amount of fertilizer,

rainfall, sunshine, revenue per sales channel,

et cetera?

Data & Analysis

What criteria should he have for the data he’s collecting?

• It should be ACCURATE/VALID

• It should be RELEVANT

• It should be HIGHLY PREDICTIVE

• It should be COST EFFECTIVE

• Most importantly, it should enable his organization to make its most important DECISIONS!

Data & Analysis

How are we doing in collecting data that:

• Is ACCURATE/VALID

• Is RELEVANT

• Is HIGHLY PREDICTIVE

• Is COST EFFECTIVE

• Helps us make our most important DECISIONS!

Learning Professionals

Data & Analysis



• Is RELEVANT





What are our most important decisions?

What about our smile‐sheet (learner‐feedback) data?

What about our quiz and knowledge‐check data?

What about our post‐training on‐the‐job performance data?

LearningIntervention

PerformanceSituation

LearningOutcomes

IndividualResults

OrganizationalResults

On-the-Job Learning

Prompting

The Learning Landscape

On-the-Job Performance

© Copyright Work-Learning Research, Inc.

YouTube: http://is.gd/LearningLandscape

Remembering

Learning

Later on the job, learners remember

what they’ve learned.

The learners get a return on their efforts.

Performance can be prompted through job aids, signage,

intuitive cues, performance

support, management, etc.

Learners can learn on-the-job through trial &

practice, insight learning, help from others, social media, studying on their

own, etc.

Working Memory

Demonstrate the Value

Improve theLearning

Three Reasons to Use Measurement in Learning

# 1

# 2# 3

Measurement

Measurement illuminates

some things.

Measurement doesn’t

illuminate everything.

There is no perfect measurement tool!

“Our measures are not perfect, but they should instead be thought of as approximations.”

Deborah L. Bandalos, Author of the 2018 book:

Measurement Theory and Applications for the Social Sciences

“Metric fixation is in fact often

counterproductive, with costs to individual satisfaction with work,

organizational effectiveness, and economic growth.”

Jerry Muller, In Interview with

Princeton University Press

MeasurementLet us be humble,skeptical,

and wise!!

“The evaluation of training programs in terms of ‘results’ is progressing at a very slow rate.”

Donald Kirkpatrick1960

“For the most part, the benefits of industrial training have been taken on faith. Few demands have been made to evaluate it in a rigorous manner.”

Ronald BurkeResearcher

1969

“With some notable exceptions… relatively little work has been devoted to making evaluation more useful and worthwhile.”

Robert Brinkerhoff1981

“ASTD’s research revealed that the actual practice of evaluation doesn’t often follow the strict recommendations of evaluation literature.”

1990

Only 20% were able to do the learning measurement they wanted to do…

2007

“In every year [from 2005‐2007], more than 90 percent rated measurement as the number one or number two area they would like to improve.”

Josh BersinPrinciple and Founder

Bersin, Deloitte Consulting LLP2008

Dixon, G., & Overton, L (2017). Making an impact: How L&D leaders can demonstrate value. Available at: www.towardsmaturity.org/impact2017

The Kirkpatrick Four-Level Model

Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

The Kirkpatrick-Katzell Four-Level Model

Read about Raymond Katzell’s role:

https://is.gd/originator

Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

Three Biases in the WayWe Measure Level 2 Learning

Remembering

LearningIntervention

PerformanceSituation

LearningOutcomes

IndividualResults

OrganizationalResults

The Learning Landscape

On-the-Job Performance

© Copyright Work-Learning Research, Inc.

Remembering

Learning

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

After LearningDuring Learning

Learning and Forgetting Curves

© Copyright by Work-Learning Research, Inc. (www.work-learning.com)

Less Remembering

More Remembering

Learners

Learners

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15



© Copyright by Work-Learning Research, Inc. (www.worklearning.com)

On-the-JobLearning Curves

On-the-JobForgetting

Curves

LearningCurve

YouTube: http://is.gd/LearningForgettingCurves

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




What does an end-of-course

assessment tell us?

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




If we measure

here?BiasedMetric!!

Understanding

Remembering

Application

DelayedTests

WHEN did you

Measure Learning?

90 %

WHERE did you Measure Learning?

91 %

Aggleton, J. P., & Waskett, L. (1999). The ability of odours to serve as state-dependent cues for real-world memories: Can Viking smells aid the recall of Viking experiences? British Journal of Psychology, 90, 1-7.

Balsam, P. D. (1985). The functions of context in learning and performance. In P. D. Balsam & A. Tomie (Eds.) Context and Learning. Hillsdale, NJ: Erlbaum.Bjork, R. A., & Richardson-Klavehn, A. (1989). On the puzzling relationship between environmental context and human memory. In C. Izawa (Ed.) Current Issues in Cognitive Processes: The Tulane Floweree

Symposium on Cognition (pp. 313-344). Hillsdale, NJ: Erlbaum.Bouton, M. E. (1993). Context, time, and memory retrieval in the interference paradigms of Pavlovian learning. Psychological Bulletin, 114, 80-99.

Bower, G. H., Monteiro, K. P., and Gilligan, S. G. (1978). Emotional mood as context for learning and recall. Journal of Verbal Learning and Verbal Behavior, 17, 573-585.Bransford, J. D., Franks, J. J., Morris, C. D., & Stein, B. S. (1979). Some general constraints on learning and memory research. In L. S. Cermak & F. I. M. Craik (Eds.), Levels of processing in human memory

(pp. 331-354). Hillsdale, NJ: Erlbaum.Cassaday, H. J., Bloomfield, R. E., Hayward, N. (2002). Relaxed conditions can provide memory cues in both undergraduates and primary school children. British Journal of Educational Psychology, 72(4), 531-

547.Cousins, R., & Hanley, J. R. (1996). The effect of environmental context on recall and category clustering scores following relational and individual item processing: A test of the outshining hypothesis.

Memory, 4, 79-90.Dalton, P. (1993). The role of stimulus familiarity in context-dependent recognition. Memory & Cognition, 21, 223-234.

Davies, G. (1986). Context effects in episodic memory: A review. Cahiers de Psychologie Cognitive, 6, 157-174.Dijkstra, K., Kaschak, M. P., & Zwaan, R. A. (2007). Body posture facilitates retrieval of autobiographical memories. Cognition, 102, 139-149.

Dulsky, S. G. (1935). The effect of a change of background on recall and relearning. Journal of Experimental Psychology, 18, 725-740.Eich, E. (1985). Context, memory, and integrated item/context imagery. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 764-770.

Eich, E. (1995). Mood as a mediator of place dependent memory. Journal of Experimental Psychology: General, 124(3), 293-308.Eich, J. E. (1980). The cue dependent nature of state dependent retrieval. Memory and Cognition, 8, 157-173.

Fernandez, A., & Glenberg, A. M. (1985). Changing environmental context does not reliably affect memory. Memory & Cognition, 13, 333-345.Gartman, L. M., & Johnson, N. F. (1972). Massed versus distributed repetition of homographs: A test of the differential encoding hypothesis. Journal of Verbal Learning and Verbal Behavior, 11, 801-808.

Godden, D. R., and Baddeley, A. D. (1975). Context dependency in two natural environments: on land and underwater. British Journal of Psychology, 91, 99-104.Godden, D., & Baddeley, A. (1980). When does context influence recognition memory? British Journal of Psychology, 71, 99-104.

Grant, H. M., Bredahl, L. C., Clay, J., Ferrie, J., Groves, J. E., McDorman, T. A., & Dark, V. J. (1998). Context-dependent memory for meaningful material: Information for students. Applied Cognitive Psychology, 12, 617-623.

Herz, R. S. (1997). The effects of cue distinctiveness on odor-based context-dependent memory. Memory & Cognition, 25(3), 375-380.Jacoby, L. L. (1983). Remembering the data: Analyzing interactive processes in reading. Journal of Verbal Learning and Verbal Behavior, 22, 485-508.

Johnson, A. J., Miles, C. (2008). Chewing gum and context-dependent memory: The independent roles of chewing gum and mint flavour. British Journal of Psychology, 99(2), 293-306.Marian, V., & Kaushanskaya, M. (2007). Language context guides memory content. Psychonomic Bulletin & Review, 14(5), 925-933.

Marian, V., & Neisser, E. (2000). Language-dependent recall of autobiographical memories. Journal of Experimental Psychology: General, 129, 361-368.Mead, K. M. L., & Ball, L. J. (2007). Music tonality and context-dependent recall: The influence of key change and mood mediation. European Journal of Cognitive Psychology, 19(1), 59-79.

Pan, S. (1926). The influence of context upon learning and recall. Journal of Experimental Psychology, 9, 468-491.Parker, A., & Gellatly, A. (1997). Moveable cues: A practical method for reducing context-dependent forgetting. Applied Cognitive Psychology, 11, 163-173.

Prestera, G. E., Clariana, R., & Peck, A. (2005) Memory-Context Effects of Screen Color in Multiple-Choice and Fill-in Tests. Journal of Educational Multimedia and Hypermedia, 14(4), 2005, 415-436.Riccio, D. C., Richardson, R., & Ebner, D. L. (1984). Memory retrieval deficits based upon altered contextual cues: A paradox. Psychological Bulletin, 96, 152-165.

Roediger, H. L., III, & Guynn, M. J. (1996). Retrieval processes. In E. L. Bjork & R. A. Bjork (eds.), Memory (pp. 197-236). San Diego, CA: Academic Press.Russo, R., Ward, G., Geurts, H., & Scheres, A. (1999). When unfamiliarity matters: Changing environmental context between study and test affects recognition memory for unfamiliar stimuli. Journal of

Experimental Psychology: Learning, Memory, and Cognition, 25, 488-499.Schab, F. R. (1990). Odors and remembrances of things past. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 648-655.

Schroers, M., Prigot, J., & Fagen, J. (2007) The effect of a salient odor context on memory retrieval in young infants. Infant Behavior & Development. 30(4), 685-689.Smith, S. M. (1979). Remembering in and out of context. Journal of Experimental Psychology: Human Learning and Memory, 5, 460-471.

Smith, S. M. (1982). Enhancement of recall using multiple environmental contexts during learning. Memory & Cognition, 10, 405-412.Smith, S. M. (1984). A comparison of two techniques for reducing context-dependent forgetting. Memory & Cognition, 12, 477-482.

Smith, S. M. (1985). Background music and context-dependent memory. American Journal of Psychology, 98, 591-603.Smith, S. M. (1988). Environmental context-dependent memory. In G. M. Davies & D. M. Thomson (eds.) Memory in Context: Context in Memory (pp. 13-34), Chichester, UK: Wiley.

Smith, S. M. (1995). Mood is a component of mental context: Comment on Eich (1995). Journal of Experimental Psychology: General, 124(3), 309-310.Smith, S. M., & Rothkopf, E. Z. (1984). Contextual enrichment and distribution of practice in the classroom. Cognition and Instruction, 1, 341-358.

Smith, S. M., & Vela, E. (2001). Environmental context-dependent memory: A review and meta-analysis. Psychonomic Bulletin & Review, 8, 203-220.Smith, S. M., Glenberg, A., and Bjork, R. A. (1978). Environmental context and human memory. Memory and Cognition, 6, 342-353.

Spear, N. E. (1978). The processing of memories: Forgetting and retention. Hillsdale, NJ: Erlbaum.Thompson, L. A., Williams, K. L., L'Esperance, P. R., Cornelius, J. (2001) Context-dependent memory under stressful conditions: The case of skydiving. Human Factors, 43(4), 611-619.

Tulving, E., & Thompson, D. M. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352-373.Vela, E. (1984). Memory as a function of environmental context. Paper presented at the 30th annual meeting of the Southwest Psychological Association, New Orleans, LA.

Weiss, W., & Margolis, G. (1954). The effect of context stimuli on learning and retention. Journal of Experimental Psychology, 48, 318-322.Wright, D. L., & Shea, C. H. (1991). Context dependencies in motor skills. Memory & Cognition, 19, 361-370.

Tested in the learning room or in a different room. Does this matter?

Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

0

5

10

15

20

25

Tested ina

DifferentRoom

Tested inthe

LearningRoom

MeasurementLevels

Level AReal World

Level C Scenario-Based

Decisions

Level DMemorization

Questions

Level B Simulation

Level EAttendance

Level FAffiliation

Certification

Quasi-Certification

“Level C represents the last level of certification that can be considered to assess an ability to perform on the job. Level D represents the first quantum

jump away from fidelity in assessment and should be used with caution.”

Scenario‐Based Question Example

Alena wants to start a firm that helps farmers grow food organically. She has a degree in sustainable agriculture and has worked for a non‐profit organization for seven years doing similar work. She’s developed a marketing plan, a financial plan, and has found several farmers who would pay her if she went out on her own. What should Alena do first—before she tells her boss that she’s quitting to start her own firm?

A. Create a cash flow statement to determine whether her predicted income will support the business through the first year.

B. Form a group of advisors with experience in both small‐business management and agriculture.

C. Determine whether she has enough seed money and start‐up capital to get started.

D. Analyze her values and goals to ensure that the proposed business will support them.

Three Biases in the WayWe Measure Level 2 Learning

1. Measuring only at end of learning

2. Measuring in the learning context

3. Utilizing knowledge questions

How would you rate your learner‐feedback questions?

Terrible

1

Bad

2

Okay

3

Good

4Given, that there is NO perfect

measurement…

Trevor was recently promoted into a new position, the Director of Learning Evaluation for an integrated health network. After a month of talking with key stakeholders and examining the past few years of evaluation results, Trevor gathers his team for strategic planning.

After several weeks of researching options, Trevor’s team decides to improve their evaluations by:

• Reframing all their survey questions to be learner‐centric (stated from learner’s viewpoint).

• Focusing particularly on how likely learners are to recommend the learning to others.

Trevor was recently promoted into a new position, the Director of Learning Evaluation for an integrated health network. After a month of talking with key stakeholders and examining the past few years of evaluation results, Trevor gathers his team for strategic planning.

After several weeks of researching options, Trevor’s team decides to improve their evaluations by:


• Focusing particularly on how likely learners are to recommend the learning to others.

A. NOT AT ALL effective.

B. SOMEWHAT effective.

C. MOSTLY effective.

D. VERY effective.

How would you rate the effectiveness of Trevor’s team’s new measurement approach as an indicator of learning success?

Smile Sheetsto

Learning

r=.09

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997).

A meta-analysis of the relations among training criteria.

Personnel Psychology, 50, 341-357.

Smile Sheetsto

Learning

r=.09

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D.

(2008). A review and meta-analysis of the nomological network of

trainee reactions. Journal of Applied Psychology, 93, 280-295.

1990’s 2000’s

Weak Relationship is below .30 and .09 is VERY WEAK

So…SMILE SHEETS tell us VERY LITTLE about Learning

Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree

54321

Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:

On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):

“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)

Research shows that learnersdon’t always know their own learning…

Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).

Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).

Learners Fail to Properly Use ExamplesRenkl (1997).

Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).

Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).

Trevor was recently promoted into a new position, the Director of Learning Evaluation for an integrated health network.

Measured:

Learner Satisfaction

Reputation of the Learning

Question 3How much are learner‐

feedback questions correlated with learning outcomes?

A. High marks indicate that the training was likely to be VERY SUCCESSFUL in creating learning.

B. High marks indicate that the training was likely to be at least SOMEWHAT SUCCESSFUL in creating learning.

C. High marks on smile sheets tell us VERY LITTLE about the success of our training programs in creating learning.

1. Red‐flagging training programs that are not sufficiently effective.

2. Gathering ideas for ongoing updates and revision of a learning program.

3. Judging strengths and weaknesses of a pilot program to enable revision.

4. Providing instructors with feedback to aid their development.

5. Helping learners reflect on and reinforce what they learned.

6. Helping learners determine what (if anything) they plan to do with their learning.

7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.

8. Upholding the spirit of common courtesy by giving learners a chance for feedback.

9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.

Modified based on work by Robert Brinkerhoff

Reasons for Smile Sheets

Effectiveness

of the

Learning

Reputation

of the

Learning

Supporting Learners

Percent of Companies Using“To Any Extent”

ATD and i4cp

TechnologyClassroom

0 20 40 60 80 100

1.

2.

3.

4.

Classroom – Reaction 81%

5.

Technology – Reaction 52%

Classroom – Learning 50%Technology – Learning 43%

Behavior 25%Behavior 15%

Results 14%Results 10%

ROI 6%ROI 6%

Our Measurement Blind Spots:

1. Meaningless Data on Smile Sheets2. Biased Data on Learning3. Hardly Any Performance Data

Measurement

We are

here!









Other names for smile sheets?

reaction sheets

happy sheets

level 1’s

Ultimate Goal Primary Goals Secondary Goals Tertiary Goals

HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.

A. I am NOT AT ALL ready to use the skills taught.

B. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to put the skills into practice.

C. I need MORE HANDS‐ON EXPERIENCE to be GOOD at using these skills.

D. I am FULLY COMPETENT in using these skills.

E. I am CAPABLE at an EXPERT LEVEL in using these skills.

A Better Smile Sheet Question

0 10 20 30 40 50 60


Unacceptable

Acceptable

Deplorable

Superior

Percentage of Respondents

Unlikely

NOT AT ALL READY

HAVE AWARENESS,NEED MORE GUIDANCE

NEED MOREHANDS ON EXPERIENCE

FULLY COMPETENT

COMPETENTAT EXPERT LEVEL


After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.

A. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.

B. I will have someone available TO COACH OR MENTOR ME in applying the learning.

C. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.

D. I will have JOB AIDS to guide me in applying the learning to real job tasks.

E. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.

F. I will NOT get much direct support, but will rely on my own initiative.

A Question About Follow-Through…

But won’t learners dislike

these new type of

questions?

Percent saying NEW QUESTIONS BETTERthan traditional questions:

80%SmileSheets.com

“The Kirkpatrick framework has a number of theoretical and practical shortcomings.”

“[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders…” (p. 91)

http://is.gd/TrainingResearch2012

Special Thanks:

• Julie Dirksen• Clark Quinn• Roy Pollock• Adam Neaman• Yvon Dalat• Emma Weber• Scott Weersing• Mark Jenkins• Ingrid Guerra‐Lopez• Rob Brinkerhoff• Trudy Mandeville• Mike Rustici

Industry Empathy

Grok Problem

Triggering Event

Generate Quick

Solution

Gather Blind‐Spot Feedback

Improve Solution

Reality‐Test

Solution

A

D

D

I

E

Assessment

Design

Development

Implementation

Evaluation


Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

What Messages Does the Four‐Level Model Send?

Messaging of Four-Level Model of Learning Evaluation

Harmful Messages (Sent or Missed)

• Does Not Warn Us Against Ineffective Evaluation Practices

• Ignores the Role of Remembering

• Level 2 Learning is Mashed into One Bucket

Beneficial Messages

• Don’t Just Focus on Learning! Focus on Results too!

• Learner Opinions Are Not Most Important

Messaging our Evaluation Model Should Have: A Few Examples

• Just because learners ENGAGE IN LEARNING doesn’t mean they will have learned. Therefore, measuring attendance is an inadequate way of evaluating learning.

• Just because learners PAY ATTENTION doesn’t mean they learned. Measuring attention is an inadequate way of evaluating learning.

• Just because learners ACTIVELY PARTICIPATE in learning doesn’t mean they learned. Measuring participation is inadequate

• Just because learners say they LIKE A LEARNING EVENT doesn’t mean they learned. Therefore, surveying learners on their general satisfaction is an inadequate way of evaluating learning.

• Just because learners REPORT THEY HAVE EXPERIENCED EFFECTIVE LEARNING METHODS doesn’t guarantee they learned. Therefore, surveying learners on their experience with learning methods, must be augmented with objective measures of learning.

• Just because learners CAN RECITE FACTS AND TERMINOLOGY doesn’t mean they know what to do. Therefore, measuring knowledge recitation is an inadequate way of evaluating learning.

• Just because learners COMPREHEND A CONCEPT doesn’t mean they will be able to use that concept in a work situation. Therefore, measuring knowledge retention is an inadequate way of evaluating learning.

• Just because learners DEMONSTRATE COMPETENCY during a learning event doesn’t mean they’ll remember how to use the competency later. Therefore, measuring competency during or soon after a learning event is an inadequate way of evaluating learning.

• There are a NUMBER OF GOALS WE SHOULD HAVE as learning designers, including supporting our learners in building: comprehension, remembering, decision making competence, task competence, and perseverance in applying what they’ve learned to their job or other performance situations.

Understanding‐‐‐‐‐‐‐‐

Remembering


Remembering


Remembering

Work Performance

Performance In Learning

Compared to

Kirkpatrick‐Katzell

Four‐Level Model of Learning Evaluation

Level 1 ‐‐ Reaction

Level 2 ‐‐ Learning

Level 3 ‐‐ Behavior

Level 4 ‐‐ Results

Two Ways to Use LTEM1. Assessing Your Evaluations2. Learning Design & Development:

Working Backward from Your Goals

2018

2018

2019

2020

Two Ways to Use LTEM1. Assessing Your Evaluations2. Learning Design & Development:

Working Backward from Your Goals

1. Sales Increase by 5%

2. Managers Coach Better

3. Simulated Coaching

4. Scenario Questions

5. IF‐THEN Decisions

6. Perf‐Focused Questions


LTEM is not a panacea!

Do your evaluations with wisdom!!

Read the Report:https://is.gd/LTEM999

Data & Analysis



• Is RELEVANT





Questions?

• LTEM Report and Model — https://is.gd/LTEM999

• Katzell’s Contribution — https://is.gd/Katzell

• Updated Smile‐Sheet Questions 2018 — https://is.gd/Questions2018

• A Better Net Promoter Question — https://is.gd/replaceNPS

• Be Careful When Benchmarking — https://is.gd/DoNotBenchmark

• Debate About Kirkpatrick Model —https://is.gd/epicbattle

• Better Responses on Smile Sheets — https://is.gd/betterresponses

Subscribe to my Newsletter: — https://www.worklearning.com/sign‐up/

Measuring Learning to Drive Improved Learning Results


Phone: +1‐617‐718‐0767Email: [email protected]: WorkLearning.comBook: SmileSheets.comTwitter: @WillWorkLearn

https://www.worklearning.com/contact/

Slides available at:www.worklearning.com/

add/will‐for‐cfa87/


Measuring Learning to

Drive Improved Learning Results

Afternoon Session

Schedule

Fundamentals of Learning EvaluationMorning

Building Learner‐Feedback Questions Afterward

How to Get Helpful Data from Learner SurveysAfternoon

Bridging Gap between Research and Practice

Research Translation










“Our measures are not perfect, but they should instead be thought of as approximations.”

Deborah L. Bandalos, Author of the 2018 book:

Measurement Theory and Applications for the Social Sciences

“The Kirkpatrick framework has a number of theoretical and practical shortcomings.”

“[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders…” (p. 91)



Read about Raymond Katzell’s role:

https://is.gd/originator

Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

Measurement

We are

here!

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




If we measure

here?BiasedMetric!!

Understanding

Remembering

Application

DelayedTests

Trevor had been recently promoted into a new position, the Director of Learning Evaluation for an integrated health network. After a month of talking with key stakeholders and examining the past few years of evaluation results, Trevor gathered his team for strategic planning. They then decided to do the following:


• Focusing particularly on how likely learners were to recommend the learning to others.

We were called in as consultants and we… well, we basically told Trevor and his team that they needed to start over, and develop something better.

What was wrong withTrevor’s team’s approach?

Trevor thanks you all for shedding light on his team’s learning‐evaluation approach and he and his team come up with an improved plan.

Instead of just focusing on learner satisfaction ratings and the reputation of the course, they will ask questions at all levels of the four‐level model, in other words asking questions about satisfaction, learning, on‐the‐job performance, and business results to get a fuller picture.

A. NOT AT ALL effective.

B. SOMEWHAT effective.

C. MOSTLY effective.

D. VERY effective.

How would you rate the effectiveness of Trevor’s team’s new measurement approach as an indicator of learning success?

INSTRUCTOR

• Instructor was knowledgeable

• Instructor cared about me as a learner

• Instructor helped me learn

Level 1 ENVIRONMENT

• Classroom setup conducive to learning

• Sufficient breaks were given

CONTENT

• The topics were relevant to my work

• Examples presented helped me learn

LEARNING EFFECTIVENESS

• I learned new knowledge and skills

• The learning design was engagingLevel 2

JOB IMPACT

• I will be able to apply what I learned to my work

• I am motivated to use what I learned

• What percentage of what you learned from this training do you estimate you will directly apply to your job?

Level 3

BUSINESS RESULTS

• The training will improve my job performance

• What percentage of your job improvement over the next month will be attributable to the training?

Level 4

ROI

• This training was a worthwhile investment in my personal development

• The time I spent in the training will provide organizational benefits equal or greater than the costs

Level 5

Asking Learners Questions Gathers ONLY Level 1 Data: https://is.gd/BiggestLies

My Journey in Tryingto Create a Better

Smile Sheet

My “New” Smile Sheet

Specific Concepts

Helping Learners Calibrate

Best Feedback Comes from Comments

What else can we do to ensure learners engage our smile sheets?

1. Ask with urgency and passion

2. Follow‐through with changes

3. Let people know the changes

4. Use better smile‐sheet questions

Better Responses on Smile Sheets — https://is.gd/betterresponses

Smile Sheetsto

Learning

r=.09

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997).

A meta-analysis of the relations among training criteria.

Personnel Psychology, 50, 341-357.

Smile Sheetsto

Learning

r=.09

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D.

(2008). A review and meta-analysis of the nomological network of

trainee reactions. Journal of Applied Psychology, 93, 280-295.

1990’s 2000’s

Weak Relationship is below .30 and .09 is VERY WEAK

So…SMILE SHEETS tell us VERY LITTLE about Learning

Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree

54321

Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:

On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):

“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)

Research shows that learnersdon’t always know their own learning…

Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).

Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).

Learners Fail to Properly Use ExamplesRenkl (1997).

Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).

Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).

Transmogrify

Strongly AgreeAgree

Neither Agree Nor DisagreeDisagree

Strongly Disagree

We Start with Fuzzy

Adjectives

54321

Magically We Turn Adjectives

Into Numbers

We Average

Responses, Losing

More Info

3.9

4.2

3.7

4.0

4.1

4.0

4.2

3.4

We Choose One

Question and Report Results

4.1

Performance-FocusedSmile Sheets


http://is.gd/ddResearch

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET

Primary Goals Secondary Goals Tertiary Goals

LEARNINGEFFECTIVE?

Will the learning be effective in supporting

on-the-job performance?

RESULTS ACTIONABLE?

Will the Smile-Sheet results communicate

with clarity and urgency to guide

action?

LearnersUnderstand?

LearnersRemember?

LearnersMotivated to Apply?

Do Smile Sheet Results Distinguish between Different

Levels of Success?

After-Training Supports in Place?

Learners Smile Sheet Decisions

Accurate? Avoiding GIGO?

Are We Measuring the Things that

Matter?

Are We Using the Smile Sheet

Opportunity to Educate Our

Stakeholders?

LearnersEngaged

Cognitive Supports Effective

Realistic Retrieval

SpacingSit-Action Triggers

Belief in Value of Concepts

Self-Efficacy in Skill Area

InoculatedJob Aids

Supervisors Follow-up

Do learners remember enough to answer the questions, are the questions focused on most

important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?

From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding

numeric averages that discourage a standards-based decision on success and failure?

While smile sheets are not capable on their own to determine effectiveness, we should at

least try to examine the four goals, (1) understanding, (2) remembering, (3)

motivation to apply, (4) after-training support.

We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,

instructional designers.

QUATERNARYGOALS

Traditional Smile Sheets

Learners rate instructors as credible and

engaging?

Learners say that classroom environment

was conducive to learning?

Learners satisfied with experience?

Learners think course was well‐organized?

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET


LEARNINGEFFECTIVE?



RESULTS ACTIONABLE?



action?

LearnersUnderstand?

LearnersRemember?



Levels of Success?





Matter?



Stakeholders?

LearnersEngaged


Realistic Retrieval




InoculatedJob Aids











QUATERNARYGOALS



engaging?





What Most Smile Sheets Provide

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET


LEARNINGEFFECTIVE?



RESULTS ACTIONABLE?



action?

LearnersUnderstand?

LearnersRemember?



Levels of Success?





Matter?



Stakeholders?

LearnersEngaged


Realistic Retrieval




InoculatedJob Aids











Three Key Goals:

1. Are the Questions Well-Designed?

2. Are Learners Making Good Smile Sheet Decisions?

3. Are The Data we’re Getting Clear and Actionable?

Ensuring that we are focused on the science-

of-learning factors that matter!

QUATERNARYGOALS



engaging?








B. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to put the skills into practice.




A Better Smile Sheet Question

0 10 20 30 40 50 60


Unacceptable

Acceptable

Deplorable

Superior

Percentage of Respondents

Unlikely

NOT AT ALL READY

HAVE AWARENESS,NEED MORE GUIDANCE

NEED MOREHANDS ON EXPERIENCE

FULLY COMPETENT

COMPETENTAT EXPERT LEVEL


In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?

A. I will NOT MAKE THIS A PRIORITY when I get back to my day‐to‐day job.

B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day‐to‐day job.

C. I will make this a MODERATE PRIORITY when I get back to my day‐to‐day job.

D. I will make this a HIGH PRIORITY when I get back to my day‐to‐day job.

E. I will make this ONE OF MY HIGHEST PRIORITIES when I get back to my day‐to‐day job.

A Question About Motivation…


After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.

A. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.

B. I will have someone available TO COACH OR MENTOR ME in applying the learning.

C. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.

D. I will have JOB AIDS to guide me in applying the learning to real job tasks.

E. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.

F. I will NOT get much direct support, but will rely on my own initiative.

A Question About Follow-Through…

But won’t learners

dislike these new type of questions?

We’re using a new type of question to get your feedback. Instead of using a scale from 1 to 5 or choices from strongly disagree to strongly agree, we’re presenting questions with more specific answer choices. (This question is an example)

We’d like to know how you feel about these new questions?

They HAVE ABOUT THE SAME LEVEL OF EFFECTIVENESS as the 1‐to‐5 questions or the strongly‐disagree to strongly‐agree questions.A 11%

They are BETTER because they GIVE ME MORE CLARITY about the choices I’m making.B 61%

They are WORSE because they TAKE MORE TIME to think through the answer choices.C 6%

Their BENEFITS (BRINGING MORE CLARITY) OUTWEIGH their DOWNSIDES (TAKING MORE TIME).D 19%

Their BENEFITS (MORE CLARITY) ARE NOT WORTH the extra time required.E 4%

Percentage saying NEW QUESTIONS EQUAL OR BETTER:

90%

Percentage saying NEW QUESTIONS BETTER:

80%

Pilot (20 courses)

Leadership, Technical, Change, PowerPoint, Coaching, Writing, Harassment Awareness, Gender & Masculinity


What aspects of the training made it MOST EFFECTIVE FOR YOU? What should WE DEFINITELY KEEP as part of the training?

Three Open-Ended Questions to End Your Survey

What aspects of the training COULD BE IMPROVED?

Remember, your feedback is critical, especially in providing us with constructive ideas for improvement.

Is there anything else we should have asked about? Is there anything you want to tell us?

Now that you’ve taken the course, how well do you feel you understand the concepts taught in the course?

A. I have some significant CONFUSIONS AND/OR BLIND SPOTS.

B. I have a BASIC FAMILIARITY with the concepts.

C. I have a SOLID UNDERSTANDING of the concepts.

D. I have a COMPREHENSIVE UNDERSTANDING of the concepts.

E. I have an EXPERT‐LEVEL UNDERSTANDING of the concepts.

Standards

Unacceptable

Unlikely

?

Acceptable

?

AcceptableUnacceptable

AcceptableSuperior

Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts

Smile Sheets

Understanding

Remembering

Motivation to Apply

After Supports

Meeting Target Goals?

Job Performance

Organizational Results

Learner Expectations

Other Expectations

Supports:

Management Support?

Workplace Obstacles?

Reinforcement?

Reminders?

Learners able to:

Understand?

Remember?

Make Decisions?

Apply the Learning?

eLearning

Self‐StudyLearning Program

Employee Exit Survey

I am clear about what is expected of me as a result of going through this training.

A. Strongly DisagreeB. DisagreeC. NeutralD.AgreeE. Strongly Agree

Good

Okay

Bad

This training will have a significant impact on: (check all that apply):

A. Increasing quality

B. Increasing productivity

C. Increasing employee satisfaction

D. Decreasing costs

E. Increasing sales

F. Increasing customer satisfaction

G. Decreasing cycle time

H. Decreasing risk

Good

Okay

Bad

How READY are you TO WRITE SCENARIO‐BASED QUESTIONS?

CIRCLE ONE OR MORE ANSWERS AND WRITE YOUR REASONING BELOW

A. I’m STILL NOT SURE WHERE TO BEGIN.

B. I KNOW ENOUGH TO GET STARTED.

C. I CAN TELL A GOOD QUESTION FROM A BAD ONE.

D. I CAN WRITE MY OWN QUESTIONS, but I’d LIKE to get SOME FEEDBACK before using them.

E. I CAN WRITE MY OWN QUESTIONS, and I’m CONFIDENT they will be reasonably WELL DESIGNED.

In your own words, how ready do you feel you are in being able to write scenario‐based questions?

Good

Okay

Bad

What messages does a

traditional smile sheet

send?

What messages does a

performance‐focused smile sheet send?

What Messages Are Being Sent?

What Messages Are Being Sent?

Compared to most webinars (online meetings), how well did the session keep

YOUR attention? Select ONE choice.

‐‐‐

A. I had a HARD TIME STAYING FOCUSED.

B. My attention WANDERED AT A NORMAL LEVEL.

C. My attention RARELY WANDERED.

D. I was very much SPELLBOUND throughout the session.

RESEARCH/EVIDENCE

For each column,

SELECT ONE CHOICE.

HOW MUCH RESEARCH/EVIDENCE

WAS SHARED?

⃝ ZERO OR VERY LITTLE RESEARCH/EVIDENCE

was shared.

⃝ A SMALL AMOUNT OF RESEARCH/EVIDENCE

was shared.

⃝ A SUBSTANTIAL AMOUNT OF RESEARCH/

EVIDENCE was shared.

QUALITY OF RESEARCH/EVIDENCE SHARED?

⃝ I DON’T KNOW THE RESEARCH well enough to

answer this question.

⃝ Too often, the concepts presented SEEMED TO

CONFLICT WITH THE MOST‐TRUSTED RESEARCH.

⃝ The concepts presented WERE GENERALLY

SUPPORTED BY SOLID RESEARCH.

What Message Is Being Sent?

1. There is no perfect measurement system, approach, model, test, quiz, or question!

2. Our measures illuminate some things, but leave others hidden.

3. Most traditional smile sheets provide misleading feedback about learning effectiveness. Therefore, we are getting very poor feedback about the strengths and weaknesses of our learning interventions.

4. Likert‐like scales and numeric scales are problematic in three ways: (a) make it hard for learners to make clear decisions, (b) discourage learners from engaging our evaluations, and (c) fail to create data that is clear and clearly useful.

5. Learners don’t always know their own learning, so we should support their smile‐sheet decision making.

6. We should utilize the science of learning in our learning metrics.

7. Most traditional smile sheets can be significantly improved.

8. Smile Sheets should NOT be our sole source of feedback.

9. Open‐ended questions can provide very rich data.

10. In the past, we have NOT prioritized improving our learning designs and delivery approaches. But we should!

Key Points

John is the Director of Learning Measurement for a large consumer electronics company. He has recently redesigned his company’s e‐learning evaluations to keep up with industry benchmarks.

The new evaluations utilize the Net Promoter Score, which asks learners to rate the likelihood that they’d recommend the elearning course to others.

Is John’s change likely to produce a more effective evaluation?

John is the Director of Learning Measurement for a large consumer electronics company. He has recently redesigned his company’s e‐learning evaluations to keep up with industry benchmarks.

The new evaluations utilize the Net Promoter Score, which asks learners to rate the likelihood that they’d recommend the elearning course to others.

Is John’s change likely to produce a more effective evaluation?

A. Yes, the change will increase effectiveness.

B. No, the change will keep effectiveness about the same.

C. No, the change will actually hurt effectiveness.

REPLACEMENT FOR NET PROMOTER SCORE

If someone asked you about the effectiveness of the learning experience, would you recommend the learning to them? CHOOSE ONE.

A. The learning was TOO INEFFECTIVE to recommend.

B. The learning was INEFFECTIVE ENOUGH THAT I WOULD BE HESITANT to recommend it.

C. The learning was NOT FULLY EFFECTIVE, BUT I would recommend it IF IMPROVEMENTS WERE MADE to the learning.

D. The learning was NOT FULLY EFFECTIVE, BUT I would still recommend it EVEN IF NO CHANGES WERE MADE to the learning.

E. The learning was EFFECTIVE, SO I WOULD RECOMMEND IT.

F. The learning was VERY EFFECTIVE, SO I WOULD HIGHLY RECOMMEND IT.

https://is.gd/replaceNPS

Cynthia was tasked with designing a new assessment to measure learner readiness to troubleshoot electronic circuit failures. She knows it’s complicated but she pushes ahead.

Based on historic data, she and her team determine the 20 most common circuit failures and create a visually‐realistic simulation to enable learners to test themselves on the last day of the classroom training program.

How well have Cynthia and her team avoided the three main biases in measuring learning results?

A. They avoided all three biases.

B. They failed on at least one bias.

Cynthia was tasked with designing a new assessment to measure learner readiness to troubleshoot electronic circuit failures. She knows it’s complicated but she pushes ahead.

Based on historic data, she and her team determine the 20 most common circuit failures and create a visually‐realistic simulation to enable learners to test themselves on the last day of the classroom training program.

How well have Cynthia and her team avoided the three main biases in measuring learning results?

The Three Biases:

• Measuring During Learning (or right after)

• Measuring in Same Context as Learning

• Measuring with Knowledge Questions

Delayed Smile Sheet

Pivot Question:

Have you used what you learned in the workshop to make a significant improvement in your work?

• No, and I doubt that I will use what I learned.

• No, but I probably will use what I learned.

• No, but I have a plan to use what I learned.

• Yes, I have already used what I learned.

Are You Using What You Learned?

Yes!!

What Enabled You to Apply

What You Learned?

No

What is Holding You Back?

Feedback to:ManagementLearnersUs

Delayed Smile Sheet – Feedback for Improvement

What is holding you back from teaching others what you have learned?

• The learning content is not relevant to my colleagues’ work.

• I don’t think the learning content is valid.

• I don’t see how teaching others will benefit me.

• I can’t remember the learning content well enough.

• I’m not very good at teaching others.

• I haven’t had the time.

• I have had higher priorities.

• I have not had the resources.

• The risk of teaching this to others is too high.

• I’m still working to persuade management of the value.

• Other, please specify ____________________________

Delayed Smile Sheet

―――Feedback

toManagement,

Learners,and/or Us.

What are the top 3 to 4 reasons that have enabled you to put the workshop information into practice?

• I remember (or can look up) the workshop’s learning points.

• I believe strongly in the workshop’s learning points.

• I have the authority to make the necessary changes.

• I have decided to lead a change effort to make this happen.

• I have the time to put this into practice.

• I have the resources to put this into practice.

• I will be acknowledged or rewarded if I do this.

• I will be sanctioned or punished if I do NOT do this.

• My management has made this a high priority.

• My management is fully supportive of my efforts.

• My coworkers are fully supportive of my efforts.

• My Team (or our whole unit) is working together on this.

50%

50%75%

50%

Message toManagement

“More management-

driven than self-driven.”

Immediately After Learning Delayed Several Weeks

When would you use both?

Which is most important?

John, who is grateful for your advice on that Net Promoter thing, now seeks you out for more wisdom.

He is especially interested in how to share smile‐sheet results with his CLO and senior staff. He knows they’re very numbers‐oriented so he wants to provide data that will resonate – and data that will provide a valid perspective.

Which is his best strategy? What would you recommend to John?

A. Show them the raw data in a bar chart. Let them draw conclusions without biasing them.

B. Show them the bar chart but add labels with the standards clearly stated.

A. Show them the raw data in a bar chart. Let them draw conclusions without biasing them.

B. Show them the bar chart but add labels with the standards clearly stated.

Version 2

A. I’m NOT AT ALL READY.

B. I’m ALMOST READY BUT I will need more GUIDANCE before I can even begin to get started using the new skills that I learned.

C. I’m READY BUT I will need MORE EXPERIENCE to be good at using the new skills that I learned.

D. I am FULLY COMPETENT in using the new skills that I learned.

E. I am COMPETENT AT AN EXPERT LEVEL in using the new skills that I learned.

Version 1

A. I’m ALMOST READY BUT I will need more GUIDANCE before I can even begin to get started using the new skills that I learned.

B. I’m READY BUT I will need MORE EXPERIENCE to be good at using the new skills that I learned.

C. I am FULLY COMPETENT in using the new skills that I learned.

HOW ABLE ARE YOU to put what you’ve learned into practice in your work? CHOOSE ONE.

Version 4




D. I am FULLY READY to use the new skills that I learned.

Version 3






Version 6




D. I am FULLY CAPABLE in using the new skills that I learned.


Version 5





After training, my manager and I will discusshow I will use the learning on my job.

strongly disagree ‐‐‐‐‐ 1 2 3 4 5 6 7 ‐‐‐‐‐ strongly agree

What’s Wrong?

What plans, if any, do you have for talking with your manager in the next 10 days about how you will use the learning in your work?

A. My manager and I have made plans for at least TWO meetings.B. My manager and I have made plans for at least ONE meeting.C. My manager HAS MENTIONED the idea, but we don’t yet have firm plans.D. I will SEEK OUT my manager and ask for at least one meeting on this.E. We are LIKELY TO DISCUSS my use of the learning as we work together.F. It is DOUBTFUL that we will spend much time discussing my use of the learning.

Better!

I was able to relate each of the learning objectives to the learning I achieved.

Strongly Agree ‐‐‐‐‐ Agree ‐‐‐‐‐ Neutral ‐‐‐‐‐ Disagree ‐‐‐‐‐ Strongly Disagree

What’s Wrong?

The scope of the material was appropriate to meet my needs.

strongly disagree ‐‐‐‐‐ 1 2 3 4 5 6 7 ‐‐‐‐‐ strongly agree

What’s Wrong?

What percentage of new knowledge and skills learned from this training do you estimate you will directly apply to your job?

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

What’s Wrong?

In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?

A. I’m NOT AT ALL ABLE to put the concepts into practice.

B. I have GENERAL AWARENESS of the concepts taught, but I WILL NEED MORE TRAINING / PRACTICE / GUIDANCE / EXPERIENCE TO DO ACTUAL JOB TASKS using the concepts taught.

C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS‐ON EXPERIENCE to be fully competent in using the concepts taught.

D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY‐COMPETENT LEVEL in using the concepts taught.

E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.

What’s Wrong?

What’s Wrong?



B. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to begin to put the skills into practice.





A. My CURRENT ROLE DOES NOT ENABLE me to use what I learned.

B. I am NOT AT ALL ready to use the skills taught.

C. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to begin to put the skills into practice.

D. I need MORE HANDS‐ON EXPERIENCE to be GOOD at using these skills.

E. I am FULLY COMPETENT in using these skills.

F. I am CAPABLE at an EXPERT LEVEL in using these skills.


A. My CURRENT ROLE DOES NOT ENABLE me to use what I learned.

B. I AM STILL UNCLEAR about what to do, and/or why to do it.

C. I NEED MORE GUIDANCE before I know how to use what I learned.

D. I NEED MORE EXPERIENCE to be good at using what I learned.

E. I CAN BE SUCCESSFUL NOW in using what I learned (even without more guidance or experience).

F. I CAN PERFORM NOW AT AN EXPERT LEVEL in using what I learned.

Now that you’ve completed the workshop, HOW ABLE ARE YOU to use PowerPoint to create an effective set of presentation slides? Choose One.

A. I am NOT AT ALL ready to use PowerPoint effectively.

B. I am STILL UNCLEAR about too many aspects of PowerPoint.

C. I WILL NEED ADDITIONAL HELP to create a fully effective set of slides.

D. I can create an EFFECTIVE set of slides, but I WILL NEED MORE EXPERIENCE to be good at this.

E. I can create a FULLY EFFECTIVE set of slides without additional help or experience.

F. I can create an AMAZINGLY EFFECTIVE set of slides, quickly and at an EXPERT LEVEL.

Questions?

• LTEM Report and Model — https://is.gd/LTEM999

• Katzell’s Contribution — https://is.gd/Katzell

• Updated Smile‐Sheet Questions 2018 — https://is.gd/Questions2018

• A Better Net Promoter Question — https://is.gd/replaceNPS

• Be Careful When Benchmarking — https://is.gd/DoNotBenchmark

• Debate About Kirkpatrick Model —https://is.gd/epicbattle

• Better Responses on Smile Sheets — https://is.gd/betterresponses

Subscribe to my Newsletter: — https://www.worklearning.com/sign‐up/

PLEASE COMPLETE TODAY:www.surveymonkey.com/r/CFA‐2018

Measuring Learning to Drive Improved Learning Results


Phone: +1‐617‐718‐0767Email: [email protected]: WorkLearning.comBook: SmileSheets.comTwitter: @WillWorkLearn

https://www.worklearning.com/contact/

Slides available at:www.worklearning.com/

add/will‐for‐cfa87/

PLEASE COMPLETE:

www.surveymonkey.com/r/CFA‐2018

thalheimer -- evaluation workshop -- chick-fil-a workbook

Documents