Guiding Adult Learning with Assessments
Theresa Dolbert, Compuware & William Heinz, Openskylearning.
Produced by
December 7 & 8, 2006
301
Designing & Developing Online Assessments & Evaluations
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 1Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Guiding Adult Learning with Assessments
“Testing is the most important intervention in learning”
Theresa Dolbert
William Heinz
Important: While you wait, please go to:
http://www.openskylearning.com/Game/GuidingLearning.html
Send us a chat (if you like) with your score on the game. The highest score gets a copy of the authoring tool for creating these games.
Agenda
Compuware Case StudyThe Value of AssessmentsJob Task AnalysisStoryboarding AssessmentsTools Tracking
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 2Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
How We Came To Guided Learning
We were charged with creating a Certification Program.We have a background in integrated testing and training and we knew the value of testing first.We had an audience with diverse backgrounds and needs and with the ability for learning without much assistance.We did not have the resources to create any unnecessary training.These factors led us to draw from external resources and use assessments to guide and track learning.
What are the chief limitations of Computer Based Training as compared with Instructor Led?
Answer with Chat
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 3Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Why can’t people just go learn on their own?
Most people are not good at evaluating their own skill level, and are therefore poor at deciding whether or not training is required.Without help, most individuals that need training have no way of knowing what the requirements are for success.Most people are unable to learn without a structure that continually supports their motivation.Most people are busy, and it can be a bit frightening to take time away from today's problems in order to possibly improve tomorrow.
The Argument for Guidance
One can learn to tile a kitchen, program in Flash, play the recorder, or make sushi from the web.SkillSoft and other vendors offer thousands of courses.Information is available with 2 or 3 searches from Google on virtually any subject.
The future for Learning Professionals lies less in creating training and more in guiding learners through the right learning path.
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 4Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Certification as Guidance
The Compuware Certification Program is one way to guide learning•It adheres to adult learning principles•It accommodates learning styles•It uses employee training time efficiently•It promises to produce a continuous learning program for 5000 employees.
Compuware Certification Goals
Provide a learning compass for Compuware Professional Staff:
• Indicate which skills are most marketable.• Provide a means to assess the minimum
requirements associated with those skills.Help to ensure that Professional Staff are qualified for assignment.Educate all professional staff about Compuware’s products so that even if they are not using those products they can become a passive sales force for the products.
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 5Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
The Compuware Certification Components
External Certifications (e.g. MCSE or BrainBench)Web Based Training (SkillSoft)Web Based AssessmentsInstructor Led TrainingVirtual Performance Based Exercises
Testing Can be Funhttp://www.openskylearning.com/Game/GuidingLearning.html
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 6Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
The Value of Testing
A study from Black and William on formative testing has shown that testing can increase learning more than any other single educational intervention, and formative testing, when applied correctly, increased the learning rate by 70%. [1]Putting the Task First doubles learning speed (Instant Context and Motivation).
•[1]Assessment in Education, Vol. 5, No. 1,March 1998
The 3 Stages To Guided Learning
Determining what people should learn•Job task analysis•Assessments to see what is already known
Providing learning opportunities and incentives Tracking to make sure that the learning occurred
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 7Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
What Are The Costs Of Training With The Wrong Material?
Three types of wrong material•Already known•Not related to success•Aimed at incorrect Mastery Level.
The cost is a workforce that does not want to learn from your training – not to mention the wasted resources invested in developing irrelevant materials.The cost to you is a devaluation of your training department.A Job Task Analysis can help to align your material with Job success.
How To Conduct A Job Task Analysis
Put together a potential skills listGet acceptance from SMEsProduce a survey that gathers data for each skill for•Importance•Frequency•Difficulty•Mastery level required
Analyze data for mean and standard error for each skill.Define a decision criteria for which skills you will test and train and which ones may be omitted.
www.asptrain.com
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 8Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Creating the SurveyJTAAS lets you import from Excel.
•What obstacles prevent Learning Professionals from performing a Job Task Analysis.
Answer in chat.
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 9Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Performance Based Testinghttp://www.openskylearning.com/Production/word2003/
Considerations For Good Assessments
Performance-based testing• Simulate the activity performed on the job
Bloom’s taxonomy• Knowledge: Terms• Comprehension: Concepts• Application: apply • Analysis: discriminate • Synthesis:• Evaluation:
The assessment storyboard spreadsheetEvidence centered design
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 10Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Entering Data in the Test Storyboard• Follow ECD
•Adhere to blooms Taxonomy
•Create 2 equivalent forms simultaneously
•Make it possible for SMEs to design the tests
The Assessment Storyboard Spreadsheet
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 11Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Terminology Questions•SMEs list the terms
•Rank for Importance, Frequency and Difficulty
•Sort by importance and keep top 20.
•Sort by difficulty
•Odd items are form A; Even are Form B
Analysis Questions
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 12Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Testing Tools Used By Compuware
•QA Assessment•http://www.openskylearning.com/QA%20Test%20v2/start.htm
•Captivate•Right click
•http://www.adobe.com/devnet/captivate/articles/rightclick.html
•(New version for Captivate 2)•Autoscroll
•Hatsize (the virtual sandbox)
Low Stakes Test Analysis
Expert Review – Allow comments on each question.Pass Rates - Make sure that pass rates are reasonableP-Values (Probability of a right answer on each question)Point-Biserial
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 13Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Point-Biserial Analysis
Provides a correlation for how individuals do on each item relative to how they do on the whole test.Helps to see items that:•Have more than one factor
•Clever but not practiced•Practiced but not clever
•Are too easy or too hard•Are not good indicators of success (bad items)
How to do a Point-Biserial with Excel•http://www.eddata.com/resources/publications/EDS_Point_Biserial.pdf
Compuware’s Approach to Tracking
OnTrack Online•Classroom Training•External Certifications•Internal Assessments•SkillSoft CBT•Employee and Management View
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 14Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Adult Learning Principles
Need to have a reason for learning something Learn best when the topic is of immediate value Value autonomy
Need to learn experientially Approach learning as problem solving Have a variety of backgrounds Value self-esteem
Knowles, M. The Adult Learner: A Neglected Species
Learner OrientationsDr. Maggie Martinez
TransformingPerformingConformingResistant
Transforming Learners: Use personal strengths, self-motivation, strong emotions, persistence, and abstract or holistic thinking abilities to self-manage their learning.
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 15Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Compuware's Guided Learning Approach
Most Compuware employees are transforming learners.A certification program presents a set of appropriate problems.The program will direct and compel a continuous learning environment.This will put new demands on training.The result will be a more knowledgeable work force.
Which great man made these statements?•“You cannot help men permanently by doing for them what they could and should do for themselves”.
•“You cannot build character and courage by taking away a man's initiative and independence.”
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 16Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Our OpportunityThe web makes it easy to get information.It is easier now to create interactive learning.Our job is to create guidance and motivation for learning.Tracking enables motivation. Tracking cannot occur without some form of testing.Testing needs to be focused on job success and the best tests are the ones where the tasks are similar to what is required on the job. With regard to learning, formative Testing is more effective and less threatening than summative Testing.
•From the perspective of Guided Learning…•What might we have done better?•What did we forget to do?
Answer in chat.
Designing & Developing Online Assessments and Evaluations
December 7 & 8, 2006
Page 17Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning
Links
Materials from Presentation (and more) •http://www.OpenSkyLearning.com•William Heinz (530) 265-3213•[email protected]
Compuware –•www.Compuware.com
Job Task Analysis (JTAAS)–•http://www.asptrain.com