Download - MC Item Writing Workshop
Nursing Workshop 2/12/2013
Educational
TESSERACT
Nursing Workshop 2/12/2013
Multiple Choice Item Writing Workshop
Doug Mauger EdDSchool Of Instructor [email protected]
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Introductions
• Name:
• What topics do you teach?
• How long have you been teaching?
• One thing you know about Item writing
• What do you want to get out of the workshop?
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Put each rider on the top
of a horse
Nursing Workshop 2/12/2013
Put each rider on the top
of a horse
Nursing Workshop 2/12/2013
Put each rider on the top
of a horse
Nursing Workshop 2/12/2013
Put each rider on the top
of a horse
Nursing Workshop 2/12/2013
Put each rider on the top
of a horse
Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
What are your attitudes to creating exams?
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders• Validity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Stakeholders Gathering
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Instrument……Confidence
Nursing Workshop 2/12/2013
There were flaws in 46% of the total questions, with the number of flaws per test ranging from 36% to 65%. In a sample of examinations distributed to first and second year medical students(Downing, 2005)
Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the con- sequences of using flawed test items on achievement ex- aminations in medical education. Advances in Health Science Education 10 (2), 133–143.
Points to Ponder
Nursing Workshop 2/12/2013
Process ValidityActivity
Nursing Workshop 2/12/2013
Validity and Reliability
Nursing Workshop 2/12/2013
Reliable but Not Valid
Nursing Workshop 2/12/2013
NOT Valid and NOT Reliable
Nursing Workshop 2/12/2013
Valid and Reliable
Nursing Workshop 2/12/2013
Validity Content
Predictive
Construct
Consequences
Process
Nursing Workshop 2/12/2013
Reliable Test – Retest
Alternate Form
Inter Rater
Nursing Workshop 2/12/2013
Types of ValidityActivity
Nursing Workshop 2/12/2013
1. Process, Content Validity (inadequate sample)
2. Process Validity (stress)
3. Content Validity (inadequate preparation)
4. Predictive Validity (inconsistent results)
5. Consequence Validity (induced anxiety)
6. Reliability/Process Validity (possible bias)
Nursing Workshop 2/12/2013
7. Process Validity (students not able to provide best performance)
8. Content Validity (different instructor’s test reflects different instruction)
9. Construct Validity (improper form of assessment)
10. Content Validity/Alignment (instruction and assessment do not match)
Nursing Workshop 2/12/2013
Why is Validity and Reliability important to you as a professional?
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability
• Alignment• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Alignment
Content and Assessment
Alignment
Are Critical
For Valid
Assessment
Intended
Curriculum
WHAT we plan
on teaching
Implemented
Curriculum
HOW we teach
the curriculum
Achieved
CurriculumAssessment
Evidence of
Student
Learning
Assessment
Perfect Alignment
Co
mp
lex
ity o
f c
on
ten
t
HowWhat
Outcomes: Intended Implemented Achieved
Assessment
Scenario 1
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 2
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 3
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 4
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 5
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 6
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 7
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Scenario 8
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Assessment
Perfect Alignment
Co
mp
lex
ity o
f c
on
ten
tHowWhat
Nursing Workshop 2/12/2013
How has Alignment
Impacted You?
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Blooms Taxonomy revised
Nursing Workshop 2/12/2013
Sorting skills/outcomes
fromSimple to Complex
Nursing Workshop 2/12/2013
Blooms Taxonomy revised
Nursing Workshop 2/12/2013
Remembering/Understanding Level
Describes the circulation through the heart
Defines the term systole
Nursing Workshop 2/12/2013
Application (AP)theories in providing care to clients (e.g., a
Application Level
Apply the concepts of aging in developing interventions for the elderly
Nursing Workshop 2/12/2013
Blooms Taxonomy revisedCritical Thinking Level
Develop a plan for delivering services to elderly persons who are homebound
Evaluate nursing research based on predetermined criteria
Analyzes the organizational structure of the community health agency and its impact on client services.
Nursing Workshop 2/12/2013
Sorting skills/outcomes
into 3 classifications
Nursing Workshop 2/12/2013
Factual KnowledgeThe basic elements students must know to be acquainted with a discipline or solve problems in it.
Conceptual KnowledgeThe interrelationships among the basic elements within a larger structure that enable them to function together.
Procedural KnowledgeHow to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.
Meta-Cognitive KnowledgeKnowledge of cognition in general as well as awareness and knowledge of one's own cognition
Four Knowledge Dimensions
Nursing Workshop 2/12/2013
Verbs
Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Identify skills/outcomes
By
Knowledge Dimension & Cognitive Process
Nursing Workshop 2/12/2013
Writing a MC ItemWhat is the biggest challenge for you in sorting outcomes?
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Writing a MC Item
Nursing Workshop 2/12/2013
Writing a MC Item
Nursing Workshop 2/12/2013
MC Item Anatomy
Nursing Workshop 2/12/2013
1. Selecting the competency/outcome
2. Construct the stem
3. Write plausible distractors
4. Peer edit
5. Item analysis
Steps to writing a MC Item item
Nursing Workshop 2/12/2013
ATOA and NOTA
Nursing Workshop 2/12/2013
Results showed that when “all-of-the-above” is used as the correct answer, the item is more difficult for all students, despite the literature assumption that it provides a cueing effect to students. Research findings corroborate literature assumptions that high ability students score significantly higher than other ability students in this type of option.
Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learninghttp://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University
Constructing a MC ItemPoints to Ponder
Nursing Workshop 2/12/2013
Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learninghttp://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University
Constructing a MC Item
Nursing Workshop 2/12/2013
2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint submitted to Advances in Health Science Education
The present study examined the effect of NOTA on multiple choice questions in a large classroom setting. Specifically, we manipulated the place- ment of NOTA across versions of the test such that, for each of 20 experimental questions, NOTA was either not present, replaced the key, or re- placed the high, medium, or low frequency dis- tractors respectively. We found that NOTA sig-nificantly increased the difficulty of multiple choice questions only when it was used as the key. We found no effect of NOTA replacing any of the dis- tractors, regardless of their effectiveness. Addi-tionally, we found no significant effect of NOTA on discrimination scores, regardless of its place- ment.
Constructing a MC ItemPoints to Ponder
Nursing Workshop 2/12/2013
Constructing a MC Item
Nursing Workshop 2/12/2013
Correct Answer
Best Answer
Correct or Best
Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Content Free Test
Nursing Workshop 2/12/2013
Varieties of MC Items
1. Single Correct Answer
2. Multiple Response
3. Combined Response
4. Multiple Response
Nursing Workshop 2/12/2013
Writing a MC Item
1. Design each item to measure an important learning outcome
2. Present a single clearly formulated problem in the stem of the item
3. State the stem of the item in simple, clear language
4. State the stem in positive form wherever possible
5. Word the alternatives clearly and concisely
6. Keep the alternatives mutually exclusive
Nursing Workshop 2/12/2013
7. Keep the alternatives homogeneous in content
8. Keep the alternatives free from clues as to which response is correct
9. Avoid the alternatives AOTA and NOTA
10. Use as many functional distractors as are feasible.
11. Include one and only one correct or clearly best answer for each item
Writing a MC Item
Nursing Workshop 2/12/2013
Writing a MC Item
12. Present the answer in each of the alternative positions approximately an equal number of times, in a random order
13. Layout the items in a clear and consistent manner
14. Use proper grammar, punctuation, and spelling
15. Avoid using unnecessarily difficult vocabulary
16. Analyze the effectiveness of each item after each administration of the test.
Writing a MC Item
Nursing Workshop 2/12/2013
Rules Broken in the
Content Free Test
Nursing Workshop 2/12/2013
Writing a MC Item
Item writing……
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint • Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Blueprint or TOS
Nursing Workshop 2/12/2013
Remembering/Understanding Level
Nursing Workshop 2/12/2013
Application (AP)theories in providing care to clients (e.g., a
Application Level
Nursing Workshop 2/12/2013
Blooms Taxonomy revisedCritical Thinking Level
Nursing Workshop 2/12/2013
Blueprint or TOS
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away• Next Workshop
Nursing Workshop 2/12/2013
Writing a MC Item
WhatSo what Now what
Nursing Workshop 2/12/2013
Writing a MC Item
Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
Writing a MC ItemThank You
Nursing Workshop 2/12/2013
References
http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxonomy%20Question%20Stems.pdf
Anderson, Lorin W. & Krathwohl, David R. (2001). A Taxonomy for Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy. New York. Longman Publishing.