writing strong multiple-choice items alan carter, it instructor [email protected] 1
TRANSCRIPT
Agenda 9:00 – 9:45 Welcome
Presentation on Writing StrongMultiple-Choice Exam Items
9:45 – 10:30 Break, Participants Write 3 Items10:30 – 11:00 Meet as a Group
Introduce Multiple-Choice Item Rubric11:00 – 11:30 Break, Participants Analyze Their
Own Items, Groups Analyze EachOthers’ Items Against the Rubric
11:30 – 11:45 Meet as a Group to Discuss Reflection/Reactions/Results, Q & A, Wrap-Up
2
In the Beginning, There Was Assessment...
What Is Assessment? According to Stephen Brookfield, “assessment is a value-free ascertainment of the extent to which objectives determined at the outset of a program have been attained by participants. [It] requires no value judgment as to their worthwhileness. It is simply a nonjudgmental checking as to whether or not certain purposes have been attained.”
3
Why Do We Assess Student Learning?
To determine whether students have met the learning outcomes specified in the CAR
For grading purposes
For accreditation purposes
4
Exams: A Common Assessment Technique
Exams are often used to assess student learning
Discussion Question:What general types of questions or elementsdo you use in your exams (multiple-choice,perform a hands-on task, etc.)?
5
Focus of This Session: Writing Strong Multiple-Choice Test Items
Multiple-choice items are test items in which students select the correct answer from a list of several choices. (Haladyna, 1994)
Multiple-choice items can be used to “measure knowledge and complex mental acts, such as reasoning, critical thinking, and problem solving” (Haladyna, 1994).
6
Components of a Multiple-Choice Item
What is H2O?
A. FireB. WaterC. AirD. Earth
Stem
Correct Answer
Distractors
7
Components of a Complex Multiple-Choice Item
Jack, who is 48, took his 78-year-old mother to see a movie. General admission tickets cost $10, and the cost of senior admission is $8.How much did Jack spend for movie tickets?
A. $ 10B. $ 16C. $ 18D. $ 20
StemCorrect Answer
Distractors
Scenario
8
What Is a Strong Multiple-Choice Item?
Measures the learning outcomes for the course
Follows the best practices for writing multiple- choice items
Assesses student learning at the appropriate level of Bloom’s taxonomy (which we will discuss later in this session)
9
Learning Outcomes The course content learning outcomes in the Course Adoption Revision document (CAR) should drive assessment
For example, if an outcome states: “Student identifies the functions of the four classes of biomolecules” then the exam items relating to this outcome should require students to identify the functions, as opposed to analyzing or evaluating the functions
10
Best Practices for Constructing Multiple-Choice Items
There should be only one correct answer
Avoid “trick” questions
The wording in the stem should not telegraph the correct answer
Avoid negatives (no, not, etc.) in the scenario and stem. If they cannot be avoided, consider calling attention to the word by using bold and/or italic font
11
Best Practices for Constructing Multiple-Choice Items
All distractors should be plausible answers
Distractors should be of parallel construction and length
Avoid “all of the above,” “none of the above,” and “A and B, but not C”
Avoid testing trivial material
Items should focus on a single problem, concept, or idea
12
Best Practices for Constructing Multiple-Choice Items
Balance the answer key: If there are four possible answers, approximately 25% of the correct answers should be A, 25% should be B, and so on
Items should not provide the answer for other items on the exam
When possible, the stem should be a question, and end with a question mark
Avoid textbook verbatim phrasing unless asking a knowledge-based question
13
You Be the Judge
Which sport involves players shooting a ball through a hoop?
A. SoccerB. CricketC. Ping pongD. Basketball
Which best practice of item writing does this item violate?
14
You Be the Judge
What is a paraplegic?
A. An animalB. A fishC. A person suffering from paralysis of both lower limbsD. A plant
Which best practice of item writing does this item violate?
15
You Be the Judge
The textbook, Operations Management, was written in:
A. 2000B. 2002C. 2004D. 2006
Which best practices of item writing does this item violate?
16
Introduction to Bloom’s Taxonomy
Benjamin Bloom, an educational psychologist, proposed this classification of learning levels in 1956
Breaks cognitive thinking into six categories
These categories can be thought of as degrees of difficulty
17
Levels of Bloom’s Taxonomy
Category/Level Description
Evaluation Make judgments
Synthesis Put parts together to form a whole
Analysis Distinguishes between facts and inferences
Application Use a concept in a new situation
Comprehension Understand the meaning
Knowledge Recall data or information
18Clark, D. (2001)
Bloom’s Examples and Key Words Knowledge Examples: Recite a policy. Quote prices from memory
to a customer. List the safety rules.Key Words: defines, describes, identifies, knows, labels, lists, matches, names, outlines, recalls, recognizes, reproduces, selects, states.
Comprehension Examples: Explain in your own words the steps for performing a complex task. Translate an equation into a computer spreadsheet.Key Words: comprehends, converts, defends, distinguishes, estimates, explains, extends, generalizes, gives examples, infers, interprets, paraphrases, predicts, rewrites, summarizes, translates.
19Clark, D. (2001)
Bloom’s Examples and Key Words Application Examples: Use a manual to calculate an employee’s
vacation time. Apply laws of statistics to evaluate the reliability of a written test. Key Words: applies, changes, computes, constructs, demonstrates, discovers, manipulates, modifies, operates, predicts, prepares, produces, relates, shows, solves, uses.
Analysis Examples: Troubleshoot a piece of equipment by using logical deduction. Recognize logical fallacies in reasoning. Gather information from a department and select the required tasks for training. Key Words: analyzes, breaks down, compares, contrasts, diagrams, deconstructs, differentiates, discriminates, distinguishes, identifies, illustrates, infers, outlines, relates, selects, separates.
20Clark, D. (2001)
Bloom’s Examples and Key Words
Synthesis Examples: Write a company operations or process manual. Design a machine to perform a specific task. Revise a process to improve the outcome.Key Words: categorizes, combines, compiles, composes, creates, devises, designs, explains, generates, modifies, organizes, plans, rearranges, reconstructs, relates, reorganizes, revises, rewrites, summarizes, tells, writes.
Evaluation Examples: Select the most effective solution. Hire the most qualified candidate. Explain and justify a new budget.Key Words: appraises, compares, concludes, contrasts, criticizes, critiques, defends, describes, discriminates, evaluates, explains, interprets, justifies, relates, summarizes, supports.
21Clark, D. (2001)
Bloom’s Level?
The Social Security Act of 1935 provides:
A. Emergency relief to farmersB. Funding for small businessesC. Financial assistance to retired peopleD. A guaranteed minimum wage for workers
22
Bloom’s Level?
Your car suddenly stops running. When you attempt to restart the car by turning the key in the ignition to the start position, nothing happens.
What is the most likely cause of the problem?
A. The gas tank is emptyB. The spark plugs are fouledC. The battery is deadD. The fuel injectors are clogged
23
What About Test Banks?
What is your experience with test banks? Do you use them? Do you like the items?
Test bank items often violate many of the best practices for writing strong multiple-choice items
You may find that you need to edit or revise test bank questions
24
How Many Questions Per Exam?
Depends on: Complexity of exam itemsLength of exam itemsAmount of analysis requiredAmount of time available for exam
Consider taking the exam yourself, and then allowing students 2–3 times as long as it took you to complete the exam
25
Assignment: Item Writing Take a short break!
Write three multiple-choice items:Create a scenario for each item, or one scenario that is used for multiple itemsTry to target the Application or Analysis level of learningUse best practices for item writing
Reconvene at 10:30 a.m.
26
Item Writing Debrief
Was everyone able to write three items?
Did you find it difficult to write items that followed the best practices?
Did you find it difficult to write items that targeted the higher levels of learning?
27
Using Multiple-Choice Items to Assess Critical Thinking
Critical thinking is one of the college’s campus-wide learning outcomes:
“Critical thinking finds expression in all disciplines and everyday life. It is characterized by an ability to reflect upon thinking patterns, including the role of emotions on thoughts, and to rigorously assess the quality of thought through its work products. Critical thinkers routinely evaluate thinking processes and alter them, as necessary, to facilitate an improvement in their thinking and potentially foster certain dispositions or intellectual traits over time.”(Green River Community College, 2005, Critical Thinking Community Rubric)
28
Using Multiple-Choice Items to Assess Critical Thinking
You can use multiple-choice items to assess a student’s ability to think critically about a topic
Items that assess critical thinking will fall in the Application, Analysis, Synthesis, or Evaluation level of Bloom’s taxonomy
29
Must We Always Assess Critical Thinking?
In general, not all multiple-choice items should assess critical thinking, because not all course learning outcomes are at the Application, Analysis, Synthesis, or Evaluation levels
For example, if a learning outcome says the student will comprehend a topic, an item at the Comprehension level would be appropriate
This item should not measure critical thinking
30
A Rubric for Assessing How Well Multiple-Choice Items Measure Critical Thinking
Competency Poor/No Critical Thinking Required
Adequate/Easy Good/Intermediate
Excellent/Difficult
2.1 Apply relevant criteria and standards when evaluating information, claims, and arguments
The item has only 1 discrete fact or statement that must be evaluated when determining how to answer the question
The item has two discrete facts or statements that must be evaluated when determining how to answer the question
The item has three discrete facts or statements that must be evaluated when determining how to answer the question
The item has four or more discrete facts or statements that must be evaluated when determining how to answer the question
31
A Rubric for Assessing How Well Multiple-Choice Items Measure Critical Thinking
Competency Poor/No Critical Thinking Required
Adequate/Easy Good/Intermediate
Excellent/Difficult
2.2 Use appropriate reasoning to evaluate problems, make decisions, and formulate solutions.
The item:Is knowledge basedHas only one fact or statement for students to analyze
The item:Requires students to analyze multiple facts or statements that are interrelatedPartially or directly telegraphs the correct answer
The item:Requires students to analyze multiple facts or statements that are partially discretePartially telegraphs the correct answer
The item:Requires students to analyze multiple facts or statements that are discreteDoes not telegraph the correct answer
32
You Be the Judge
What is H2O?
A. FireB. WaterC. AirD. Earth
Using the rubric, does this item do a poor, adequate, good, or excellent job of
assessing critical thinking for each competency?
33
You Be the Judge Your car suddenly stops running. When you attempt to restart the car by turning the key in the ignition to the start position, nothing happens.
What is the most likely cause of the problem?
A. The gas tank is emptyB. The spark plugs are fouledC. The battery is deadD. The fuel injectors are clogged
Using the rubric, does this item do a poor, adequate, good, or excellent job of
assessing critical thinking for each competency?34
Assignment: Determining Whether Items Assess Critical Thinking
As we discussed previously, not all multiple- choice items should assess critical thinking
However, in this exercise we will practice determining how well a multiple-choice item assesses critical thinking
35
Assignment: Determining Whether Items Assess Critical Thinking
Form teams of two persons First, individually evaluate the three items you wrote by using the rubric. Rate each item as Poor, Adequate, Good, or Excellent for each competency. Next, exchange items with your teammate and evaluate his or her items using the rubric. Meet with your teammate and discuss any differences in your ratings. If time, take a short break! Reconvene at 11:30 a.m. for discussion and final wrap-up
36
Reflections, Reactions, and Questions
37
Resources for Writing Multiple-Choice Test Items
Bloom, B. S. (1956). Taxonomy of educational objectives, Handbook I: The cognitive domain. New York: David McKay Co. Inc.
Haladyna, T. M. (1994). Developing and validating multiple-choice test items. Hillsdale, NJ: Lawrence Erlbaum Associates.
38
References
Brookfield, S. D. (1986). Understanding and facilitating adult learning: A comprehensive analysis of principles and effective practices. San Francisco: Jossey-Bass.
Clark, D. (2001). Learning domains or Bloom’s taxonomy. Retrieved September 9, 2006, from http://www.nwlink.com/~donclark/hrd/bloom.html
Haladyna, T. M. (1994). Developing and validating multiple-choice test items. Hillsdale, NJ: Lawrence Erlbaum Associates.
39