writing multiple choice questions
DESCRIPTION
Writing Multiple Choice Questions. Amy S. Oxentenko, MD, FACP, FACG, AGAF Mayo Clinic, Rochester. Disclosures. Relevant Relationships: Consulting work for MKSAP 14-17 On the IM-ITE exam committee. Outline/Objectives. Define the test taker level before writing - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/1.jpg)
Writing Multiple Choice Questions
Amy S. Oxentenko, MD, FACP, FACG, AGAFMayo Clinic, Rochester
![Page 2: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/2.jpg)
• Relevant Relationships:
Consulting work for MKSAP 14-17
On the IM-ITE exam committee
Disclosures
![Page 3: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/3.jpg)
Outline/Objectives
• Define the test taker level before writing
• List components of a multiple-choice question Objective, stem, lead line, options Annotated description of answers
• Evaluate your questions
![Page 4: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/4.jpg)
Who Are You Writing For?
• Med school GI pathophysiology course? More details; physiology on USMLEs
• Resident end-of-rotation exam? Presentation, testing, management
• GI Fellow board review course? More advanced understanding of above
• IM or GI CME course? Very general vs very specific
![Page 5: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/5.jpg)
What are the Stakes of the Questions?
• Board certification?
• Passing a class/rotation?
• Simple knowledge assessment?
• Solely for interaction?
![Page 6: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/6.jpg)
The Stakes
ReproducibleAccurate
Learning
![Page 7: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/7.jpg)
What Should Be Tested?
• For a clerkship/CME course Match goals/objectives
• For a standardized exam Match the targeted learner level
![Page 8: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/8.jpg)
How Often Will Learner Encounter Information
• Frequent More relevant to test often Atypical manifestations
• Infrequent Test the “can’t miss” things Life threatening Typical manifestations
![Page 9: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/9.jpg)
Types of Questions
• Single best answer With clinical scenario
• Factoid questions• All of the following EXCEPT• Matching• True/False• A and B, A and C, all of the above
![Page 10: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/10.jpg)
What are the Components of a Multiple Choice Question (MCQ)?
![Page 11: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/11.jpg)
STEM
OPTIONS
LEADLINE
Question Components
![Page 12: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/12.jpg)
EXPLANATION
OBJECTIVEANSWER
Question Components
![Page 13: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/13.jpg)
PLEASE TAKE THE PRETEST.
![Page 14: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/14.jpg)
Steps to Writing the MCQ
1. Determine the objective
2. Create the lead line and options
3. Draft the stem
4. Write the explanation, if required
![Page 15: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/15.jpg)
The Objective
![Page 16: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/16.jpg)
Example: Objective
• Recall alcoholic hepatitis
What is wrong with this?
![Page 17: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/17.jpg)
Example: Objective
• Treat a patient with alcoholic hepatitis with corticosteroids in the absence of contraindications
More focused
![Page 18: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/18.jpg)
The Objective
• Succinct sentence• Specifies a clinical skill to be learned• Uses action verbs to delineate a goal
Recognize, diagnose, treat, manage• Avoid vague action verbs
Remember, recall, know• Allows you to match curriculum
![Page 19: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/19.jpg)
VERB+
TOPIC+
MODIFIER
Objective Format
![Page 20: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/20.jpg)
Action Verbs for Objectives
![Page 21: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/21.jpg)
Examples of Objectives:The Good, the Bad, the Ugly
• Good: Diagnose niacin deficiency in a patient
with carcinoid syndrome• Bad:
Recall features of niacin deficiency• Ugly:
Understand niacin deficiency
![Page 22: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/22.jpg)
Other Objective Examples
• Select appropriate testing needed before beginning a biologic agent on an IBD patient.
• Evaluate for splenic vein thrombosis in a patient with isolated gastric varices.
• Manage a patient with celiac disease who has persistent symptoms on a GFD.
• Identify peptic ulcers with stigmata for rebleeding that require endoscopic therapy.
![Page 23: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/23.jpg)
Lead Line
![Page 24: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/24.jpg)
Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic,
with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen.
The next step would be to administerA. NiacinB. CT scan of the headC. ThiamineD. InsulinE. Dismissal to home
What iswrong
with this?
![Page 25: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/25.jpg)
Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic,
with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen.
What is the next best step in management?A. NiacinB. CT scan of the headC. ThiamineD. InsulinE. Dismissal to home
Keep lead line general; avoid syntax errors
![Page 26: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/26.jpg)
Lead Line
• Most commonly used lead lines: Which of the following is the most likely diagnosis? What is the next best step in management?
Half treatment options, half diagnostic options
• Avoid negative lead lines: All of the following EXCEPT Which of the following is LEAST likely Which would you NOT do
![Page 27: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/27.jpg)
Lead Line• Keep the lead line generic
“Next step” or “management” Allows diagnostic & therapeutic options
• Avoid writing syntax or grammatical errors that help exclude answers
• “Cover the options” rule Should be able to guess the answers
![Page 28: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/28.jpg)
The Options
![Page 29: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/29.jpg)
Example #1: Options
What is the next best step in management?A. Add IV metronidazoleB. Continue po vancomycinC. Start cholestyramineD. Begin rifaximinE. Stop the po vancomycin
What is wrong
with this?
![Page 30: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/30.jpg)
Example #1: Options
What is the next best step in management?A. Add IV metronidazoleB. Continue po vancomycinC. Start cholestyramineD. Begin rifaximinE. Stop the po vancomycin
Mutually exclusive
![Page 31: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/31.jpg)
Example #2: Options
What is the next best step in management?
A. Metronidazole 500 mg po TID x 14 daysB. Vancomycin 250 mg IV QID x 10 daysC. Metronidazole 500 mg IV QID x 14 daysD. Vancomycin 125 mg po QID x 14 days
What is wrong with this?
![Page 32: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/32.jpg)
Example #2: Options
What is the next best step in management?
A. Metronidazole 500 mg po TID x 14 daysB. Vancomycin 250 mg IV QID x 10 daysC. Metronidazole 500 mg IV QID x 14 daysD. Vancomycin 125 mg po QID x 14 days
Testing too many concepts!Drugs, dose, route, duration
![Page 33: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/33.jpg)
Example #3: OptionsThe cause of diarrhea in most patients with Zollinger-Ellison syndrome is:
A. Antacid useB. High glucagon levelsC. Fat malabsorption from pancreatic lipase
inactivation by acid productionD. Bacterial overgrowth What is wrong
with this?
![Page 34: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/34.jpg)
Example #3: OptionsThe cause of diarrhea in most patients with Zollinger-Ellison syndrome is:
A. Antacid useB. High glucagon levelsC. Fat malabsorption from pancreatic lipase
inactivation by acid productionD. Bacterial overgrowth Correct answer
longer, detailed
![Page 35: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/35.jpg)
The Options
• 4 or 5 options (less may be better)• One is not right and the rest 100% wrong
C E A B DWrong Right
![Page 36: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/36.jpg)
The Options
• 4 or 5 options (less may be better)• One is not right and the rest 100% wrong• One is correct, the rest plausible
C E A B DLeast correct Most correct
![Page 37: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/37.jpg)
Options
• Avoid mutually exclusive options Increases/decreases/same Always/never; stop/continue
• Each option should test one concept Drug OR dose OR route OR duration
• Keep options brief; similar in length• Options should be homogeneous
Length, complexity
![Page 38: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/38.jpg)
Stem
![Page 39: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/39.jpg)
The Stem
• Chronologic order of presentation Age, gender (avoid race unless needed) Site of visit (ER, clinic, hospital) Chief complaint (add features) PMH/Meds/FH/SH (relevant or distracter) Vitals/Exam/Labs/Tests (pertinent)
![Page 40: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/40.jpg)
The Stem
• Fine to have details to make the other options attractive; avoid extraneous info
• It takes time to read; make sure it is fitting in with time to administer test
• Organize labs if many
![Page 41: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/41.jpg)
The Explanation
![Page 42: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/42.jpg)
The Explanation
• Explain why right answer correct first• Explain why wrong answers incorrect
– Do this chronologically
• Example for explanation order:– If correct answer is B– B, A, C, D, E
![Page 43: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/43.jpg)
How Do You Analyze Your Questions?
![Page 44: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/44.jpg)
Question Performance
• Goal for correct answer 50-70% 40-90% may be allowed at times Difficulty score of > 90% (too easy) Difficulty score of < 40% (too hard)
• See if it discriminates: Top test takers Bottom test takers
![Page 45: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/45.jpg)
Performs well; difficulty ok and discriminating
(A) (B) (C) (D)
Answer *
Top 50% 7 80 9 4
Bottom 50% 19 44 28 9
Total 13 62 18 7
Option 0.62
![Page 46: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/46.jpg)
Performs poor; difficulty ok, non-discriminating
Option (A) (B) (C) (D)
Answer *
Top 50% 62 0 7 31
Bottom 50% 80 0 7 13
Total 71 0 7 22
Difficulty 0.71
![Page 47: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/47.jpg)
Performs poor; difficulty low (hard), non-discriminating
Option (A) (B) (C) (D)
Answer *
Top 50% 3 45 2 50
Bottom 50% 6 51 5 38
Total 5 48 3 44
Difficulty 0.44
![Page 48: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/48.jpg)
http://www.nbme.org/publications/item-writing-manual-download.html
![Page 49: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/49.jpg)
![Page 50: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/50.jpg)
The Pretest: How Did You Do?
![Page 51: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/51.jpg)
Avoid Duplicating Terms
1. The primary role of the denter is to move the a) ormeb) lattenc) denter pind) stagle
![Page 52: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/52.jpg)
Avoid Repeat Digits, Look at Pattern
2. How many steps are in an average carrum?a) 3.7b) 370c) 240d) 37
![Page 53: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/53.jpg)
Avoid Absolute Terms:Never, Always, Every, All, None
3. The grotto is frequently humid becausea) every ictal is fabliterousb) langers are always ganteringc) reslins are often dewinestd) culers are never liblis
![Page 54: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/54.jpg)
Avoid Inhomogeneous Options
4. Rigniter is aa) form of glatnickb) prestigious subset of the family of honorary gloutninsc) signal by a sellantd) crown of slessings
![Page 55: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/55.jpg)
Avoid Repeating the Correct Option More Than Others
5. Which of the following pairs contributed the Sterlik track?
a) Smith & Johnsonb) Jones & Smithc) Johnson & Thomsond) Smith & Davis
![Page 56: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/56.jpg)
Avoid Syntax or Grammatical Errors
6. The tarnon is proceeded by aa) oflerb) arglingc) isterind) glantern
![Page 57: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/57.jpg)
Questions?
![Page 58: Writing Multiple Choice Questions](https://reader035.vdocuments.net/reader035/viewer/2022081502/5681660b550346895dd944fa/html5/thumbnails/58.jpg)
Summary
• Know your learner level
• Writing order: objective, lead line, options, stem
• Avoid errors in MCQ writing
• Evaluate the effectiveness of your MCQs