Selected-Response Test Items

Multiple-Choice (MC) Multiple-Choice (MC) and and

Multiple-Response (MR) Multiple-Response (MR)

Dr. Belal Hijji, RN, PhDDr. Belal Hijji, RN, PhD

10/11/201010/11/2010

2

Learning OutcomesLearning Outcomes

At the end of this presentation, and due to time limits,At the end of this presentation, and due to time limits,participants will be able to:participants will be able to:• Discuss the advantages and disadvantages of multiple-Discuss the advantages and disadvantages of multiple-

choice items.choice items.• Identify the parts of a multiple-choice item.Identify the parts of a multiple-choice item.• Discuss the challenges that face nursing educators when Discuss the challenges that face nursing educators when

writing multiple-choice items.writing multiple-choice items.• Identify the rules and principles of writing a MC item’s Identify the rules and principles of writing a MC item’s

stem and alternatives. stem and alternatives. • Identify suggestions for writing multiple-response items Identify suggestions for writing multiple-response items

333

Rationale for the Presentation

We know that nursing educators from the United States of

America had problems with constructing multiple-choice

items. In a study by Masters et al. (2001), 2,233 minor and

major violations of item-writing guidelines were identified.

Our question would then be:

What is it like when a nursing educator with limited English language ability embarks on constructing multiple-choice

items?

4

Advantages of Multiple-Choice ItemsAdvantages of Multiple-Choice Items

• Knowledge of facts, specific information, and principlesKnowledge of facts, specific information, and principles• Definitions of termsDefinitions of terms• Understanding of contentUnderstanding of content• Application of concepts in clinical situationsApplication of concepts in clinical situations• Analysis of data and clinical situationsAnalysis of data and clinical situations• Comparison and selection of varied interventionsComparison and selection of varied interventions• Judgments and decisions about actions to take in clinical Judgments and decisions about actions to take in clinical

situationssituations

5

Disadvantages of Multiple-Choice ItemsDisadvantages of Multiple-Choice Items

• These items are difficult to construct at higher cognitive These items are difficult to construct at higher cognitive levels. As such, many of them are written at the lower levels. As such, many of them are written at the lower cognitive levels.cognitive levels.

• Teachers often experience difficulty developing plausible Teachers often experience difficulty developing plausible distractors.distractors.

• It is often difficult to identify only one correct answer for It is often difficult to identify only one correct answer for an item.an item.

• As a result, these items are time-consuming to construct.As a result, these items are time-consuming to construct.• What would further compound these disadvantages is the What would further compound these disadvantages is the

fact that, in our region, multiple-choice items could be fact that, in our region, multiple-choice items could be written by nursing educators whose level of English written by nursing educators whose level of English language proficiency is inadequate. language proficiency is inadequate.

6

Parts of Multiple-Choice ItemParts of Multiple-Choice Item

An early and common sign An early and common sign STEM in form of STEM in form of of pregnancy is: of pregnancy is: incomplete statementincomplete statement

Options or AlternativesOptions or Alternativesa. amenorrheaa. amenorrhea AnswerAnswerb. morning sickness b. morning sickness DistractorDistractorc. spotting c. spotting DistractorDistractord. tenderness of the breasts d. tenderness of the breasts DistractorDistractor

7

What is the Major Challenge for Nursing What is the Major Challenge for Nursing Educators in Palestine and other ME countries?Educators in Palestine and other ME countries?

• Nursing educators should demonstrate their ability to adhere Nursing educators should demonstrate their ability to adhere to the various principles and rules of writing each of the to the various principles and rules of writing each of the above-mentioned parts. This is a critical issue for them to above-mentioned parts. This is a critical issue for them to infer that the scores a test yields are reliable and infer that the scores a test yields are reliable and meaningful, and is a true reflection of students’ knowledge meaningful, and is a true reflection of students’ knowledge and abilities. and abilities.

• We need to look at the parts as one unit; any violation of the We need to look at the parts as one unit; any violation of the said rules in any part may result in a weakness that said rules in any part may result in a weakness that increases the chances for students to guess the right answer. increases the chances for students to guess the right answer.

8

What are the Rules and Principles to Consider What are the Rules and Principles to Consider When Writing MCQWhen Writing MCQ

(Stem)(Stem)

1.1. The stem should present clearly and explicitly the problem The stem should present clearly and explicitly the problem to be solved. Options should be similar. For example:to be solved. Options should be similar. For example:

The causes of cataracts include:The causes of cataracts include:a. aging.*a. aging.*b. arteriosclerosis.b. arteriosclerosis.c. hemorrhage.c. hemorrhage.d. iritis.d. iritis.

A violation of this rule is provided next.A violation of this rule is provided next.*: Correct answer*: Correct answer

9

Cataracts:Cataracts:a. are painful.a. are painful.b. may accompany coronary artery disease.b. may accompany coronary artery disease.c. occur with aging.*c. occur with aging.*d. result in tunnel vision.d. result in tunnel vision.

The stem of the question does not clearly present the problemThe stem of the question does not clearly present the problemassociated with cataracts that the alternatives address. In associated with cataracts that the alternatives address. In

addition,addition,the options are dissimilar.the options are dissimilar.

10

2.2. The stem should be brief including ONLY necessary The stem should be brief including ONLY necessary information. An example showing a violation of this rule information. An example showing a violation of this rule follows.follows.

You are caring for an elderly man who lives alone butYou are caring for an elderly man who lives alone but

has frequent visits from his daughter. He has congestivehas frequent visits from his daughter. He has congestive

heart failure and some shortness of breath. Your patientheart failure and some shortness of breath. Your patient

was told recently that he has cataracts.was told recently that he has cataracts.

The causes of cataracts include:The causes of cataracts include:

a. aging.*a. aging.*

b. arteriosclerosis.b. arteriosclerosis.

c. hemorrhage.c. hemorrhage.

d. iritis.d. iritis.

The background information about the patient is irrelevant The background information about the patient is irrelevant to the problem addressed.to the problem addressed.

11

3.3. Avoid inserting information in the stem for instructional Avoid inserting information in the stem for instructional purposes. The goal of testing is not to teach new Information:purposes. The goal of testing is not to teach new Information:

Cataracts are an opacity of the lens or capsule of the eye leading toCataracts are an opacity of the lens or capsule of the eye leading toblurred and eventual loss of vision. The causes of cataractsblurred and eventual loss of vision. The causes of cataractsinclude:include:a. aging.*a. aging.*b. arteriosclerosis.b. arteriosclerosis.c. hemorrhage.c. hemorrhage.d. iritis.d. iritis.

12

4.4. Avoid the use of negatively stated stems, including words such Avoid the use of negatively stated stems, including words such as “no,” “not,” and “except.”as “no,” “not,” and “except.”

All of the following are realities about health education except? All of the following are realities about health education except?

a.a. It does It does not need special skills need special skills b.b. It depends on theories of learningIt depends on theories of learningc.c. It should be planned in participation with the learnerIt should be planned in participation with the learnerd.d. It is a continuing education that can happen in many settingsIt is a continuing education that can happen in many settings

((From a second exam))

All of them are an example for the Biological dimension except.All of them are an example for the Biological dimension except. a. genetica. genetic b. sexb. sex c. geographyc. geography ((From a final examFrom a final exam))

13

5.5. The stem and alternatives that follow should be consistent The stem and alternatives that follow should be consistent grammatically. If not, clues may be provided as to the grammatically. If not, clues may be provided as to the correct or incorrect responses. A consistent verb form should correct or incorrect responses. A consistent verb form should be used with the alternatives. For example,be used with the alternatives. For example,

Your patient is undergoing a right carotid endarterectomy. WhichYour patient is undergoing a right carotid endarterectomy. Which

pre-operative information would be most important to collect as apre-operative information would be most important to collect as a

baseline for the early recovery period? Her ability to:baseline for the early recovery period? Her ability to:

a. follow movements with her eyesa. follow movements with her eyes

b. move all four extremities*b. move all four extremities*

c. rotating her head from side to sidec. rotating her head from side to side

d. swallow and gagd. swallow and gag

14

6. Avoid ending stems with “a” or “an” because these often provide grammatical clues as to the option to select. It is usually easy to rephrase the stem to eliminate the “a” or “an.” For instance,

Narrowing of the aortic valve in children occurs with an:

a. aortic stenosis.*

b. atrial septal defect.

c. coarctation of the aorta.

d. patent ductus arteriosus.

15

''any physiological or psychological factors necessary for a healthy

existence '' is a.

a. adaptation

b. human needs

c. concepts

Feelings toward a person, object or idea is an.

a. values

b. attitudes

c. ethics

(Both items were taken from a final examination; they suffer several violations)

16

Alternatives (Options)Alternatives (Options)

1.1. The number of words included in each option should be The number of words included in each option should be consistent in length. Frequently the correct answer is the consistent in length. Frequently the correct answer is the longest and this may the testwise student to realize that it is the longest and this may the testwise student to realize that it is the correct answer. For example, correct answer. For example,

You are assessing an emaciated 14-year-old girl. Her motherYou are assessing an emaciated 14-year-old girl. Her mother

describes the following changes: resistance to eating and 20-lb.describes the following changes: resistance to eating and 20-lb.

weight loss over the last 6 weeks. It is most likely that the patientweight loss over the last 6 weeks. It is most likely that the patient

resists eating for which of the following reasons?resists eating for which of the following reasons?

a. Complains of recurring nausea.a. Complains of recurring nausea.

b. Describes herself as “fat all over” and fearful of gaining weight.*b. Describes herself as “fat all over” and fearful of gaining weight.*

c. Has other GI problems.c. Has other GI problems.

d. Seeks her mother’s attention.d. Seeks her mother’s attention.

The correct answer can be shortened to: Is fearful of gaining weight.The correct answer can be shortened to: Is fearful of gaining weight.

17

2.2. The options should have the same number of parts. In the The options should have the same number of parts. In the following example, including two causes in option “a” following example, including two causes in option “a” provides a clue to the answer. Revising that option to only provides a clue to the answer. Revising that option to only “aging” avoids this.“aging” avoids this.

Causes of cataracts include:Causes of cataracts include:

a. aging and steroid therapy.*a. aging and steroid therapy.*

b. arteriosclerosis.b. arteriosclerosis.

c. hemorrhage.c. hemorrhage.

d. iritis.d. iritis.

18

3.3. The alternatives should be consistent grammatically. The answer The alternatives should be consistent grammatically. The answer and distractors should be similar in structure and terminology. and distractors should be similar in structure and terminology. Without this consistency, the test-taker may be clued to the correct Without this consistency, the test-taker may be clued to the correct response or know to eliminate some of the options without being response or know to eliminate some of the options without being familiar with the content. See below how the correct answer familiar with the content. See below how the correct answer differs grammatically from the others:differs grammatically from the others:

You are making a home visit with a new mother who is breastfeeding.You are making a home visit with a new mother who is breastfeeding.

She tells you that her nipples are cracked and painful. Which of theShe tells you that her nipples are cracked and painful. Which of the

following instructions should be given to the mother?following instructions should be given to the mother?

a. Put the entire areola in the baby’s mouth during feeding.*a. Put the entire areola in the baby’s mouth during feeding.*

b. The baby should be fed less frequently until the nipples are healed.b. The baby should be fed less frequently until the nipples are healed.

c. There is less chance of cracking if the nipples are washed daily c. There is less chance of cracking if the nipples are washed daily with soap.with soap.

d. Wiping off the lotion on the nipples before feeding the baby may d. Wiping off the lotion on the nipples before feeding the baby may help.help.

19

4.4. The alternatives should sample the same domain, for instance, all The alternatives should sample the same domain, for instance, all symptoms, all diagnostic tests, all nursing interventions, varying symptoms, all diagnostic tests, all nursing interventions, varying treatments, and so forth.treatments, and so forth.

You are examining a patient who is having difficulty breathing. HisYou are examining a patient who is having difficulty breathing. His

RR is 40, HR 140, and oxygen saturation 90%. He also complains ofRR is 40, HR 140, and oxygen saturation 90%. He also complains of

headache.headache.

Which of the following nursing diagnoses is of greatest priority?Which of the following nursing diagnoses is of greatest priority?

a. Activity intolerancea. Activity intolerance

b. COPDb. COPD

c. Impaired gas exchange*c. Impaired gas exchange*

d. Paind. Pain

20

5.5. Avoid including opposite responses among the options. This is Avoid including opposite responses among the options. This is often a clue to choose between the opposites and not consider the often a clue to choose between the opposites and not consider the others. A sample item follows:others. A sample item follows:

The nurse should determine the correct placement of a nasogastricThe nurse should determine the correct placement of a nasogastric

tube by:tube by:

a. asking the patient to swallow.a. asking the patient to swallow.

b. aspirating gastric fluid from the tube.b. aspirating gastric fluid from the tube.

c. inserting air in the tube and auscultating in the epigastric area.*c. inserting air in the tube and auscultating in the epigastric area.*

d. inserting water in the tube and auscultating in the epigastric area.d. inserting water in the tube and auscultating in the epigastric area.

In this example, the correct response (c) is opposite one of theIn this example, the correct response (c) is opposite one of the

distractors (b), which clues the student to select one of thesedistractors (b), which clues the student to select one of these

alternatives. In addition, options “c” and “d” begin with “inserting,”alternatives. In addition, options “c” and “d” begin with “inserting,”

which may provide a visual clue to choose between them.which may provide a visual clue to choose between them.

21

McDonald (2007) suggested that when two sets of opposites are usedMcDonald (2007) suggested that when two sets of opposites are usedin the alternatives, there is less opportunity for guessing. Using thisin the alternatives, there is less opportunity for guessing. Using thisprinciple, the first distractor in the example could be reworded to principle, the first distractor in the example could be reworded to

formforma second pair of opposites:a second pair of opposites:The nurse should determine the correct placement of a nasogastricThe nurse should determine the correct placement of a nasogastrictube by:tube by:a. aspirating air from the tube.a. aspirating air from the tube.b. aspirating gastric fluid from the tube.b. aspirating gastric fluid from the tube.c. inserting air in the tube and auscultating in the epigastric area.*c. inserting air in the tube and auscultating in the epigastric area.*d. inserting water in the tube and auscultating in the epigastric area.d. inserting water in the tube and auscultating in the epigastric area.

22

6.6. Arrange the options in alphabetical, numerical, or chronological Arrange the options in alphabetical, numerical, or chronological order, so that the position of the correct response is randomly order, so that the position of the correct response is randomly distributed throughout the test.distributed throughout the test.

6.6. Options with numerical values should be listed sequentially, and Options with numerical values should be listed sequentially, and the values should not overlap. When alternatives overlap, a the values should not overlap. When alternatives overlap, a portion of anportion of an option may be correct, or more than one answer option may be correct, or more than one answer may be possible. An example of this problem is:may be possible. An example of this problem is:

The normal range for serum potassium level in adults is:The normal range for serum potassium level in adults is:a. 2.5 – 4.5 mEq/L.a. 2.5 – 4.5 mEq/L.b. 0.5 – 3.5 mEq/L.b. 0.5 – 3.5 mEq/L.c. 3.5 – 5.2 mEq/L.*c. 3.5 – 5.2 mEq/L.*d. 1.5 – 4.5 mEq/L.d. 1.5 – 4.5 mEq/L.To correct this problem, see next slide.To correct this problem, see next slide.

23

The normal range for serum potassium level in adults is:The normal range for serum potassium level in adults is:

• a. 0.5 – 1.5 mEq/L.a. 0.5 – 1.5 mEq/L.

• b. 2.0 – 3.2 mEq/L.b. 2.0 – 3.2 mEq/L.

• c. 3.5 – 5.2 mEq/L.*c. 3.5 – 5.2 mEq/L.*

• d. 8.5 – 10.3 mEq/L.d. 8.5 – 10.3 mEq/L.

2424

Multiple-Response Items

In this format several alternatives may be correct. Students select all of the options that apply by checking the box that precedes each option, as in the following example:The preliminary diagnosis for your patient, a 20-year-old collegestudent, is meningitis. Which signs and symptoms should you anticipate finding? Select all that apply:

❒ 1. Abdominal tenderness ❒ 2. Fever * ❒ 3. Lack of pain with sudden head movements ❒ 4. Nausea and vomiting * ❒ 5. Nuchal rigidity * ❒ 6. Sensitivity to light * ❒ 7. Sudden bruising in neck area

25

The principles for writing multiple-response items are the same as for writing multiple-choice. An additional suggestion for writing these items is:The responses should be listed in a logical order, for instance,alphabetically, for ease in reviewing. Alternatives are easier to review if shorter combinations are listed before longer ones. For example, Causes of cataracts include:1. aging.2. arteriosclerosis.3. hemorrhage.4. iritis.5. steroid therapy.a. 1, 2b. 1, 5*c. 2, 4d. 1, 3, 4e. 2, 3, 5

Where to go from here

• Conduct a research study to assess the level of compliance of Conduct a research study to assess the level of compliance of multiple-choice questions with published rules and guidelines multiple-choice questions with published rules and guidelines of item construction.of item construction.

• Design and implement hands on training for educators in Design and implement hands on training for educators in nursing and other disciplines who construct objective tests of a nursing and other disciplines who construct objective tests of a variety of format. variety of format.

272727

References

• Masters, J. C., Hulsmeyer, B. S., Pike, M. E., Leichty, K., Miller, M. T., & Verst, A. L. (2001). Assessment of multiple-choice questions in selected test banks accompanying text books used in nursing education. Journal of Nursing Education. 40, 25–32.

• McDonald, M. E. (2007). The nurse educator’s guide to assessing learning outcomes. Boston: Jones and Bartlett.

• Oermann, M. & Gaberson, K. (2008). Evaluation and Testing in Nursing Education. 3rd edition. New York: Springer.

28

Thank You Very Much

Any Questions?