setting effective mcqs€¦ · able to differentiate primary and secondary sources of information...

Setting Effective MCQs

Lim Fun Siong

Why do you use MCQs?

Why use MCQs?

1. Simple to complex learning outcomes can be measured

2. Provides highly structured and clear tasks

3. Measures a broad sample of achievements

4. Provides diagnostic information with incorrect alternatives

5. Scoring is easy, objective and reliable

Limitations of MCQs

1. Test construction relatively more time consuming

2. Difficult to find plausible distractors

3. Not appropriate for many demonstrative knowledge/skills

e.g. design, explain, conduct, etc.

4. Score can be influenced by reading ability

Source: https://cambrigags.wordpress.com/cambrigags-specials/

Comparison with other assessment types

Characteristics Selection-Type

Items e.g. MCQ

Supply-Type Items

Short Answer Essay

Measures factual information Yes Yes Wasteful

Measures understanding Yes Limited Yes

Measures synthesis Limited Limited Yes

Easy to construct No Yes Yes

Samples broadly Yes Yes No

Eliminates bluffing Yes No No

Eliminates writing skill Yes No No

Eliminates blind guessing Depends Yes Yes

Easy to score Yes No No

Scoring is objective Yes No No

Pinpoints learning errors Yes Yes No

Encourages originality No No Yes

Constructing MCQsValidity and Reliability

Test scores Reliable but invalid

Test scores Reliable & valid

Test scores Unreliable, invalid

Yellow partrepresents learning goals assessed

Green dots represent test scores of studentsof similar academic abilities in the same test

Validity versus Reliability

10

The test aims to assess attainment

of learning goals

Is the test score distribution reliable, valid?

Another way to look at validity

● Increase validity of MCQs in test design

● Address issues with MCQs such as guess work

Focus of workshop

Test Item

I. Nothing in the content or structure of the item should prevent

an informed student from responding correctly

II. Nothing in the content or structure of the item should enable

an uninformed student to select the correct answer

III. Measures what it is meant to measure

IV. Able to diagnose students’ misunderstanding of the

concepts

What is an effective MCQ question item?

● Readability

● Clarity, and

● Appropriateness of questions and distractors

Factors affecting validity in MCQs

Readability

Readability #1

Which of the following is the best

definition of seismograph?

A. An apparatus of measuring

sound waves

B. An apparatus of measuring heat

waves

C. An apparatus of measuring

ocean waves

D. An apparatus of measuring

earthquake waves

Readability #1




sound waves


waves


ocean waves


earthquake waves

Nothing in the content or

structure of the item

should prevent an

informed student from

responding correctly

Readability #1




sound waves


waves


ocean waves


earthquake waves

What type of waves does a seismograph

measure?

A. Sound

B. Heat

C. Ocean

D. Earthquake

Guideline:

Put as much of the

wording as possible in

the stem of the item

Readability #2

In what year did Singapore gain

independence?

A. 1945

B. 1819

C. 1965

D. 1914

Readability #2


independence?

A. 1945

B. 1819

C. 1965

D. 1914



should prevent an



Readability #2


independence?

A. 1945

B. 1819

C. 1965

D. 1914


independence?

A. 1819

B. 1914

C. 1945

D. 1965

Guideline:

Arrange alternatives in a

logical order

Readability #3

Green plants may lose their colour

when they

A. are forming flowers.

B. grown in the dark.

C. are placed in strong light.

D. temperature drops.

Readability #3


when they







should enable an

uninformed student to

select the correct answer

Readability #3


when they





Green plants may lose their colour when

they are

A. forming flowers.


C. placed in strong light.

D. exposed to temperature drops.

Guideline:

Use consistent, correct

punctuation and grammar

relative to the stem

Clarity

Clarity #1

Louis Pasteur

A. discovered that liquid food that

are treated with heat tend to spoil at a

lower rate

B. taught that there are

microorganisms that are responsible for

food spoilage

C. developed the process of

pasteurisation to preserve liquid food

longer

D. was a French chemist and

microbiologist born in 27 Dec 1822.

Clarity #1

Louis Pasteur



lower rate



food spoilage



longer





should prevent an



and

Measures what it is meant

to measure

Clarity #1

Louis Pasteur



lower rate



food spoilage



longer



Guideline:

Ask a direct question or

set a specific problem

Clarity #1

Which of the following was Louis

Pasteur’s greatest contribution to the

food industry?

A. His discovery of the role of

microorganisms in fermentation.

B. His invention of a method to treat

milk and wine to prevent them from

causing sickness.

C. His discovery for the basis of

asymmetry in crystals.

D. His work and discovery of

immunisation and vaccination.

Better. But can it be even

better?

Clarity #2

Given the present day utilisation of the

automobile in urban settings, which of

the following represents an important

contribution of Garrett A Morgan’s

genius?

A. Car safety belts

B. Crosswalk markers

C. Traffic lights

D. Vulcanised rubber tires

Clarity #2





genius?

A. Car safety belts


C. Traffic lights




should prevent an



Clarity #2





genius?

A. Car safety belts


C. Traffic lights


Which of the following did Garrett A

Morgan invent?

A. Car safety belts


C. Traffic lights


Guideline:

Use simple vocabulary

and sentence structure

Appropriateness

Learning Outcome:

Able to differentiate primary and secondary sources of information

Appropriateness #1

Measures what it is meant

to measure

Guideline:

Each item should

measure an important

learning outcome

Learning Outcome:


Question: Which of the following is the most reliable primary source of

information?

A. An advertisement by BMW on the specifications of their new car

B. A list of testimonies from test drivers of the new BMW car

C. A review of the new BMW car by the local paper

D. A car enthusiast’s video footage of the new BMW car on his/her YouTube

channel

Appropriateness #1

Learning Outcome:


Question: Which of the following is a secondary source of information?

A. An advertisement by BMW on the specifications of their new car

B. A list of testimonies from test drivers of the new BMW car

C. A news report on the release of the new BMW car by the local paper

D. A car enthusiast’s video footage of the new BMW car on his/her YouTube

channel

Appropriateness #1Measures what it is meant

to measure

Appropriateness #2

10 × 2(3 – 9 ÷ 3) = _____.

A. -10

B. 0

C. 10

10 × 2(3 – 9 ÷ 3) = _____.

A. -40

B. 0

C. 30

Appropriateness #2

10 × 2(3 – 9 ÷ 3) = _____.

A. -10

B. 0

C. 10

10 × 2(3 – 9 ÷ 3) = _____.

A. -40

B. 0

C. 30

Able to diagnose students’

misunderstanding of the

concepts

Appropriateness #2

10 × 2(3 – 9 ÷ 3) = _____.

A. -10

B. 0

C. 10

10 × 2(3 – 9 ÷ 3) = _____.

A. -40

B. 0

C. 30

Guideline:

Make the distractors

plausible and attractive to

the uninformed

Question: Advertisement by BMW on the specification of their new

car is a reliable primary source of information

A. Absolutely True

B. Mostly True

C. Maybe True

D. Mostly False

E. Absolutely False

Appropriateness #3

Question: Advertisement by BMW on the specification of their new

car is a reliable primary source of information

A. Absolutely True

B. Mostly True

C. Maybe True

D. Mostly False

E. Absolutely False

Appropriateness #3

Great for facilitating

discussion and class

sharing but does not

help you to diagnose

what students know or

don’t know in a test

setting

Guess Work Part I

Addressing guess work

Common ways to eliminate guess work

● Not using All of the AboveNothing in the content or


should enable an



Example

Which of the following is a category in the revised Bloom’s

taxonomy of cognitive domain?

A. Comprehension

B. Analysis

C. Evaluation

D. All of the above



should enable an




● Not using All of the Above

● Eliminate grammatical clues



should enable an



Example

17 is an example of a _____.

A. integer

B. irrational number

C. ordinary number

D. prime number



should enable an



Example

17 is an example of a _____.

A. integer


C. ordinary number

D. prime number

17 is an example of a/an _____.

A. integer


C. ordinary number

D. prime number

Activity

● Look at your own MCQ questions

● Evaluate and improve them based on the 4 criteria i.e.

I. Nothing in the content or structure of the item should prevent an informed student

from responding correctly

II. Nothing in the content or structure of the item should enable an uninformed

student to select the correct answer

III. Measures what it is meant to measure

IV. Able to diagnose students’ misunderstanding of the concepts

Guess work Part II


● Not using All of the Above

● Eliminate grammatical clues

● Testing the concept repeatedly in the MCQ paper

● Negative marking

● Allow Rationing



should enable an



Team-based Learning Way (Give students 4 marks for each question)

Question No A B C D

1 4

2 2 2

3 1 1 1 1

4 4

5 2 2

6 2 1 1

7 1 1 1 1

8 4

9 3 1

10 4

Example of a question that tests deeper level of understanding

Questions

Summary: Guidelines for MCQs

1. Each item should measure an important learning outcome

2. Present a single clearly formulated problem in the stem of

the item

3. State the stem of the item in simple, clear language

4. Put as much of the wording as possible in the stem of the

item

5. State the stem of the item in positive form, wherever

possible

6. Emphasise negative wording whenever it is used in the stem

of an item


7. Make certain that the intended answer is correct or clearly

best.

8. Make all alternatives grammatically consistent with the stem

of the item and in parallel form

9. Avoid verbal clues that might enable students to select the

correct answer or eliminate an incorrect alternative

10.Make the distractors plausible and attractive to the

uninformed

11.Vary the relative length of the correct answer to eliminate

length as a clue


12.Avoid using the alternative ‘all of the above’, and use ‘none

of the above’ with extreme caution

13.Randomize the position of the correct answers

14.Control the difficulty of the item either by varying the problem

in the stem or by changing the alternatives

15.Make certain each item is independent of the other items in

the test

16.Use an efficient item format

17.Follow the normal rules of grammar

18.Break (or bend) any of these rules if it will improve the

effectiveness of the item


Reference: Waugh & Gronlund, Assesssment of Students

Achievement (10th edition), p. 93 - 107

setting effective mcqs€¦ · able to differentiate primary and secondary sources of information...

Documents