Teaching technology:Assessment-II

Dr Naresh T Chauhan

Objective methods of evaluation

• Advantage and limitations• Enumerate the methods of objective evaluation• Types of multiple choice question• Advantage and disadvantage of MCQ• The component of MCQ item• Guideline for construction of an MCQ• Item analysis

Objective

Dale's ‘Cone of Experience’

Advantage • Objective in nature• In depth evaluation of student !• Easy to evaluate• To test the candidate at the desired level of

difficulty • Good for formative evaluation and for

screening a large number of examinees• Standard of scoring can be kept constant for

many years

Disadvantage

• Poorly framed MCQ may be misleading• Tendency among teacher to test trivial and

less important data• Questions are difficult and time consuming to

construct• Not very useful for summative evaluation

Types • The single or one best response • Multiple completion type• Relationship analysis type• Multiple true false completion type• Matching type• Case history type• Pictorial/Graphical /Tabulated type• Comparison type

Testing Item Content Levels• Recall – testing knowledge of

isolated facts• Interpret* – asking to review

and reach some conclusion• Problem solve* – asking to

take some action after reading a situation

*Higher order thinking skills.

Writing for Different Cognitive Levels

• Recall– Which of the following is the process of mitosis?

• Interpret– Which of the following can be classified as a

meiosis error?• Problem-solving– A maternal blood sample revealed decreased

levels of AFP and estriol and elevated hCG. What would be the next logical test to perform?

Purposes…

• Motivate students• Identify areas of deficiency or further learning• Determine final grades or make promotion

decisions• Identify areas where the course/curriculum is

weak

What Should Be Tested?

• Exam content should match course/clerkship objectives

• Important topics should be weighted more heavily

• The testing time devoted to each topic should reflect the relative importance of the topic

Component of MCQ

Key

Distractor

Direction for framing MCQs

• Design each item to measure an important learning outcome

• Present single, compact, clearly formulated stem

• Use simple and clear language• Stem should be a statement , not a word• State the stem in positive form as far as possible• The word ‘NOT’ and ‘EXCEPT’ should be

underlined

Direction …cont

• Grammar ,Style, length, plausible distractors • Avoid clue that suggest answers, similar length

and form• ‘All of the above’, ‘none of the above’• Item with numerical alternatives , arrange

them in rank order• Testwiseness• Irrelevant Difficulty

Issues Related to Testwiseness

• Grammatical cues• Logical cues• Absolute terms• Long correct answer• Word repeats• Convergence strategy• B or C position overused

The forbidden fruit that Eve offered Adam was an:

(a) apple (b) pear(c) banana (d) melon(e) carrot

Keep the alternatives simple• When your body adapts to your exercise load,

a. you should decrease the load slightly.b. you should increase the load slightly.*c. you should change the kind of exercise you are doing.d. you should stop exercising

• When your body adapts to your exercise load, you should

• a. decrease the load slightly.b. increase the load slightly.*c. change the kind of exercise.d. stop exercising

Implausible distractors:

• In a study of the effect of diet on risk of diabetes, the researcher can manipulate a number of variables including the amount of food, carbohydrates, proteins or fats consumed. During the experiment the amount of food, protein and fat subjects consumed remained the same. Only the amount of carbohydrates consumed changed. What was the independent variable in this study?

a. amount of food consumedb. amount of carbohydrates consumedc. amount of protein consumedd. amount of fat consumed

Plausible distractors:

• In a study of the effect of diet on risk of diabetes, the researcher measured how likely the subjects were to get diabetes and how severe their symptoms were if they developed the disease. To prevent amount of exercise from influencing the results, the researcher held it constant in the two groups he was studying. What was the independent variable in this study?

a. likelihood of developing diabetesb. severity of symptoms c. dietd. amount of exercise

dietary pattern

Possibility of a “Random Pass”

Depends on the number of answer options per questionand the number of questions!

Number of Questions

Percent Pass (≥50%) by Chance2 choices 3 choices 4 choices 5 choices

1 50 33 25 20

2 75 56 44 36

4 69 41 26 18

6 66 32 17 10

10 62 21 8 3

20 59 9.2 1.4 .3

50 56 1 .01 .0004from Brown, 2001

Adjustment for Guessing

Negative Marking…– Elimination strategy reduces odds of wrong

answer penalty

– Subtracting a percentage of the number of wrong answers obtained from the final grade

– Give a grade of 4 for a correct answer and a score of -1 for a wrong answer on a 4 choice question

Item analysis

• Difficulty index

• Discrimination index

• Distractor effectiveness

• To create a question bank• Used in classroom teaching to modify teaching content

or methodology

Purpose of Item Analysis

–Evaluates the quality of each item

–Rationale: the quality of items determines the quality of test (i.e., reliability & validity)

– May suggest ways of improving the measurement of a test

• Difficulty index– The item difficulty for item i, pi , is defined as the

proportion of examinees who get that item correct.

• Method for Dichotomously Scored Item

• Method for Polytomously Scored Item

• Grouping Method

N

RP

maxX

XP

2LU PP

P

Item Difficulty (cont.)

Interpreting the p-value...

example: 100 people take a test 15 got question 1 right

What is the p-value?Is this an easy or hard item?

Item Difficulty (cont.)

Interpreting the p-value...

example: 100 people take a test 70 got question 1 right

What is the p-value?Is this an easy or hard item?

0.5 + 10

Too EasyToo Difficult

Level of DifficultyIndex Range Difficulty Level

0.00-0.20 Very Difficult

0.21-0.40 Difficult

0.41-0.60 Average/Moderately Difficult

0.61-0.80 Easy

0.81-1.00 Very Easy

ITEM DISCRIMINATION

• The extent to which an item differentiates people on the behavior that the test is designed to assess.

• the computed difference between the percentage of high achievers and the percentage of low achievers who got the item right.

Index of Discrimination

• (used for dichotomously scored items)

D = PH - PL • We need to set one or two cutting scores to divide the

examinees into upper scoring group and lower scoring group.

• PH is the proportion in the upper group who answer the item correctly and PL is the proportion in the lower group who answer the item correctly.

• Values of D may range from -1.00 to 1.00.

Example

There are 140 students attending a world history test. If 18 examinees in upper group answer item 5 correctly, and 6 examinees in lower group answer it correctly, then calculate the discrimination index for item 5.

Example 2 50 Examinees’ Test Data on 8-Item Scale About Job

Stress.

Item 1 2 3 4 5 6 7 8

PHPL

.54 .81 .47 .32 .51 .18 .63 .56 .32 .56 .11 .05 .10 . 23 .25 .19

D .18 .25 .36 .27 .41 -.05 .38 .37

Guidelines for Interpretation of D Value

• D≥.40, the item is functioning quite satisfactorily• .30≤ D≤.39, little or no revision is required• .20 ≤ D≤.29, the item is marginal and needs revision• D≤.19, the item should be eliminated or completely

revised

0 + 1- 1

Good Discrimination

Reverse Discrimination

What’s a “Good” Distracter?A good distracter is a plausible answer:• CM = common student mistakes, a recurrent

misconception about topic• TG = too general option, that is too broad to

answer the question• TS = too specific option, option focuses on one

detail in stem• OT = off topic, option misses the answer

completely

Guidelines for constructing a MCQ paper

• Include a sufficient number of items to have validity

• Time allotted • Group according to format• Instructions for negative marking• Place easy questions first, easy moderate and

difficult questions in sufficient numbers• All part of an item should on same page

design

Pilot study

ReliabilityInternal consistency – reliability coefficients (coefficient alpha or KR 20)Equivalence - alternate form correlation

Content validity - expert panelFace validity - expert panel

Construct validity - 'key check' / item discrimination / item difficulty

Use of machine !

• Computer• OMR

Go green, Think green, Act green !http://www.go-green.ae/

Further reading…

• http://www.teambasedlearning.org/• Constructing Written Test Questions For the

Basic and Clinical Sciences Third Edition (Revised)•Multiple Choice Questions Not Considered

Harmful by

Karyn Woodford, Peter Bancroft accessed from http://crpit.com/confpapers/CRPITV42Woodford.pdf on 09 Aug14