THE FOLLOWING MATERIALS

WERE PREPARED FOR

USE BY CARS AND SASS

CONSULTANTS.

ITEM WRITING

Chris OremJerusha GerstnerChristine DeMars

AGENDA

Presentation and Discussion Item writing guidelinesExamples

Develop, Evaluate, and Revise Items

A test blueprint is a table of specifications that weights each objective according to how important it is or how much time is spent covering that objective, links objectives to test items and summarizes information

Essential for Competency Testing

Test blueprint

BLUEPRINT TIPS

- List objectives in a table

- Identify length of test

- Designate number of items per objective

-Evaluate the importance of each objective and

assign items accordingly

- Often assign one point per item

EXAMPLE OF BLUEPRINT

Objective Weighting Items

Objective 1 25% 10

Objective 2 25% 10

Objective 3 25% 10

Objective 4 25% 10

Some items are more appropriate when testing different kinds of knowledge; or when tapping into different kinds of cognitive processes.

We’ll focus on multiple choice items because recruiting faculty to score open-ended items would be difficult.

Type of Item Construction Scoring

True/False Difficult Easy

Matching Easy Easy

Completion Easy Difficult

Multiple Choice Difficult Easy

Essay Easy Difficult

Writing Items

MULTIPLE-CHOICE ITEM ANATOMY

http://testing.byu.edu/info/handbooks/betteritems.pdf

GUIDELINES FOR WRITING ITEMS

Content Style/format

Writing the stem

Writing the distracters

Focus on a single problem when writing an item

Use new situations to assess applicationAvoids memorization exercisesAllows for synthesis and evaluation

Keep content of items independentstudents shouldn’t be able to use one item to

answer another, although a set of items may tap into a shared scenario

Avoid opinion-based items Address a mix of higher-order and

lower-order thinking skills.

Content

HIGHER ORDER THINKING Bloom Haladyna

Analysis

Synthesis

Application

Comprehension

Knowledge

Evaluation

Predicting

Evaluating

Defining

Recalling

Problem Solving

Higher Order Skills

Lower Order Skills

WRITING ITEMS TO ASSESS HIGHER ORDER THINKING (BASED ON HALADYNA)

Try item stems such as "If . . ., then what happens?", "What is the consequence of . . .?", or "What would happen if . . .?" (predicting)

Ask students to make a decision based on predetermined criteria, or to choose criteria to use in making a decision, or both (evaluating).

Require the student to use combinations of recalling, summarizing, predicting, and evaluating to solve problems.

Avoid excess words – be succinct Use specific, appropriate vocabulary Avoid bias (age, ethnicity, gender,

disabilities) Write stems and options in third person Underline or bold negative or other

important words Have others review your items

Style/format

The stem should clearly state the problem Place the main idea of the question in the stem, not the

item options Keep the stem as short as possible Don’t provide clues to correct answer in stem (e.g.,

grammatical clues) If the stem is a complete sentence, end with a period and

begin all response options with upper-case letters. If the stem is an incomplete sentence, begin all response

options with lower case letters. Use negative stems rarely

Writing the stem

Writing the distracters

Make sure there is only one correct answer for each item

Develop as many effective plausible options as possible, but three are sufficient (Rodriguez, 2005) It is better to have fewer options than to write BAD

options to meet some quota! Vary the location of the correct answer when

feasible (Flip a coin), or put options in logical order (e.g. chronological, numerical)

Avoid excessive use of negatives or double negatives

Keep options independent Keep options similar (in format)

Length and wording

DO use as distracters: common student misconceptions (perhaps from

open-ended responses from previous work) words that “ring a bell” or “sound official” responses that fool the student who has not

mastered the objective DO NOT use as distracters:

responses that are just as correct as the right answer implausible or silly distracters

Use “all of the above” and “none of the above” sparingly

Don’t use “always” or “never”

Don’t give clues to the right answer

Writing the distracters

ALL THE OPTION CHOICES SHOULD USE PARALLEL GRAMMAR TO AVOID GIVING CLUES TO THE RIGHT ANSWER.

The best way to increase the reliability of a test is to:

A. increase the test length B. removing poor quality items C. Tests should be readable for all test

takers.

What’s wrong with this item?

California:A). Contains the tallest mountain in the United StatesB). Has an eagle on its state flag.C). Is the second largest state in terms of area.*D). Was the location of the Gold Rush of 1849.

What is the main reason so many people moved to California in 1849?A). California land was fertile, plentiful, and inexpensive.*B). Gold was discovered in central CaliforniaC). The east was preparing for a civil war.D). They wanted to establish religious settlements.

Stem should state the problem

“FOCUS ON A SINGLE PROBLEM”; ITEMS WITH MULTIPLE CLAUSES MAY HAVE MULTIPLE CORRECT ANSWERS DEPENDING ON WHICH ASPECT THE STUDENT FOCUSES.

Bleeding of the gums is associated with gingivitis, which can be cured by the sufferer himself by brushing his teeth daily.

A. true B. false

The United States should adopt a foreign policy based on:A). A strong army and control of the North American continent.B). Achieving the best interest of all nations.C). Isolation from international affairs.*D). Naval supremacy and undisputed control of the world’s sea lanes.

According to Alfred T. Mahan, the United States should adopt a foreign policy based on:A). A strong army and control of the North American continent.B). Achieving the best interest of all nations.C). Isolation from international affairs.*D). Naval supremacy and undisputed control of the world’s sea lanes.

What’s wrong with this item?More than One Possible Answer

Piloting and revising

Following these rules does not guarantee that items will perform

well empirically. Testing companies, using paid item-writers and detailed writing

guidelines, ultimately only use 1/3 (or fewer) of the items

operationally.

It is normal and expected to have to revise items after pilot-testing .

Piloting and revisingITEM ANALYSIS STEPS:

• Item difficulty - proportion of people who answered the item correctly- an item should not be too easy or too difficult- problems arise from poor wording, trick questions, or

speediness

• Item Discrimination - correlation of item and total test, should be higher than 0.2

- item as an indicator of the overall test score- the higher the better- can be (but shouldn’t be) negative

• Distractor Analysis - frequency of response selections- item is problematic if people with a high overall score are

selecting incorrect responses frequently

THE BIG IDEA Ultimately, you want to know

whether students are achieving your objectives

Tests are used to indirectly measure these knowledge, skills, attitudes, etc.

Items you write are a sample of all possible items to measure that objectivethe more you write (and the better your

items are!) the more reliably you can measure the objective

You want to be sure that the items you create are measuring achievement of the objective, and NOT test-wiseness, reading ability, or other factors

REFERENCES Haladyna, T. M. (1999). Developing and validating

multiple-choice test items. Mahwah, NJ: Lawrence Erlbaum Associates.

Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24, 3-13.

Downing & Haladyna (2006). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates.

Burton S.J. et al (1991). How to prepare Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University Testing Services.