F O R S TA N D A R D S - B A S E D A S S E S S M E N T S

TEST ITEM WRITING

AGENDA

8:00 – 10:00 Item writing guidelines

10:00 – 10:30 Write items

10:30 – 11:30 Vet items as a group

11:30 – 12:30 Lunch

12:30 – 1:00 Item rewrites

1:00 – 2:30 Write items

2:30 – 3:30 Vet items in like groups

*We have included slides in this presentation for you to use with your teachers, but we will not go in great detail on some of the slides with you.

ITEM WRITING GUIDELINES

3

ANATOMY OF AN ITEM

4

Use the excerpt below to answer the question. (Direction Line)

“We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.”(Stimulus – prompt/background)

Which document includes these words? (Stem - question)

A.The Magna Carta

B.The Mayflower Compact (Distractors)

C.The Articles of Confederation

D.The Declaration of Independence* (Correct Answer)

Cognitive Complexity, Item Difficulty, Sensitivity, Bias, Copyright

TEST ITEM WRITING PROCESSOVERVIEW – FOR MULTIPLE CHOICE

5

1. Look at the standard and item specifications

2. Develop a stimulus (if required)

3. Write the question (item stem)

4. Write the correct answer

5. Write distractors that are reasonable but clearly incorrect

6. Identify the cognitive complexity level for your item

7. Check the item for any flaws in logic, construction, or grammar and mechanics, bias, sensitivity, etc.

COGNITIVE COMPLEXITY

6

UNDERSTANDING COGNITIVE COMPLEXITY

• Florida measures the cognitive demand of the task that students are asked to perform by using Norman Webb’s Depth of Knowledge (DOK) levels

• Items that you develop will fall into one of the four DOK levels:

- Level 1: Recall

- Level 2: Basic Application of Concepts & Skills

- Level 3: Strategic Thinking & Complex Reasoning

- Level 4: Extended Thinking & Complex Reasoning

Reference material is available as a supplemental resource.

*Florida combines Level 3 and 4 and should be indicated as Level 3

8

EXAMPLES OF COMPLEXITY LEVEL QUESTIONS FOR U.S. HISTORY

9

MULTIPLE CHOICE QUESTIONS FOR U.S. HISTORY- LOW COGNITIVE COMPLEXITY

10

Which social issue is addressed in the cartoon?A. Industrial safetyB. The training of teachersC. Compulsory public educationD. Organization of immigrant laborers

MULTIPLE CHOICE QUESTIONS FOR U.S. HISTORY-MODERATE COGNITIVE COMPLEXITY

11

Which problem was the state action shown in the cartoon meant to resolve?

A. The influence of labor unionsB. The employment of child laborersC. The low literacy rates for immigrantsD. The poor working conditions in industry

EOC MULTIPLE CHOICE QUESTIONS FOR U.S. HISTORY- HIGH COGNITIVE COMPLEXITY

12

Based on the cartoon, what conclusion can be made about the US in the early 1900s?

A. The government assumed an active role in suppressing social activists.

B. The government assumed an active role in legislating social reforms.

C. Business monopolies controlled the means of production.D. Business owners monitored the activities of laborers.

ITEM WRITING TRAINING

13

UNIFY ITEM TYPES

•Multiple Choice

•Mutliselect

•GRID with

Action Buttons

•Hot Text

•Open Response

•Matching

•& more…

ITEM TYPES

15

Selected response (multiple choice)

Selected Response item (multiple choice):

•Should take the student approx. one minute per item to answer

•Has a question with four answer choices – three distractors and one correct answer

•The stem and answer choices can include text or images

WHAT IS A STIMULUS?

The stimulus is what prompts the stem (or question).

Stimuli can be:

• Text

• Charts and graphs

• Diagrams

• Illustrations

• Photographs

• Maps

• Numeric equations

• Tables and charts

• Timelines

• Graphic organizers

Not every question will have a stimulus. (Stem & answer choices only.)

WHAT IS THE STEM?

17

• The stem is the question being asked and the foundation of the

item.

• After reading the stem, the student should know exactly what the

problem is and what he or she is expected to do to solve it.

• If the student has to infer what the problem is, the item will likely

measure the student’s ability to draw inferences from vague

descriptions rather than his or her achievement of a course

objective.

ACTIVITY

18

Please complete the

activity and then you may

take a 5 minute break

THE STEM: CLEAR AND CONCISE

19

The problem should be stated clearly in the stem.

THE STEM: CLEAR AND CONCISE

20

From the article, the reader can tell that:A.Turtles like to hide under the rocks

B.Dogs like to play in the snow

C.Squirrels like to eat mulberries

D.Cats like to chase mice

From the article, what can the reader tell about what that turtles like to do?

A. Hide under the rocksB. Play in the snowC. Eat mulberriesD. Chase mice

Poor Example

Better Example

Does this stem provide the test taker with enough information?

THE STEM: RELEVANT MATERIAL

21

Include relevant material; avoid irrelevant material.

The local community theater in Conroe, Texas is performing a play in the Conroe High School auditorium. There are 15 rows of seats. Each row contains 28 seats.

What is the largest number of tickets that the community theater can sell to fill the auditorium for one performance?

A. 400 tickets

B. 420 tickets*

C. 450 tickets

D. 470 tickets

Answer – The stem is too lengthy, contains too much unnecessary information, and has no relation to Florida. The excessive information may confuse students as to what is actually being asked.

THE STEM: AVOID REDUNDANCY

22

Rather than repeating redundant words or phrases in the

distractors, place such material in the stem.

STEM: CLUEING

23

Lisa is writing a report on different types of ecosystems. Which source would most likely contain this information?

A. Cooking with Native Plants

B. A Guide to Rock Collecting

C. Ecosystems of North America*

D. The Big Book of Animal Habits

Key words in the stem that are also found in the answer choices may be clues to the correct answer.

This measures students cleverness, not knowledge of content.

24

THE STEM: NEGATIVE

Student achievement is more effectively measured by having students identify correct rather than an incorrect answer. Words like “is not”, “except”, “least”, etc. should be used sparingly.

25

WHAT IS A DISTRACTOR?

A distractor is an answer choice that leads a student away from the correct answer.

A good distractor has the following characteristics:

• Correct grammar

• Professional tone (avoid humor or slang)

• Unique scenario (one size does not fit all items)

• Direct and concise

• Test-savvy students easily eliminate implausible choices and improve their chances of guessing the correct answer.

• If a writer cannot develop three plausible distractors, then the question itself may be flawed.

26

STEM AND DISTRACTORS: VERBATIM

Avoid stems and distractors that are word-for-word from the pages of the textbook. This measures student’s memory rather than understanding.

27

DISTRACTORS: CLEAR AND CONCISE

Clear wording

reduces student

confusion, and

concise wording

reduces the

reading burden

placed on the

student.

28

DISTRACTORS: OVERLAPPING

If the overlap

includes the correct

answer, there may

be confusion about

which is the correct

answer and which is

the distractor.

29

DISTRACTORS: HOMOGENEOUS

Keep the

distractors

homogeneous

in content.

30

DISTRACTORS: GRAMMAR

Keep the grammar of each distractor consistent with

the stem.

31

DISTRACTORS: PARALLEL FORM

The stem and distractors should be worded similarly.

In this food chain, what will happen if there are fewer crickets in one year?

A.There will be more hawks

B. There will be fewer cattails

C.There will be fewer shrews*

D.The shrews will eat cattail

MORE ABOUT PARALLELISM

32

If the correct answer contains:Then the distractors should

NOT contain:

A long phrase or sentence Short phrases or sentences

A general rule Specific instances

A complete sentence A phrase

A means to an end End products

Simple, common language Professional jargon

Unqualified words or statements Qualifying words

Negative descriptions Positive descriptions

33

DISTRACTORS: FLAGS

Avoid words such as never, always, only, etc.; they serve as flags to alert students.

34

DISTRACTORS: PLAUSIBLE

Avoid unrealistic or humorous distractors. They are nonfunctional and increase student’s chance of guessing the correct answer.

35

DISTRACTORS: SINGLE CORRECT ANSWER

Avoid “all of the above”, “none of the above”, “both a and b”, etc. (multiple responses or combined responses)

36

TEST ITEM: CONVENTIONS

• Use proper grammar, spelling and punctuation.• Avoid using unnecessarily difficult vocabulary.• Use grade appropriate language – all items should

be written to the lower grade band for the standard.• Ex. LACC.1112.W.1.1 should be written to an 11th

grade level.• Ex. MACC.912.MP.5.1 should be written to a 9th grade

level.

TEST ITEM: SENSITIVE TOPICS

37

Be mindful of how you approach sensitive topics, such as:

• Alcoholic beverages, drugs, or tobacco

• Death and/or funerals

• Gambling

• Name brands and trademarked products

• Occult (witchcraft, etc.)

• Parapsychology

• Politics

• Religion

There may be exceptions to this rule (such as history courses).

TEST ITEM: BIAS

38

Read the following statement:

Firemen are important to the community.

They keep all of us safe.

• Which word contains bias?

• How can it be changed to eliminate the bias?

Answer: Change firemen to firefighters

CHECK ITEMS FOR BIAS

39

To ensure that test items are unbiased, include:‒Precisely defined ideas‒Language appropriateness

But avoid:‒Sensitive topics‒Gender stereotypes‒Ethnic stereotypes‒Socioeconomic bias‒Geographic bias

COPYRIGHT ISSUES

40

To avoid copyright issues:• Go to the original source and provide full source information

• Use public domain sites, such as government sites (e.g., EPA, US Census Bureau, or Library of Congress) and check for copyright information

• Avoid using permissioned work

Avoid using copyrighted names/brand names, such as:• Kleenex• Miami Dolphins• Adidas• Jell-O• Q-Tip

WEBSITES FOR ITEM INSPIRATION

41

• http://achievethecore.org/

• http://www.problem-attic.com/

• http://www.engageny.org/resource/regents-

exams-mathematics

• Florida Achieves Grade 5 and Bio (Shared by

Heidi in Google Drive)

42

QUESTIONS??

FSA MATHEMATICS ITEM TYPES

• Multiple Choice

• Mutliselect

• Equation

• GRID with Action

Buttons

• Hot Text

• Open Response

• Matching

• Table

• Editing Task

Choice

MULTIPLE CHOICE

Multiple Choice items require the student to

select a single answer option. (worth 1

point)Multiple Choice Items consist

of:

A context (mathematical or

real-world);

A question;

A single correct answer;

and

Three plausible distractors.

MULTISELECT ITEMS

Multi-select items allow the student to select more than one answer option.

Some items may ask the student to select a specific number of responses,

while others will ask them to select all correct responses. No partial credit

will be given. (worth 1 point)

Multiselect Items consist of:

A context (mathematical or

real-world);

A question;

Two or more correct

answers; and

Several plausible distractors.

Select all the statements that correctly

compare the two numbers.

Click the

checkbox next

to each option

you want to

select as a

response. You

may select

more than one

option.

EQUATION ITEMS

Equation items require the student to enter a valid statement that

answers the question. The response may be a number, an

expression, or an equation. No partial credit will be given. (worth 1

point)

Equation Items consist of:

A context (mathematical or

real-world);

A question; and

A response area.

Equivalent fractions and decimals will

be accepted except in items that are

assessing a standard where a certain

form is the expectation of the standard.

A zero in the ones place for answers

that fall between -1 and 1 (excluding 0

itself) is not required.

Students can type in an answer using a

mixed number or a fraction greater than

one.

EQUATION ITEM

GRID ITEM WITH ACTION BUTTONS

GRID items require the student to use the point, line, and/or

arrow buttons to create a response on a grid. No partial credit

will be given. (worth 1 point)

GRID Items consist

of:

A context

(mathematical or

real-world);

A question; and

A response area.

HOT TEXT ITEMS

Hot text items require the student to either click on a

response option or drag a response option to another

location.

Hot text Items

consist of:

A context

(mathematical or

real-world);

A question; and

A response

area.

In the ordered pair (0,4), what

does the 4 represent in terms of

its location on the coordinate

plane? Type your answer in the

space provided.

OPEN RESPONSE ITEMS

Open response items require the student use the

keyboard to enter a response into a text field.

Open Response Items

consist of:

A context (mathematical

or real-world);

A question; and

A response area.Click on the text box

and type your answer

using the keyboard.

Select the value of each decimal

number when it is rounded to the

nearest whole number.

MATCHING ITEMS

Matching items require the student to check a box to indicate if

information from a column header matches information from a row. No

partial credit will be given. (worth 1 point)

Matching Items consist of:

A context (mathematical

or real-world);

A question; and

A response area.

Click in the

boxes to

highlight the

desired

options.

The table shows the height of two containers, in feet..

Complete the table to show the height of each container, in

inches..

TABLE ITEMS

Table items require the student to type numeric values into a

given table. The student may complete the entire table or

portions of the table depending on the question. No partial

credit will be given. (worth 1 point)

Table Items consist

of:

A context

(mathematical or

real-world);

A question; and

A response area.

EDITING TASK CHOICE ITEMS

Editing task items require the student to click on the highlighted word

or phrase and select the replacement word or phrase from a menu. No

partial credit will be given. (worth 1 point)

Table Items consist

of:

A context

(mathematical or

real-world);

A question; and

A response area.

COMBINATION ITEMS

Items could combine more than one item type. These

may be worth more than 1 point, in which case, points

may be awarded by type.