March 11, 2013Chicago, IL

American Board of Preventive MedicineAmerican Board of Preventive MedicineClinical Informatics Examination Committee

Measurement Research Associates, a division of Measurement Incorporated

Goals and Important Measurement Considerations

Considerations

Validity

Reliability

Objectivity

Validity

Does the test measure what it is supposed to measure with regard to:

Construct Content

Validity

Exam content guidelines

Distribution of items ensures the test is representative of the field.

A sufficient number of items cover all relevant areas of practice appropriately

Reliability

Confidence in test scores

Measurement error – “uncontrolled noise”

Objectivity

Criterion-Referenced Testing Ascertains an individual’s ability level

relative to performance on the content domains as represented by the test items

The measurement of a candidate is independent of other test-takers

Criterion Standard

Collective expectation of the level of knowledge and skills needed to practice safely and effectively in the field Determined by experts

A criterion-referenced standard allows any candidate that can achieve the standard to pass. All candidates can pass All candidates can fail

Developing and Reviewing Multiple-Choice Items

Clinical Informatics Item Bank

Goal: Compile and maintain a pool of exam items which are appropriate to measure the knowledge and skills necessary for safe and effective performance in the field of practice.

Taxonomy

Taxonomy refers to the level of cognitive skill required to answer the item correctly.

1. Recall2. Interpretive3. Problem Solving

Parts to an Item

1. The stem

2. The responses Correct response Distractors

The Stem Format

1. Ask a question Which of the following microscopic subtypes of ameloblastoma

is most common?

2. Give an incomplete statement The most common microscopic subtype of ameloblastoma is:

3. Scenario with a question or an incomplete statement A 25 year-old man is brought to the emergency room. He was

found lying unconscious on the sidewalk. After ascertaining that the airway is open, the next step in management should be:

Item Options

Options are all the possible answers for a stem.

– One correct (best) answer

– Three distractors

The best answer is agreed upon by experts.

The distractors are logical misconceptions of

the best answer.

Developing Items

Items should have one best answer. Avoid items based on opinion or for which there is not an accepted answer.

Items must focus on a single issue, fact, or problem.

Developing Items

Items should test important and pertinent material while avoiding trivial facts.

Attempt to write interpretation and problem solving items.

Items should be developed utilizing good grammar, punctuation, and spelling.

Stem Construction

Avoid overly specific knowledge, excess

information, and teaching in the stem.

Include the central idea and most text in the stem

The stem should be stated positively; avoid

negative phrasing.

Stem Construction

Use terminology common to practice and avoid

verbatim textbook phrasing.

Avoid personal pronouns (i.e., you).

Response Construction

Avoid “all of the above” and “none of the above.”

Avoid absolutes such as “always” and “never.”

Responses should be

– organized in a logical order

– independent and not overlapping

– fairly consistent in length

– homogeneous or parallel in content

– plausible

Questions or Comments