Intuitive Test Theory

Robert J. Mislevy

CRESST/ University of Maryland

September 9, 2004

Presented at the 2004 CRESST Conference, September 9-10, UCLA, Los Angeles, CA. This talk is based on joint work with Henry Braun.

Where we’re going

How people think about tests

Howard Gardner quote

Intuitive physics

Scientific test theory

Some testing p-prims

Conclusion

Howard Gardner, 1993, p. 5 (1)

In most domains of knowledge, we develop very powerful theories when we are very young. School and the disciplines are supposed to reformulate those theories and to make them more comprehensive and more accurate.

As long as we stay in school, we can maintain the illusion that the effort has succeeded, but ... once we leave school, the illusion disappears and there is a 5-year-old mind dying to get out and express itself...

Howard Gardner, 1993, p. 5 (2)

No one has to tell a kid that heavy objects fall more quickly than light objects. It's totally intuitive. It happens to be wrong. Galileo showed that it was wrong. Newton explained why it was wrong. But, like others with a robust 5-year-old mind, I still believe heavier objects fall more quickly than lighter objects....

Howard Gardner, 1993, p. 5 (3)

The only people on whom these engravings change are experts. Experts are people who actually think about the world in more sophisticated and different kinds of ways. ...

In your area of expertise, you don't think about what you do as you would when you were five years of age. But I venture to say that if I get to questioning you about something that you are not an expert in, the answers you give will be the answers you would have given before you had gone to school.

Intuitive Physics

diSessa’s “p-prims”

• Heavy objects fall faster than light objects.

• Things bounce because they are 'springy’.

• An object moves in a direction if and only if there’s a force on it in that direction.

Chi, Feltovich & Glaser’s problem-sorting

• springs & pulleys vs. equilibrium & Newton’s laws

“Scientific” Test Theory (1)

Items and scores are the springs and pulleys of testing.

Assessment as evidentiary reasoning.

Want to know what students know, can do, or have accomplished,

from a handful of particular things they’ve said, done, or made.

“Scientific” Test Theory (2)

Probability models connect what you see with what you make inferences about.

• Quantify accuracy and error.

• Play “what if” games for design.

• Quality control (e.g., large system w. raters)

• Can tell when you’re wrong.

Can be any kind of data.

Can be any kind of knowledge or skill.

Testing P-Prims

A test measures what it says at the top of the page.

Any two tests that measure the same thing can be made interchangeable, with a little equating magic

A score is a score is a score.

You score a test by adding up scores for items.

821- 285

664

885- 221

664

63- 15

52

17- 9

12

93% is an A, 85% is a B, 78% is a C, and 70% is passing.

Multiple-choice questions only measure recall.

A. The truck exerts the same amount of force on the car as the car exerts on the truck.

B. The car exerts more force on the truck than the truck exerts on the car.

C. The truck exerts more force on the car than the car exerts on the truck.

D. There’s no force because they both stop.

What are the forces at the instant of impact?

20 mph 20 mph

A. The truck exerts the same amount of force on the car as the car exerts on the truck.

B. The car exerts more force on the truck than the truck exerts on the car.

C. The truck exerts more force on the car than the car exerts on the truck.

D. There’s no force because they both stop.

What are the forces at the instant of impact?

10 mph 20 mph

A. The truck exerts the same amount of force on the fly as the fly exerts on the truck.

B. The fly exerts more force on the truck than the truck exerts on the fly .

C. The truck exerts more force on the fly than the fly exerts on the truck.

D. There’s no force because they both stop.

What are the forces at the instant of impact?

10 mph 1 mph

Technology will solve testing problems by making it possible to get voluminous amounts of data.

Cisco example, if time allows

Conclusion (1)

Problems:

Assessment policy is based on intuitive theory.

Intuitive test theory is used with new forms of testing, like simulations.

Conclusion (2)

Three ways of dealing with it:

Not-fun way: Tell people what they’ve already planned is wrong, and why.

Good way: Be involved in project planning early on.

Good & fun way: Existence-proof projects that demonstrate what can be done, outside familiar forms, using scientific test theory.

Conclusion (2)

An example: Cisco Learning Institute’s NetPass Prototype

Create on-line performance assessment of networking skills

Focus on learner feedback rather than high-stakes testing

A text file is scored following detailed rules,

which result in characterizations of the

work

Conclusion (2)