1

The Many Ways of Using Tests for Educational Improvement

Edward H. Haertel

School of Education Stanford University

Learning from History: Assessment and theImprovement of Student Learning

2005 CRESST ConferenceLos Angeles, CA

September 8, 2005

2

-OR-

It seemed like such a good idea at the time...

What Can History Teach About Today’s Standards-Based Accountability Testing?

3

Hindsight is Humbling

Looking back, seeing what “just made sense” at other times, may help to uncover some hidden assumptions underlying today’s reforms

4

Purposes in Looking Back

• What was obvious at the time that now seems to have been mistaken?

• What is obvious today that was not seen at the time?

To notice...

5

Seven Ways to Make Education Better By Testing

1. Group students into fast vs. slow tracks according to their IQ.

2. Divide the curriculum into tiny testable slices; test for mastery before moving on.

3. Diagnose learner’s individual aptitudes; prescribe individualized instruction accordingly.

6

Seven Ways to Make Education Better By Testing

4. Compare curricula using randomized trials; use tests to determine the best outcome.

5. Require passing a Minimum Competency Test or Exit Exam for HS graduation.

6. Use “authentic,” hands-on performance assessments and portfolios to promote higher-order thinking & improve motivation.

A few of these ideas have been completely abandoned, but traces of most can be found in present-day practices.

7

Seven Ways to Make Education Better By Testing

7. Evaluate schools according to the percent of students scoring at-or-above judgmentally determined cut scores representing “Proficient” on tests that (by mandate) are aligned with academic content standards.

8

Group students into fast vs. slow tracks according to their IQ.

• Immigration (since late-19th Century)

• Rapid increase in numbers of students

• Efficiency and Scientific Management of Schools

• Perceived success of the Army Alpha

• Prevailing beliefs about origins of group differences

9

At the time ...—IQ Testing was a powerful new scientific tool that could improve tracking (already an established practice) by eliminating reliance on teachers’ subjective judgments

—IQ-based tracking was based on unexamined assumptions about “nature vs. nurture,” and the origins and fixity of group differences.

—Influences of language and culture on test performance were not recognized

—Negative effect of tracking on opportunity to learn was not understood

Today ...

10

Divide the curriculum into tiny testable slices; test for mastery before moving on.

• Winnetka Plan (Washburne, 1925)

• Behaviorist Psychology

• Teaching Machines and Programmed Instruction

• Mastery Learning

• Criterion-Referenced Testing

11

At the time ...—New learning requires mastery of prerequisite “cognitive entry behaviors.”

—If students master each unit before going on, all can succeed

—Assumptions of “decomposability” and “decontextualization” were faulty

—Learning is best when it serves a purpose embraced by the learner

—Most worthwhile school learning requires that learners actively construct new understandings

Today ...

12

Diagnose learner’s individual aptitudes; prescribe instruction accordingly.

• From “The Two Disciplines of Scientific Psychology” (Cronbach, 1957) to Aptitudes and Instructional Methods (Cronbach & Snow, 1977)

Teaching without first making a “detailed diagnosis ... of the initial state of the learner” would be like a physician “prescribing medication for an illness without first examining the symptoms.”

Glaser and Nitko (1971, pp. 631-632)

13

At the time ...—Stable individual aptitude differences (e.g., “learning styles”) determine which alternative instructional “treatment” is best for each learner

—Educational Measurement and Ed Psych will soon enable precise determination of relevant aptitudes

—Different instructional approaches may be best for different learners, but the “local knowledge” of experienced educators is usually a better guide than any battery of tests we have been able to devise

—Research priority is on “main effects.” (Which curriculum or instructional method has the largest overall effect?)

Today ...

14

Compare curricula using randomized trials; use tests to determine the best outcome.

• Head Start evaluations mandated by the ESEA of 1965)

• Follow Through Planned Variation Study

• Evaluations of post-sputnik NSF-sponsored math and science curricula

• The What Works Clearinghouse

15

At the time ...—Questions about the (relative) effectiveness of curricula and instructional methods have clear answers, largely independent of local conditions

—Scientific methods can reveal those answers

—Questions about the (relative) effectiveness of curricula and instructional methods have clear answers, largely independent of local conditions

—Scientific methods can reveal those answers

Today ...

16

Today ...

—Progress in educational research requires a plurality of research approaches, informed by multiple disciplines in the social sciences and also the humanities

—Dissemination of innovation is a complex, adaptive process that depends on local expertise and local “buy-in”

—“What Works” in one place, with one group of learners, to accomplish one educational goal, may not work the same way elsewhere

17

Require passing a Minimum Competency Test or Exit Exam for HS graduation.

• MCT was outgrowth of “Back to Basics” movement of the 1970s plus popular concern that HS diploma no longer stood for much of anything

• Major expansion of MCT was 1975-80

• By 1985, only 11 states still required passing MCT for HS graduation

18

At the time ...—Many high school graduates cannot read even simple text (e.g., warning on a medicine bottle)

—MCTs will restore meaning to the HS diploma, pushing students to work harder and pushing teachers to make sure all students succeeded

—Many high school graduates lack academic skills critical for college success or tomorrow’s workplace

—HS Exit Exams will push students to work harder and push teachers to make sure all students succeed, thereby assuring a HS diploma really means something

Today ...

19

Use “authentic,” hands-on performance assessments and portfolios to promote

higher-order thinking & improve motivation.

• “Assessing the Thinking Curriculum” (Resnick & Resnick, 1992)

• New Standards Project (Lauren Resnick, LRDC, & Marc Tucker, NCEE)

• Vermont Portfolio-Based Assessment

• California Learning Assessment System

20

At the time ...—MC tests pushed curriculum and instruction in the wrong direction (superficial, isolated bits of knowledge)

—The right kinds of tests, aligned with ambitious content standards and tied to clear performance standards, could push C&I in the right direction

—Measurement reality fell short of political expectations

—Changing assessments alone could not bring about successful large-scale reform

—Low reliability, high cost, rushed implementation, and neglect of other critical reform elements ended a promising educational movement

Today ...

21

Evaluate schools according to the percent of students scoring at-or-above

judgmentally determined cut scores representing “Proficient” on tests that, by mandate, are aligned with academic

content standards.

• Goals 2000: Educate America Act (PL 103-227, 1994)

• No Child Left Behind Act of 2001 (PL 107-110, 2002)

22

Lessons to be Learned?

• The educational system must acknowledge individual differences in student aptitude

• Some crucial learning outcomes cannot be assessed using multiple-choice tests

• Tests, rewards, and sanctions alone will not bring about meaningful educational reform

• Educational research to inform best practice requires multiple methods and perspectives

23

The educational system must acknowledge individual differences in student aptitude

• Confusion about sources of within- versus between-group differences in test scores has made this topic difficult to discuss

• All students must be expected to master the basics

• A sensible, humane educational system must provide meaningful education for students who may not be college-bound

24

Some crucial learning outcomes cannot be assessed using multiple-choice tests

• Testing isolated skills leads to teaching isolated skills

• Meaningful, extended projects can improve motivation and address otherwise neglected learning outcomes

• Performance assessment has a place in the classroom, but we do not yet know how to make it work very well as part of a high-stakes assessment system

25

Tests, rewards, and sanctions alone will not bring about meaningful educational reform

• If educators knew how to improve test performance and eliminate achievement gaps, they would do so

26

Educational research to inform best practice requires multiple methods and perspectives

• In medicine, for example, randomized clinical trials are just the last step, building on basic research, case studies, and clinical observations

27

Lessons to be Learned?

• This too shall pass