What kinds of assessments improve learning?

Dylan Wiliam, ETS

Raising achievement matters

For individualsIncreased lifetime salaryImproved healthFor societyLower criminal justice costsLower health-care costsIncreased economic growth

Wheres the solution?

StructureSmall high schoolsK-8 schoolsAlignmentCurriculum reformTextbook replacementGovernanceCharter schoolsVouchersTechnology

Its the classroom

Variability at the classroom level is up to 4 times greater than at school levelIts not class sizeIts not the between-class grouping strategyIts not the within-class grouping strategyIts the teacher

Teacher quality:

A labor force issue with 2 solutionsReplace existing teachers with better ones?No evidence that more pay brings in better teachersNo evidence that there are better teachers out there deterred by certification requirementsImprove the effectiveness of existing teachersThe love the one youre with strategyIt can be doneWe know how to do it, but at scale? Quickly? Sustainably?

The design challenge

Key metric:Cost of buying one standard deviation of increased student achievementConstraintsSolution must be in principle scalable

Cost/effect comparisons

InterventionEffect (sd)Cost/yrClass-size reduction(by 30%)0.1$24kIncrease teacher content knowledge by 1 sd0.1?Formative assessment/Assessment for learning0.2

Effects of formative assessment

Several major reviews of the researchNatriello (1987)Crooks (1988)Black & Wiliam (1998)Nyquist (2003)All find consistent, substantial effects

Effects of feedback

Kluger & DeNisi (1996)Review of 3000 research reportsExcluding those:without adequate controlswith poor designwith fewer than 10 participantswhere performance was not measuredwithout details of effect sizesleft 131 reports, 607 effect sizes, involving 12652 individualsAverage effect size 0.4, butEffect sizes very variable40% of effect sizes were negative

Kinds of feedback (Nyquist, 2003)

Weaker feedback onlyKnowledge of results (KoR)Feedback onlyKoR + clear goals/knowledge of correct results (KCR)Weak formative assessmentKCR+ explanation (KCR+e)Moderate formative assessment(KCR+e) + specific actions for gap reductionStrong formative assessment(KCR+e) + activity

Effect of formative assessment (HE)

NEffectWeaker feedback only310.16Feedback only480.23Weaker formative assessment490.30Moderate formative assessment410.33Strong formative assessment160.51

Feedback and formative assessment

Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way (Ramaprasad, 1983 p. 4)Formative assessment requiresdata on the actual level of some measurable attribute;data on the reference level of that attribute;a mechanism for comparing the two levels and generating information about the gap between the two levels;a mechanism by which the information can be used to alter the gap.

Formative assessment

Frequent feedback is not necessarily formativeFeedback that causes improvement is not necessarily formative Assessment is formative only if the information fed back to the learner is used by the learner in making improvementsTo be formative, assessment must include a recipe for future action

Assessment for learning &
formative assessment (Black et al., 2002)

Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged.

Such assessment becomes formative assessment when the evidence is actually used to adapt the teaching work to meet learning needs.

Feedback

Feedback is therefore formative only if the information fed back is actually used in closing the gap.Three key instructional processesEstablishing where learners are in their learningEstablishing where they are goingEstablishing how to get there

Aspects of formative assessment

Where the learner is goingWhere the learner isHow to get thereTeacherClarify learning intentionsEngineering effective discussionsProviding feedback that moves learners onPeerUnderstand/clarify criteria for successActivating students as instructional resources for one anotherLearnerUnderstand criteria for successActivating students as owners of their own learning

Five key strategies

Clarifying and understanding learning intentions and criteria for successEngineering effective classroom discussions that elicit evidence of learningProviding feedback that moves learners forwardActivating students as instructional resources for each otherActivating students as the owners of their own learning

and one big idea

Use evidence about learning to adapt instruction to meet student needs

Keeping Learning on Track (KLT)

A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc.Educational systems must do the same:Plan a carefully chosen route ahead of time (in essence building the track)Take readings along the way Change course as conditions dictate

Regulation of learning

Teaching as engineering learning environmentsKey features:Create student engagement (pedagogies of engagement)Well-regulated (pedagogies of contingency)Long feedback cycles vs. variable feedback cyclesQuality control vs. quality assurance in learningTeaching vs. learningRegulation of activity vs. regulation of learning

Regulation of learning

Proactive (upstream) regulationPlanning regulation into the learning environmentPlanning for evoking informationInteractive (downstream) regulationNegotiating the swiftly-flowing riverMoments of contingencyTightness of regulation (goals vs. horizons)Retrospective regulationStructured reflection (e.g., lesson study)

Types of formative assessment

Long-cycleFocus: between unitsLength: four weeks to one yearMedium-cycleFocus: within units, between lessonsLength: one day to two weeksShort-cycleFocus: within lessonsLength: five seconds to one hour

Questioning in math: discussion

Look at the following sequence:

3, 7, 11, 15, 19, .

Which is the best rule to describe the sequence?

n + 4

3 + n

4n - 1

4n + 3

Questioning in math: diagnosis

In which of these triangles is a2 + b2 = c2 ?

A

a

c

b

C

b

c

a

E

c

b

a

B

a

b

c

D

b

a

c

F

c

a

b

Questioning in science: discussion

Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt?

The level of the water drops

The level of the water stays the same

The level of the water increases

You need more information to be sure

Questioning in science: diagnosis

The ball sitting on the table is not moving. It is not moving because:

no forces are pushing or pulling on the ball.

gravity is pulling down, but the table is in the way.

the table pushes up with the same force that gravity pulls down

gravity is holding it onto the table.

there is a force inside the ball keeping it from rolling off the table

Wilson & Draney, 2004

Questioning in English: discussion

Macbeth: mad or bad?

Questioning in English: diagnosis

Where is the verb in this sentence?

The dog ran across the road

A

B

C

D

Questioning in English: diagnosis

Where does the subject end and the predicate begin in this sentence?

The dog ran across the road.

A

B

C

D

Questioning in English: diagnosis

Which of these is a good thesis statement?

The typical TV show has 9 violent incidents

There is a lot of violence on TV

The amount of violence on TV should be reduced

Some programs are more violent than others

Violence is included in programs to boost ratings

Violence on TV is interesting

I dont like the violence on TV

The essay I am going to write is about violence on TV

Questioning in history: discussion

In which year did World War II begin?

1919

1937

1938

1939

1941

Questioning in History

Why are historians concerned with bias when analyzing sources?

People can never be trusted to tell the truth

People deliberately leave out important details

People are only able to provide meaningful information if they experienced an event firsthand

People interpret the same event in different ways, according to their experience

People are unaware of the motivations for their actions

People get confused about sequences of events

Why research hasnt changed teaching

Misunderstanding nature of teacher expertiseLeaving teachers to translate into practiceFailure to attend to both content and process

Klein & Klein (1981)

Six video extracts of someone delivering cardio-pulmonary resuscitation (CPR)5 of the videos are of CPR students1 of the videos is of an expert paramedicVideos shown to three groups:students, instructors and expertsSuccess rate in identifying expert:Experts: 90%Students: 50%Instructors: 30%

Teacher expertise (Berliner, 1994)

Experts excel mainly in their own domainExperts often develop automaticity for the repetitive operations that are needed to accomplish their goalsExperts are more sensitive to the task demands and social situation when solving problems.Experts are more opportunistic and flexible in their teaching than novicesExperts represent problems in qualitatively different ways than novices.Experts have fast and accurate pattern recognition capabilities. Novices cannot always make sense of what they experience.Experts perceive meaningful patterns in the domain in which they are experienced.Experts begin to solve problems slower, but bring richer and more personal sources of information to bear on the problem that they are trying to solve.

Knowledge creation and transmission

After Nonaka & Tageuchi, 1995

A model for teacher learning

Content (what we want teachers to change)EvidenceIdeas (strategies and techniques)Process (how to go about change)Small stepsFlexibilityChoiceAccountabilitySupport

Discussion

What are the implications of this research for publishers of formative assessment systems?

Questions?

Dont forget to fill out your evaluations