Synthesizing Extant Knowledge

for Practitioners in a Carnegie Knowledge Network

Chris Thorn, Managing DirectorAnalytics and Program Technology

September 16, 2013⦁ Columbus, OH

Triple Aims of Educational Improvement

2

EFFICIENCY

EFFECTIVENESS

ENGAGEMENT

Context: We Live in Extraordinary Times

More EfficientSystems

Ambitious Learning For All Students

MoreRelevance

Why focus on value added?

Value-added methods are relatively new, use is increasingly wide spread, but many technical questions remain

unresolved.

The Problem We’re Trying to Address:

• The state of knowledge in the field is changing rapidly

• The vast amount of information can be overwhelming

• Most findings are written in highly technical language

• Many experts are tied to commercial interests or policy stances

What a teacher interested

in learning more about

value-added might find

through an online search.

McCaffrey, D. F., Lockwood, J. R., Koretz, D.,

Louis, T. A., & Hamilton, L. (2004). Models for

value-added modeling of teacher effects. Journal

of educational and behavioral statistics,29(1), 67-

101.

Instrument Design Actual Practices of

Use

Rules & Regulations

Economists

Applied

Researchers

Designers

Statisticians

Policy

Advocates

LegislatorsState

Education

Officials

Union Leaders

Teachers

Principals

Local Teacher

Union Officials

District

leadersExternal Service

Providers

Carnegie’s Distinctive Role: Integrative Agent

The Carnegie Knowledge Networkwww.carnegieknowledgenetwork.org

• Identifies high priority areas characterized by significant knowledge gaps between research and practice

• Builds on an R&D agenda focused on practitioner needs

• Engages the community of practitioners

• Assembles balanced technical expertise

• Acts as an integrative agent

• Builds scholarly consensus

• Informs policy

CKN Online

Most common value added models in use

Vendor Name of Model Brief Description

American Institutes for Research (AIR)

VariedUsually control for student

background

Mathematica VariedUsually control for student

background

National Center for the Improvement of Educational

Assessment (NCIEA)

Student Growth Percentile (SGP) Models

Models a descriptive measure of student growth within a

teacher’s classroom

SAS EVAASModels control for prior test scores but not other student

background variables

Value Added Research Center (VARC)

VariedUsually control for student

background

Highlights of the recommendations

• Teachers of advantaged students benefit from models that do not control for student background factors, while teachers of disadvantaged students benefit from models that do control for student background factors

• Even when correlations between models are high, different models will categorize many teachers differently

• Rules for combining measures should reflect the qualities of those measures

Highlights of the recommendations

• High quality linkage is critical (dosage/teams/mobility)

• Consider the level of precision and balance the risks

• Bias may arise when comparing the value-added scores of teachers who work in different schools

• The properties of value-added measures differ across grades and subjects

• There is only a moderate, and often weak,

correlation between value-added calculations for the same teacher based on different tests

What’s on the Horizon for Carnegie

• We have little research to draw upon for designing systems or for predicting the effects of emerging

evaluation systems

• The Foundation leveraging the pressure of

accountability as the gateway drug to improvement

• Variation in effectiveness is the problem to solve

An Interesting Case Example

• First year results from a large randomized field trial

of Reading Recovery

(I3 initiative)

• Key: a multi-site trial

12

0

2

4

6

8

10

12

14

16

-0.5 -0.3 -0.1 0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9

Effect Size

RCT (average) Treatment Effect: Reading RecoveryN=141 schools

It is a success

0

2

4

6

8

10

12

14

16

-0.5 -0.3 -0.1 0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9

Co

un

t

Effect Size

Distribution of RCT Treatment Effects: Reading RecoveryN=141 schools

Undesirable/Weak Outcomes

Positive Deviants

0

200

400

600

800

1000

1200

F D C B A

Distribution of Letter grade of

Overall Value-Added for Ohio Schools

See the System to Improve it

We cannot improve outcomes without understanding the processes that generate them and the interconnections

between the processes.