value-added systems presentation to the isbe performance evaluation advisory council

Value-Added SystemsPresentation to the ISBE Performance

Evaluation Advisory Council

Dr. Robert H. MeyerResearch Professor and DirectorValue-Added Research Center

University of Wisconsin-Madison

February 25, 2011

Attainment and Gain• Attainment – a “point in time” measure of student

proficiency– compares the measured proficiency rate with a

predefined proficiency goal.

• Gain – measures average gain in student scores from one year to the next

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Gain

Gain

Gain

Attainment versus Gain

4

Growth: Starting Point Matters

School 2006 Grade 4Scale Score Avg.

2007 Grade 5Scale Score Avg.

Average Scale Score Gain

A 455 465 10

B 425* 455* 30

Reading results of a cohort of students at two schools

Grade 4 Proficient Cutoff 438Grade 5 Proficient Cutoff 463*Scale Score Average is below Proficient

Example assumes beginning of year testing

Value-Added• A kind of growth model that measures the

contribution of schooling to student performance on the standardized tests

• Uses statistical techniques to separate the impact of schooling from other factors that may influence growth

• Focuses on how much students improve on the tests from one year to the next as measured in scale score points

Value-Added Model Definition• A value-added model (VAM) is a quasi-experimental

statistical model that yields estimates of the contribution of schools, classrooms, teachers, or other educational units to student achievement, controlling for non-school sources of student achievement growth, including prior student achievement and student and family characteristics.

• A VAM produces estimates of productivity under the counterfactual assumption that all schools serve the same group of students. This facilitates apples-to-apples school comparisons rather than apples-to-oranges comparisons.

• The objective is to facilitate valid and fair comparisons of productivity with respect to student outcomes, given that schools may serve very different student populations.

A More Transparent (and Useful) Definition of VA

• Value-added productivity is the difference between actual student achievement and predicted student achievement.

• Or, value-added productivity is the difference between actual student achievement and the average achievement of a comparable group of students (where comparability is defined by a set of characteristics such a prior achievement, poverty and ELL status).

In English

Posttest PretestPost-on-Pre

Link

StudentCharacteristics

School Effects

UnobservedFactors

= x

+ + +

ValueAdded

VARC Philosophy

• Development and implementation of a value-added system should be structured as a continuous improvement process that allows for full participation of stakeholders

• Model Co-Build; Complete customization– Analysis – Reporting

• Value–added is one tool in a toolbox with multiple indicators

VARC Value-Added Partners• Design of Wisconsin State Value-Added System (1989)• Minneapolis (1992)• Milwaukee (1996)• Madison (2008)• Wisconsin Value-Added System (2009)• Milwaukee Area Public and Private Schools (2009)• Racine (2009)• Chicago (2006)• Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010)• New York City (2009)• Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts

(2009)• Illinois (2010)• Hillsborough County , FL (2010)• Broward County, FL (2010)• Atlanta (2010)• Los Angeles (2010)• Tulsa (2010)

Districts and States working with VARC

New York City

BrowardCounty

HillsboroughCounty

Atlanta

TulsaLos Angeles

Minneapolis

Madison

Chicago

Milwaukee

Racine

Measuring knowledge

• Many factors influence what a student learns and how their knowledge is measured

• A variety of measures, including (but not limited to) assessments, tell us what a student knows at a point in time.

• What are some ways we measure knowledge?

Measuring knowledgeLarge scale assessments Local assessments used by the district

Daily teacher assessments Observations

MAP WKCE Diagnostic Test End-of-course Exam

Daily Journal Unit Project After-school Activities Hands-on Project

The Simple Logic of Value-Added Analysis

• School Value-Added Report– School specific data– Grade level value-added

• Comparison Value-Added Reports– Compare a school to other schools in the district,

CESA, or state– Also allows for grade level comparisons

• Tabular Data available for School Report and Comparison Reports

Attainment and Value-Added

How complex should a value-added model be?

• Rule: "Simpler is better, unless it is wrong.“

• Implies need for “quality of indicator/ quality of model” diagnostics.

Model Features

• Demographics• Posttest on pretest link• Measurement error• Student mobility: dose model• Classroom vs. teacher: unit vs. agent• Differential effects• Selection bias mitigation: longitudinal data• Test property analysis

MAP vs. ISAT

• MAP dates: September, January, May• MAP: uses Rasch equating

– ISAT: 3PL• MAP: slightly higher reliability - ~0.96 in

math, ~0.94 in reading– ISAT math ~0.93, reading ~0.9

• Cut scores on MAP are determined by equipercentile equating to ISAT

means

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

GRADE_LEVEL

02 03 04 05 06 07 08 09 10 11 12

Hillsborough FCAR Average Math Developmental Scale Scores

Cohort year is fall of first grade

cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007

means

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

GRADE_LEVEL

02 03 04 05 06 07 08 09 10 11 12

Hillsborough Average FCAT Reading Developmental Scale Scores

Cohort year is fall of first grade

cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007

stdevs

160

170

180

190

200

210

220

230

240

250

260

270

280

290

300

310

320

330

340

GRADE_LEVEL

02 03 04 05 06 07 08 09 10 11 12

Hillsborough Standard Deviation of FCAT Math Developmental Scale Scores

Cohort year is fall of first grade

cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007

stdevs

230

240 250

260

270

280

290

300 310

320

330

340

350

360 370

380

390

400

410

GRADE_LEVEL

02 03 04 05 06 07 08 09 10 11 12

Hillsborough Standard Deviation of FCAT Reading Developmental Scale Scores

Cohort year is fall of first grade

cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007

Minimal correlation between initial status and value-added

Grade-Level Statewide Results

Subject State Grade NMean Score

SD of Score

Reliability of Value-Added

Math MN 3 59460 200.0 13.9 0.901

Math MN 4 58346 210.8 14.6 0.916

Math MN 5 57053 219.9 16.2 0.907

Math MN 6 52400 226.7 16.5 0.873

Math MN 7 47985 232.1 17.4 0.883

Math MN 8 44227 236.4 17.9 0.823

Math MN 9 26512 238.8 18.2 0.826

Grade-Level Statewide Results

Subject State Grade NMean Score

SD of Score

Reliability of Value-Added

Math WI 3 43289 199.9 13.2 0.820

Math WI 4 44140 209.3 13.7 0.842

Math WI 5 43822 217.3 14.8 0.849

Math WI 6 47004 222.7 15.2 0.836

Math WI 7 44549 228.4 16.0 0.837

Math WI 8 43246 233.1 16.8 0.865

Math WI 9 26427 234.0 17.7 0.862

Grade-Level Statewide Results

Subject State Grade NMean Score

SD of Score

Reliability of Value-Added

Reading WI 3 43139 194.8 15.1 0.736

Reading WI 4 43671 202.9 14.4 0.780

Reading WI 5 43668 209.7 13.8 0.737

Reading WI 6 46233 214.0 14.2 0.719

Reading WI 7 44616 218.3 14.0 0.792

Reading WI 8 43251 221.7 14.1 0.826

Reading WI 9 28066 223.3 14.4 0.843

School/District VA Productivity Parameters in WKCE Scale Score Units (Relative to State)

MPS School Effects

MMSD School Effects

6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects

MPS and MMSD Value-Added compared to Wisconsin

Visit the VARC Website

http://varc.wceruw.org/

for more information about VARC and value-added