How Does MAP Predict State Test Performance?

Understanding, Conducting, and Using Alignment Studies

June 27, 2012 11:15 am

Vancouver Public Schools and Highline Public Schools in Washington State

Presenters:Paul Stern Paul.Stern@vansd.orgSarah JohnsonSarah.Johnson@highlineschools.org

Overview•Background/The Value of Alignment Studies•Highline’s Regression Study•NWEA’s Linking Study•Multi-District Regression Study•Conclusions•Applying the Results

Learning Objectives•Learn how to define proficiency using MAP cut

scores. •Understand the alignment of MAP to

Washington’s State Assessments. •Learn how alignment studies can be conducted

and used to inform instruction.

Value of Alignment StudiesResearchers align scales for one of two purposes:• Use results from measure “X” to predict the value of

a harder-to-observe measure or outcome “Y”.• Use results from measure “X” to predict the value of

a future measure or outcome “Y”.

In our case, faculty and administration are interested in identifying students who are likely to struggle on future state performance measures. By intervening early, we can target resources to students who may not meet “proficiency”.

http://kingsburycenter.org/gallery

About Vancouver Public Schools•About 22,000 enrolled students•6 High Schools (4 comprehensive, 1 magnet, 1

alternative)•18% of students speak a language other than

English at home•49% eligible for free or reduced price lunch•The district serves half of the city of

Vancouver, WA (across the river from Portland)

About Highline Public Schools•About 18,000 enrolled students•15 High Schools (2 comprehensive, 6 small

learning community, 1 magnet, 5 alternative, 1 skills center)

•43% of students speak a language other than English at home - 21% are ELL.

•67% eligible for free or reduced price lunch•The district serves neighborhoods of White

Center, Burien, Des Moines, SeaTac and Normandy Park just south of Seattle.

Washington’s State Assessments• Measures of Student Progress (MSP) is given in

grades 3-8 in math and reading.• High School Proficiency Exam (HSPE) is given in

grade 10 in math. There is not a 9th grade test.• End of Course Exam (EOC1 and EOC2) given at

the end of Algebra and Geometry courses regardless of the student’s grade. (Some middle school students take both the math MSP and an EOC).

• The Writing and Science MSP and HSPE were not included in any of the following analyses.

• A score of 400 is proficient in Reading & Math/EOC.

Highline’s Regression Study• In 2007, School and District Administration had

been requesting ways to interpret student MAP scores in context of (then) WASL testing. One concern in particular was that students had been above average on the national norms, but yet were not meeting standard on the state assessment.

•School staff also requested a way to quickly identify if a student was on track or not.

Highline’s Regression Study•Decided to do a regression analysis to predict

WASL performance. •Ran correlations on multiple variables, and

found that “HiMap” (max of last 3 test administrations) had a higher correlation with WASL than a single MAP score. •Weeds out test “bombs” and missing data

SPRINGHIMAP

5th Grade

WinterHIMAP

5th Grade

FallHIMAP

5th Grade

“HIMAP” Variable Defined

Highline’s Regression Study•Rather than make a straight out prediction of

whether a student will meet/not meet standard, we wanted to emphasize the possible prediction error.

•Decided to find a cut on the MAP assessment to predict 400 on WASL, and then generate an error band around that where students would be considered “too close to call”

•Used 4 points as a generous estimate of the standard error of the assessment (usually between 3-3.5)

Intervention Categories: 3 “Bands”• “Above Benchmark” students were those who performed

more than 4 RIT points above the cut score. These students are considered on track to meet standard.

• “Strategic” students were those who performed within 4 RIT points of the cut. These students are “too close to call” and should receive strategic intervention to meet standard.

• “Intensive” students were those who performed more than 4 RIT points below the cut score. These students are unlikely to meet standard without intensive intervention.

Cuts for Fall, Winter and Spring• When the study was first done in 2008, regression

analyses were performed using Spring MAP scores and WASL.

• Growth norms were utilized to back track to get cuts for Fall and Winter

• Cut scores and ranges were disseminated to teachers and administrators, along with an explanation of the scores.

• Excel files for schools began including MAP scores, along with each students’ “BSI Indicator”, color coded in Red, Yellow and Green.

Predictive Validity• When a student’s indicator is compared to their actual

performance: • Approximately 90% of students identified as “Above

Benchmark” actually met standard. • Approximately 50% of students identified as

“Strategic” actually met standard. • Approximately 10% of students identified as

“Intensive” actually met standard.

• These were generally true within about 10 percentage points

2010 MSP•The analysis was re-run in 2010 following the

first year of transition from WASL to MSP. •During the second analysis, regressions were

run on each test window individually in each grade level, finding individual cuts, rather than using growth norms.

•District budget cuts made high school MAP testing optional, and therefore High School was excluded.

NWEA’s Linking Study•Most recently updated in Feb, 2011•Based on a sample of 271 schools in the Spring

of 2010•NWEA uses an Equi-percentile method to

equate test results

Equipercentile Method of Alignment

• NWEA used a sample of students from 271 schools taking the 2010 spring assessment in WA.

• For each grade and subject, identify the percentage of students in the study sample that met standard.

• For each grade and subject, identify the RIT associated with the equivalent percentile from within the study sample.

“If 40% of the study population in grade 3 math performed below the proficient level on the state test, we would find the RIT score that would be equivalent to the 40th percentile for the study population”

Multi-District Regression Study• Included 7 districts including Seattle,

Bellingham, Vancouver, Highline, Sumner, Auburn, and Clover Park

•Data covered the 2009-10 and 2010-11 academic years

•The “cut score” for proficiency was consistent across both years at each grade level, so data from both years was pooled

•Overall N of approximately 80,000

Independent Variables Created•Math Spring RIT (Winter and Fall as well)•Math Spring HIMAP (Winter and Fall as well)•Combined Spring HIMAP (sum of Read & Math)

(Winter and Fall as well)•Math Winter HIMAP + Math MSP•Math Fall HIMAP + Math MSP

(Comparable variables were also created for Reading)

Quality of Correlation

Best: (Corr: 0.78) • Spring RIT (but no predictive value, so Spring

indicators will be ignored)Next Best: (Corr: 0.73-0.75)• Winter RIT• Winter HIMAP + MSP scale score (275-500)• Winter HIMAP

Third Best: (Corr: 0.70)• Read Winter HIMAP + Math Winter HIMAP

Rationale for Selecting Winter HIMAP•Spring MAP test window overlaps MSP/HSPE

test window.•Prior Year MSP scores not available for grades 3

and 10.•New students in district are missing MSP

scores.•Not all students perform to their best ability on

every test. •Many students do not take the Winter MAP.

Rationale for Selecting Winter HIMAPWinter HIMAP …• Is not very different in the quality of the

correlation as compared to other options, •Maximizes the number of students for whom it

can be applied, and • Is relatively easy to explain

Failed MSP Passed MSP Total

Predicted Would Fail 3,699 790 4,489

Predicted Would Pass 1,036 7,298 8,334

Total 4,735 8,088 12,823

Failed MSP Passed MSP Total

Predicted Would Fail 29% 6%

Predicted Would Pass 8% 57%

Total 100%

Predictive Validity, using Multi-Dist ModelFourth Grade Reading Red circles on

students that were predicted accurately

Blue circle on students that were “under-estimated”

Purple circle on students that were “over-estimated”

Predictive Validity of Winter Score

READING Multiple Districts NWEA Highline

AccurateOver-Est. State Perf

Under-Est. State Perf Accurate

Over-Est. State Perf

Under-Est. State Perf Accurate

Over-Est. State Perf

Under-Est. State Perf

Grade 3  85% 7% 8% 84% 10% 6% 84% 10% 6%Grade 4  85% 6% 9% 85% 7% 8% 84% 10% 6%Grade 5  84% 6% 10% 84% 7% 9% 83% 10% 7%Grade 6  84% 7% 9% 84% 10% 7% 82% 12% 6%Grade 7  82% 10% 9% 82% 7% 12% 82% 10% 9%Grade 8  84% 8% 8% 84% 7% 9% 82% 13% 6%Grade 10  86% 4% 10% 85% 9% 6% n/a n/a n/a

MATH Multiple Districts NWEA Highline

AccurateOver-Est. State Perf

Under-Est. State Perf Accurate

Over-Est. State Perf

Under-Est. State Perf Accurate

Over-Est. State Perf

Under-Est. State Perf

Grade 3  83% 7% 9% 83% 10% 7% 83% 10% 7%Grade 4  84% 7% 10% 83% 12% 5% 84% 10% 6%Grade 5  85% 7% 8% 83% 13% 4% 83% 10% 7%Grade 6  86% 6% 9% 85% 10% 5% 86% 7% 7%Grade 7  85% 5% 10% 86% 9% 6% 84% 11% 4%Grade 8  85% 6% 9% 85% 8% 7% 85% 10% 5%

Predictive Validity: Percent of Students Meeting Standard by Band

READINGMulti-District NWEA Highline

Likely Not Proficient 15% 17% 20%At Risk 55% 59% 65%Likely Proficient 92% 93% 94%

MATHMulti-District NWEA Highline

Likely Not Proficient 10% 14% 14%At Risk 50% 62% 61%Likely Proficient 92% 95% 95%

10%-20% of “Likely Not Proficient” Students Met Standard.

50%-65% of “At Risk Students Met Standard.

90%-95% of “Likely Proficient” Students Met Standard.

Pro and Con of Do-It-YourselfPro:• Data are based on “our kids” (this is an emotional

argument, not a statistical one).• Winter and prior spring estimates can be computed

rather than estimated.

Con:• It is a lot of work.• Controlling for test windows is complex.• NWEA results are very similar to DIY results.• Teachers who encounter the NWEA linking study will be

confused about why our cut points are different.• … and did I say it was a lot of work?

Vancouver’s Plan to Move Forward•Use NWEA-published linking study to identify

cut-point targets in each grade/ testing window.• Identify students as likely to meet standard, at

risk, and not likely to meet standard based on their HIMAP RIT for that period and a 4 point band around the NWEA targets.

•Estimate winter values based on the mid point between fall and spring. Estimate prior spring equal to subsequent fall value (no summer drop-off).

Applying the Results: Vancouver•Teacher tables with color coding to identify

which students are likely to meet or not meet standard on the MSP/HSPE

•Predictions of the number of students the district might expect to meet standard if no changes are made to the pace of student learning during the year.

•Maintain a higher priority in the use of MAP to identify individual student learning needs and target instruction (using DesCartes)

ID#LAST NAME

FIRST NAME Grade

Read Met Growth Target 2012

Read Fall RIT

Read Fall

Pctile

MSP 2013 Read Categ.

MSP 2013 Read Odds

Read Spring Target

Math Met Growth Target 2012

Math Fall RIT

Math Fall

Pctile

MSP 2013 Math

Categ.

MSP 2013 Math Odds

Math Spring Target

10944 5 Yes 202 34 46% 207 No 196 12 17% 203

13455 5 No 202 34 46% 207 Yes 208 40 36% 216

13980 5 Yes 215 73 75% 219 No 228 88 81% 235

17713 5 No 217 78 80% 221 No 215 61 52% 223

17716 5 No 192 14 24% 199 Yes 184 3 5% 192

17719 5 No 206 45 54% 211 No 204 29 27% 212

17728 5 Yes 211 61 66% 215 Yes 208 40 36% 216

17732 5 Yes 213 67 70% 217 Yes 208 40 36% 216

17736 5 Yes 216 76 78% 220 Yes 227 87 80% 234

17804 5 Yes 203 36 48% 208 Yes 217 66 56% 224

18312 5 Yes 205 42 52% 210 Yes 201 21 22% 208

18328 5 Yes 212 64 68% 216 Yes 202 24 24% 210

18578 5 No 203 36 48% 208 Yes 201 21 22% 208

18624 5 Yes 216 76 78% 220 Yes 206 34 32% 214

19057 5 No 225 93 89% 228 Yes 212 52 46% 220

19128 5 Yes 217 78 80% 221 Yes 222 78 68% 229

21036 5 No 176 2 5% 186

24125 5 No 215 73 75% 219 Yes 210 46 40% 218

26414 5 No 194 17 27% 201 Yes 209 43 38% 217

27807 5 No 180 4 8% 189 Yes 185 3 5% 193

30737 5 No 205 42 52% 210 No 209 43 38% 217

36075 5 No 201 31 43% 206 No 185 3 5% 193

36376 5 Yes 171 1 3% 182 191 7 9% 199

41166 5 No 184 6 12% 192 No 197 14 18% 204

43584 5 No 230 97 93% 233 Yes 224 82 72% 231

46978 5 Yes 197 22 34% 203 Yes 197 14 18% 204

How Does MAP Predict State Test Performance?

Understanding, Conducting, and Using Alignment Studies

Vancouver Public Schools and Highline Public Schools in Washington State

Presenters:Paul Stern Paul.Stern@vansd.orgSarah JohnsonSarah.Johnson@highlineschools.org