A few things you might not know

about the Michigan School

Accountability Index

MERA Spring 2018 Conference

Dave Treder

Adequate Growth Percentiles – how accurate are the On Track to Proficiency growth projections?

We have 6 years where we gave the SAME test –MEAP 2008 to 2013, so….

We can investigate the Accuracy of the On Track To Proficiencydesignation by looking at:

1) 2008-2010, Grades 3 to 5, and compute which kids are projected to be On Track to Proficiency;

and then (because have the data),

2) determine the accuracy of these On Track to Proficiencypredictions, based on actual Grade 8 proficiency in 2013.

Scal

e Sc

ore

Visually, what this would look like

Proficiency

Adequate Growth Targets(trajectory needed to be proficient in 3 years)

Student Growth Percentiles (SGPs)(actual "growth" from gr. 3 to gr. 5)

Gr. 3 Gr. 4 Gr 52008 2009 2010

Gr. 6 Gr. 7 Gr 82011 2012 2013

Scal

e Sc

ore

Proficiency

Adequate Growth Targets(trajectory needed to be proficient in 3 years)

Student Growth Percentiles (SGPs)(actual "growth" from gr. 3 to gr. 5)

Projected Catching Up: NO(missed target)

Gr. 3 Gr. 4 Gr 52008 2009 2010

Gr. 6 Gr. 7 Gr 82011 2012 2013

Visually, what this would look like

Scal

e Sc

ore

Proficiency

Adequate Growth Targets(trajectory needed to be proficient in 3 years)

Student Growth Percentiles (SGPs)(actual "growth" from gr. 3 to gr. 5)

Projected Catching Up: NO(missed target)

Gr. 3 Gr. 4 Gr 52008 2009 2010

Gr. 6 Gr. 7 Gr 82011 2012 2013

Actual gr. 8 Score(student met proficiency)

ACTUAL Catching Up: YES

(student IS proficient)

Visually, what this would look like

Gr 5, 2010, w/math score: 32,333w/SGPs: 29,504

Gr 8, 2013, math score: 32,254W/2010 Gr 5 SGPs: 25,901

CATCH_UP_KEEP_UP_

STATUS Frequency Percent

Catch Up: No 8,494 33%

Catch Up: Yes 3,823 15%

Keep Up: No 6,954 27%

Keep Up: Yes 6,630 26%

Descriptive Statistics

Grade 5, 2010, Catch Up / Keep Up Status

Students with MATH MEAP scores, Fall 2008 – Fall 2013• Data from Genesee, Lapeer, Macomb, Oakland, & Ottawa ISDs

Counts

MI tests around 100,00 kids per grade, so we're looking at (around) 1/3 of the state.

How many kids, classified in gr. 5 as Catching Up (or Not Catching Up) did, indeed, catch up (or not catch up)?

Gr 8 Proficiency N PCT N PCT

Prof (level 1 or 2) 2709 71% 676 8%

Not Prof (level 3 or 4) 1114 29% 7818 92%

"Catching Up" (N = 3,823)

Accuracy of On Track to Proficiency growth projections

Looking at Students' Proficiency Status on Grade 8 Spring 2013 MEAP Test

"Not Catching Up" (N = 8,494)

Gr 8 Proficiency N PCT N PCT

Prof (level 1 or 2) 5557 84% 3743 54%

Not Prof (level 3 or 4) 1073 16% 3211 46%

"Keeping Up" (N = 3,823) "Not Keeping Up" (N = 6,954)

|---------------------- Grade 5 Projections -------------------------------|

How many kids, classified in gr. 5 as Keeping Up (or Not Keeping Up) did, indeed, keep up (or not keep up)?

Reasonable Hit Rate…

ACTUAL

PROJECTEDCatch Up/Keep Up

Not Catching-up/Keeping up

Catch/Keep Up 32% 17%

Not Catching-up/Not Keeping up

8% 43%

Accuracy of On Track to Proficiency growth projections

Classification Accuracy

What do we see when we look at the accuracy rates, between projected and actual, looking as individual grade 5 scale scores?

Which non-proficient kids had a greater probability of reaching proficiency, those who were projected to "catch up" or those who weren't?

Probability of Gr. 8 Proficiency, based on Gr 5 Scale Scoreand Gr 5 "Catch Up Status" (YES or NO)

CATCH_UP

_STATUS*

N

Gr 8 Proficiency Gr. 5 SCALE_SCORE

Mean Std Dev Mean Std Dev

Catch Up: No 8494 0.08 0.27 505 13.5

Catch Up: Yes 1573 0.12 0.32 523 6.1

Least Squares Means, adjusting for grade 5 Scale Score

CATCH_UP

STATUS*

Gr 8 Proficiency

LSMEAN

H0:LSMean1=LSMean2

Pr > |t|

Catch Up: No 0.094 <.0001

Catch Up: Yes 0.042

• Kids are 2.3 times more likely to reach proficiency in Grade 8, if they were classified AS NOT catching up, compared the to kids who were classified AS catching up.

or, as medical research often frames it:

• Kids expected NOT to catch up have, in fact, a 126% greater chance of catching up than the kids expected TO catch up.

*Non-proficient 5th graders

*Non-proficient 5th graders

Analysis, "Catch Up: No" vs "Catch Up: Yes" and Grade 8 Proficiency

Did we do any better with the Keep-Up Projections?

Which proficient kids had a greater probability of staying proficient, those who were "keeping up" or those who weren't?

Probability of Gr. 8 Proficiency, based on Gr 5 Scale Scoreand Gr 5 "Keep Up Status" (YES or NO)

KEEP_UP

_STATUS*

N

Gr 8 Proficiency Gr. 5 SCALE_SCORE

Mean Std Dev Mean Std Dev

Keep Up: No 4040 0.71 0.45 545 10.8

Keep Up: Yes 6630 0.83 0.37 572 24.2

Least Squares Means, adjusting for grade 5 Scale Score

KEEP_UP

_STATUS*

Gr 8 Proficiency

LSMEAN

H0:LSMean1=LSMean2

Pr > |t|

Keep Up: No 0.82 <.0001

Keep Up: Yes 0.77

*Proficient 5th graders

*Proficient 5th graders

Not nearly as dramatic:

Analysis, "Keep Up: No" vs "Keep Up: Yes" and Grade 8 Proficiency

• Kids are 1.1 times more likely to stay proficiency in Grade 8, if they were classified as NOT keeping up, compared the to kids who were classified AS keeping up.

-- But, it's nearly a crap-shoot, when looking at ALL kids who are classified as NOT Keeping Up, as to whether they will be proficient or not: 46% Not Proficient, 54% Proficient

(the analysis above only includes grade 5 proficient kids; there were around three thousand kids NOT proficient in grade 5, but proficient in grade 4, so classified as NOT Keeping Up)

-- And, still, kids projected NOT to be Keeping Up are MORE likely to be proficient than kids projected to be proficient.

Before I go on, thought I would stop and see if I could elicit some Questions/Comments…

So, what is it we're measuring?

• 97% of the Variance between-School "Growth" can be explained by Percent Proficient

Statewide, Middle Schools (n=471)

The Michigan Index System labels this a "Growth Index"

r = .983

Let me repeat:97% of the variance in between-School "Growth" can be explained by Percent Proficient

• To put this in perspective, a summary of 28 alternate-form reliability estimates of the SAT showed reliability values of .88 to .91 -- so, 80% to 83% of the between-test variance,

on parallel forms of the SAT, is reflected in the other – on two tests purported to measure the exact same thing.

Almost axiomatic (given that 97% of stuff measured by "growth" can be explained by proficiency): the relationship between growth and poverty is pretty much indistinguishable – i.e., the same, not different, statistically equivalent, etc. – from the relationship between proficiency and poverty

r = .80 r = .83

The relationship between growth and poverty and the relationship between proficiency and poverty

Subject = Math Type = Middle School

So, could we please stop the pretense that this Index is

measuring "growth"?

Probably running out of time, but one probability plot

What's the probability of meeting the Growth Target, based on Scale Score?Grade = 8 Subject = Math

PCT MET ProbabilityTarget at Cut Score

5 kids, SS > 1800 (1801-04) ,not meeting target (out of 8646)

40 kids, SS < 1800, meeting Target(all 40 had SGP = 99)

SCHOOL_EMPHASIS Frequency Percent

CumulativeFrequency

Alternative Education 57 35% 57

General Education 104 64% 161

PSA_LEA Frequency

ALT 57

LEA 71

PSA 33

PSA_LEA Type Frequency Percent

Cumulative

Frequency

LEA ELEMENTARY 9 8% 9

LEA ELEMENTARY/MIDDLE SCHOOL 34 33% 43

LEA MIDDLE SCHOOL 3 3% 46

LEA MIDDLE SCHOOL/HIGH SCHOOL 2 2% 48

LEA HIGH SCHOOL 21 20% 69

LEA K-12 2 2% 71

54 of the 71 (76%) are Detroit/EEA schools

28 "official" Alt Ed schools,another 29 pretty much for sure

(i.e.,Dream Academy Alternative School of Choice,

Battle Creek Alternative School,Davison Alternative Education)

Who are We Identifying as CIS Schools?

one more, if we have time

All Done…