multiple measures, assessment, and the re-imagination of student capacity february 16, 2016 craig...

MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT

CAPACITYFebruary 16, 2016

Craig HaywardDirector of Research, Planning and

AccreditationIrvine Valley College

OVERVIEW• Standardized assessment, in its current form, has led us to

systematically underestimate student capacity

• Underestimating students is not cost-free

• Evidence-based, strong multiple measures provide a key tool in the effort to transform basic skills in California and nationally

• Powerful completion, equity, and real world implications

• Based on years of substantial, convergent research in California and in other states

• Multiple measures can greatly increase placement accuracy

3

CURRENT PRACTICE• CCCs reliant on standardized assessment (REL West, 2011) http://bit.ly/20wYEZT

• 97% of CCCs place students with a standardized assessment

• Majority of students are placed below college-level• 68% of students in two year institutions take developmental education course

(Scott-Clayton & Belfield, 2015). http://bit.ly/CCRCPlacementAccuracy

• >85% of CCC students place into basic skills http://bit.ly/11_12_Basic_Skills_Report

• Most CCCs collect multiple measures data. Few leverage it.

• 81% collect information on high school performance

• 65% collect high school GPA

• MMs are typically assigned a relatively minor weight (e.g., Ngo & Kwon, 2014) http://bit.ly/1KP8NPZ

WHAT DOES THIS MEAN FOR STUDENTS?

• First interaction with college is basically a statement of our lack of confidence in them

• Implied message: You are not ready for college and are likely to fail

• ~30% of those assigned to remediation never attempt a course in the sequence

COULD CONVENTIONAL WISDOM BE WRONG?

• Research increasingly questions effectiveness of standardized assessment as the only measure to understanding student capacity• Little relation to college course outcomes

• (e.g., Belfield & Crosta, 2012; Edgecombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012): bit.ly/CCRCAssess

• Underestimates capability of students of color, women, first generation college students, low SES• Hiss & Franks, 2014; bit.ly/DefiningPromise

A NEW HOPE

• There is a powerful, easily deployed multiple measures system that will work seamlessly with your current placement test and with the incoming common assessment system

• A team of researchers from the RP Group, Educational Results/Cal-PASS Plus and the University of Michigan has spent three years developing and refining this system

• There is a $1.5 million grant to help you get this system going

MULTIPLE MEASURES ASSESSMENT PROJECT (MMAP)

• Predicting successful completion of English & math in the CCCs based on high school performance data

• Focus on predictive validity (success in course) and improving student completion of foundational skills

• Statewide support• Research base, predictive analytics, decision tree models• Pilot colleges and faculty/staff engagement

• Webinars, convenings/summits, professional development• K-12 outreach and data population• Data warehouse and tool development• Integration with Common Assessment System

GROWING INTEREST AND SCALE

• Colleges continue to join the project and enthusiastically inquire about participating• 41 pilot colleges now committed, 8 more at various stages of

exploration, representing more than:• >900,000 community college students• >40% of community college students statewide• >8% of all community college students nationally

• 11 had pilots in place for Fall 2015• 10 colleges submitted prospective applicants for Spring 2016

English (n=103,510)

Math (n=143,253)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

38%31%

61%

42%

Placement into Transfer-level with Current Placement Systems vs. Strong Multiple Measures

(Statewide)

Current Disjunctive MM

11

3 Levels below 2 Levels below 1 Level below Transfer+0%

10%20%30%40%50%60%70%80%

7%

25% 23%

44%

2%

21%16%

61%

Math(incoming cohort Fall

2015, placement)

Test State MMAP disjunctive

Afr Am Latino Asian White0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

24%30%

41%

53%

40%

51%

73% 74%

Transfer Level English Placement by EthnicityCurrent Disjunctive MM

Afr Am Latino Asian White0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

15%21%

41%

51%

22%32%

53%

65%

Transfer Level Math Placement by EthnicityCurrent Disjunctive MM

SAMPLE PHASE II RULE SETS

English Level Rule (NDMs)Transfer HS 12 GPA >= 2.6

One level below Transfer

HS 12 GPA >= 2.2 ANDC or better in 12th grade English course

Math Level Rule (DMs)College Algebra

HS 12 GPA >=3.2 OR HS 12 GPA >=2.9 ANDPre-Calculus C or better

Intermediate Algebra

HS 12 GPA >=2.9 OR HS 12 GPA >=2.5ANDAlgebra II CST >= 302

Find the full rule set here: http://bit.ly/PhaseII_MMAP_rules

STATISTICS TREE – NDM

DISJUNCTIVE APPLICATION OF MULTIPLE MEASURES

Disjunctive Placement

Test score

OR

High School Transcript

OR

AP score

OR

EAP score

OR

Essay score

Conjunctive Placement

Test score

AND

(High School Transcript

OR

AP score

OR

EAP score

OR

Essay Score)

PROJECTED IMPACT ON COURSE SUCCESS RATES

(COMPLETION OF COURSE WITH C OR BETTER)

Transfer-level Math Transfer-Level English0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

62%72%

62%71%

Historic success rate Projected success rate

Succ

essf

ul c

ompl

etion

of

tran

sfer

-leve

l cou

rse

SAN DIEGO CCD MMAP F2015 PILOT

English Math0%

10%

20%

30%

40%

50%

60%

70%

80%

24%28%

58%

68%

Accuplacer Accuplacer + MM

http://bit.ly/MMAPPilot

LBCC MULTIPLE MEASURES RESEARCH• Five cohorts tracking more than 7,000 HS grads who matriculated to

LBCC directly

• Examined predictive utility of wide range of high school achievement data for predicting:

• How students are assessed and placed• How students perform in those classes

RE-IMAGINED STUDENT CAPACITY

• Starting in Fall 2012, students were provided an alternative assessment• Reverse-engineered the analysis to place students using:• Last high school course in discipline• Grade in last course in discipline• Overall HSGPA• Last standardized test in discipline (and level)

• Placed students in highest course where projected success rate higher than average success rate for that course.

LBCC COHORT 3 (FALL 2014): SUCCESS RATES IN TRANSFER-LEVEL COURSES

English difference, p < .001

WHAT MIGHT THIS MEAN FOR STUDENTS?

• LBCC saved students over 10,000 semesters of unneeded remediation in first three years.

• $250 per course for student (plus books!), $750 per course for state

• At scale (113 CCs) this could lead to 250,000 years of time, each year, saved for our students

• Dramatic opportunity costs of college reduced• Median 2012 salary of “some college” is ~$30,000/year• Don’t lose their first year or median salary though, they lose their last

year.

THANK YOU!• For more information about MMAP visit: http://

bit.ly/About_MMAP

• And/or contact:• Mallory Newell, Project Manager, MMAP (

newellmallory@fhda.edu)

or• Ken Sorey, CalPASS Plus (ken@edresults.org)