multiple measures, assessment, and the re-imagination of student capacity february 16, 2016 craig...

23
MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation Irvine Valley College

Upload: alan-harper

Post on 18-Jan-2018

217 views

Category:

Documents


0 download

DESCRIPTION

3

TRANSCRIPT

Page 1: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT

CAPACITYFebruary 16, 2016

Craig HaywardDirector of Research, Planning and

AccreditationIrvine Valley College

Page 2: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

OVERVIEW• Standardized assessment, in its current form, has led us to

systematically underestimate student capacity

• Underestimating students is not cost-free

• Evidence-based, strong multiple measures provide a key tool in the effort to transform basic skills in California and nationally

• Powerful completion, equity, and real world implications

• Based on years of substantial, convergent research in California and in other states

• Multiple measures can greatly increase placement accuracy

Page 3: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

3

Page 4: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

CURRENT PRACTICE• CCCs reliant on standardized assessment (REL West, 2011) http://bit.ly/20wYEZT

• 97% of CCCs place students with a standardized assessment

• Majority of students are placed below college-level• 68% of students in two year institutions take developmental education course

(Scott-Clayton & Belfield, 2015). http://bit.ly/CCRCPlacementAccuracy

• >85% of CCC students place into basic skills http://bit.ly/11_12_Basic_Skills_Report

• Most CCCs collect multiple measures data. Few leverage it.

• 81% collect information on high school performance

• 65% collect high school GPA

• MMs are typically assigned a relatively minor weight (e.g., Ngo & Kwon, 2014) http://bit.ly/1KP8NPZ

Page 5: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

WHAT DOES THIS MEAN FOR STUDENTS?

• First interaction with college is basically a statement of our lack of confidence in them

• Implied message: You are not ready for college and are likely to fail

• ~30% of those assigned to remediation never attempt a course in the sequence

Page 6: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

COULD CONVENTIONAL WISDOM BE WRONG?

• Research increasingly questions effectiveness of standardized assessment as the only measure to understanding student capacity• Little relation to college course outcomes

• (e.g., Belfield & Crosta, 2012; Edgecombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012): bit.ly/CCRCAssess

• Underestimates capability of students of color, women, first generation college students, low SES• Hiss & Franks, 2014; bit.ly/DefiningPromise

Page 7: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

A NEW HOPE

• There is a powerful, easily deployed multiple measures system that will work seamlessly with your current placement test and with the incoming common assessment system

• A team of researchers from the RP Group, Educational Results/Cal-PASS Plus and the University of Michigan has spent three years developing and refining this system

• There is a $1.5 million grant to help you get this system going

Page 8: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

MULTIPLE MEASURES ASSESSMENT PROJECT (MMAP)

• Predicting successful completion of English & math in the CCCs based on high school performance data

• Focus on predictive validity (success in course) and improving student completion of foundational skills

• Statewide support• Research base, predictive analytics, decision tree models• Pilot colleges and faculty/staff engagement

• Webinars, convenings/summits, professional development• K-12 outreach and data population• Data warehouse and tool development• Integration with Common Assessment System

Page 9: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

GROWING INTEREST AND SCALE

• Colleges continue to join the project and enthusiastically inquire about participating• 41 pilot colleges now committed, 8 more at various stages of

exploration, representing more than:• >900,000 community college students• >40% of community college students statewide• >8% of all community college students nationally

• 11 had pilots in place for Fall 2015• 10 colleges submitted prospective applicants for Spring 2016

Page 10: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

English (n=103,510)

Math (n=143,253)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

38%31%

61%

42%

Placement into Transfer-level with Current Placement Systems vs. Strong Multiple Measures

(Statewide)

Current Disjunctive MM

Page 11: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

11

3 Levels below 2 Levels below 1 Level below Transfer+0%

10%20%30%40%50%60%70%80%

7%

25% 23%

44%

2%

21%16%

61%

Math(incoming cohort Fall

2015, placement)

Test State MMAP disjunctive

Page 12: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

Afr Am Latino Asian White0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

24%30%

41%

53%

40%

51%

73% 74%

Transfer Level English Placement by EthnicityCurrent Disjunctive MM

Page 13: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

Afr Am Latino Asian White0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

15%21%

41%

51%

22%32%

53%

65%

Transfer Level Math Placement by EthnicityCurrent Disjunctive MM

Page 14: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

SAMPLE PHASE II RULE SETS

English Level Rule (NDMs)Transfer HS 12 GPA >= 2.6

One level below Transfer

HS 12 GPA >= 2.2 ANDC or better in 12th grade English course

Math Level Rule (DMs)College Algebra

HS 12 GPA >=3.2 OR HS 12 GPA >=2.9 ANDPre-Calculus C or better

Intermediate Algebra

HS 12 GPA >=2.9 OR HS 12 GPA >=2.5ANDAlgebra II CST >= 302

Find the full rule set here: http://bit.ly/PhaseII_MMAP_rules

Page 15: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

STATISTICS TREE – NDM

Page 16: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

DISJUNCTIVE APPLICATION OF MULTIPLE MEASURES

Disjunctive Placement

Test score

OR

High School Transcript

OR

AP score

OR

EAP score

OR

Essay score

Conjunctive Placement

Test score

AND

(High School Transcript

OR

AP score

OR

EAP score

OR

Essay Score)

Page 17: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

PROJECTED IMPACT ON COURSE SUCCESS RATES

(COMPLETION OF COURSE WITH C OR BETTER)

Transfer-level Math Transfer-Level English0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

62%72%

62%71%

Historic success rate Projected success rate

Succ

essf

ul c

ompl

etion

of

tran

sfer

-leve

l cou

rse

Page 18: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

SAN DIEGO CCD MMAP F2015 PILOT

English Math0%

10%

20%

30%

40%

50%

60%

70%

80%

24%28%

58%

68%

Accuplacer Accuplacer + MM

http://bit.ly/MMAPPilot

Page 19: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

LBCC MULTIPLE MEASURES RESEARCH• Five cohorts tracking more than 7,000 HS grads who matriculated to

LBCC directly

• Examined predictive utility of wide range of high school achievement data for predicting:

• How students are assessed and placed• How students perform in those classes

Page 20: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

RE-IMAGINED STUDENT CAPACITY

• Starting in Fall 2012, students were provided an alternative assessment• Reverse-engineered the analysis to place students using:• Last high school course in discipline• Grade in last course in discipline• Overall HSGPA• Last standardized test in discipline (and level)

• Placed students in highest course where projected success rate higher than average success rate for that course.

Page 21: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

LBCC COHORT 3 (FALL 2014): SUCCESS RATES IN TRANSFER-LEVEL COURSES

English difference, p < .001

Page 22: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

WHAT MIGHT THIS MEAN FOR STUDENTS?

• LBCC saved students over 10,000 semesters of unneeded remediation in first three years.

• $250 per course for student (plus books!), $750 per course for state

• At scale (113 CCs) this could lead to 250,000 years of time, each year, saved for our students

• Dramatic opportunity costs of college reduced• Median 2012 salary of “some college” is ~$30,000/year• Don’t lose their first year or median salary though, they lose their last

year.

Page 23: MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation

THANK YOU!• For more information about MMAP visit: http://

bit.ly/About_MMAP

• And/or contact:• Mallory Newell, Project Manager, MMAP (

[email protected])

or• Ken Sorey, CalPASS Plus ([email protected])