Multiple Measures Assessment Project

Welcome WebinarSpring 2014

• Multiple Measures and Common Assessment• Overview of analysis • Student Transcript Enhanced Placement Study (STEPS)

findings leading to Multiple Measures Project (MMAP)• Working with K-12 partners• Compass or Accuplacer data• Role of the pilot colleges • Projected timeline • Questions and Answers

Agenda

Multiple Measures and Common Assessment Initiative• Role of the Common Assessment Advisory

Group

Overview of Analysis• Examine the value of using transcripts as part of the

assessment process• Create predictive models using intersegmental data

to study students who had already taken community college courses

• Use models to analyze how well transcript data predicts the first English & math courses students take and how well they do in them

• Recruit colleges to do local analyses to generate campus-specific insights and trigger CCC/K-12 conversations

4Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

Student Transcript Enhanced Placement Study (STEPS) –

Statewide Findings

In English tests predict tests, grades and courses matter but vary by college

6

7

Cox & Snell pseudo R-square ~ 0.35

Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

0123456789

1011

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HS GPA*

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce

Predictor Variable Category

Predicting College English Level

Weakest

Intermediate

Strongest

In math tests predict tests,but high school level is also important

8

9

Cox & Snell pseudo R-square ~ 0.50

Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

0123456789

1011

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HS GPA*

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce

Predictor Variable Category

Predicting College Math Level

Weakest

Intermediate

Strongest

In English, grades predict grades

10

11

Cox & Snell pseudo R-square ~ 0.20

Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

0123456789

1011

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HSGPA*

CollegeCourseLevel

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce

Predictor Variable Category

Predicting College English Success

Weakest

Intermediate

Strongest

In math, success predictors vary by college

12

13

Cox & Snell pseudo R-square ~ 0.20

Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

0123456789

1011

CST's A-GCourses

HSCourseLevel

HSCourseGrade

HSGPA*

CollegeCourseLevel

Coun

t of C

olle

ges

Show

ing

Sign

ifica

nce

Predictor Variable Category

Predicting College Math Success

Weakest

Intermediate

Strongest

What Does This Mean?

• Using high school transcript data could help inform alignment/articulation and refine assessment policies

• The relative weight of variables might be influenced by local factors, due to considerations such as articulation and curriculum

• Barriers to implementation must be addressed such as processing transcripts

14Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

What Happens Next?• Updating the statewide analysis• Encouraging more colleges to replicate the study so

they understand the value of transcript data for their own students

• Starting conversations about how the study relates to other aspects of developmental education reform—such as alignment, course offerings, and curriculum

• Helping to develop a tool that enables colleges to access high school data to inform multiple measures assessment

15Harnessing the Power of Alternative Assessment to Advance the Completion Agenda

Working with K-12 partners to upload data• Key aspect of analysis using high school transcript data is

getting local K-12 data into Cal-PASS Plus

• Cal-PASS Plus can assist in outreach, though initially coming from a college helps

• Cal-PASS Plus and RP Group can help provide materials and templates

• Entire process to submit data is streamlined – drag and drop files they already create for the CA Dept. of Education

Compass or Accuplacer data- Does your college use Accuplacer or Compass

for student placements?- If not, we ask that you submit you college’s data

to Cal-PASS Plus to include in the analysis.

Projected timeline Jan – Mar: Build data warehouse for application to multiple measures analysis – collect various data sourcesMay – July: Complete research on system wide multiple measures models and algorithmsJuly – Sept: Develop and deploy online analytic tools and decision tree modelsOct – Dec: Provide PD and direct support to pilot colleges to test multiple measures tools and application; develop and test local user interface to re-identify data for placement decisionsDec – Ongoing: Provide cohort tracking data to pilot colleges on course outcomes; integrate multiple measures project with Common Assessment Initiative

Role of the pilot colleges • Test interactive online placement tools• Provide feedback on user experience• Compare student placements using MMAP analysis to

your college’s current system– How are students placing when comparing the two?– What new conversations have occurred about improving

the placement process?• Work with Cal-PASS Plus to track cohort outcomes

after applying multiple measures through the project

Questions?

Contacts:Mallory NewellThe RP Groupnewellmallory@fhda.edu

Ken Sorey,Cal-PASS Plusken@calpass.org

Amanda AvalloneCal-PASS Plusamanda@calpass.org

STEPS Report and Summaryhttp://www.rpgroup.org/projects/steps

MMAP Project Sitehttp://www.rpgroup.org/projects/multiple-measures-assessment-project

Cal-PASS Plushttps://www.calpassplus.org/