national alliance of concurrent enrollment partnerships - multiple … slides... · 2017. 10....

Multiple Measures Assessment of Student Readiness for College Courses

Elisabeth Barnett, CCRC and NCRESTJennifer Kim, NCREST

NACEPOctober 2017

Agenda

• The college-readiness assessment landscape and the emerging use of multiple measures assessment

• Middle and early colleges’ current assessment practices

• Discussion and Q&A

Organization of CAPR

CCRC MDRC

Descriptive Study of Developmental

Education

Evaluation of The New Mathways Project

(RCT in TX)

Evaluation of New Assessment Practices

(RCT in NY)

Supplemental Studies

Research on an Alternative Placement Strategy using Multiple Measures

• 5 year study; seven colleges; 10,000 students

• Alternative placement system

• Random assignment: half of students placed using algorithm

• Analysis of student outcomes and implementation.

4

Slides available at: bit.ly/capr_ashe16 5

RAPS – Partner Sites

A – CAPR/CCRC/MDRCB – Cayuga CCC – Jefferson CCD – Niagara County CCE – Onondaga CCF – Rockland CCG – Schenectady County CCH – Westchester CC

Predictive analytics used to develop an algorithm

Use data from previouscohorts

Develop formula to

predict student performance

Use formula to place entering

cohort of students

6

Why Use Multiple Measures Assessment?

7

Community college 8-year graduation rates (Attewell, Lavin, Domina, and Levey, 2006)

28%

43%

0%

10%

20%

30%

40%

50%

Students Needing Remediation Students Not Needing Remediation

8

Under-placement and Over-placement (severe)

Placement According to Exam

Developmental College Level

Stud

ent A

bilit

y

Developmental Over-placed(English – 5%)(Math – 6%)College Level

Under-placed(English – 29%)(Math – 18%)

9

COLLEGE 2: ENGLISH COLLEGE 2: MATH

10

9.9%

2.7%

12.0%

14.5%

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

GPA only Test only GPA and test Full model

3.8%

1.0%

4.8%

7.5%

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

GPA only Test only GPA and test Full model


Model R-Squared StatisticsEnglish

0

0.02

0.04

0.06

0.08

0.1

0.12

College 1 College 2 College 3 College 4 College 5 College 6 College 7

R-Squared Statistics – Graphical Representation

GPA ACCUPLACER GPA + ACCUPLACER Full Model


Model R-Squared StatisticsMath

0

0.05

0.1

0.15

0.2

0.25

College 1 College 2 College3 College 4 College 5 College 6 College 7

R-Squared Statistics – Graphical Representation

GPA ACCUPLACER GPA + ACCUPLACER Full Model

Conclusions so far

• Better assessment systems are needed.

• HS GPA is the best predictor of success in college math and English.

13

Multiple Measures Assessment

14

Why Use Multiple Measures

• Existing placement tests are not good predictors of success in college courses.

• More information improves most predictions.• Different measures may be needed to best

place specific student groups.

15

Current assessment practices in the US (preliminary; CAPR 2017)

% using….. Math English Standardized tests 88% 87%High school performance 40% 37%Planned course of study 29% 18%Other indicators of motivation 11% 14%No assessment done 6% 6%

16

Possible Measures

Type Examples

Placement test AccuplacerALEKS

High school GPA, course grades, test scores

From transcriptSelf-reportFrom SAT, SAT, SB, etc.

Non-cognitive assessments GRIT QuestionnaireSuccessNavigator or Engage

Career inventory, computer skills Kuder Career AssessmentHome grown computer skills test

Writing examples Faculty-assessed portfolioHome-grown writing assessment

Individual advisement

17

http://my.mdrc.org/http://my.mdrc.org/

Non-cognitive assessments

Development of non-cognitive skills promotes students’ ability to think cogently about information, manage their time, get along with peers and instructors, persist through difficulties, and navigate the landscape of college…(Conley, 2010).

Non-cognitive assessments may be of particular value for:

• Nontraditional (older) students.

• Students without a high school record.

• Students close to the cut-off on a test.

18

Multiple Measures Options (Barnett and Reddy, 2017)MEASURES SYSTEMS OR APPROACHES PLACEMENTS

Administered by college:1. Traditional or alternative

placement tests2. Non-cognitive assessments3. Computer skills or career

inventory4. Writing assessments5. Questionnaire items

Obtained from elsewhere:1. High school GPA2. Other HS transcript information

(courses taken, course grades)3. Standardized test results (e.g.,

ACT, SAT, Smarter Balanced)

• Waiver system• Decision bands• Placement formula

(algorithm)• Decision rules• Directed self-placement

• Placement into traditional courses

• Placement into alternative coursework

• Placement into support services

19

But what about…. • Students placed via multiple measures will not be successful.

• Our test is different/better/more awesome.

• High school GPA is only predictive for recent graduates.

• High schools grade differently.

• It’s too hard to get or use transcripts/it’s not worth it.

20

Davidson County, NC, 2013-15 Ivy Tech, IN, 2014-15

Students placed via multiple measures will likely be successful.

21

59%48%

76%65%

0%

20%

40%

60%

80%

100%

English Math

Comparison HS Data

Rules used for English and Math: HSGPA >=2.6 and completion of four years of mathematics including one year beyond Algebra 2 in HS

Rules used for English and Math: HSGPA >=2.6

FROM HETTS, 2016

57% 59% 57%64% 68% 64%

0%

20%

40%

60%

80%

100%

English Math Reading

Accuplacer HS Data

NC ENGLISH NC MATH

Our test is different/better/more awesome.

From Bostian (2016), North Carolina Waves GPA Wand, Students Magically College Ready adapted from research of Belfield & Crosta, 2012 – see also Table 1)

HS GPA is a better predictor than test results for long time (Hetts, 2016)

MMAP (in preparation): correlations b/w predictor and success (C or better) in transfer-level course by # of semesters since HS

For the most part, college grades stay parallel with feeder high school grades. (Bostian, 2016)

Sources of HS transcript data Self-report research

• The students bring a transcript.

• The high school sends.• Obtained from state data files.• Self report.

Note: Consider using the 11thgrade GPA.

• UC admissions uses self-report but verifies after admission. In 2008, at 9 campuses, 60,000 students. No campus had >5 discrepancies b/w reported grades and student transcripts (Hetts, 2016)

• College Board: Shawn & Matten, 2009: “Students are quite accurate in reporting their HSGPA”, r = .73.

• ACT research often uses self-reported GPA and generally find it to highly correlated with students actual GPA: ACT, 2013: r = .84.

25

Examples of alternative assessment systems

• Indiana

• California

• North Carolina

• New York

• Minnesota

• Wisconsin

26

IVY TECH, INDIANA – WAIVERS (CCCSE, 2016)

Ivy Tech Community College (IN) has been using a multiple measures placement policy for degree-seeking students since 2003.

• Students may submit any of the following documents: ACT, SAT, or PSAT scores for tests taken within the past four years

• The college added the acceptance of high school GPA to the policy for students entering the college in fall 2014.

• Students who do not have the above documentation are required to take the college’s custom ACCUPLACER diagnostic assessment.

27

IVY TECH PASS RATES

28

CALIFORNIAMMAP Project (Hetts, 2016)• Collaborative effort of CCCCO, Common Assessment Initiative (CAI), Cal-

PASS Plus, RP Group and ~45 CCC pilot colleges*• Identify, analyze, & validate multiple measures data

• For English, Mathematics, ESL and Reading• Focus on predictive validity (success in course) • Key variables included HSGPA, last course in discipline, course grade

or level, AP course-taking, CST scores, etc.

bit.ly/MMAP2015 and http://bit.ly/MMAPRules

29

http://bit.ly/MMAP2015http://bit.ly/MMAPRules

Example– CA Math PlacementLevel Direct Matriculants (from HS) Non-Direct MatriculantsCalculus I

Passed Precalculus or Trigonometry (or better)

11th-grade GPA ≥ 3.6

11th-grade GPA ≥ 3.2 and Precalculus C (or better)

12th-grade GPA ≥ 3.1 and took Calculus


Precalculus

Passed Algebra II (or better)


11th-grade GPA ≥ 2.6 and took Calculus


12th-grade GPA ≥ 3 and Algebra II California Standards Test ≥ 340

12th-grade GPA ≥ 3 and Calculus C (or better)

Trigonometry

Passed Algebra II (or better)


11th-grade GPA ≥ 3 and Precalculus C+ (or better)

11th-grade GPA ≥ 3 and Algebra II B (or better)



College Algebra

( )




12th-grade GPA ≥ 3.0 and (

30

NORTH CAROLINA (Bostian, 2016)

The North Carolina Community College System has adopted a multiple measure placement hierarchy.• The 58 colleges began implementing in 2013.• All colleges required to implement by fall 2016.

CRITERIA1. Unweighted HS GPA 2.6 + 4 college prep math courses2. ACT/SAT at national benchmark scores3. NCDAP placement test

31

Relevance for dual/concurrent enrollment

• There may be opportunities to improve access

– Can more students enroll in DE?

– Can analyses be done of your students’ high school GPA and DE performance?

– Can students be prepared to meet new criteria for inclusion?

Good time for a conversation at the college…...

Maps and Tools from Other Settings

34

Middle and Early College Partnerships (NCREST)

Outcomes andMultiple Assessments

NCREST• National Center for

Restructuring Education, Schools and Teaching

• Focused on school change, mainly in high schools

• Our team – college/high school partnership programs, especially Early Colleges

36

Related projects:

• MCNC – Middle College National Consortium schools

• MEMCA – Michigan Early Middle College Association schools

• MEMPHIS – District Early College initiative

• NEW YORK – Smart Scholars initiative

• BARD – High School Early College model

• SECEP – STEM Early College Expansion Partnership (I3 Grant: Michigan, Connecticut)

Middle and Early CollegeDesign Principles

• Rigorous college-focused academic program

• Comprehensive student support

• Robust high school – college partnerships

• Culture of continuous improvement

37

MCNC Middle and Early College High SchoolsData Project 2015-16

38

• 6 – California

• 4 – Michigan

• 4 – New York

14 – Three States

• 1 – Colorado• 1 – Connecticut• 1 – Illinois• 1 – Maryland• 1 – Tennessee• 1 – Texas

6 – Six States

MCNC Students Served

39

African American

26%

Hispanic46%

Caucasian15%

Asian 7%

Other 6%

Male42%

Female58%

69% Free/Reduced Lunch

Criteria used by Middle and Early College High Schools for Readiness for College Course-taking

40

32%

68%

47%

16%

16%

0% 20% 40% 60% 80% 100%

High school GPA

College placement exams

High school counselor,teacher recommendation

SAT or ACT

High school state exam

College Coursework Outcomes:Access and Success

41

Access to College Courses

42

54%

79%85%

92% 94% 95%85% 86% 87%

95%

0%

20%

40%

60%

80%

100%

2006 2007 2008 2009 2010 2011 2012 2013 2014 2015MCNC Graduating Class (12th Graders)

Percent of 12th GradersWho Enrolled inCollege Courses

Credit Completion in College Courses

43

25.4

30.827.0

31.5 31.7 31.833.7 33.8 37.6 32.8

0.0

10.0

20.0

30.0

40.0

2006 2007 2008 2009 2010 2011 2012 2013 2014 2015College Course-Taking Students, MCNC Graduating Class

Average CollegeCredits Earned (cumulative)

Success in College Courses

44

87% 86% 87% 89% 90%90%

0%

20%

40%

60%

80%

100%

2010 2011 2012 2013 2014 2015MCNC Graduating Class (12th Graders)

Percent of course gradesresulting in

C grade or higher(includes A, B, C, D, F grades in calculation)

Percentage of 2014-15 MCNC Students Taking College Courses

45

100%

95%

92%

80%

44%

0% 20% 40% 60% 80% 100%

13th graders

12th graders

11th graders

10th graders

9th graders

College Coursework Summaryby 2014-15 Grade Cohort

2015-16Grade Cohort

Numberof Students

Taking College Courses

AverageGPA

AverageCreditsEarned

Percentage of Courses Passed(C grade or higher)

9th graders 597 3.08 5.4 94%10th graders 1,293 3.10 11.9 94%11th graders 1,297 2.88 21.1 91%12th graders 1,379 2.90 32.7 90%13th graders 220 2.68 40.6 84%Total 4,786 2.96 20.9 91%

46

College Course Grade Distribution: ENG & MATHAll MCNC 2014-15 StudentsSubject CourseEnrollments A B C D F W Other

ENGLISH 3777(87% A-C) 42.3% 30.7% 13.9% 3.7% 6.2% 2.7% .6%

MATH 2520(79% A-C) 34.7% 25.0% 19.0% 6.5% 8.8% 5.4% .6%

47

Assessment Practices2 Middle and Early College High School – College Partnerships

48

1. Washtenaw Community College & Washtenaw Technical Middle College High School - Assessments

49

• High school course grades (As and Bs at first eligibility point)

• Academic cut scores on Accuplacer

• Soft Skills credential (successful completion of high school 6 modules by end of term; high school teachers must sign off that student has mastery and proficiency in these skills in action

Soft Skills College Readiness and Success• Similar to college course, College 101/Prep… but starting 9th grade

• In addition to academic skills, students must learn and practice a number of "soft" skills needed for transition to and success in college-level work.

– Time management and organization

– Goal setting

– Decision-making

– Conflict resolution

– Persistence and seeking help50

Soft Skills Assessment Grades• Grade ‘R’ = Recommended for college coursework. This student has

demonstrated effective self-management skills and academic proficiency in this subject area.

• Grade ‘S’ = Soft skill credentialed. This student has demonstrated effective self-management skills, but has not yet demonstrated academic proficiency in this subject area.

• Grade ‘N’ = Needs improvement. This student needs to improve in one or more soft skill area before being ready to self-manage in a college class.

51

2. Buffalo State University & Buffalo Middle Early College High School - Assessments• Interview with entering students – assess whether kids are committed

to full summer and college coursework program

• MATH: Accuplacer (2-year partner), SAT scores (4-year partner)

• ENGLISH: 65 score or better on state high school exam

• Calculated approach on how college classes are introduced Arts or Music college course in early years.

• Preparing students from the start of high school Advisory (FOCUS)

52

9th Grade Advisory (FOCUS) Objectives and Topics

53

9th Grade Advisory (FOCUS) – Parent Engagement

54

School Principal on Early Preparation

• Support support support….The more you can introduce and talk about ‘college’ to the kids, how possible it is, just starting with freshman class and talking about their grades and being realistic with them about college work. So that later, our kids are mentally prepared.

• Guidance, seminar teachers are all saying the same messages, and mentioning this multiple times from multiple teachers. Even principal knows and so the kids take it seriously.

55

Assessments for College Readiness vs. Assessment + Early Prep/Support for College Readiness

• How do we help students be successful in high school courses early on… to increase college access and success?

• How do we equip students, academically and socially, with the confidence ‘to do’ and be successful in college?

– Middle Early College High Schools Comprehensive supports –academic and social

56

Sample MCNC Schools:Starter and Final Year College CoursesSchool Course names – Grade 10 Course names – Grade 12

School A • THEA 1001: INTRO THEATER ARTS• THEA 1101: INTRO TO ACTING• CORC 1130: MUSIC LANG HIST & CULT• SPEC 1103: COMMUNICAT TECHNIQ

• CHEM 1050: GENERAL CHEM 1A• ENGL 2301: INTRO CREATIVE WRITING• MATH 1011: PRECALC or MATH 1201: CALCULUS • PSYC 2810: ABNORMAL PSYCHOLOGY

School B • CPT101: Intro Comp or MUS105: Music App• PSY201: Gen Psych or SOC-101: Intro Soc• SPA101: Elementary Spanish I• COL03: College Skills

• ENG101: English Composition I• ENG102: English Composition II• BIO101: Bio Science I • MAT110: College Algebra or MAT120: Statistics

School C • FA 205: DRAWING• FA 305: PAINTING• History or Political Science course

• EN 110: COLLEGE COMPOSITION• EN 114: PUBLIC SPEAKING• PS 100: GENERAL PSYC or SO 100: INTRO TO SOC• GS 111: COLLEGE SUCCESS SKILL• MT 112: SURVEY OF MATH

School D • COM 115: Public Speaking• HPR 104: Health Career Opt & Readiness• CIS 118: Intro PC Applications• Criminal Justice or Health Careers course

• ENG 121: English Composition I • ENG 122: English Composition II• HPR 178: Medical Terminology• PSY 101: General Psychology I• ACC 101: Fundamentals of Accounting• MAT 135: Intro to Statistics

57

MCNC Students: The best part of my middle early college experience has been…• Being able to take a oral communications class at the college I

attend. It's what interested me in my current career path and I would not know that without my dual enrollment experience.

• The best part of my Middle Early College experience would have to be that my love for learning came back. I am forever grateful that I was given the teachers, counselors and support staff at the school. Coming to an early college gave me hope that I would actually be able to make it in a college environment.

• Not having to worry about a crippling college debt. I know that I will have one, but because of this school it will be less of a debt. I am also way more prepared for university life thanks to this program.

58

Community College Research Center \ Institute on Education and the Economy \ Teachers College \ Columbia University

525 West 120th Street, Box 174 New York, NY 10027 \ E-mail: [email protected] \ Telephone: 212.678.3091

Contact Us Visit us online:

Elisabeth Barnett: [email protected]

Jennifer [email protected]

ccrc.tc.columbia.eduwww.mdrc.orgTo download presentations, reports, and briefs, and sign-up for news announcements. We’re also on Facebook and Twitter.

59

mailto:[email protected]:[email protected]

Multiple Measures Assessment of Student Readiness for College CoursesAgendaOrganization of CAPRResearch on an Alternative Placement Strategy using Multiple MeasuresRAPS – Partner SitesPredictive analytics used to develop an algorithmWhy Use Multiple Measures Assessment?Community college 8-year graduation rates (Attewell, Lavin, Domina, and Levey, 2006)Under-placement and Over-placement (severe)Slide Number 10Model R-Squared Statistics�EnglishModel R-Squared Statistics�MathConclusions so farMultiple Measures AssessmentWhy Use Multiple MeasuresCurrent assessment practices in the US �(preliminary; CAPR 2017)Possible MeasuresNon-cognitive assessmentsMultiple Measures Options (Barnett and Reddy, 2017)But what about…. Slide Number 21Slide Number 22HS GPA is a better predictor than test results for long time (Hetts, 2016)For the most part, college grades stay parallel with feeder high school grades. (Bostian, 2016)�Slide Number 25Examples of alternative assessment systemsIVY TECH, INDIANA – WAIVERS (CCCSE, 2016)IVY TECH PASS RATESCALIFORNIAExample– CA Math PlacementNORTH CAROLINA (Bostian, 2016)Slide Number 32Relevance for dual/concurrent enrollmentMaps and Tools from Other SettingsMiddle and Early College Partnerships (NCREST)��Outcomes and�Multiple AssessmentsNCRESTMiddle and Early College�Design PrinciplesMCNC Middle and Early College High Schools�Data Project 2015-16MCNC Students ServedCriteria used by Middle and Early College High Schools for Readiness for College Course-takingCollege Coursework Outcomes:�Access and SuccessAccess to College CoursesCredit Completion in College CoursesSuccess in College CoursesPercentage of 2014-15 MCNC Students Taking College CoursesCollege Coursework Summary�by 2014-15 Grade CohortCollege Course Grade Distribution: ENG & MATH�All MCNC 2014-15 StudentsAssessment Practices�2 Middle and Early College High School – College Partnerships1. Washtenaw Community College & Washtenaw Technical Middle College High School - AssessmentsSoft Skills College Readiness and SuccessSoft Skills Assessment Grades2. Buffalo State University & Buffalo Middle Early College High School - Assessments9th Grade Advisory (FOCUS) Objectives and Topics9th Grade Advisory (FOCUS) – Parent EngagementSchool Principal on Early PreparationAssessments for College Readiness vs. �Assessment + Early Prep/Support for College ReadinessSample MCNC Schools:�Starter and Final Year College CoursesMCNC Students: The best part of my middle early college experience has been…Slide Number 59

national alliance of concurrent enrollment partnerships - multiple … slides... · 2017. 10....

Documents