DEVELOPING & USING INTERMEDIATE MEASURES:ASKING NEW & DIFFERENT QUESTIONS TO SUPPORT STUDENT SUCCESSJames Sass, Rio Hondo CollegeAgi Horspool, Fullerton CollegeRP Conference, 4/9/2015

DEVELOPING AND USING INTERMEDIATE MEASURES

Overview to Session

Intermediate Measures 101

• Introduction to intermediate measures of student success.

• Techniques for using intermediate measures in research and planning.

• Group-process approach for integrating intermediate measures at your college.

• Steps for adapting Scorecard measures for local use.

Focus: Organizational Change at Your College or District

• Jim Sass• Rio Hondo College• Research Analyst• PhD, Arizona State

University, Organizational Communication

• Working with program evaluation since 1998

• JSass@riohondo.edu

• Agi Horspool• Fullerton College• Project Manager, Research• PhD, Claremont Graduate

University, Organizational Behavior

• Working with program evaluation since 2005

• ahorspool@fullcoll.edu

STUDENT SUCCESS DATAChallenges and Opportunities

Student Outcomes Scenarios

• Volunteers:– Please read aloud each of the scenarios about the

challenges of measuring student outcomes in California community colleges.

Student Outcomes Scenarios

• Introduce yourself to others at your table• For a few minutes, discuss the scenarios you

listened to:– In what ways did these scenarios get you thinking

about the measures you currently/typically use on your campus?

• Select an individual from your table group to report out

Gap between Course-Level and Completion Measures

Course-Level Measures• Retention & Success

• Little change across time• Not addressing student

progress

Completion Measures• Degrees, Certificates &

Transfer

• Not about current students• Not addressing students

who did not complete

Solution: Intermediate Measures

• Measures of progress toward completion– Completing course sequences– Achieving specific milestones

• Progress by current students– One-year and three-semester cohorts– Three-year cohorts

• Support campus decision making and evaluation of initiatives

INTERMEDIATE MEASURES OF STUDENT SUCCESS

A Brief History

Development of Intermediate Measures: Publications

• Reaching Consensus on Common Indicators (Ewell, 2006)

• Using Longitudinal Data to Increase Community College Student Success (CRCC, 2008)

• Taking the Next Step (IHELP, 2010)– Steps to Success (IHELP, 2009) – Advancing by Degrees (IHELP, 2010)

Development of Intermediate Measures: Themes

• “Information about student progression . . . is an important policy tool” (Ewell, 2006).

• There are significant markers of student progression toward completion.

• Tracking these markers and disaggregating by demographic and enrollment groups can be useful for planning and improvement.

Development of Intermediate Measures: Terms

• Milestones: “Necessary intermediate educational achievements”– Persisting for two or three semesters– Earning one year of college-level credits– Completing the General Education curriculum

• Success Indicators: “Academic behaviors that predict success”– Completing a college success course– Attempting/completing courses within a timeline– Earning summer units

Intermediate Measures in the California Community College System

• Student Success Task Force (2012)• Student Success Scorecard (2013)– Completion Outcomes

• Degree/Transfer (SPAR)• Career Technical Education

– Momentum Points• Basic Skills (“Remedial”)• Persistence • 30 Units

INTERMEDIATE MEASURESImplications for Research and Planning

Using Intermediate Measures: Some Examples

• Rio Hondo College– Added Scorecard measures to Institution-Set Standards– Focus on 3-year cohorts

• Creative Ideas from Anonymous Colleges Responding to Survey– At 30 units, survey on ILOs and non-cognitive factors.– Awarding “badges” to students achieving milestones

Intermediate Measures in Research & Planning

• Some colleges incorporating Scorecard measures as is.• Some confusion on nature of intermediate measures.• Some good examples of tracking intermediate measures

with recent cohorts. – Persistence on yearly basis to inform strategic planning. – Persistence and 30 units for first-time college students.– Gateway course performance and 15-, 30-, and 45-unit

thresholds. – Persistence, basic-skills sequence, and 30 units on an annual

basis and strategizing where not improving.

Potential Uses for Intermediate Measures

• Conversations about goals and objectives.• Setting institutional standards.• Developing logic models for programs and

initiatives.• Evaluating programs and initiatives.

INTERMEDIATE MEASURES AT YOUR COLLEGE OR DISTRICT

A Group-Process Activity

Local Intermediate Measures

• For this activity, work in small groups at your tables to discuss the two questions on your “Activity #2: Group Process” worksheet.– Take notes at your table so we can collect and

share with participants after the conference.– Record your notes on the flipchart. These will be

posted so others can see after the session.– Designate a speaker who will report out.

Debrief: Local Intermediate Measures

• From your “Activity #2: Group Process” discussion, report out on the following question:

– How might you use a similar group process to open conversation at your campus or district about important and/or relevant intermediate measures?

DEVELOPING INTERMEDIATE MEASURES

Adapting Scorecard Metrics for Local Use

Scorecard Momentum Points

• Persistence• 30 Units• Basic Skills Sequence Completion– English– ESL/ENLA– Math– Reading

Persistence and 30 Units

• Use first-time students.• Window: fall semester or full academic year.• Cohort entry criteria:– Attempting 9 units in first term.– Attempting 6 units, including math or English, in

first year.– Earning 3 units and attempting math or English in

first year.• Outcomes: Use Scorecard criteria.

Basic Skills Sequence Completion

• Follow Scorecard procedures for entry to cohorts.

• Window: fall semester or full academic year.

• Outcomes: Use Scorecard criteria• For Reading outcome, use transfer-level

Reading or English course.

Time to Outcome Achievement

• Set cohort lengths appropriate to the outcome.– Persistence takes 3 semesters.– 30 units can be reasonable after 2 years (15 units

after 1 year)– Basic skills: 1 to 5 years

• Consider using 3 years’ progress as a standard for planning.

CONCLUSION

Measurement Issues

• Traditional measures of student success can be of limited usefulness.

• The Scorecard’s use of intermediate measures is a step ahead, but 6-year cohorts are not practical for decision making.

• Intermediate measures for recent cohorts have great potential to support planning and student success.

Practical Implications

• Colleges can and should develop their own intermediate measures.

• Cohorts can be created to meet the needs of the college.

• Group process is vital to developing and integrating intermediate measures.

• Scorecard measures can be adapted for local use.

QUESTIONS?James SassJsass@riohondo.edu

Agi Horspoolahorspool@fullcoll.edu

Link to Published Resources on Intermediate Measures

https://www.dropbox.com/sh/0he8zy6ephirqa5/AACjgICHTtHwFYYRLr2dmmSPa?dl=0