developing & using intermediate measures: asking new & different questions to support...
Embed Size (px)
TRANSCRIPT
DEVELOPING & USING INTERMEDIATE MEASURES:ASKING NEW & DIFFERENT QUESTIONS TO SUPPORT STUDENT SUCCESSJames Sass, Rio Hondo CollegeAgi Horspool, Fullerton CollegeRP Conference, 4/9/2015
DEVELOPING AND USING INTERMEDIATE MEASURES
Overview to Session
Intermediate Measures 101
• Introduction to intermediate measures of student success.
• Techniques for using intermediate measures in research and planning.
• Group-process approach for integrating intermediate measures at your college.
• Steps for adapting Scorecard measures for local use.
Focus: Organizational Change at Your College or District
• Jim Sass• Rio Hondo College• Research Analyst• PhD, Arizona State
University, Organizational Communication
• Working with program evaluation since 1998
• Agi Horspool• Fullerton College• Project Manager, Research• PhD, Claremont Graduate
University, Organizational Behavior
• Working with program evaluation since 2005
STUDENT SUCCESS DATAChallenges and Opportunities
Student Outcomes Scenarios
• Volunteers:– Please read aloud each of the scenarios about the
challenges of measuring student outcomes in California community colleges.
Student Outcomes Scenarios
• Introduce yourself to others at your table• For a few minutes, discuss the scenarios you
listened to:– In what ways did these scenarios get you thinking
about the measures you currently/typically use on your campus?
• Select an individual from your table group to report out
Gap between Course-Level and Completion Measures
Course-Level Measures• Retention & Success
• Little change across time• Not addressing student
progress
Completion Measures• Degrees, Certificates &
Transfer
• Not about current students• Not addressing students
who did not complete
Solution: Intermediate Measures
• Measures of progress toward completion– Completing course sequences– Achieving specific milestones
• Progress by current students– One-year and three-semester cohorts– Three-year cohorts
• Support campus decision making and evaluation of initiatives
INTERMEDIATE MEASURES OF STUDENT SUCCESS
A Brief History
Development of Intermediate Measures: Publications
• Reaching Consensus on Common Indicators (Ewell, 2006)
• Using Longitudinal Data to Increase Community College Student Success (CRCC, 2008)
• Taking the Next Step (IHELP, 2010)– Steps to Success (IHELP, 2009) – Advancing by Degrees (IHELP, 2010)
Development of Intermediate Measures: Themes
• “Information about student progression . . . is an important policy tool” (Ewell, 2006).
• There are significant markers of student progression toward completion.
• Tracking these markers and disaggregating by demographic and enrollment groups can be useful for planning and improvement.
Development of Intermediate Measures: Terms
• Milestones: “Necessary intermediate educational achievements”– Persisting for two or three semesters– Earning one year of college-level credits– Completing the General Education curriculum
• Success Indicators: “Academic behaviors that predict success”– Completing a college success course– Attempting/completing courses within a timeline– Earning summer units
Intermediate Measures in the California Community College System
• Student Success Task Force (2012)• Student Success Scorecard (2013)– Completion Outcomes
• Degree/Transfer (SPAR)• Career Technical Education
– Momentum Points• Basic Skills (“Remedial”)• Persistence • 30 Units
INTERMEDIATE MEASURESImplications for Research and Planning
Using Intermediate Measures: Some Examples
• Rio Hondo College– Added Scorecard measures to Institution-Set Standards– Focus on 3-year cohorts
• Creative Ideas from Anonymous Colleges Responding to Survey– At 30 units, survey on ILOs and non-cognitive factors.– Awarding “badges” to students achieving milestones
Intermediate Measures in Research & Planning
• Some colleges incorporating Scorecard measures as is.• Some confusion on nature of intermediate measures.• Some good examples of tracking intermediate measures
with recent cohorts. – Persistence on yearly basis to inform strategic planning. – Persistence and 30 units for first-time college students.– Gateway course performance and 15-, 30-, and 45-unit
thresholds. – Persistence, basic-skills sequence, and 30 units on an annual
basis and strategizing where not improving.
Potential Uses for Intermediate Measures
• Conversations about goals and objectives.• Setting institutional standards.• Developing logic models for programs and
initiatives.• Evaluating programs and initiatives.
INTERMEDIATE MEASURES AT YOUR COLLEGE OR DISTRICT
A Group-Process Activity
Local Intermediate Measures
• For this activity, work in small groups at your tables to discuss the two questions on your “Activity #2: Group Process” worksheet.– Take notes at your table so we can collect and
share with participants after the conference.– Record your notes on the flipchart. These will be
posted so others can see after the session.– Designate a speaker who will report out.
Debrief: Local Intermediate Measures
• From your “Activity #2: Group Process” discussion, report out on the following question:
– How might you use a similar group process to open conversation at your campus or district about important and/or relevant intermediate measures?
DEVELOPING INTERMEDIATE MEASURES
Adapting Scorecard Metrics for Local Use
Scorecard Momentum Points
• Persistence• 30 Units• Basic Skills Sequence Completion– English– ESL/ENLA– Math– Reading
Persistence and 30 Units
• Use first-time students.• Window: fall semester or full academic year.• Cohort entry criteria:– Attempting 9 units in first term.– Attempting 6 units, including math or English, in
first year.– Earning 3 units and attempting math or English in
first year.• Outcomes: Use Scorecard criteria.
Basic Skills Sequence Completion
• Follow Scorecard procedures for entry to cohorts.
• Window: fall semester or full academic year.
• Outcomes: Use Scorecard criteria• For Reading outcome, use transfer-level
Reading or English course.
Time to Outcome Achievement
• Set cohort lengths appropriate to the outcome.– Persistence takes 3 semesters.– 30 units can be reasonable after 2 years (15 units
after 1 year)– Basic skills: 1 to 5 years
• Consider using 3 years’ progress as a standard for planning.
CONCLUSION
Measurement Issues
• Traditional measures of student success can be of limited usefulness.
• The Scorecard’s use of intermediate measures is a step ahead, but 6-year cohorts are not practical for decision making.
• Intermediate measures for recent cohorts have great potential to support planning and student success.
Practical Implications
• Colleges can and should develop their own intermediate measures.
• Cohorts can be created to meet the needs of the college.
• Group process is vital to developing and integrating intermediate measures.
• Scorecard measures can be adapted for local use.
QUESTIONS?James [email protected]
Link to Published Resources on Intermediate Measures
https://www.dropbox.com/sh/0he8zy6ephirqa5/AACjgICHTtHwFYYRLr2dmmSPa?dl=0