who owns student success? a case for institution-wide...

Who Owns Student Success? A Case for Institution-wide

Engagement

Steve Robbins, Director, Research Innovation , Center for Academic and Workforce Readiness and Success, Educational Testing Service, Princeton, NJ

Kevin Li, Dean of Instruction, Wilbur Wright College, Chicago, IL

Copyright © 2013 by Educational Testing Service. All rights reserved.

2 Copyright © 2013 by Educational Testing Service. All rights reserved.

Consultant in higher education and Executive Director of The Leadership Board for CIO's in higher education. Dr. Zastrocky examines and analyzes emerging higher education IT trends and assesses the implications of those trends on educational institutions globally. Dr. Zastrocky has served as Managing Vice President, Education Research Leader and Research Director for Academic Strategies at Gartner Inc., a faculty member at several universities, assistant dean, chief information officer, vice president of CAUSE, and vice president of the Kaludis Consulting Group.

Dr. Michael Zastrocky

Moderator


Steven Robbins

Speaker

Steven Robbins is Director of Research Innovation in the Center for Academic and Workforce Readiness and Success at Educational Testing Service in Princeton, N.J. He received his Ph.D. degree in counseling psychology and his M.S. degree in rehabilitation counseling — both from the University of Utah. He received his B.A. degree in psychology from the University of Illinois. Before joining ETS, Robbins was Vice President for Research at ACT, Inc.


Kevin Li holds a Master of Arts degree in economics and is a Ph.D. candidate at the University of Illinois at Chicago in the field of educational psychology. He has served in various roles in the City Colleges of Chicago for the past fifteen years. Since 2009, he has served as Dean of Instruction at Wilbur Wright College, one of the City Colleges of Chicago. Li is responsible for overseeing and coordinating the college’s accreditation process.

Kevin Li

Speaker

• 50 percent of community college

students need remedial – or below

college level – courses

• 20 percent of 4-year college students

also start in remediation

• 40 percent of these students –

frustrated by their low placement –

don’t even bother to enroll in classes

But many are unprepared


More than 3.4 million students begin college each year.

• There is convincing evidence that

many college students are not

prepared for college and don’t have

the behavioral skills needed for

success.

• “Preparedness” encompasses a

holistic look at both cognitive and

noncognitive skills.

First-year college students are often confused and overwhelmed by the complexity of navigating their college experience


And the path to college-level work has detours.


The Challenges

• Students are not able to complete credit bearing general education courses in a timely fashion

• Even incremental success in improving student retention a success; 1% improvement means more students succeed

• The swirl (students coming and leaving) reflects the increasing complexity of student goals and barriers to success

• Institutions must reach across natural silos and structures to effectively drive student success


The Solution

How do we . . .

• Strategically accelerate students into their credit bearing courses?

• Consider individual differences in background, skills, abilities, and experiences to promote success?

• Develop and target sound and practical interventions?

• Best disseminate that information to students, faculty, staff, and administrators?


The Solution: From the “so what” to the “now what”…

Holistic Assessment

Data Integration & Use Across Silos

Student Commitment & Classroom Success


Adding a Noncognitive Assessment Allows for a Holistic View

• Respects the whole student (examines cognitive, noncognitive, behavioral factors)

• Acknowledges the unique individuality of students

• Focuses on factors educators can control and empower students to modify: motivation, social connectedness, as opposed to socio-economic, situational factors

• A common language, equitable model, for early warning, advising, and acceleration decisions


Domain Facet Definition Example Items

Academic Skills Tools and strategies for academic success

Organization Strategies for organizing work and time. I make a schedule for getting my school work done.

Meeting Class Expectations

Doing what’s expected to meet the requirements of your course including assignments and in-class behaviors.

I attend almost all of my classes.

Commitment Active pursuit toward an academic goal

Commitment to College Goals

Perceived value and determination to succeed in and complete college.

One of my life goals is to graduate college.

Institutional Commitment

Attachment to and positive evaluations of the school.

This is the right school for me.

Self-Management Reactions to academic and daily stress

Sensitivity to Stress

Tendency to feel frustrated, discouraged, or upset when under pressure or burdened by demands.

I get stressed out easily when things don't go my way.

Academic Self-Efficacy

Belief in one’s ability to perform and achieve in an academic setting.

I'm confident that I will succeed in my courses this semester.

Test Anxiety General reactions to test-taking experiences, including negative thoughts and feelings (e.g., worry, dread).

When taking a test, I think about what happens if I don't do well.

Social Support Connecting with people and students resources for success

Connectedness A general sense of belonging and engagement.

I feel connected to my peers.

Institutional Support

Attitudes about and tendency to seek help from established resources.

If I don't understand something in class, I ask the instructor for help.

Barriers to Success

Financial pressures, family responsibilities, conflicting work schedules, and limited institutional knowledge.

Family pressures make it hard for me to commit to school.

Risk Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive , and psycho-social information and supported by statistical relationships with success.

Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.

Feedback: Determine by more specific “facet” scores (see next page).

Action Plans: Suggested interaction with programs and services on campus.

Background Information: Communicate key student information from both SuccessNavigator and SIS to faculty/advisor.

Academic Success: Classroom success broken into ability to meet expectations and organizational skills.

Self-management: Ability to handle stress, testing anxiety and academic self-efficacy.

.

Social Support: Tendency to seek help, attitude toward barriers to success and ability to relate to peers in school.

Commitment: Measures both commitment to college and commitment to the specific institution.

Biographical Background Information: Student-reported background information for use in context of advising.

Education plans: Helps advisor understand student’s commitment to college, goals.

Education Background Information: Communicates key student information about high school performance.

Influences : Understand outside influences that may affect student performance, yet go unmentioned.


Early Intervention is key

Three basic implications: • Educators have a profound influence on students

– Faculty can motivate and inspire students

– Educators can also shape and foster their intentionality and academic determination

• Observable behaviors are indicative of internal states

– E.g., Motivation, engagement

• Educators have the opportunity and capability to guide students towards desirable classroom/ academic behaviors


Data Integration and Effective Coordination

Rethinking institutional enterprise systems:

• A data warehouse that populates by student rather than transaction

• Ability to connect student information across critical decision points, whether course scheduling and placement, advising, classroom monitoring, or semester and yearly status

• A vehicle for communicating across groups: testing, faculty, student, advising

• Business Intelligence capability to support dashboards and reporting


TACTICAL

Holistic Placement Decision Support Systems

SuccessNavigator

Profile

Student Education

Plan

Student Bio,

Registration

Student Active

Course Information

Success Navigator

GradesFirst PeopleSoft Blackboard

Advisor Feedback

Report

Accelerated Students # of Students

% of Students

Non-Accelerated Students # of Students

% age of Students

Non-SuccessNavigator – College Advising Interventions

253 79% Non-Accelerated College Advising Interventions

10,775 85%

INFORMA- TIONAL

STRATEGIC

Data Warehouse Reports

Reports Business

Intelligence (Zogotech)


What we have learned at CCC

• Evidence-based solutions

• ETS/CCC Collaboration Objectives

• Lessons Learned and Next Steps

19

Noncognitive Assessment Experience at Wright College

semester

Note. Results from Wright College (Spring 2009).

n = 453; All paths are statistically significant.


20

High Medium Low

Math Readiness

High 96% 91% --

Medium 94% 77% 82%

Low 86% 61% 38%

n=411

Developmental Education Class Success Rates Based on Math Readiness and In-Course Behavior Levels

Success rate not

calculated due to

insufficient sample size

Effort level was measured as whether a student 1) actively participated in group work 2) actively participated in lecture 3) attended class 4) completed homework

Success rates for students with low readiness but high effort more

than twice as high as for low effort, low readiness

Future area of focus

Research shows that behavioral monitoring and intervention are key to student success


ETS/CCC Collaboration Objectives

• Combine COMPASS and Psychosocial Measure to accelerate student progress and college completion

– Students placed within the upper-band with high psychosocial scores can enroll in the next level course

• Allow more individualized, actionable plans for students, tied to curricular goals

• Noncognitive survey and in-class behavioral ratings will be used to trigger Early Alert and intervention strategies

• Create supplementary services tied to Early Alert and to behavioral monitoring

21



Resource Library

• Also available to advisors, the Resource Library will include such topics as:

– Guidelines for working with students

– Sample advising scripts

– Helping a student choose a planner/calendar

– Goal setting worksheets

– Costs and benefits of a college degree

– Conflict resolution tips

– Financial/budget planner for students

– Communication skills quick reference

– Anxiety symptoms sheet


Comprehensive Early Interventions

• Serve to model and coach at-risk students toward a set of desirable behaviors.

• Findings from Li et al. (2013 or in press) indicate that low-level developmental education math students can increase the likelihood of passing the course by 30% if they exhibit desirable in-class behaviors:

Active participation in group work

Actively engaged during group work Helps other students with assignments Does his/her fair share of the work

Active participation in lecture Displays alertness Attentive during class Asks and answers questions

Attendance Attends class Stays for whole period

Completion of homework assignments

Completes assignments thoroughly Submits assignments on time


Recommendations: The 3 Pillars of Success

• Place students more accurately using multiple measures, including noncognitive skills for possible acceleration

• Smarter, strategic interventions: coordinate efforts to support the whole student

• Institutional commitment: Faculty, advisors, and staff collaborate in early alert program

• Generate fine-grained reporting to empower frontline workers and drive further research and program development

Holistic Assessment

Instructional and Advising Supports

Academic Readiness


Next steps in Chicago

• CCC is balancing faculty, staff and administration perspectives to achieve buy-in and reach effective solutions to emerging problems.

• Coordination across instruction, advising and other support is critical. Classroom monitoring flags early alert for proactive advising.

• The need for integration of data that enhances process and resource planning.


Leadership and Change: Final Thoughts and Perspective

• Who “owns” this change agenda?

• Can this system be replicated at other institutions?

• Likely challenges and issues to address.

For more information:

Web: www.ets.org/SuccessNavigator/chronicle

Phone: 1-800-745-0269

Email: [email protected]

Questions?


http://www.ets.org/SuccessNavigator/chronicle

mailto:[email protected]


Useful Background References

Allen, J., & Robbins, S. (2010). Effects of interest-major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23-35.

Allen, J., Robbins, S., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: effects of academic performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647-664.

American Association of Community Colleges (AACC). (2009). Community college fast facts. Retrieved July 10, 2009 from http://www2.aacc.nche.edu/research/index.htm

Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of placement tests and High school transcripts. CCRC Working Paper No. 42. New York: Community College Research Center, Teachers College, Columbia University.

Berkner, L., & Choy, S. (2008). Descriptive summary of 2003-04 beginning postsecondary students: Three years later. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

Bragg, D. D. (2001). Community college access, mission and outcomes: Considering intriguing intersections and challenges. Peabody Journal of Education. 76(1), 93-116.

Community College Research Center. (2002a). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8

Community College Research Center. (2002b). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8

http://www2.aacc.nche.edu/research/index.htm

http://ccrc.tc.columbia.edu/Seminar.asp?uid=8


References (cont.)

Habley, W., Bloom, J, & Robbins, S. (2012). Increasing persistence: Research-based strategies for college student success. San Francisco: Jossey-Bass

Habley, W., & McClanahan, R. (2004). What works in student retention – All survey colleges. Iowa City, IA: ACT, Inc.

Harris, S. (1991) Can’t you guys read? Cartoons on academia. New Brunswick, NJ: Rutgers University Press.

Herzog, K., & Miller, M. P. (Eds.). (1985). Scholarship: More great cartoons from the Kappan. Bloomington, IN: Phi Beta Kappa International Inc.

Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and Psychological Measurement, 65, 482-508.

Li, K., & Cordes, C. (2010). Individualizing the Delivery of Math Developmental Education. Talk presented at the 34th Annual National Association of Developmental Education Conference, Columbus, OH.

Li, K., Zelenka, R., Buonaguidi, L., Beckman, R., Casillas, A., Allen, F., Crouse, J., Hanson, M. & Robbins, S. (In press). Modeling Effects of Math Readiness and Student Behavior on Foundational Skills Course Success. Journal of Developmental Education.

Lotkowski, V., Robbins, S., & Noeth, R. (2004). The role of academic and non-academic factors in improving college retention. Retrieved from ACT, Inc. website: www.act.org/research/policy/index.html.

http://www.act.org/research/policy/index.html


References (cont.)

Markle, R., & Robbins, S., (2013). Breaking the developmental education logjam: A holistic approach to acceleration. Princeton, NJ: Educational Testing Service.

Markle, R., & Robbins, S., (2013). A holistic view of course placement fecisions – Avoiding the HS GPA trap. Princeton, NJ: Educational Testing Service.

McCabe, R. H. (2000). No one to waste. Washington, DC: Community College Press.

NCES (2004). Remedial education at degree-granting postsecondary institutions in fall 2000 (NCES 2004-010).

NISOD (2012, March). Assessing college-readiness assessments. Webinar delivered at the annual NISOD Conference.

Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco: Jossey-Bass.

Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five: Examining social desirability and college academic performance. Personality and Individual Difference, 41, 663-673.

Porchea, S., Allen, J., Robbins, S., & Phelps, R. (in press). Predictors of long-term enrollment and degree outcomes for community college students: Integrating academic, psychosocial, socio-demographic, and situational factors. Journal of Higher Education.

President’s Commission on Higher Education. (1947). Organizing higher education. Washington, DC: Author.


References (cont.)

Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Campbell, R., Mahoney, E., & Gore, P.A. (2009). Associations of resource and service utilization, risk level, and college outcomes. Research in Higher Education, 50(1), 101-118.

Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616.

Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.

Robbins, S., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163-1184.

Scott-Clayton, J. (2012). Do High-Stakes Placement Exams Predict College Success? Predicting Success in College: The Importance of Placement Tests and High School Transcripts. CCRC Working Paper No. 41. New York: Community College Research Center, Teachers College, Columbia University.

Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The Chronicle of Higher Education, B16.

who owns student success? a case for institution-wide...

Documents