qualitative & quantitative perspectives research and planning group 40 th annual conference...
TRANSCRIPT
Qualitative & Quantitative Perspectives
Research and Planning Group 40th Annual Conference
Pacific Grove, CA
May 1-3, 2002
Esau Tovar, M.S.Retention Counselor
Santa Monica College
(310) 434-4012
2
Need for Study
Increased faculty concerns with current program
High rates of unsuccessful grades awarded to “EA” students
No clear evidence that EA works
Need to systematically assess EA
High probationary rates & low persistence rates at SMC
3
Why Early Alert?
4
Matriculation Components
Admissions
Orientation
Assessment
Counseling & Advisement
Student Follow-Up
Coordination & Training
Research & Evaluation
Prerequisites, Corequisites, Advisories
5
Student Follow-Up
Post enrollment evaluation of non-exempt students
Early intervention
Appropriate referrals
Positive and non-positive feedback
6
“Early Intervention” Model
Early Alert Program Coordinated by Matriculation Office Instructors identify “at-risk” students through EA rosters Typically between 4th & 6th week All instructor responses merged for each student identified Notices sent to students informing them of status and possible referrals No additional follow-up
7
Goals of Matriculation
Two primary goals Student success Institutional effectiveness
• Student services and instruction partnership• Systemwide accountability• Efficient use of resources• Institutional research• Increased participation of underrepresented
students
8
Studying Early AlertDriven by limited research
9
Research Questions
How effective is the EA program in the identification of “at-risk” students?
To what extent has EA impacted student success?
What do instructional and counseling faculty think of EA?
What components should EA incorporate to improve its effectiveness?
10
EA Evaluation Approaches
Institutional research Academic performance outcomes
Student interviews Tutoring use
Counseling/Instructional faculty interviews & survey
Major findings
11
EA Academic Performance Outcomes
Grade Received X EA Status
0.00%5.00%
10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%
A B C D F W
Grade Received
EA Students
All GradedStudents
Source: Geltner, P. (2001). The Characteristics of Early Alert Students: Fall 2000. Santa Monica College Office of Institutional Research.
EA Grade Records
7,913
All Grade Records
72,912
EA students succeed to a lesser extent than All students (28% vs. 67%)
Withdrawal rates are significantly higher for EA students 47% vs. 19%
Despite differences in grades awarded, conclusions are hard to draw
12
Fall 2000 EA Messages
Already contacted 11.9%
Absent or tardy 24.5%
Turn in assignments 16.3%
Tutoring needed 19.1%
Study skills needed 13.0%
Meet with instructor 10.4%
Meet with counselor 4.8%
Source: Geltner, P. (2001). The Characteristics of Early Alert Students: Fall 2000. Santa Monica College Office of Institutional Research.
13
EA Faculty InterviewsFaculty not satisfied with EA
Use and Understanding of EA
No clear link between EA & student success
How is it helping students?
Problems with Existing EA Program
Timeliness is crucial if students are to succeed
Exclusion of short-term & online classes
The current EA program does not provide any feedback to instructors
Once notices are sent out students do not respond
Negative image of EA & increased load factor
Suggestions for Improvement Use EA to identify at-risk students at any point during semester
Letters should be customized by instructor; hand-delivered in class, or emailed
Feedback to instructors of services provided to students and outcomes
Phone calls to students
14
“The Survey”
15
SMC EA Survey
Online & Printed Form
Background (8 items)
EA Effectiveness (15 items X 2 scales)
Student Academic Information (7 items X 2 scales)
Revamped EA Scenario (9 items)
Reporting Features (5 items)
Open-ended questions (5 items)
16
Summary of FindingsParticipants
N = 80
Position Held61% Full-time teaching (n = 48)32.5% Part-time teaching (n = 26)4% Full-time counseling (n = 3)1% Part-time counseling (n = 2)1% Other (n = 1)
FT Faculty Status43% Tenured (n = 30)57% Non-tenured (n = 21)
Years EmployedFT: M = 11.6PT: M = 10.3
Department ParticipationCross-sectional representation (20 departments)
Frequency of EA Use1% Never1% Rarely11% Sometimes13% Often74% Always
Average Time Per Class96% 1 hour or less
Preference for EA Use14% At any point, Ongoing5% 1st to 2nd week23% 3rd to 4th week46% 5th to 6th week11% 7th to 8th week1% 9th week and beyond
17
Assessing Importance & Satisfaction with
Early Alert Components
18
Importance of EA Components
Referring students easily to college survival workshops (e.g., study skills, time management)
Supporting increased student contact with instructional resources
Making students aware of academic support resources: tutoring
Supporting increased student contact with counselors
Providing identified students with timely feedback and suggested interventions to improve performance
Providing identified students with accurate feedback on identified problems
Supporting increased student contact with instructors
Supporting increased student contact with special programs (e.g., Latino Center, EOPS)
Providing counselors with appropriate training on use of EA and follow-up 2
Monitoring individual and/or group student academic performance
Providing or improving communication among instructors, counselors, and students
Tracking students after initial identification
Providing instructors with appropriate training on use of EA and follow-up 1
Aiding instructors to identify students at risk of performing poorly academically
Assisting instructors by providing a means by which to evaluate student classroom performance
72 4.26 0.96
69 4.26 1.22
76 4.25 1.12
67 4.22 1.18
77 4.22 1.13
75 4.05 1.27
75 3.97 1.33
67 3.9 1.28
54 3.7 1.27
69 3.65 1.39
73 3.64 1.34
69 3.45 1.41
69 3.26 1.45
76 3.16 1.55
75 2.91 1.45
n M SD
Scale: Importance of Early Alert Components 1 = Not at all important, 5 = Very important
Part-Time vs. Full-Time Faculty Member Mean Importance1 PT = 3.77 (SD = 1.31), FT = 3.02 (SD = 1.47); t(67) = 2.05, p < .052 PT = 4.38 (SD = 0.89), FT = 3.42 (SD = 1.31); t(52) = 2.67, p < .01
19
Satisfaction with EA Components
Making students aware of academic support resources: tutoring 1
Referring students easily to college survival workshops (e.g., study skills, time management)
Supporting increased student contact with special programs (e.g., Latino Center, EOPS) 2
Supporting increased student contact with instructional resources
Supporting increased student contact with instructors
Providing identified students with timely feedback and suggested interventions to improve performance
Aiding instructors to identify students at risk of performing poorly academically
Providing counselors with appropriate training on use of EA and follow-up
Supporting increased student contact with counselors
Monitoring individual and/or group student academic performance
Providing identified students with accurate feedback on identified problems
Assisting instructors by providing a means by which to evaluate student classroom performance
Providing or improving communication among instructors, counselors, and students
Tracking students after initial identification
Providing instructors with appropriate training on use of EA and follow-up
68 3.50 1.19
63 3.27 1.17
50 3.16 1.00
58 3.14 1.15
70 3.11 1.27
71 3.03 1.44
71 2.89 1.36
39 2.82 1.19
51 2.80 1.37
61 2.80 1.26
72 2.76 1.45
67 2.67 1.17
67 2.61 1.28
60 2.60 1.15
64 2.55 1.18
n M SD
Scale: Satisfaction of Early Alert Components 1 = Not at all satisfied, 5 = Very satisfied
Part-Time vs. Full-Time Faculty Member Mean Satisfaction1 PT = 4.00 (SD = 1.02), FT = 3.23 (SD = 1.20); t(66) = 2.67, p < .012 PT = 3.81 (SD = 0.91), FT = 2.85 (SD = 0.89); t(48) = 3.53, p < .001
20
EA Priorities Matrix*
Very Important
Not At All Important
Very SatisfiedNot At All Satisfied
High Status ItemsImprovement Needed
(large performance gaps)
Greatest Strengths(smallest performance gaps)
Low Status Items Opportunity to Study/Redirect
(medium performance gaps)
Low Status ItemsOpportunity to
Redirect Efforts (medium performance gaps)
* Priorities matrix based on Noel-Levitz’s Matrix for Prioritizing Action
Importance – Satisfaction = Performance Gap
21
Performance Gaps inEA Components*
Supporting increased student contact with counselors
Providing identified students with accurate feedback on identified problems
Providing identified students with timely feedback and suggested interventions to improve performance
Providing or improving communication among instructors, counselors, and students
Supporting increased student contact with instructional resources
Referring students easily to college survival workshops (e.g., study skills, time management)
Supporting increased student contact with instructors
Tracking students after initial identification
Monitoring individual and/or group student academic performance
Providing counselors with appropriate training on use of EA and follow-up
Making students aware of academic support resources: tutoring
Supporting increased student contact with special programs (e.g., Latino Center, EOPS)
Providing instructors with appropriate training on use of EA and follow-up
Assisting instructors by providing a means by which to evaluate student classroom performance
Aiding instructors to identify students at risk of performing poorly academically
49 1.41 1.47
70 1.29 1.44
70 1.19 1.70
65 1.06 1.56
57 1.04 1.59
62 0.94 1.24
67 0.91 1.62
56 0.89 1.42
60 0.87 1.42
36 0.78 1.24
67 0.72 1.40
47 0.70 1.35
59 0.69 1.32
66 0.29 1.41
70 0.29 1.45
n M SDImportance – Satisfaction = Performance Gap
22
Additional Sources of Information Beneficial to
Assist Students
23
Information Beneficial to Tailor Courses for All Students & to Identify At-Risk
Students
High school academic history (e.g., GPA, courses completed)* t(67) = 2.49, p < .01
English—writing—placement information
English—reading—placement information
Directory of campus referral sources
College academic history (e.g., SMC or other college GPA, courses completed) t(69) = -2.71, p < .01
Mathematics placement information t(65) = -2.60, p < .01
Training on referral skills
68 4.78 2.64 3.70 1.37
69 4.30 1.09 4.38 1.02
69 4.29 1.11 4.41 0.99
67 4.13 1.17 4.19 1.17
70 3.93 1.30 4.16 1.22
66 3.39 1.48 3.71 1.32
55 3.35 1.35 3.42 1.33
All At-Risk Students Students
n M SD M SD
Scale: Beneficial to assist ALL STUDENTS & Beneficial to IDENTIFY AT-RISK STUDENTS 1 = Definitely not beneficial, 5 = Definitely beneficial
24
Revamping Early Alert Program at SMC
25
Revamped EA: ScenarioRevamped Early Alert Program to address shortcomings expressed by instructors and students. Among the potential solutions being explored are the following:
Moving from a paper-and-pencil student evaluation roster to an online-based student evaluation allowing individual customization (for instructor and student);
Allowing instructors to identify and assess only at-risk students when necessary;
Allowing instructors to provide timely feedback directly to students by means of email or printed letters that can be delivered to the student in class;
Allowing instructors, counselors, and other campus programs to follow up with instructor recommendations and maintain contact with identified students;
Tracking all sources of student contact (e.g., meetings with counselors, workshops attended, tutoring attendance) after initial identification to prepare and easily present reports on such things as the impact of early alert on student success indicators individually, by group, etc.
26
Level of Agreement to Revamped/Integrated EA
ComponentsObtaining handouts at the beginning of the semester addressing interventions that are likely to help at-risk students (before they experience difficulty and are identified for EA purposes)I would be willing to hand out the EA letter directly to the student at a subsequent class meetingReports pertaining to interventions suggested and those providedReports pertaining to student success indicators (retention, persistence, grades)The college should design and implement an online-based Early Alert systemEarly alert evaluations should be completed only for students at risk of performing poorly or experiencing problem in classesI would be willing to have my name appear on the EA letter or email sent to studentsReports pertaining to how students are contacted and by whomI would be willing to use an online-based early alert systemThe college should continue to use a paper-and-pencil-based early alert system 1
Reports pertaining to the success of specific groups (gender, ethnicity, age, etc.)Early alert evaluations should be completed for every student in every class 2
77 4.71 1.32
79 4.00 1.24
72 3.97 1.0570 3.96 1.0876 3.88 1.2478 3.77 1.49
78 3.76 1.50
70 3.64 1.2778 3.03 1.5378 2.76 1.3670 2.64 1.3277 2.42 1.57
n M SD
Scale: Level of Agreement Given Scenario 1 = Strongly Disagree, 5 = Strongly Agree
Part-Time vs. Full-Time Faculty Member Mean Agreement1 PT = 3.57 (SD = 1.26), FT = 2.30 (SD = 1.20); t(76) = 4.41, p < .012 PT = 2.97 (SD = 1.61), FT = 2.08 (SD = 1.46); t(75) = 2.47, p < .01
27
Open-Ended Questions
28
Additional Components/Feedback Desired: For Students
Increase Student/Instructor/Counselor Contact
Formalized Referrals/Resources
Assessment/Prerequisites Completion & Accuracy
Student Unpreparedness Lack Study Skills College Survival Workshops
29
Additional Components/Feedback Desired: For Faculty
Implement Follow-Up ComponentProvide feedback on what happens with the identified student, & the services provided
Timeliness of EA NoticesAssessment/Prerequisites
Completion & Accuracy Student Background Information
Assessment InformationProgress in Other Courses
Increased Instructor/Counselor CollaborationFacilitate Student Learning
Office VisitsClassroom ManagementFaculty Training
Formalized Referrals/Resources
30
Next Steps
31
Discussions
Expanding EA Task Force
Consulting with MIS Online EA System
Reviewing Products: RetentionTRAX by Noel-Levitz
Student Correspondence System by Precision Dynamics
32
Timeline
Finalize discussion Late May
Develop/Adopt Product Summer ‘02
Pilot Test Fall ’02
Assess Effectiveness Fall ‘02
Institutionalize Spring ‘03