using technology to efficiently and effectively gather information from recent alumni and employers...
TRANSCRIPT
Using Technology to Efficiently and Effectively Gather Information from
Recent Alumni and Employers
April 2009
Florida Association of Community CollegesInstitutional Effectiveness, Planning, & Professional Development Commission2009 Joint Annual Spring Conference
April 16, 2009 Florida Association of Community Colleges 2
2009 Joint Annual Spring Conference April 2009
Alumni and Employer Surveys
Presenters Nicole Carr, Assessment Coordinator for
Academic Programs Maggie Tymms, Assessment Coordinator
for Academic Programs Dr. James Coraggio, Director of
Institutional Research and Effectiveness
April 16, 2009 Florida Association of Community Colleges 3
2009 Joint Annual Spring Conference April 2009
SPC Background
SPC, established in 1927, is the oldest 2-year college in Florida First Community College in Florida to offer
4 year degrees (2002) 9 Campuses throughout the county 2007-08 FTE: 16,086 (LD), 1,415 (UD) Opening Fall 2008 credited enrollment:
27,076 Annual 2007-08 headcount: 62,200
April 16, 2009 Florida Association of Community Colleges 4
2009 Joint Annual Spring Conference April 2009
Institutional Effectiveness
“Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.”
Operating ResultsBudget PlanningStrategic & Unit
PlanningEvaluations/ AssessmentsMission
“Closing the Loop”
April 16, 2009 Florida Association of Community Colleges 5
2009 Joint Annual Spring Conference April 2009
Performance Improvement
From Compliance to Performance Improvement
Data Information ImprovementKnowledge
Improved OutcomesSPCReportsAssessment Data
CommunityCollege LeadershipProgram Directors
FacultyStudents
April 16, 2009 Florida Association of Community Colleges 6
2009 Joint Annual Spring Conference April 2009
Agenda
Student Assessment Points Purpose of Follow-up Surveys Alumni/Employer Survey Process Dissemination of Survey Results and
Reports
April 16, 2009 Florida Association of Community Colleges 7
2009 Joint Annual Spring Conference April 2009
Student Assessment Points
Students surveyed multiple times
April 16, 2009 Florida Association of Community Colleges 8
2009 Joint Annual Spring Conference April 2009
Purpose of Follow-up Surveys
Recent Alumni Survey was designed to: Gain insight into alumni’s perceptions of
the quality of general education and career preparation
Determine the current activities of former students such as employment information and educational status
April 16, 2009 Florida Association of Community Colleges 9
2009 Joint Annual Spring Conference April 2009
Purpose of Follow-up Surveys
Employer Survey was designed to: Gain insight into an employer’s perceptions of
hired SPC graduates in the areas of key general education competencies and foundational skills
Provide the employee’s major job responsibilities and wage information, as well as employer’s willingness to hire another SPC graduate
Identify employer’s willingness to support various college activities
April 16, 2009 Florida Association of Community Colleges 10
2009 Joint Annual Spring Conference April 2009
Purpose of Follow-up Surveys
State Requirements Survey data are also collected for state
required annual Placement & Follow-up Report (FETPIP: Florida Education Training Placement Information Program) and supplemental information to meet workforce criteria
April 16, 2009 Florida Association of Community Colleges 11
2009 Joint Annual Spring Conference April 2009
Alumni/Employer Survey Process
Previous model Labor intensive Manually tabulated paper surveys Telephone interviews conducted
using overtime hours
April 16, 2009 Florida Association of Community Colleges 12
2009 Joint Annual Spring Conference April 2009
Alumni/Employer Survey Process
Issues with previous model Costly survey process Declining college budget Increasing demand for accountability Outdated contact information Students over-surveyed Declining response rates Conducted in isolation
April 16, 2009 Florida Association of Community Colleges 13
2009 Joint Annual Spring Conference April 2009
Alumni/Employer Survey Process
Goals of new survey processA. Improve data collection processB. Enhance quality of survey formsC. Employ Industry Best PracticesD. Include discipline specific
informationE. Increase response rates
April 16, 2009 Florida Association of Community Colleges 14
2009 Joint Annual Spring Conference April 2009
A. Improve Data Collection Process
Data Collection Process Eliminated telephone calls Automated the process (email merge) Employ online survey submission
(ZoomerangTM) Obtain accurate email addresses Use multiple points of contact
April 16, 2009 Florida Association of Community Colleges 15
2009 Joint Annual Spring Conference April 2009
A. Improve Data Collection Process
Survey Schedule Recent Alumni Survey
Graduates surveyed six months after the end of session (Three times per year)
Employer Survey Employer of graduates surveyed at the
end of academic year (Once per year)
April 16, 2009 Florida Association of Community Colleges 16
2009 Joint Annual Spring Conference April 2009
A. Improve Data Collection Process
First email Second email Third email
First mailing Second mailing
2 weeks
2 weeks
2 weeksRecent Alumni Survey Multiple Points of Contact
2 weeks
April 16, 2009 Florida Association of Community Colleges 17
2009 Joint Annual Spring Conference April 2009
B. Enhance Quality Of Survey Forms
Improved Survey Forms Better “look and feel” of survey format Developed scannable paper forms Use online branching models to reduce
survey length Design items to gather only pertinent
data
April 16, 2009 Florida Association of Community Colleges 18
2009 Joint Annual Spring Conference April 2009
C. Employ Industry Best Practices
Don Dillman’s Five Elements Respondent- friendly questionnaire Up to five contacts with the respondent Inclusion of stamped envelope Personalized correspondence A token financial incentive that is sent
with the survey request
April 16, 2009 Florida Association of Community Colleges 19
2009 Joint Annual Spring Conference April 2009
Online Survey: Email and Form
April 16, 2009 Florida Association of Community Colleges 20
2009 Joint Annual Spring Conference April 2009
Paper Survey: Letter and Form
April 16, 2009 Florida Association of Community Colleges 21
2009 Joint Annual Spring Conference April 2009
C. Include Program Specific Information
Program specific information Meet with Deans/Program Directors Discuss requirements for program
accreditation and timelines Include their necessary program specific
items
April 16, 2009 Florida Association of Community Colleges 22
2009 Joint Annual Spring Conference April 2009
D. Increase Response Rates
Create awareness and convey importance Involve program faculty in the process Request faculty assistance to inform
students Include a flyer with graduation materials Coordinate with programs that hand-
carry surveys to local employers
April 16, 2009 Florida Association of Community Colleges 23
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Summer 07
April 16, 2009 Florida Association of Community Colleges 24
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Fall 07
April 16, 2009 Florida Association of Community Colleges 25
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Spring 08
April 16, 2009 Florida Association of Community Colleges 26
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
*Summer 2008 in progress
0.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
1st E-mail 2nd E-mail 3rd E-mail 1st Mailing 2nd Mailing
Summer/ Fall 06 Spring 07 Summer 07 Fall 07 Spring 08 Summer 08
April 16, 2009 Florida Association of Community Colleges 27
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Source of Responses 2007-08
29%
15%
8%
30%
18%1st E-mail
2nd E-mail
3rd E-mail
1st Mailing
2nd Mailing
April 16, 2009 Florida Association of Community Colleges 28
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Percent of Responses Online
51.5%45.5%
23.5%
6.6%
0%
20%
40%
60%
80%
100%
2004-05 2005-06 2006-07 2007-08
April 16, 2009 Florida Association of Community Colleges 29
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Overall Response Rate
*Summer 2008 in progress
23.4%28.7%25.7%
21.2%13.7%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Fall &Summer
2006
Spring 2007 Summer2007
Fall 2007 Spring 2008
April 16, 2009 Florida Association of Community Colleges 30
2009 Joint Annual Spring Conference April 2009
Alumni Survey Response Rates
Graduate Year
Surveyed
Number Surveyed
Number Responded
Percent Respond
ed
Number of Online
Responses
Percent of Responses
Online
2007-08 3,701 1,013 26.5% 522 51.5%
2006-07 3,953 658 16.7% 300 45.5%
2005-06 3,756 982 26.1% 231 23.5%
2004-05 3,417 1,494 43.7% 100 6.6%
April 16, 2009 Florida Association of Community Colleges 31
2009 Joint Annual Spring Conference April 2009
Response Rates - Employer
OtherResponses
Recent Alumni Surveys Sent
Employer Surveys Sent
Is a 44% Employer Response Rate Good?
COE Responses
With Permission Responses
Without Permission
April 16, 2009 Florida Association of Community Colleges 32
2009 Joint Annual Spring Conference April 2009
Disseminating Information
College wide access to reports http://www.spcollege.edu/Central/Surveys/SurveyList.htm
Program Specific Reports Published on the Educational Outcomes
website http://it.spcollege.edu/edoutcomes
April 16, 2009 Florida Association of Community Colleges 33
2009 Joint Annual Spring Conference April 2009
Disseminating Information
April 16, 2009 Florida Association of Community Colleges 34
2009 Joint Annual Spring Conference April 2009
Disseminating Information
April 16, 2009 Florida Association of Community Colleges 35
2009 Joint Annual Spring Conference April 2009
Disseminating Information
April 16, 2009 Florida Association of Community Colleges 36
2009 Joint Annual Spring Conference April 2009
Program Specific Reports
April 16, 2009 Florida Association of Community Colleges 37
2009 Joint Annual Spring Conference April 2009
Lessons Learned
Do not expect that students will remember their ID#s
Administer the survey soon after the end of the session (six months)
Use multiple points of contact (two weeks) Find the most recent contact information Partner with programs to acquire up-to-date
information
April 16, 2009 Florida Association of Community Colleges 38
2009 Joint Annual Spring Conference April 2009
Lessons Learned
Make students aware of the importance of the survey process (survey announcements)
Provide contact updates to college data systems (PeopleSoft, Alumni Foundation)
Monitor results and make changes as necessary More accurate data (quicker turn around of
results) with automated processes
April 16, 2009 Florida Association of Community Colleges 39
2009 Joint Annual Spring Conference April 2009
Future Direction
Ultimate goal is to provide Provosts (Deans) and Program Directors ‘timely’, ‘relevant’ ‘accurate’, and ‘interpretable’ information regarding their programs and their performance outcomes Multi-measure ‘Dashboard’ style reports
(Business Intelligence/Performance Measurement)
April 16, 2009 Florida Association of Community Colleges 40
2009 Joint Annual Spring Conference April 2009
Questions/Next Steps
Using Technology to Efficiently and Effectively Gather Information from
Recent Alumni and Employers
April 2009
Florida Association of Community CollegesInstitutional Effectiveness, Planning, & Professional Development Commission2009 Joint Annual Spring Conference