conversion of faculty evaluations to an on- line format catherine hackett renner suny geneseo larry...
Post on 15-Jan-2016
215 views
TRANSCRIPT
Conversion of Faculty Evaluations to an On-Line FormatCatherine Hackett Renner SUNY Geneseo
Larry PiegzaGap Technologies, Inc.OnlineCourseEvaluations.com
The Paper Process ~22,000-25,000 forms per term
were hand sorted into course sections
Distributed to faculty the last 2 weeks of classes Faculty returned them to IR
Office Forms were separated, scanned,
then hand sorted back into course sections Invariably evaluations would
be turned in late Faculty received the results
back mid-February
Complaints with Process Faculty
Time delay in getting results back Late returns were not processed
Students Anonymity Detail in comments
Classroom time did not allow for thoughtful detail in comments
Convenience Would like to see results online
Conversion to On-line Format
Internal conversion was not feasible Migration to new system Internal survey software was not secure
enough for this process Decision was made to outsource
Constraints on company Had to use our own instrument Had to be able to conform to the levels of
access that were dictated by Faculty Senate
Company Choice
Best met our needs Could take our current evaluation system and upload it Could reproduce our levels of access Very customer-oriented
Spring 2006 Pilot
Students enrolled in Psychology classes and Astronomy Laboratories provided their Student Evaluation of Faculty Instruction (SOFI) ratings in an on-line format rather than in-class paper format 51 course sections 1,199 individual students providing1,806
evaluations These 1,199 students were also able to
compare the on-line version to in-class paper format (used in their other classes)
Getting the Word Out Students were informed of the pilot by their course
instructors at the beginning of the semester One week prior to the SOFI evaluation period
informational flyers were posted within the classrooms of the 51 sections
Article on the process was published in the student newspaper
Student Government Association endorsed the process and was involved with disseminating information
On the first evaluation day faculty were given handouts to distribute to their students explaining the process
The evaluation period started on April 17th and ended on May 3rd
Pilot Procedure
Students received an e-mail with their Login ID and password E-mail contained direct link the login webpage
E-mail reminders 48 hours Reminders stopped once all evaluations were
performed
Faculty received e-mails requesting they remind the students of the evaluation period Within the e-mail faculty also received the
response rate for each of their courses
Pilot Results
The response rate for the pilot was 76%! This was consistent
with the response rate for the in-class SOFI (70-73%)
Some bumps in the road All occurred within the
first 2 days All were minor All were resolved in
less than 24 hours
Student Satisfaction Survey
Seven survey questions designed to assess the following: Anonymity Detail of Comments Convenience Choice
This final question was designed to be the “litmus” test of the pilot
This question asks students which format they would prefer to use if given the choice
Anonymity
In general, I believe in-class SOFIs are: Percent
In general, I believe on-line SOFIs are: Percent
not anonymous 3.7 not anonymous 6.2
slightly anonymous 13.6 slightly anonymous 7.4
somewhat anonymous 57.4 somewhat anonymous 34.6
completely anonymous 25.3 completely anonymous 51.9
Detail of Comments
Was the amount of detail you provided in your comments different in the on-line version
compared to the in-class version? Percent
Yes, I was able to provide more detail than in class 27.8
Yes, I provided less detail than in class 13.6
No, it was about the same 57.4
Convenience
How convenient was it to use the on-line method compared to the in-class method? Percent
Not convenient at all 4.9
somewhat inconvenient 14.8
somewhat convenient 32.1
very convenient 47.5
Results
How often have you referred to the SOFI results in any given term? Percent
never 70.4
occasionally 22.2
often 3.7
very often 3.1
Results
SOFI results will be available for students within 3 weeks of the last day of completion and will be accessible on-line. Given this, how likely will you refer to the on-line SOFI results?
Percent
never 13.6
occasionally 38.3
often 28.4
very often 19.1
Choice
If you could choose which SOFI format
would you prefer to use? Percent
on-line 53.7
in-class paper 17.3
doesn't matter 27.2
Decision
We went to a full implementation Fall 2006
80% response rate Smooth run
What SUNY Geneseo Did Well
Create a login link from their web portal Erased student confidentiality concerns
by selecting an off-site vendor Communicated with their teachers Shared results to their students
What SUNY Geneseo Did Well
Sent out regular emails to their students Have teachers hand out directions Ran ad in school newspaper Mobilized their student government
Top 10 Things Not To Do When Going Online:
10) Have a one week evaluation window for your first semester
9) Decide on a vendor two weeks before your evaluation starts
8) Spam your students
7) Change your course of action based on the teacher suggesting things at the current time
Top 10 Things Not To Do When Going Online:
6) Run concurrent with paper
Actual school's response rates
2004
20062005
0%
10%
20%
30%
40%
50%
60%
70%
Top 10 Things Not To Do When Going Online:
5) Buy a system that doesn't allow your students to click a "I dropped this class" button.
4) Don't communicate to your teachers
3) Ignore your vendor’s advice / Go it alone
2) Ask 100 questions on each survey
Top 10 Things Not To Do When Going Online:
1) Send a two page email lecturing your students on their civic responsibility to fill out course evaluations
8 Must Haves
1) Patience to change your school’s culture
Response Rates For Edgewood College
0%
10%
20%
30%
40%
50%
60%
70%
2004 2005 2006
Year
Fall
Spring
Summer
8 Must Haves
2) Eliminate common questions for team taught courses
3) Prizes to give away to students
4) Share results to students
5) Allow teachers to email to their students
6) Have a contest
8 Must Haves
7) Follow-up question technology
Did the instructor speak audibly and clearly?
What could the instructor have done to speak more clearly?
Get a microphone Not face the blackboard
Speak slower Speak faster
8 Must Haves
8) Dropped class survey
What was the primary reason why you dropped this class?
Course related Instructor related Financial issue
Scheduling issue Something about me Other