transiting to an online course evaluation model: the online student survey of instruction

Transiting to an Online Course Evaluation Model: The Online Student Survey of

Instruction

February 2010

Florida Association of Institutional Research2010 Annual Conference

http://www.spcollege.edu/

February 25, 2010 Florida Association of Institutional Research 2

Transiting to an Online Course Evaluation Model February 2010

Student Survey of Instruction

Presenters Dr. James Coraggio, Director, Academic

Effectiveness and Assessment Magaly Tymms, Assessment Coordinator,

Academic Effectiveness and Assessment

With contributions from…Debbie Montalvo, SSI CoordinatorRichard Jakubowski, SSI DB Administrator



SPC Background SPC, established in 1927, is the oldest 2-year college in Florida First Community College in Florida to offer

4 year degrees (2002) Nine Campuses throughout Pinellas County 2008-09 FTE: 16,967 (LD), 1,677 (UD) Opening Fall 2009 credit enrollment:

29,560 Annual 2008-09 headcount (credit and

non-credit): 61,414



Purpose This presentation describes the

development and implementation of a college-wide online process to administer course evaluations, the Student Survey of Instruction (SSI), which takes advantage of existing technology to efficiently collect student responses, disseminate the results in a more timely manner, and save the institution time and money.



Agenda Issues with the prior process Intended outcomes New online process Dissemination of results and reports Response rates Next steps



Issues with Prior Process Expenses related to Scantron forms, and

numerous hours of sorting, organizing, and transporting forms

Time consuming process of transcribing open-ended items

Not all Scantrons were scanned and processed (scanning issues, data integrity issues, etc.)

Scantron results and online results captured separately and reported separately

Process conducted by multiple departments with no central ownership



Original SSI Items Original SSI forms were developed by a college

committee a number of years ago Items were reviewed for content and a lecture

form was aggregated into four categories (e.g., faculty student interaction, organization, presentation, and evaluation)

There were four SSI forms (Scantron): Lecture, Non-lecture, Clinical A*, and Clinical B*

Results were scanned by a third party (PCSB) and entered into PeopleSoft



eCampus SSI Items Online SSI form was developed later by

eCampus to assess online courses This form was similar to the Lecture SSI

form with some slightly modified items and the same four categories

Original online survey tool created to capture summative information in database



Performance Improvement

From Compliance to Performance Improvement

Data Information ImprovementKnowledge

Improved OutcomesSPCReportsAssessment Data

CommunityCollege LeadershipProgram Directors

FacultyStudents



Intended Outcomes New SSI online process was designed to:

Provide stakeholders (Faculty, Program Administrators, etc.) information to assist in making decisions that will result in performance improvement

Provide more reliable and valid course evaluation information

Create centralized coordination of the process Save the college time and money (est. $45,000

per year) Eliminate the need for in-class administration

(more course time for instruction)



SSI Online Process Timeline

Fall 2007 Established an SSI Team Factor analysis (exploratory and confirmatory cross

validation) determined that lecture and online SSI forms had three factors (e.g., faculty engagement, preparation and organization, and course instruction)

Spring 2008 Pilot of 100 FTF course sections conducted

Fall 2008 First collegewide administration (All course ANGEL shells

enabled) Created an online reporting tool (program administrators

only) Revised older survey administration tool




Spring 2009 Created customized usage reporting tools Created new survey interface with an opt-out

capability Added Faculty Representative to SSI Team Created online reporting tool access for Faculty

(No more paper)




Summer 2009 First collegewide summer administration Modified faculty email notices to include link to

real-time response data Began working with HEC on Clinical Solutions Evaluated current SSI items and forms (SSI

Revision Committee) Fall 2009

Implement self-service module so faculty can revise their own administration dates




Spring 2010 College-wide Pilot of new SSI items set Pilot Clinical SSI in the areas of MLT, EMS and

COE Add standard deviation information to reports Identify best practices from faculty members

and programs with the highest response rates



SSI Link in ANGEL Course instructors are required at the start of

each term to: enable their ANGEL course shells, upload their course syllabus, and attached their SSI link into their course.



Faculty Notification Emails

Faculty are provided four informational emails during their SSI administration. Five days before the beginning At the 50% mark At the 75% mark Two days after the end

Emails contain course, form, date, and response information.



Faculty Notification Emails



SSI Link in ANGEL



Accessing Information

A site was developed to provide faculty and program administrators information about the SSI online process: http://www.spcollege.edu/central/ssi/



Information about the Process



Accessing Results

All SSI survey reports are available on the online SSI Administration site: http://it.spcollege.edu/ssi/

Faculty and Program Administrators can login by entering their SPC network username and password

Access is limited to only those courses under their area of responsibility.

http://it.spcollege.edu/ssi/





Accessing Results



Accessing Results

Instructors and Program Administrators can select ‘Term’ from the drop-down menu (i.e. 0405) and select how they would like to view the results Each individual section with its own results OR Combined sections by

instructor



Accessing Results

After selecting a term and indicating ‘Each Individual Section with its own results’, users can view each individual section with its own results, as selected on the previous screen for Term: 0405.

By selecting any of the courses, users can view the SSI report for that course.



Accessing Results



Accessing Results

For the ‘Combined sections by Instructor’ options, users can view combined sections with results, as selected on the term selection screen (Term: 0405 in this example).

By selecting any of the instructor names, users can view a combined SSI report for that instructor.

Note: multiple reports for a single instructor may be listed if the instructor taught courses with either different SSI survey forms, different academic orgs, or on different campuses.



Accessing Results



Accessing Results For an electronic version of the report (pdf), users can

select ‘View in Adobe Reader’ ‘Each Individual Section with its own results’ report

produces individual SSI results, three comparative measures, graph, and student comments.

‘Combined sections by Instructor’ report produces an instructor’s combined sections SSI results with three comparative measures, graph, and student comments by course. This report is somewhat similar to the SSI reports generated by AIS in the past.



Administration Dates SSI is intended to be an evaluation of the instruction

provided during the course. It is administered after the students have participated in the

majority of the course (no sooner than the 60% mark of the course) in order to provide the student an opportunity to experience the course instruction.

Length of the administration period is approximately the last 20%-25% of the course.

Faculty are provided default SSI administration start and end dates, and are permitted to make minor adjustments to these dates within a predetermined timeframe. This ensures a more standardized and consistent data collection process.



Changing Administration Dates

Users can locate the course that they would like to change and select the Calendar icon




They then click on “Next Month” on the upper right hand side of the calendar to select the next month. The sample calendar above shows the course ends on 12/18 and the SSI closes on 12/11.




To extend the SSI to remain open until 12/18… Select 12/18. It will turn blue through 12/18. This changes in real-time. Should you make a mistake, simply click back on the original survey end

date of 12/11 or select another. The latest you can extend the survey to is 12/24.



Response Rates Student response rates are calculated and

compared each term by program (academic org), campus, form type, and overall college.



Response Rates

Submission Entries by Campus for

0400 Term Submission Entries by Campus for

0405 Term SSI Submissions Split by Campus

0410 TermCampus Roster Entries Opt-outs* Average Roster Entries Opt-outs* Average Roster Entries Opt-outs AverageAllstate Center 1811 1388 0 76.6% 2540 1537 23 60.5% 1226 624 25 50.9%Clearwater Campus 16001 7194 0 45.0% 15292 5469 48 35.8% 6527 2209 35 33.8%Electronic Campus 9563 5937 0 62.1% 10658 4785 100 44.9% 5679 2674 57 47.1%EpiCenter 908 451 0 49.7% 1022 295 1 28.9% 271 113 2 41.7%

Health Education Center 8484 5121 0 60.4% 9271 4438 74 47.9% 4242 2091 97 49.3%Open Campus 654 151 0 23.1% 556 132 3 23.7% 52 24 0 46.2%Seminole Campus 8461 4911 0 58.0% 8101 3610 58 44.6% 4226 1942 41 46.0%St Pete/Gibbs Campus 20991 9332 0 44.5% 20706 7786 71 37.6% 8336 2739 23 32.9%Tarpon Springs Campus 15736 7034 0 44.7% 12634 6006 51 47.5% 4691 2310 48 49.2%Totals 82609 41519 0 50.3% 80780 34058 429 42.2% 35250 14726 328 41.8%*Opt-outs are included in the number of Entries and were not available during Fall Term



Response Rate Comparison

Comparison with another Florida public community college Another community college debuted its online course

evaluation system in Fall 2007. Until the Spring 2007 term, this community college relied on a paper-based course evaluation system for its traditional classroom courses.

Response rates:Pilot testSpring 2007: 30% classroom, 26% distance learningCollegewideFall 2007: 20% classroom, 26% distance learningSpring 2008: 24% classroom, 29% distance learning



SSI Revision Committee

An SSI Revision Committee was established in the Summer of 2009 to review the current SSI forms and make suggestions for revisions to the current items sets.

The new modality-neutral form was designed to address faculty and student engagement in the learning process and is being administered this term as a college-wide pilot.



Increasing Response Rates

While not the primary focus of the SSI Revision Committee, there was considerable interest in identifying and implementing ideas and processes to increase the response rate of the SSI.

Through creative brainstorming at the meetings and online, a list of “best practices” was developed.




SSI Revision Committee Recommendations

1. Send Constant Reminders (‘inescapable messaging’)2. Stress Confidentiality3. Imbed link in announcement page -- “inescapable messaging”4. Move every week to top of unit5. Link to “content” – graded or non-graded6. Bring students to lab /use of lab7. Give extra credit8. Genuinely convey the message that the information will be used

to make this class better9. Hold activities on campus, e.g., a table highlighting importance

for accreditation




There was some concern that item #5 (linking the SSI to accessing exams or other course materials) was not acceptable from a legal standpoint.

The question was posed to the Office of General Counsel, and, in brief, the answer was ‘yes’, the faculty could block access to course materials assuming that the content ‘blocked’ is just the next module or assignment the student would see, not specifically an exam.

The Office of General Counsel also advised against giving extra credit for taking the SSI.



Next Steps Work with programs to improve

response rates Finish creating forms for programs

using clinical items Develop reports which provide trend

information across terms and years Finalize new items



Questions

Transiting to an Online Course Evaluation Model: The Online Student Survey of

Instruction

February 2010

Florida Association of Institutional Research2010 Annual Conference

http://www.spcollege.edu/

transiting to an online course evaluation model: the online student survey of instruction

Documents

online course evaluation

online results

online coursesthis form

course evaluations

lecture ssi form

collegewide online process

ssi forms scantron

ssi db administratortransiting