student evaluation of teaching and course (setc) on ... · student evaluation of teaching and...

Student Evaluation of Teaching and Course (SETC)

On-boarding Package

Prepared by Kiran Bisra, Director, Learning Experiences Assessment & Planning (LEAP)

2

Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc

Contents 1. Introduction:......................................................................................................................................... 3

1.1 Objective ...................................................................................................................................... 3

1.2 SETC Administration ................................................................................................................. 3

1.3 Background ................................................................................................................................. 3

2. Cascading Framework ....................................................................................................................... 4

3. Question Form .................................................................................................................................... 4

3.1 Instructor Question Selection (IQS) ......................................................................................... 5

4.1.2 Aggregated Faculty Report ............................................................................................... 5

4.1.3 Aggregated Evaluating Unit Report ................................................................................. 5

4.1.4 Course Report ..................................................................................................................... 5

4.1.5 Individual Report ................................................................................................................. 6

4.2 Instructor Narrative ..................................................................................................................... 6

4.3 Report Access ............................................................................................................................. 7

5. Semester Workflow ............................................................................................................................ 8

Step 1: Data Verification ....................................................................................................................... 8

Step 2: Instructor Question Selection (IQS) ....................................................................................... 9

Step 3: Evaluation Period ..................................................................................................................... 9

Step 4: Release of Reports ................................................................................................................... 9

6. On-boarding ...................................................................................................................................... 10

Step 1: Gather information .................................................................................................................. 10

Step 2: Data Verification Training ...................................................................................................... 10

Step 3: Implementation of Pilot Project ............................................................................................. 10

Step 4: Assessment of Pilot ................................................................................................................ 11

Step 5: Complete roll-out .................................................................................................................... 11

Appendix A: Evaluation Form for FASS ................................................................................................ 12

Appendix B: Evaluation Form for FENV ............................................................................................... 14

Appendix C: Evaluation Form for FCAT ............................................................................................... 16

Appendix C: Join Form ............................................................................................................................ 18

3


1. Introduction: The Student Evaluation of Teaching and Courses (SETC) initiative is a new online system that has replaced pen-and-paper course evaluations in many SFU academic units. The SETC team is currently scaling up the program to the remainder of the University.

Please visit our website to find out more: www.sfu.ca/setc

1.1 Objective

SFU has a strong commitment to teaching and learning, and the SETC initiative is a reflection of this commitment. The goal of the SETC program is to meaningfully collect and disseminate feedback from students on teaching, learning, and course delivery at SFU.

1.2 SETC Administration

SETC is in the AVP Learning and Teaching portfolio. The SETC Team is situated in the Learning Experiences Assessment & Planning (LEAP) Division of the Centre for Educational Excellence (CEE) and handles the operational aspects of the program (i.e. administering evaluations, generating reports, research and measurement).

1.3 Background

The SETC initiative has been a long time in the making. The process started with a community engagement process back in 2012. The online evaluation instrument was piloted in Summer and Fall 2015, and the following academic units are currently in the system:

Education Health Sciences Applied Science Science Arts and Social Sciences (7 departments) Communication, Arts and Tech (3 units) Environment

There are several circumstances in which a course will be excluded from the SETC system. Sections taught by Teaching Assistants are excluded due to specific considerations outline in their Collective Agreement. The SETC team is working with the TSSU to resolve this situation and on-board their members. As well, Center for Online and Distance Education (CODE) courses, CO-OP Placements, and sections with fewer than five students are not evaluated using the SETC form.

4


SFU has purchased third-party software called “blue by eXplorance” to administer the

evaluations. Members must login using their SFU Central Authenticating Service (CAS) to

“blue” to fill out evaluations, view reports, or write questions.

The SETC program is in a state of review. There may be changes to the structure, questions

and reports in the near future.

2. Cascading Framework It is a challenge to meet the needs of all stakeholders with a single evaluation form or report. Therefore, SETC evaluation forms and reports are customizable based on by a 4-tier cascading framework (Figure 1). An Evaluating Unit is either a Department, a School, or, for Faculties that do not have academic units, the Graduate/Undergraduate Program.

Figure 1: Cascading Framework

3. Question Form SETC evaluation forms are customizable based on the cascading framework described above. At SFU, there are 12 questions chosen by the VPA’s office which appear on all forms. Then, there are four questions chosen by the Dean’s Office which appear on all courses within that respective faulty. The Evaluating Unit Head chooses a further four more questions to appear within their unit. Lastly, each instructor has the option of selecting up to four questions per course section they taught.

Usually the questions chosen at the various levels are the result of consultation. For

example, the institution-level questions were constructed by a Senate Committee on

Teaching and Learning (SCUTL) Working Group. Some Faculties/Evaluating Units have

chosen their questions after consultations with instructors, while others task a handful of

faculty members to make this decision.

All current SETC forms are available here online.

Institution

Faculty

Evaluating Unit

Instructor

https://www.sfu.ca/setc/for-instructors/setc-forms-by-department.html

5


3.1 Instructor Question Selection (IQS) Instructors are emailed a link every semester to select or create up to four questions per course. This link redirect them to “blue” where they have access to an item bank. Currently, no one examines these questions before they are sent out to students.

Instructions on how to select questions are available here online.

3.2 Sessional Instructors As of right now, Sessional Instructors only receive the University-level and Instructor-selected questions. The reason for this is related to the TSSU Collective Bargaining Agreement.

4. Reporting

Reporting follows the same four-tier cascading model used to select questions. Five reports

are generated and an email notification is sent out to all Report Viewers at the end of week

1 of the subsequent semester. Report Viewers log into “blue” to access their reports. For

an overview, see Table 1.

4.1 Reports

4.1.1 SFU Report

The SFU Report is sent to the AVP Teaching and Learning and contains only the Institution-level questions. Frequencies, scores and response rates of all respondents are aggregated for this report.

4.1.2 Aggregated Faculty Report

The Aggregated Faculty Report is sent to the Dean’s Office and contains the Institution and Faculty-level questions. Frequencies, scores and response rates of students who took instruction within the faculty are aggregated for this report.

4.1.3 Aggregated Evaluating Unit Report

The Aggregated Evaluating Unit Report is sent to the Evaluating Unit’s Office and contains the Institution, Faculty, and Evaluating Unit-level questions. Frequencies, scores and response rates of students who took instruction within the Evaluating Unit are aggregated for this report.

4.1.4 Course Report

The Course Report is sent to the Evaluating Unit’s Office and contains the Institution, Faculty, and Evaluating Unit-level questions. Frequencies, responses, and response rates of

https://www.sfu.ca/setc/for-instructors/instructors-how-to-select-your-questions.html

6


students who took instruction within the course are aggregated for this report. This recipient may receive dozens of Course Reports per semester.

4.1.5 Individual Report

The Individual Report is sent to the instructor of the section, and contains the Institution, Faculty, Evaluating Unit, and instructor selected questions. Frequencies, responses, and response rates of students who took instruction within the course are aggregated for this report. This recipient will receive the same number of reports as courses taught each semester.

4.2 Instructor Narrative

During the evaluation period, instructors are able to submit a brief narrative or make note of any circumstances or self-reflections they would like appended to their Individual and Course reports; this narrative is completely optional.

7


Table 1: Summary of Reports

Report Name Recipients Question-level Instructor Narrative

Institution Faculty Evaluating Unit

Instructor

SFU AVP - LT Aggregated Faculty Dean Aggregated Evaluating Unit Evaluating Unit Head Course Evaluating Unit Head Individual Instructor

4.3 Report Access

Each unit is tasked with keeping an up to date Report Viewer lists. If collating and archiving reports are within one’s job description, then access is granted. Chair’s Secretary, Manager of Administration, and Program Assistant are three common personnel who have access. In order to complete their tenure, promotion or salary appeal duties, the VPA and Faculty Deans will be given access to Course Reports. Besides the instructor, access to Individual Reports are never granted to another Report Viewer.

We require that you identify a Report Viewers List steward who can login to our Report Viewers Application and make edits to the list as required. Course evaluation reports contain sensitive information about instructors. Failure to remove viewers who no longer require access could result in a serious breach of privacy.

The application can be found at: setcreportviewers.its.sfu.ca

https://setcreportviewers.its.sfu.ca/

8


5. Semester Workflow

There are four main steps in each semester.

Step 1: Data Verification

During the first step, information is inputted into “blue” from several sources.

Often, the questions and Report Viewers will be the same term to term, and no change will be required. If a change to the questions is required, email [email protected] at the beginning of the term.

Course lists, course end dates, and instructor information is pulled from the Student Information Management System (SIMS). This data needs to be checked by Evaluating Unit Administrative personnel. Common corrections include:

Instructor change after the term has started (i.e. emergency, sickness); Accounting for team or co-taught courses; Combining course sections when an instructor teaches several small sections and

one report is required; Combining cross-listed courses to generate one report for the course; Editing the course end date to align with course practices.

Figure 2: Fall 2018 Semester Workflow Overview

September October November December

4. Release of Reports

1. Data Verification

2. Instructor Question Selection

3. Evaluation Period

Last Day of Classes

mailto:[email protected]

9


These changes are made in a SIMS component called “SETC EDITS”. Each unit decides who is responsible for verifying this data. Commonly chosen positions include Manager of Administration, Scheduler, and Program Assistant. Once this decision has been made, please email [email protected] to schedule a training session. The session will go into detail on how to make changes in SETC EDITS and which issues to keep an eye on as the semester proceeds.

Evaluating Units are not required to provide enrollment information. Enrollment lists are updated each day from SIMS. Please note that it is possible for a student to be enrolled at the time the evaluation period starts, but then deregister before the reports are sent out. In this case, it is possible for the report to list a higher number for respondents than enrolled.

Step 2: Instructor Question Selection (IQS)

Instructors are emailed a link every semester to select or create up to four questions per course (see Section 3.1). Instructors select/create their questions right before the evaluation period.

Instructions on how to select questions are available here online.

Step 3: Evaluation Period

For most classes, the evaluation period will be the last two weeks before the end of classes. If the class ends early in the term, it may be the week before the last instruction day.

Instructors receive an email when the evaluations open up with:

1. Evaluation end dates 2. A link to access a response rate monitor which updates in real time, and 3. An invitation to submit a brief narrative (see Section 4.2).

For more information on the response rate monitor click here.

Step 4: Release of Reports

An email notification is sent out to all Report Viewers a day or two after the grade submission deadline. Report Viewers log into “blue” to access their reports.

mailto:[email protected]

https://www.sfu.ca/setc/for-instructors/instructors-how-to-select-your-questions.html

https://www.sfu.ca/setc/for-instructors/instructor-response-rate-monitor-.html

10


6. On-boarding

As the on-boarding process gets underway, it will be important to regularly communicate with instructors. While the SETC Team can support these communications, they are best and most effective if sent by the Evaluating Unit Head.

Step 1: Gather information

There are three key pieces of information the SETC team needs in order to proceed with

setting-up your unit:

1. The name of 1 or 2 data verification personnel 2. Report Viewers list for the Course Reports and identification of steward 3. Unit-level questions

Your unit may choose up to four questions. The institution and faculty-level questions that your students will receive can be found in Appendix A or B. Often one or a handful of faculty members are tasked with selecting the questions. It is recommended that the questions be chosen after some level of consultation with instructors.

If your future plans are to use the same questions for both Sessional Instructors as well as

with continuing faculty, then you must consult with the Teaching Support Staff Union

(TSSU).

Questions are categorized as either a course question or a teaching question. If a course is

taught by multiple instructors, course questions appear once on the evaluation form,

whereas teaching questions are asked about each instructor.

If you would require any help with selecting or formulating your questions, please contact

Lana at [email protected].

Once you have gathered the three pieces of information, please email them to Kiran at

[email protected].

Step 2: Data Verification Training We will contact the data verification personnel and set-up a SETC EDITS (see Section 5.1) training session. This training is crucial for implementation success.

Step 3: Implementation of Pilot Project

Next, a pilot project will be administered. Whether all courses or just a handful of courses are in the pilot will be determined by the SETC Team. It will depend on a myriad of factors, including the complexity of evaluating unit course offerings. This small scale preliminary implementation will allow us the opportunity to evaluate processes and improve on them before the complete roll-out.

11


Step 4: Assessment of Pilot

A meeting will be held between the SETC Team and the evaluating unit to discuss issues or concerns that arose from the pilot. Topics may include timelines, revamping the questions, changes in SIMS data processes, or alterations to the report viewers list.

Step 5: Complete roll-out

Lastly, a complete roll-out will be implemented.

12


Appendix A: Evaluation Form for FASS

University-level Questions

General

1.

I attended class…

All the time

Most of the

time

About half of

the time

Rarely

Never

2.

How easy was the course?

Very Easy

Easy

Medium

Hard

Very Hard

3.

Overall, the quality of my learning experience in this

course was…

Very Good

Good

Fair

Poor

Very Poor

Course

Strongly Agree

Agree No Opinion

Disagree Strongly Disagree

4. The different course parts/activities (lectures, labs,

tutorials, online forums, discussions, etc.) were

connected.

5. Course materials (textbook, readings, handouts,

assignments, etc.) improved my understanding of the

course content.

6. The assessments in this course (tests, assignments, essays,

etc.) allowed me to demonstrate my understanding of

the course content.

7. Course activities (lectures, discussions, group work, labs,

etc.) were engaging.

Teaching

Strongly Agree

Agree No Opinion


8. The course instructor explained course concepts clearly.

9. The course instructor explained grading criteria clearly.

10. The course instructor created a respectful learning

environment.

13


11. The course instructor was approachable when students

asked for guidance.

Open-ended

12. Do you have any further comments?

Faculty-level Questions: Arts and Social Sciences (FASS)

Course

Strongly Agree

Agree No Opinion


13. We are interested in knowing to what extent courses

promote critical thinking. To think critically means to

carefully evaluate an idea or hypothesis on the basis of

the available evidence, and to try to reach a justifiable

conclusion using rational analysis within the context of a

specific discipline.

My experience in this course has encouraged me to think

critically.

14. My experience in this course has motivated me to learn

more about the subject.

Teaching

Strongly Agree

Agree No Opinion


15. When explaining course concepts, the instructor

connected the concepts to the major themes of the

course.

16. The course instructor made it clear what students were

expected to learn in the course.

14


Appendix B: Evaluation Form for FENV

University-level Questions

General

1.

I attended class…

All the time

Most of the

time

About half of

the time

Rarely

Never

2.


Very Easy

Easy

Medium

Hard

Very Hard

3.


course was…

Very Good

Good

Fair

Poor

Very Poor

Course

Strongly Agree

Agree No Opinion




connected.



course content.



the course content.



Teaching

Strongly Agree

Agree No Opinion





environment.


asked for guidance.

15


Open-ended


Faculty-level Questions: Environment

Course

Strongly Agree

Agree No Opinion


13. This course improved my understanding of the built,

social, and/or natural environment

14. This course made me more aware of the challenges and

opportunities for creating a more just and/or sustainable

world.

15. I was able to link course concepts to current issues and

real-life situations.

16. This course inspires me to take more courses in the

Faculty of Environment.

16


Appendix C: Evaluation Form for FCAT

Evaluation Form: University-level Questions

General

1.

I attended class…

All the time

Most of the

time

About half of

the time

Rarely

Never

2.


Very Easy

Easy

Medium

Hard

Very Hard

3.


course was…

Very Good

Good

Fair

Poor

Very Poor

Course

Strongly Agree

Agree No Opinion




connected.



course content.



the course content.



Teaching

Strongly Agree

Agree No Opinion





environment.


asked for guidance.

17


Open-ended


Faculty-level Questions: Communication, Art and Technology (FCAT)

School of Interactive Arts and Technology

General

Very Light

Light Average

Heavy

Very Heavy

13. Compared to other courses, the workload in this course

was:

Course

Strongly Agree

Agree No Opinion


14. The course assignment, projects, tests and/or papers

helped me to develop skills I can use in other courses.

15. The course provided opportunity for me to enhance my

problem-solving skills.

16. The course assignments, projects, tests, and/or papers

highlighted important concepts of the course.

17. The course expanded my understanding on important

issues in the subject matter.

Teaching

Strongly Agree

Agree No Opinion


18. The instructor related course concepts to professional

practices in the field.

19. The feedback I received in this course on assignments,

projects, tests, and/or papers provided guidance on how

to improve my performance in the course.

Very Good

Good Fair Poor Very Poor

20. Overall, the quality of instruction provided by the

instructor in the course was:

18


Appendix C: Join Form

A) Data Verification Personnel

Course lists, course end dates, and instructor information is pulled from the Student Information

Management System (SIMS). This data needs to be checked by Evaluating Unit Administrative

personnel.

We need three key pieces of information about your data verification personnel:

- Name - Position - Email

B) Course and Aggregated Evaluating Unit Report Viewers List

We need three key pieces of information about your Report Viewers.

- Name - Position - Email

As well, we need the name of your Report Viewers List steward (the person who updates your

viewers list).

C) Evaluation Form Questions Please submit the following information about your unit’s evaluation questions.

Question

Type Course

Instructor

Response Scale Strongly Agree, Agree, No Opinion, Disagree, Strongly Disagree

Very Good, Good, Fair, Poor, Very Poor

Very Heavy, Heavy, Neither Heavy/Light, Light, Very Light

Other:

Email information to: [email protected]

student evaluation of teaching and course (setc) on ... · student evaluation of teaching and...

Documents