the role of course evals in faculty reviews : one perspective

The role of course evalsin faculty reviews:

One perspective

John PetraitisCAFE

March 25, 2011

Summative vs. Formative

• Summative: –Course evals as ends–1st page of IDEA report

• Formative/Developmental–Course evals as means–Remaining pages of IDEA

Logic of IDEA• One size does not fit all

–Allows faculty to identify key objectives–E.g., exploration of personal value, vs.

creativity vs. understanding principles• Advertise the warts

–Reliability and representativeness

Three Key Pieces

• 12-item Faculty Info Form (FIF)

• 36-item Student Reaction to Instruction Form (SRIF)

• Multi-page Report

FIF

• 12 learning objectives

FIF• Which of the 12

objectives do I chose?– Depends on course– Depends on instructor– Depends on section– Depends on dept

• Recipes for disaster– Choosing all of them– Choosing none of them

(by not filling out FIF)

SRIF• Two kinds of Q’s

– 36 questions linked to 12 learning objectives.

– Extraneous influences, e.g.,

• Discipline• Student motivation• Class size

Marriage of FIF &SRIF• Weighting of

students responses (on SRIF) for results/report

• M = 0• I = 1• E = 2

The Report (page 1)

• Reliability and Representativeness– Reliability

• Based on number (not percent) of respondents• Would adding a few more respondents potentially alter the results

dramatically?

– Representativeness• Based on percent (not number) of respondents• 65% response rate is considered representative

The Report (page 1)

• Reliability and Representativeness– Reliability

• Based on number (not percent) of respondents• Would adding a few more respondents potentially alter the results

dramatically?

– Representativeness• Based on percent (not number) of respondents• 65% response rate is considered representative

The Report (page 1)

• Reliability and Representativeness– Representativeness is key– If representative, reviewers could use page 1 as

summative and formative– If not representative, reviewer might use if it

helps the candidate

• Averages:– Section A: “I” and “E” objectives

• Adjustments to IDEA Ratings– Discipline (national comparisons)– “Extraneous Influences” on

student ratings are used to adjust scores or ‘level the playing field’

• SRIF Item #39: I really wanted to take this course regardless of who taught it

• SRIF Item #43: As a rule, I put forth more effort than other students on academic work

• Class size

The Report (page 1)

• Improving Teaching Effectiveness (food for thought)

The Report (p. 2-3)

IDEA and your review file• Do your own formative evaluation

– What can you learn from pages 2-3?– Show your interest in professional

development– Do it even if IDEA is not reliable or not

representative• Do your own summative evaluation

– Make it easy. Make it clear.

PSY A260

PSY A375

PSY A412

All Courses

0 10 20 30 40 50 60 70 80

Comparison of IDEA evaluations for J. Petraitis to National Averages in Psychology

National Average in PSY A. Progress on relevant objectives B. Overall Rating: Excellent teacherC. Overall Rating: Excellent course D. Average of B & C Summary Evaluation (Av. of A & D)

…instructor clearly communicates objectives…

…agreement betw course objectives & what's taught.

…communicates how students are evaluated.

…agreement betw announced and used eval methods.

…instructor gives periodic feedback.

…prepared for class.

…meets/dismisses class on time.

…assignments are helpful…

…exams represent course objectives/materials.

…instructor is helpful with problems/questions…

…instructor is available for consultation…

…teaching aids (e.g., A/V, materials) are helpful…

…questions/comments are respected…

…instructor creates enthusiasm/desire to learn.

…instructor speaks clearly…

…instructional materials are excellent.

…conceptual learning, not rote memorization.

…recommend instructor to other students.

…glad I took course; would recommend it…

…instructor uses class time efficiently.

…attendence is critical for success.

…friendly, relaxed atmosphere….

…easily follow instructor & take notes.

…instructor is sensitive to student problems.

…workload (1=much lighter, 3=same, 5=much more)

…hours/wk (1=0, 2<2, 3=3to4, 4=4to7, 5>7)

2.5 3 3.5 4 4.5 5

CAS-SSPsy DeptAll Petraitis