c.e.l.t developing a comprehensive faculty evaluation system jeff schlicht, ph.d. associate...
TRANSCRIPT
![Page 1: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/1.jpg)
C.E.L.T
Developing a Comprehensive Faculty Evaluation System
Jeff Schlicht, Ph.D.Associate Professor
Department of Health Promotion and Exercise [email protected]
![Page 2: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/2.jpg)
Measurement vs. Evaluation
Measurement… is the process of systematically assigning numbers to the individual members of a set of objects or persons for the purpose of indicating differences among them in the degree to which they possess the characteristics being measured.
Robert L. Ebel, “Measuring Educational Achievement”, 1965
![Page 3: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/3.jpg)
Measurement vs. Evaluation
Evaluation is the process of interpreting a measurement by comparing it to a value (or set of values) that represent an agreed upon standard. This process involves judgment.
![Page 4: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/4.jpg)
A Comprehensive System = Measurement and Evaluation
In order to develop a comprehensive faculty evaluation system, we must:
1) Develop technically good (valid & reliable) measurement instruments
2) Agree upon a university standard for desirable faculty performance
![Page 5: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/5.jpg)
How Do We Currently Measure Faculty Teaching Performance?
In the HPX department we use:
1) Student rating forms
2) Peer observation (in classroom)
Are either of these forms of measurement valid and/or reliable?
The answer is 1) we don’t know and 2) no
![Page 6: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/6.jpg)
Student Ratings of Faculty Performance
The development of a valid and reliable questionnaire, i.e. a student rating form, is a technical process that has a body of science behind it.
Our department does not know who created the current rating form, and whether it has ever been tested for validity and reliability.
![Page 7: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/7.jpg)
Student Ratings of Faculty PerformanceDos and Don’ts
DO include a balanced response scale• equal number of positive and negative responses• mirror-image opposites
– Strongly agree agree disagree strongly disagreeDO develop items (questions) important to faculty AND
studentsDO conduct field tests of the items ( = collect data) to do
reliability and validity analysesDO develop norms (a year’s worth of data) to allow fair
and appropriate interpretation (i.e. comparison)DO consider using a professionally designed student
rating form
![Page 8: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/8.jpg)
Student Ratings of Faculty PerformanceDos and Don’ts
DON’T• Have two items in one question (double-
barreled response item)• Simply hand back a summary sheet of student
ratings to instructor – include personal consultation to foster instructional improvement
• Consider individual student responses in faculty evaluation
![Page 9: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/9.jpg)
Peer Observation
The short answer: DON’T do it for personnel decisions
(can be useful for faculty enrichment)
The addition of another faculty member or administrator into the classroom ALWAYS alters the classroom dynamic in such a way as to provide an abnormal view of what usually happens there
THOSE DATA ARE NEVER VALID
![Page 10: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/10.jpg)
Peer Observation
If you must do it for faculty evaluation consider:
• multiple visits by multiple (4) peers– 8 visits = adequate sampling– develop valid, reliable checklist
• train observer team in use
– prepare students, instructor– schedule post-observation conference
• videography
![Page 11: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/11.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 1: Define the faculty role model
• What should be evaluated?– Start at the department level
• Have faculty write down everything they do at WCSU to create a master list of responsibilities
• Determine faculty roles– What are the things we are accountable for?
• Assign the items from the master list of faculty responsibilities to the various faculty roles
![Page 12: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/12.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 2: Determine faculty role model parameters on a University (system?) level
• How important should each role be?– Static vs. dynamic role model
• Static = fixed weights
• Dynamic = range of weights
This is missing from our current system, and therefore we do not control subjectivity as well as we could
![Page 13: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/13.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 2: Determine faculty role model parameters on a University (system?) level
• Static Role Model– Load Credit = 50%– Creative Activity = 25%– Productive Service = 15%– Professional Service = 10%
![Page 14: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/14.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 2: Determine faculty role model parameters on a school, then University (system?), level
• Dynamic Role ModelMinimum
Maximum
– Load Credit 35% 75%– Creative Activity 20% 50%– Productive Service 5% 45%– Professional Service 0% 40%
![Page 15: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/15.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 2: Determine faculty role model parameters on University (system?) level
• Dynamic Role Model– The advantage of a dynamic role model is that it
allows various levels of the University to play to their strengths
• Allow each department to set dynamic range values, then select extreme max. and min. as University standards
![Page 16: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/16.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 3: Use observable or otherwise documentable achievements/products/performances to define all the roles in the faculty role model
• Teaching (student interactions that promote learning)– Content appropriateness/currency
– Instructional design skills
– Instructional delivery skills
– Instructional assessment skills
– Course management skills
![Page 17: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/17.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 4: Define weights for each component of a single role – departments should be allowed to have their own weights, as long as they fit into the University structure. This embeds department values into the evaluation design:
• Teaching min max
– Content appropriateness/currency 10% 40%
– Instructional design skills 10% 30%
– Instructional delivery skills 20% 60%
– Instructional assessment skills 20% 50%
– Course management skills 0% 30%
![Page 18: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/18.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 5: Identify where you will collect the data to evaluate the roles• Use sources that have first hand experience with the performance in
question – preferably multiple sources of input• Teaching
– Content appropriateness/currency• syllabus review by peer• Self-review
– Instructional delivery skills• Student evaluation• Peer evaluation???• Self-review
– Course management skills• Chair or administrative assistant evaluation• Self-review
![Page 19: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/19.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 6: Define weighting for information provided by different sources
• Teaching min max– Students 15% 50%– Self 15% 40%– Peer 15% 30%– Chair 15% 30%
![Page 20: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/20.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 7: Determine how to gather information
• Questionnaire
• Interview
• Checklist
![Page 21: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/21.jpg)
8 Steps to a Comprehensive Faculty Evaluation System
Step 8: Design data collection instruments
![Page 22: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/22.jpg)
Evaluation for Promotion and Tenure
Evaluation is the process of interpreting a measurement by comparing it to a value (or set of values) that represent an agreed upon standard.
This process involves judgment.
In a system like the one described today, evaluation is an on-going process involving multiple sources and evaluators. We are constantly comparing ourselves against the WCSU Faculty Role Model, so we have immediate feedback about our performance.
![Page 23: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/23.jpg)
Evaluation for Promotion and Tenure
Promotion and tenure decisions can be based on comparing our performance against a
pre-determined, minimum required composite Faculty Role Model score. In
other words, prior to becoming eligible for advancement, we would already know the
outcome.
![Page 24: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/24.jpg)
Comprehensive Faculty Evaluation
Practical steps we can take to improve the system:• Initiate a discussion about what it means to be a
faculty member at WCSU in the 21st century• Discover new ways to measure faculty performance,
tests those ways to make sure they are valid and reliable, and create a feedback loop that uses that information to enhance faculty performance
• Lobby the union and the University to more actively support faculty enrichment
![Page 25: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/25.jpg)
Final Thoughts About Faculty Evaluation
Why should we do it?
• Promotion and tenure is important, but…
the true value of assessment lies in its ability to direct faculty enrichment. If evaluation is not tied to enhancing faculty performance, it will be perceived as punitive, a hurdle, rather than as a path toward self-improvement.
![Page 26: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/26.jpg)
Acknowledgements
These materials are summarized from the ideas of Raoul A. Arreola, University of Tennessee
Health Science Center, presented at the Center for Educational Development and Assessment
professional enrichment seminar in Orlando, FL. March 10-11, 2008, and available in the text:
“Developing a Comprehensive Faculty Evaluation System,”
3rd edition, Arreola RA, Anker Publishing Co., Bolton, MA 2007
![Page 27: C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences](https://reader031.vdocuments.net/reader031/viewer/2022032803/56649e265503460f94b153a4/html5/thumbnails/27.jpg)
C.E.L.T
Developing a Comprehensive Faculty Evaluation System
Jeff Schlicht, Ph.D.Associate Professor
Department of Health Promotion and Exercise [email protected]