1

Background on Student Evaluation of Teaching

“There have been more than 70 years of research on student ratings. Even today

the design, development, use, and interpretation of student ratings continue to be

one of the most heavily researched topics in the general area of faculty

evaluation.” [Arreola, p. 83]

“The earliest studies were by H. Remmers [at Purdue in 1927] and were

concerned with exploring student opinions as one way to find out more about

teaching/learning for „self-improvement of instruction‟ and for psychometric

reasons.” In 1949 Remmers “concluded that „there is warrant for ascribing

validity to student ratings not merely as measures of student attitude toward

instructors . . . but also as measured by what students learn of the content of the

course.‟”[Nuhfer, p. 6]

“In the 1960s teaching evaluation was conducted in response to student demands

for public accountability.” In the 1970s, evaluations were more often intended “to

help faculty improve and develop.” “In the 1980s, teaching evaluation has been

driven by administrative rather than faculty needs.” [Ory, p. 30]

“There are now more than 1,500 references dealing with research on student

evaluations of teaching.” [Cashin, p, 1.]

2

Reliability and Validity of Evaluations

Reliability of student ratings means that they are consistent for whatever it is they

are measuring. The internal consistency of student ratings is quite good and

becomes excellent as the number of students doing the rating increases.”[Wankat

& Oreovicz, p. 312] That is, they are reliable if students largely agree in

evaluating a professor.

“Students‟ ratings are considered to be very reliable in terms of consistency and

stability over time.” [Rotem and Glasman, p. 498]

“Validity means that student ratings are measuring what they are supposed to be

measuring.” “There is broad agreement in the literature that a reasonably strong

positive correlation exists between student achievement and student ratings.”

This is drawn from five large-scale studies. The most important study “was the

meta-analysis of Cohen” which showed “sections where students learned more

rated the instructor and the course higher than sections where students learned

less.” [Wankat and Oreovicz, p. 313.]

3

KEY QUESTIONS ADDRESSED IN THE LITERATURE ON EVALUATIONS

Are student evaluations essentially a popularity contest?

“In reviewing both written and objective student comments, Aleamoni (1976)

found that students frankly praised instructors for their warm, friendly, humorous

manner in the classroom, but if their courses were not well organized or their

methods of stimulating students to learn were poor, the students equally frankly

criticized them in those areas.” The author also found similar conclusions in six

studies published between 1971 and 1983. [Arreola, p. 84]

Citing studies from 1928-1960, Costin argued that there is “little direct evidence

to the viewpoint” that students “judge instruction on the basis of its

„entertainment‟ value rather than on its information, contribution to learning, or

long-term usefulness.”[Costin, pp. 516-517]

“The assumption that popularity somehow means a lack of substance or

knowledge or challenge is totally without merit. There are not studies to support

this view.”[Theall, p. 49]

Is it true that only alumni can make accurate assessments of a professor’s teaching

abilities?

Five studies between 1951 and 1979 reveal that “alumni who have been out of

school 5 to 10 years rated instructors much the same as students currently

enrolled.” [Arreola, p. 84]

“Student ratings correlate with alumni ratings . . . .” “This belies the conventional

wisdom that the students will come to appreciate our teaching after they get into

the real world as working adults.” [Cashin, p. 3 and Costin, p. 516]

4

Citing a study by Scriven, Kulik concludes alumni surveys are not very helpful.

“They usually have extremely low response rates and relate to „ancestors‟ of

current performance. Alumni perspectives are sometimes dated—teachers

change.”[Theall, p. 11]

Are peer evaluations superior to student evaluations of teaching?

“Student ratings correlate with colleague‟s ratings.” “There were some

differences in emphasis between the two groups. Students tended to place more

weight on the instructor being interesting, having good speaking skills, and being

available to help; students also focused more on the outcomes of instruction, e.g.,

what they learned. Faculty placed relatively more weight on intellectual

challenge, motivating students, setting high standards, and fostering student self-

initiated learning.”[Cashin, p. 3]

“The visits themselves alter teaching, Scriven points out, and the number of

experts and visits is usually too small to yield a reliable measure of teaching

effects.”[Theall, p. 11]

Does class size affect student evaluations?

Despite a number of studies (he cites eight), “the research literature does not

support the belief that a consistent relationship between class size and student

ratings of any sort exists.” [Arreola, p. 85]

Based upon two studies, the authors conclude, “There is evidence of a small but

consistent relationship between course size and student ratings—with small

classes . . . receiving higher ratings.” However, they also note one study that

found that ratings were higher for large classes. [USAF Academy Study, p. 7]

5

“Although there is a tendency for smaller classes to receive higher ratings, it is a

very weak inverse association.” [Cashin, p. 5]

“When there are fewer than fifteen students in a class, the ratings are significantly

higher than they are otherwise.” “As the class gets larger the ratings decrease.”

“For very large classes (more than 200 students), several studies show that ratings

go back up. This may occur because departments assign their best teachers to

large classes.” [Wankat & Oreovicz, p. 315]

Costin, after examining several studies, found that the evidence is not real strong

on this. [Costin, p. 521]

In a study done in twelve medical schools in China with 24,757 student ratings of

597 teachers, the authors concluded: “the smaller the class, the higher the ratings

teachers receive from students.” [Min and Baozhi, p. 14]

“Those who teach large classes are generally at some slight disadvantage in their

ratings, . . .” ”[Nuhfer, p.13]

Does the age of the instructor have an effect on evaluations?

“In general age, and also years of teaching experience, are not correlated with

student ratings. However, where small differences have been found, they tend to

be negative.” [Cashin, p. 4]

Does the race of the instructor have an effect on evaluations?

There are few studies on this question. [Cashin, p. 4]

In her study of student‟s evaluations in economics classes at San Diego State in

the 1990s, Harris found no bias against “non-white” professors. [Harris, p. 22]

6

Does the gender of the instructor play a role in evaluations?

After an examination of eight studies, Arreola concluded, “Clearly, no consistent

relationship between gender of the student and the instructor in student ratings has

emerged in the literature.” [Arreola, p. 85]

Based upon four studies, the authors concluded that gender of student or

instructor played no role. [USAF Academy study, pp. 4 and 5]

At most, there is a small gender bias. A review of the studies led Cashin to

conclude, “female students rated female teachers higher, and male students rated

male instructors higher.”[Cashin, p. 4]

Gender of students “not related to student ratings.” [Cashin, p. 5; Costin, p. 520;

and Harris, p. 22]

“Reviews of gender studies . . . have reached similar conclusions: there is no

strong or regular pattern of gender-based bias in ratings.”[Theall, p. 50]

Does the rank of the instructor affect evaluations?

After examining thirteen studies, he concludes there is “no consistent relationship

between faculty rank and student ratings.” [Arreola, p. 85; and USAF Academy

study, p. 5]

“regular faculty tend to receive higher ratings than graduate teaching assistants”

[Cashin, p. 5]

Does a professor’s appearance affect evaluations?

“Daniel Hamermesh, a professor of economics at the University of Texas at

Austin, and Amy Parker, one of his students, found that attractive professors

consistently outscore their less comely colleagues by a significant margin on

7

student evaluations of teaching. The findings, they say, raise serious questions

about the use of student evaluations as a valid measure of teaching

quality.”[Montell]

Does taking the course as a requirement or an elective affect student evaluations?

Five of seven studies cited show “that students who are required to take a course

tend to rate it lower than students who elect to take it.” [Arreola, p. 85. Also, see

Costin, p. 520]

Is there a difference in ratings from majors and non-majors?

Four studies reveal “there are no significant differences and no significant

relationships between student ratings and whether they were majors or

nonmajors.” [Arreola, p. 85; and USAF Academy study, p. 7]

Professors who teach courses in students‟ majors “have a major advantage in their

ratings over other teachers.” ”[Nuhfer, p. 13]

Does the academic discipline have an effect on teaching evaluations?

They are slightly higher in the humanities, arts, and education than in the “social

sciences, physical sciences, math, engineering, and business administration.”

[USAF Academy study, p. 7; Cashin, p. 6; and Wankat & Oreovicz, p. 315]]

Does the course level have an impact on student evaluations?

Based only upon one study, the authors conclude, “Higher level courses tend to

get higher ratings.” [USAF Academy study, p. 6]

8

Does the grade level of the student affect evaluations?

Most studies report “that graduate students and/or upper division students tended

to rate instructors more favorably than did lower division students.” [Arreola, p.

85. Also, see USAF Academy study, p. 4]

Does perceived workload or difficulty of class have an effect on evaluations?

“contrary to faculty belief, they are correlated positively, i.e., students give higher

ratings in difficult courses where they have to work hard.” [Cashin, 6 and Wankat

& Oreovicz, p. 316]

“research has shown that students choose courses in which they will learn more

and rate courses according to the result of their learning gains.” [Haynes and

Hunts, p. 135]

Are student ratings on single general items an accurate measure of teaching

effectiveness?

Based upon two studies, he concludes “the use of single general items should be

avoided, especially for tenure, promotion, or salary considerations.” [Arreola, p.

86. Also, see Nuhfer, p. 20]

Citing several studies in the 1980s and 1990s, Cashin concluded the studies

“suggested that one or a few global or summary type items might provide

sufficient student rating data for personnel decision.” [Cashin, p. 2]

“For administrative purposes, global questions on teaching effectiveness should

be used since they have the highest correlations with student achievement.”

[Wankat & Oreovicz, p. 309]

9

“the correlations of the global ratings with achievement were generally higher

than those of the most specific instructional practices for both the random and

nonrandom groups.” [Centra, “Student Ratings,” p. 22]

Does prior subject interest have an effect on evaluations?

“Instructors are more likely to obtain higher ratings in classes where students had

a prior interest in the subject matter.” “Prior subject interest correlates most

highly of all background variables with teacher effectiveness.” [USAF Academy

study, p. 5. Also, see Wankat & Oreovicz, p. 315]

How important are instructor traits to teaching evaluations?

Citing two studies, Harris found “enthusiasm,” “interest stimulated,” “interaction

with students,” and “clarity of communication” as the most important teacher

characteristics affecting student evaluations. [Harris, p. 4]

Based upon evaluations of all economics courses taught at San Diego State

between 1994 and 2001, Harris concluded, “The most influential determinants of

student evaluation ratings are communication, organization and presentation, and

knowledge of the subject.” [Harris, p. 22]

Do student evaluations reflect effective teaching?

In a review of several studies: “In the typical study, different instructors teach

different sections of the same course, using the same syllabus, and textbook, and

most importantly using the same external final exam, i.e., an exam developed by

someone other than the instructors.” Cashin concluded, “the classes in which the

students gave the instructor higher ratings tended to be the classes where the

10

students learned more . . . .” [Cashin, p. 3. Also, see Centra, “Student Ratings,”

pp. 17-24]

“Michael Theall, an associate professor of educational administration at the

University of Illinois at Springfield, calls the evaluations, „valid measures of

students‟ satisfaction with their experience.‟ He and other researchers point to a

set of about four dozen studies that have tested the validity of student ratings.

Researchers compared the quality of teaching in several sections of the same

course and gave students in each section the same final exam. The studies found

that sections of students who did well overall tended to give higher ratings to the

instructor than did those of students who did poorly.” [Wilson, p. 5]

“It seems that much attention has been given to the reliability and validity of

students‟ ratings as a measure of teaching effectiveness . . ., but not a sufficient

amount of attention has been directed to their effectiveness in improving

teaching.” [Rotem and Glassman, p. 507]

Drawing upon “dozens of studies of student ratings in multisection college

courses,” Kulik concluded “students generally give high ratings to teachers from

whom they learn most, and they generally give low ratings to teachers from whom

they learn least.”[Theall, p. 12]

Has there been grade inflation?

Elite schools like Harvard, Princeton, Stanford, Duke, and Dartmouth have indeed

had substantial grade inflation. Yet, the author cites a study by Clifford Adelman

for the Department of Education. He compared grades recorded on transcripts

11

between 1972 and 1984 and from 1982 to 1993 from 3000 institutions of higher

education. Adelman found that grades declined. [Scocca, p. 2.]

Adelman Study:

Grades from 1972-84 Grades from 1982-93

GPA 2.71 2.65

As 27.3% 25.2%

Bs 31.2% 31.9%

Cs 21.9% 22.2%

Ds 5.4% 6.0%

Fs 3.8% 4.5%

(W, Repeat,

Pass) 10.4% 10.2%

[Adelman, p. 200]

1995-96 study:

C‟s and D‟s or lower Mostly C‟s B‟s and C‟s Mostly B‟s A‟s/B‟s Mostly A‟s

13.9% 11.1% 18.6% 24.1% 19.0% 13.3%

[Profile of Undergraduates in U.S. Postsecondary Institutions: 1995-96, p. 74]

1999-2000 study:

--“a detailed statistical overview of the approximately 16.5 million

undergraduates enrolled in all postsecondary institutions in 1999-2000.” This reveals:

D‟s or lower Mostly C‟s B‟s and C‟s Mostly B‟s A‟s and B‟s Mostly A‟s

14% 14.9% 21.1% 24.6% 10.9% 14.5%

[Profile of Undergraduates: 1999-2000, pp. iii and 70]

12

A Chronicle of Higher Education article concluded that in 1999-2000, 33.5% of

undergraduates made C‟s and lower nationally. [Shoichet, p. A37]

What is the nature of the debate over the impact of grades on evaluations?

“This is the single most frequently researched issue on student ratings with over

400 studies on this question having been conducted to date.” “. . . the general

feeling being that students tend to rate courses and instructors more highly when

they expect to, or actually receive, good grades.” [Arreola, pp. 85-86] “The clear

outcome from the studies on this issue is that, at best, the relationship between

grades and ratings is extremely weak, . . . .” [Arreola, p. 86]

In discussing “Variables that are mildly related to student ratings,” the authors

conclude, “One of the most consistent findings in the literature is that class

average expected grades are moderately correlated with student ratings of teacher

effectiveness.” However, they note “the evidence seems to indicate that students

do not „penalize‟ a teacher whom they think will give them a low grade, although

some studies have shown that students may give lower ratings when they expect a

grade lower than is usual for them. [USAF Academy study, p. 4]

“there tend to be positive, but low correlations (.10 to .30) between students

ratings and expected grades.” [Cashin, p. 5]

Drawing upon 15 studies dating from 1928 to 1962, Costin argued: “A number of

studies, however, found no relationship between students‟ ratings of instruction

and their expected or actual grades in a course.” [Costin, p. 518] Yet, he found 12

studies dating from 1950-1970, that “have found significant positive relationships

between students‟ grades and their ratings of instructors and courses.”

13

Nonetheless, he finds “these relationships were relatively weak.” [Ibid, pp. 518-

519] On balance, Costin concludes “the positive correlations which were

obtained between student ratings and grades were typically low weakens this

claim as a serious argument against the validity of student ratings.” [Ibid, p. 519]

“My results suggest that grades affect course evaluations, but have no impact on

instructor evaluations.” [Harris, p. 2] “Grades have a significant, positive impact

on the course evaluation rating.” [Ibid, p. 28]

Anthony Greenwald and Gerald Gilmore “examined student ratings of hundreds

of courses at Washington and found that professors who are easy graders receive

better evaluations than do professors who are tougher.” [Wilson, p.2] Yet, even

Gilmore acknowledges, “The research literature provides evidence that an

instructor in a poorly taught class cannot „buy‟ high ratings by merely eschewing

all grading standards.” [Ibid]

“In and of themselves, students do not necessarily rate a course as better if they

expect a higher grade or if the course is easier.” [Haynes and Hunts, p. 135]

“The relationship between grade expectations and student satisfaction is weak.”

”[Nuhfer, p. 13]

“In addition to perceived rigor, students‟ prior interest in the course topic also is

more influential (.28) in their evaluation than their expected grade.” [Boretz, p.

45]

“Popular myth implies that teachers can manipulate students into giving them

favorable ratings by offering less demanding courses and grading more

leniently.”[Marsh and Roche, p. 223] Yet, their review of the literature reveals a

14

different conclusion. “A number of features of this literature argue that any

potential effect of grading leniency must necessarily be very small.”[Ibid, p. 224]

“. . . teachers cannot get higher than average SETs merely by offering easier

courses and giving students higher than deserved grades. Indeed, courses

demanding the least amount of work tend to receive lower ratings—not higher

grades that are above the mean grade.” “. . . the most effective ways for teachers

to get high SETs are to provide demanding and challenging materials, to facilitate

student efforts to master the materials, and to encourage them to value their

learning—in short, to be good teachers.”[Ibid, p. 226]

15

ONLINE V. PAPER STUDENT EVALUATIONS

How are online evaluations administered?

“In a typical online evaluation, students are provided with a web site address

where they can gain access to the survey instrument. Prior to giving their

responses, students are informed that professors will not have access to any

student‟s individual responses and that professors will receive the results of the

survey in summary form only after the final grades have been posted. After

students log on to the online system, typically using a student ID number, they are

able to indicate their response to multiple response items and to type their answers

to open-ended questions. After students submit their responses, they can receive a

printed document that verifies that they have completed the evaluation. Students

are generally given at least 2 weeks in which to provide their evaluations, usually

near the end of the term.” [Dommeyer, “Gathering faculty teaching evaluations,”

p. 612]

How many campuses conduct evaluations on-line?

A 2000 report on higher education conducted by Hmieleski noted that of the 200

institutions ranked as the most „wired,‟ only 2 reported institution-wide uses of

online evaluation systems, . . .” “At an international conference in 2002, Thorpe

stated, „The use of online course evaluation systems is relatively limited in higher

education.‟” [Anderson, p. 34]

What are the advantages of online evaluations?

“On-line data collection eliminates paper costs; requires less class time; permits

efficient processing of the data; is less vulnerable to influence of the faculty, and

16

can be a fast, easy, and convenient method for students to submit their

evaluations.”[Dommeyer, “Students‟ Attitudes,” p. 14]

“Online evaluations also allow students more time when entering written

comments.” Citing two studies, the authors write “online evaluations tend to

produce more written comments than traditional, in-class evaluations.”

[Dommeyer, “Attitudes of Business Faculty,” p. 456] Citing four studies,

Sorensen noted, “Research indicates that students provide more and longer

responses online than they do using a traditional paper-pencil system.” [Sorensen,

p. 5]

“While several studies have demonstrated that electronic and paper surveys

produce similar distributions of answers to survey questions . . ., Baum et al.

(2002) showed that online surveys may produce more favorable ratings of faculty

than paper surveys.” [Dommeyer, “Attitudes of Business Faculty,” p. 461]

St. Louis College of Pharmacy compared the two approaches with their students

and found “students using the online survey submitted more comments.”

[Anderson, p. 36 and Dommeyer, “Gathering faculty teaching evaluations,” p.

613]

“In an online rating systems, instructors are given the flexibility to adapt and

personalize the rating forms.”[Sorensen, p. 5]

There is a quicker “turn-around time because of the reduced time needed to

collect and enter data in the paper-pencil system.”[Sorensen, p. 7]

In an experiment in online ratings at Northwestern the “average numerical scores

for the online ratings have been shown to be about the same as for the paper

17

ratings. Although response rates have been somewhat lower using the online

system, those students who do respond write more detailed comments online.”

[Sorenson, pp. 36-37]

“The online accessibility of ratings and reports is one advantage of an online

system.” “Web-based forms are also useful for online and distance-education

courses.” “An online system is also attractive to students because of its „anytime-

anyplace‟ convenience.” [Sorenson, p. 77]

“Having students complete the ratings outside of class not only minimizes

disruption, but also tends to facilitate the production of needed information.

When filling out paper forms in class, students must do so within a narrow time

span (usually 10 to 20 minutes). In contrast, online ratings can be completed

anytime over a period of weeks, and students can take as long as they like to

complete individual rating forms. Several studies have shown that students using

an electronic course-rating format of course-rating forms tend to provide longer

and more frequent written responses than students using the paper-based system

in class.” [Williams, p. 68]

In the first trial at Yale in 2003, the students favored it with 86% responding.

[Yale Daily News]

What are the cost savings in switching to online evaluations?

“Once an online evaluation system is established, many of the costs of the

traditional method can be avoided, i.e. the costs of printing, distributing,

collecting, scanning and storing the paper surveys, the costs of typing students‟

18

responses to open-ended questions and the costs of delivering hard copy summary

reports to faculty.” [Dommeyer, “Gathering faculty teaching evaluations,” p. 612]

“When transitioning from an established paper system (where development costs

are already largely met) to an online system, the initial out-of-pocket costs to

develop the online system are substantial.” [Sorensen, p. 9]

Study of costs at BYU and Washington State at Pullman

--Development Costs: These “include time spent in consultation and

meetings, education about the data collection methods and procedures,

research, design, computer programming, and coordinators‟ management

of the development project.” This study included salary and wages of the

employees involved as well as “the purchase of equipment, software, and

hardware.”

--Operating Costs: “the costs of all materials . . . plus employee time to

assist in distributing, administering, and collecting forms. Operating

expenses also include salary and wages for delivering evaluation forms to

students, collecting forms, scanning evaluation forms, inputting data,

tabulating data, and creating and delivering reports.” [Sorenson, p. 72]

--Annual Operating Costs: $1.06 per student rating form for paper-based

and $.47 per student rating form for the online system. [Ibid., p. 76]

“Colleges considering the implementation of an online course-rating system

should conduct a careful cost-benefit analysis to determine whether or not the

advantages of online course-ratings outweigh the initial set-up costs and other

challenges associated with online course-rating.” [Williams, p. 69]

19

What are the disadvantages of online evaluations?

“only limited research has been published on online evaluations” [Dommeyer,

“Attitudes of Business Faculty,” p. 456]

St. Louis College of Pharmacy compared the two approaches with their students

and found “students spent approximately 10 minutes or less on the online

evaluation versus 25 minutes on the paper evaluation.” [Anderson, p. 36]

“the on-line method produces a lower response rate to the teaching evaluations

than the traditional paper-and-pencil method.” Students “fear that responses to

the on-line survey may not be anonymous . . . .” The on-line approach “can be

inconvenient, time-consuming, and prone to technical problems.” “. . .the on-line

survey may be the victim of student apathy, as the majority of the nonrespondents

to the on-line survey forgot or missed the deadline date.”[Dommeyer, “Students‟

Attitudes,” pp. 14-15 and Dommeyer, “Attitudes of Business Faculty,” p. 456]

The low response rate is a critical concern. “Centra (1979) concluded that about

two-thirds of students must respond if the ratings for any single class are to be

valid.” [Dommeyer, “Attitudes of Business Faculty,” p. 456] Studies of four

universities, including Rutgers, California State—Northridge, and Kansas

Medical School, found responses on-line lower than paper evaluations (9-17%

less). [Anderson, p. 35]

“Response rates to online faculty evaluations have ranged anywhere from 23 to

92%.” [Dommeyer, “Gathering faculty teaching evaluations,” p. 614]

20

At Cal. Northridge, without incentives the response rate in the experiment they

conducted was 29%. [Dommeyer, “Gathering faculty teaching evaluations,” p.

618]

In research done in 2002 and 2003, researchers found “the University of

Colorado, Duke University, Georgia Institute of Technology, the Air Force

Academy, Kansas State University, Northwestern University, and the University

of Idaho—have identified response rates as a challenge to online ratings.”

[Sorenson, p. 50]

At BYU students completion time of evaluations “for all their courses . . . was

thirteen minutes forty-seven seconds.” [Sorenson, p. 53]

At Murdoch University in Western Australia, when the School of Engineering

switched to online evaluations, the response rates varied depending upon the

effort made to persuade students to participate: 1999-30%; 2000-72%, 2001-

50%. [Sorenson, p. 105]

What can be done to improve the response rate on online evaluations?

“provide a „live‟ demonstration” “direct students to a computer laboratory”

“professors could continually remind students of the deadline date and of the

importance of the evaluations” use rewards like “extra credit or an early

notification of their course grade” [Dommeyer, “Students‟ Attitudes,” p. 15]

At Brigham Young University: “Response rates were 40% in 1997, 51% in 1998,

and 62% in 1999.” Though not as high as the paper rate (71%), it improved. The

author of that study suggested: (1) encouraging students to complete the

evaluations; (2) providing explanations of how the evaluations are use; (3)

21

counting the evaluation as an assignment; and (4) withholding early access to

grades.” [Anderson, p. 36]

The St. Louis College of Pharmacy, after getting response rates to 75% and

above, has decided to “make completion of the course evaluations mandatory for

each class.” “noncompliance of an evaluation would result in a grade of

incomplete (I) for the course.” [Anderson, p. 41]

At BYU “In the four sections that gave grade incentives, the overall online

response rate was 86.67.” “Thus, if one wishes to achieve online response rates

that are similar to in-class response rates, a very mild grade incentive should be

offered.” [Dommeyer, “Gathering faculty teaching evaluations,” p. 619] One

challenge to this notion of grade incentives is the question of whether it is either

“legal or ethical.” [Sorenson, p. 108]

“It appears that giving reminder messages to students is an effective means of

enhancing the response rate to an online faculty evaluation.” The instructor can

announce reminders in class or “they can have the survey web site programmed to

automatically Email reminder notices to survey non-respondents on a regular

basis.” [Dommeyer, “Gathering faculty teaching evaluations,” p. 619]

“Polytechnic University in New York enters participating students in a drawing

for handheld computers. Murdoch University in Perth, Western Australia, sends

students multiple e-mail reminders and enters participants in a drawing for a cash

prize.” [Sorenson, p. 50]

22

At BYU, besides “e-mail communication, the online student ratings were

publicized through campus posters and newspaper advertisements during

campuswide implementation.” [Sorenson, p. 52]

BYU completed a pilot study and tried four different approaches and response

rates:

Instructors made it a formal assignment and gave students points for

completing evaluations—87%

“Encouraged students to complete the online forms but did not make it a

formal assignment.”—32%

Instructors “did not mention the online student-rating forms to students.” 20%

[Sorenson, p. 53]

“Research on student ratings suggests that an important factor in response

rates is students‟ belief that rating results are used for important decisions

about courses and faculty.” “Students wanted to be heard and to know that

their responses make a difference.” [Sorenson, p. 58]

What are the problems to consider with online evaluations?

“Selection of software with capabilities of tracking students for completion while

maintaining their anonymity is extremely important. Another component to

consider for successful online evaluation is student computer capabilities and

access. Although most students have access to computers from home or school,

often there can be software compatibility issues that cause problems.”

[Anderson, p. 41]

23

Faculty Fears: “Faculty fear that the online method will produce a lower

response rate and a less accurate response than the traditional method. Moreover,

they fear not only that the online method may attract responses from students

who rarely attend class but also that some students will be influenced by their

peers during online evaluation.” [Dommeyer, “Gathering faculty teaching

evaluations,” p. 613]

Student Fears: “some wonder if the origin of their online comments remains

confidential because they have to identify themselves when logging into the

system.” [Sorensen, p. 10]

Standardizing Procedures: At Cal. Northridge “instructors were requested to read

a prepared statement explaining the online procedure for the particular treatment

assigned to the instructor‟s section.” “Each student doing an online evaluation

had to log on to the system using a unique student ID number and a personal

password.” Students had “to complete the online evaluation on their own time.”

[Dommeyer, “Gathering faculty teaching evaluations,” p. 615] “The total

absence of control over how and with whom the evaluations are completed, fly in

the face of good methodology.” [Theall, “Electronic Course Evaluation”]

“As students complete their ratings outside of class, much less control can be

exerted over the conditions under which they do so. Some teachers express

concern that students may be influenced by peer pressure if they discuss their

ratings with others before filling out their forms. Others are concerned that

students who are registered for a class can fill out the rating form online even if

they never attended class.”[Sorensen, p. 10]

24

“Participants in an online ratings system undoubtedly have different computers,

operating systems, types and sizes of screens, browsers, connection speed to the

Internet, graphics cards, and so forth. Therefore, authors and administrators of an

online system face a difficult challenge concerning the appearance of online

components on participants‟ computer screens.” [Sorenson, p. 78]

Inconvenience for students: “If students have to wait in line in a computer lab to

fill out their rating forms, if they run into computer problems when filling out or

submitting the form, or if they have a hard time navigating the system, they will

be less likely to complete Web-based ratings.” [Sorensen, p. 9]

“It appears that the fear of low response rates may be something that prevents

some institutions from adopting an online evaluation system; . . . .” [Sorenson,

p. 105]

“Faculty need to be assured that only those students enrolled in the class

complete the evaluations and that each student completes only one form.”

[Sorenson, p. 105]

“Given the legal and ethical implications of conducting research that involves

human subjects, colleges should clarify whether researchers can legitimately

access and use results from course ratings.” [Williams, p. 64]

“More research is needed to clarify whether online course-rating introduces a

systematic increase of negative written comments in comparison to the paper-

based method.” [Williams, p. 72]

25

Bibliography

Adelman, Clifford. The New College Course Map and Transcript File: Changes in

Course-Taking and Achievement, 1972-1993. Washington: U.S. Department of

Education, 1999.

Anderson, Heidi M., Cain, Jeff, and Bird, Eleanora. “Online Student Course Evaluations:

Review of Literature and a Pilot Study.” American Journal of Pharmaceutical

Education 69 (2005): 34-43.

Arreola, Raoul A. Developing a Comprehensive Faculty Evaluation System. Boston,

1995.

Boretz, Elizabeth. “Grade Inflation and the Myth of Student Consumerism.” College

Teaching 52 (Date?): 42-46.

Cashin, William E. “Student Ratings of Teaching: The Research Revisited.” Center for

Faculty Evaluation & Development 32 (September 1995): 1-8.

Centra, John A. “Student Ratings of Instruction and Their Relationship to Student

Learning.” American Educational Research Journal 14 (Winter 1977): 17-24.

Centra, John A. “Colleagues as Raters of Classroom Instruction.” The Journal of Higher

Education 46 (May-June 1975): 327-337.

Costin, Frank, Greenough, William T. and Menges, Robert J. “Student Ratings of

College Teaching: Reliability, Validity, and Usefulness.” Review of Educational

Research 41 (1971): 511-535.

Course and Instructor Critique Process Action Team. “Summary of Research on Course

and Instructor Critiques.” U.S. Air Force Academy.

http://www.usafa.af.mil/dfe/summary_of_research_on_critiques.htm

26

Dommeyer, Curt J., Baum, Paul, Chapman, Kenneth S., and Hanna, Robert W.

“Attitudes of Business Faculty Towards Two Methods of Collecting Teaching

Evaluations: Paper vs. Online.” Assessment & Evaluation in Higher Education

27 (2002): 455-462.

Dommeyer, Curt J., Baum, Paul, and Hanna, Robert W. “College Students‟ Attitudes

Toward Methods of Collecting Teaching Evaluations: In-Class Versus On-Line.”

Journal of Education for Business (September/October 2002): 11-15.

Dommeyer, Curt J., Baum, Paul, Hanna, Robert W., and Chapman, Kenneth S.

“Gathering Faculty Teaching Evaluations by In-Class and Online Surveys: Their

Effects on Response Rates and Evaluations.” Assessment & Evaluation in Higher

Education 29 (October 2004): 611-623.

Felder, Richard M. “What Do They Know Anyway? Chemical Engineering Education

26 (1992): 134-135.

Harris, Barbara D. “An Aggregate Approach to Measuring Teacher Performance.” M.A.

thesis, San Diego State University, 2001.

Haynes, Deborah C. and Hunts, Holly. “Using Teaching Evaluations as a Measurement

of Consumer Satisfaction.” Consumer Interests Annual 46 (2000): 134-139.

Marsh, Herbert W. and Roche, Lawrence. “Effects of Grading Leniency and Low

Workload on Students‟ Evaluations of Teaching: Popular Myth, Bias, Validity, or

Innocent Bystanders?” Journal of Educational Psychology 92 (2000): 202-238.

McGourty, Jack; Scoles, Kevin; and Thorpe, Stephen. “Web-Based Student Evaluation

of Instruction: Promises and Pitfalls.” Paper presented at the 42nd

Annual Forum

of the Association for Institutional Research, Toronto, CA, June 2002.

27

Min, Qiao and Baozhi, Sun. “Factors Affecting Students‟ Classroom Teaching

Evaluations.” Teaching and Learning in Medicine 10 (1998): 12-15.

Montell, Gabriela. “Do Good Looks Equal Good Evaluations?” Chronicle of Higher

Education, October 15, 2003 http://chronicle.com/jobs/2003/10/2003101501c.htm

Nuhfer, Edward B. “Fractals and the Value of Student Evaluations.” Center for Teaching

and Learning, Idaho State University (2004).

“Online Course Evaluations Meet Early Success, Prof Says.” www.yaledailynews.com

Ory, John C. “Changes in Evaluating Teaching in Higher Education,” Theory into

Practice 30 (Winter 1991): 30-36.

Rotem, Arie and Glasman, Naftaly S. “On the Effectiveness of Students‟ Evaluative

Feedback to University Instructors.” Review of Educational Research 48

(Summer 1979): 497-511.

Scocca, Tom. “The Great Grade-Inflation Lie.” The Boston Phoenix (1998).

http://www.bostonphoenix.com/archive/features/98/04/23/GRADE_INFLATION.

html

Shoicet, Catherine E. “Reports of Grade Inflation May Be Inflated, Study Finds.”

Chronicle of Higher Education, July 12, 2002, A37.

Sorenson, D. Lynn and Johnson, Trav D. Online Student Ratings of Instruction. San

Francisco: Jossey-Bass, 2003.

Theall, Michael. “Electronic Course Evaluation Is Not Necessarily the Solution.” The

Technology Source (November/December 2000)

http://ts.mivu.org/default.asp?show=article&id=823

28

Theall, Michael, Abrami, Philip C., and Mets, Lisa A., eds. The Student Ratings

Debate Are They Valid? How Can We Best Use Them? San Francisco:

Jossey-Bass, 2001.

U.S. Department of Education. Profile of Undergraduates in U.S. Postsecondary

Institutions: 1995-1996, NCES 98-084. Washington: U.S. Department of

Education, 1998 .

U.S. Department of Education. Profile of Undergraduates in U.S. Postsecondary

Institutions: 1999-2000, NCES 2002-168. Washington: U.S. Department of

Education, 2002 .

Wankat, Phillip and Oreovicz Frank. Teaching Engineering. New York: McGraw-Hill,

1993.

Williams, David D., Howell, Scott L., and Hricko, Mary. Online Assessment,

Measurement, and Evaluation: Emerging Practices. Hershey, PA, 2006.

Wilson, Robin. “New Research Casts Doubt on Value of Student Evaluations of

Professors.” Chronicle of Higher Education

http://chronicle.com/colloquy/98/evaluation/background.htm

“Online Course Evaluations Meet Early Success, Prof Says.” www.yaledailynews.com