reputation, rankings, and ratings kyle sweitzer data analyst office of planning & budgets...
TRANSCRIPT
Reputation, Rankings, and Ratings
Kyle Sweitzer
Data Analyst
Office of Planning & Budgets
Presentation for the University Communicators Network
May 14, 2009
Numerous Rankings & Ratings
RANKINGS OF ENTIRE INSTITUTION (Ungergraduate)• US News & World Report “America’s Best Colleges”• The Center for Measuring University Performance• Kiplinger’s Best Values in Public Colleges• Forbes “America’s Best Colleges”
World Rankings:• The Times Higher Education Supplement World
University Rankings
• Shanghai Jiao Tong Academic Ranking of World Universities 2
Numerous Rankings & Ratings
RANKINGS OF SPECIFIC COLLEGES, SCHOOLS, AND PROGRAMS (Graduate)
• US News & World Report “America’s Best Graduate Schools”
• National Research Council (1982, 1995, 2009)
3
Numerous Rankings & Ratings
AAU Data Exchange shows which rankings are important among AAU institutions
US News and the soon-to-come NRC ratings are clearly tops on the list
4
Let’s start with the US News Undergraduate Rankings
(Ranking of institution as a whole)
5
Criteria Used in US News RankingPeer Assessment (Reputation) 25% Financial Resources 10%
Graduation & Retention Rates 20% Exp’s per student
6-year grad rate(80%) Research Exps
Frosh retention rate (20%) Public Service Exps
Faculty Resources 20% Instructional Exps
Avg faculty salary (35%) Acad Support Exps
% classes < 20 (30%) Stud Services Exps
% fac highest deg (15%) Instit Support Exps
% classes ≥ 50 (10%) Op / Maintenance Ex
% faculty full-time (5%)
Student-faculty ratio (5%) Alumni Giving Rate 5%
Admissions Selectivity 15% % alumni who donate
SAT/ACT score (50%)
Frosh Top 10% (40%) Grad Rate Performance 5%
Acceptance rate (10%) Actual vs predicted rate
How MSU does in each category
See handout:
“US News Rankings Criteria”
• Shows MSU’s performance on each of the rankings criteria over the past five years.
• Also shows averages for Big 10 schools
7
MSU’s ranking & reputation over time
Overall Ranking
99 00 01 02 03 04 05 06 07 08 09
T2 T2 T2 T2 T2 73 71 74 70 71 71
Peer Reputation Rating
99 00 01 02 03 04 05 06 07 08 09
3.5 3.5 3.6 3.6 3.6 3.5 3.5 3.5 3.5 3.5 3.4
8
Research on reputation
What variables relate to reputation rating?
Volkwein, J.F., & Sweitzer, K.V. (2006). Institutional prestige and reputation among research universities and liberal arts colleges. Research in Higher Education, Vol. 47, No. 2, pp. 129-148.
9
US News peer assessment rating
• The measure of reputation employed in the study (DV) is the US News peer assessment rating, which is a reputation score that US News reports from the surveys they administer each spring.
• US News surveys the president, provost, and admissions director of each four-year institution in the country, asking them to rate the academic reputation of their peer institutions on a scale from 1 to 5
1 = Marginal
2 = Adequate
3 = Good
4 = Strong
5 = Distinguished
Reputation Rating for Research UniversitiesVariables that relate to US News Peer Assessment Rating
Age of Institution ------Govern / Control ------Total Enrollment .268*** Standardized Beta CoefficientsExpend / Student ------ from Blocked Set-Wise RegressionStudent-Fac Ratio -.156***Avg Prof Salary .268***Pct Fac Full-time .052* * .05 level of significanceMedian SAT .142* ** .01 level of significanceFaculty Productivity .103*** *** .001 level of significanceGraduation Rate .267***Alumni Giving Rate .101**
ADJ R-SQUARE .905
Comparing US News to Guidebooks• Barron’s Profiles of American Colleges
– College Admissions Selector (Noncompetitive to Most Competitive)
• Peterson’s Four-Year Colleges– Entrance Difficulty Index (Noncompetitive to Most Difficult)
• Princeton Review’s Complete Book of Colleges– Selectivity Rating (60 to 99)
• The Fiske Guide to Colleges– Academic Rating (1 to 5)
Correlations with US NEWS Peer Assessment Rating
Barron’s Admissions Selector .76
Peterson’s Entrance Difficulty Index .74
Princeton Review Selectivity Rating .81
Fiske Academic Rating .85
Can We Improve Our Reputation?The Variables that Relate to Change Over Time
in US News Peer Ratings
Kyle SweitzerData Analyst
Michigan State University
Fred VolkweinProfessor Emeritus of Education
Penn State University
Background• Most studies of reputation have examined which
variables relate to prestige for a given year.
• Few studies have explored change over time, and those that do look at changes in the overall rank, as opposed to specifically examining change in reputation.
• One such study found that the same 47 schools were ranked in the US News Top 50 every year from 1999 to 2006 in the National University category (research universities)
Research Question
What variables, if any, relate to changes in US News’ peer assessment ratings for those institutions which have experienced significant changes in the ratings over the nine-year period from 1999-2007?
Population
• There are four broad categories in the US News hierarchy, roughly based on Carnegie classifications.
• Institutions that remained in the same US News category between 1999 & 2007 were the starting point for inclusion in the study.
• Almost 1100 institutions (1095) remained in the same US News category over those nine years.
• This ensures that these schools had the same group of peer institutions to rate them (if not the same person, at least the same position at those schools).
Schools that were analyzed• Across the 1095 schools remaining in the same US
News category all nine years, the mean difference between an institution’s high and low peer reputation rating over the nine years was 0.24
• Only those schools that had an above-average difference between their high and low score were included in the analysis (difference of 0.3 or more between their high and low peer rating).
• 418 schools had a difference of at least 0.3 (412 analyzed)
Stability in reputation, 1999-2007Difference between Number of
low and high score institutions
1.0 1
0.9 2
0.8 1
0.7 10
0.6 6
0.5 46
0.4 107
0.3 245
0.2 403
0.1 260
0.0 14
18
Independent VariablesTotal of 22 predictor variables collected, in 1 of 5 categories:
SIZE VARIABLESStudent Pop, Faculty Pop, Combined Size Variable
FINANCE VARIABLESTotal Exps, Total Revs, Exp/Stud, Rev/Stud, Tuition
SELECTIVITY VARIABLESSAT, Top 25% HS, Accept Rate, Combined Selectivity Var
FACULTY VARIABLESPubs, Pubs/Fac, Salary, % Fac FT, S-F ratio, % classes <20
STUDENT OUTCOMES VARIABLES Frosh Retn Rt, Grad Rt, Avg Fr & Grad Rt, Alum Giving Rt
Change in ReputationOne immediate finding just from data collection:
The incredibly disproportionate number of schools that changed in reputation based on which US News (Carnegie) category a school is in.
Of the 412 institutions with above-average change in their peer assessment rating over the nine years: 14 “National Universities” (Research Universities) 25 “Liberal Arts Colleges” (Baccalaureate Colleges—Arts & Sciences)180 “Universities-Masters” (Master’s Colleges & Universities)193 “Comprehensive Colleges—Bachelors” (Baccalaureate Colleges –
Diverse Fields)
Results of the StudyThe only variable that remained significant in explaining
changes over time in reputation ratings is the admissions selectivity variable (Pct of freshmen in Top 25% of HS class), with coefficients of 0.2 to 0.3
Thus, a 10 percentage-point increase in the percent of freshmen that were in the Top 25% of their HS class relates to an increase in the US News peer assessment rating by 0.02 to 0.03 (10 percentage point decrease means a 0.02 to 0.03 decrease)
Thus, if a school improves from 60% to 70% of incoming students who were in the Top 25% of their HS class, the school would improve, for example, from a 2.4 to a 2.6 in the US News peer assessment rating
Overview of ResultsAcross ALL 1300+ schools that were rated by US News between 1999
and 2007, the average peer assessment rating has ranged from a low of 2.85 to a high of 2.90 (thus, no change rounded to one decimal)!!
This is despite the fact that over 400 schools have seen above-average change (at least 0.3) during the period. (Note: 62% of schools show relative stability over time in reputation)
Examining all institutions over nine years, it’s clear that upward movers have balanced downward movers, resulting in a nullification effect in reputation change.
Apparently, the data suggests that academic reputation in US News is a zero-sum game. Raters are (either intentionally or unintentionally) only rating a certain number or percentage of schools at a given rating, and if they rate one school higher than the year before, they rate another lower than the previous year.
What does it all mean?
Academic reputation changes very little, if at all, especially for research universities and liberal arts colleges.
Reputations change slowly, and where reputations do change, admissions selectivity seems to be the single most-important influence.
The pool of talented students is limited, and practically every institution is competing for them!!
What does it all mean?
If changes in academic reputation boil down simply to changes in the ability of the students coming in the door, how well does the US News peer assessment rating measure quality in higher education?
Perhaps the title of the annual US News magazine, rather than “America’s Best Colleges”, would more accurately be called “America’s Most Selective Colleges.”
25
Reputation Among Graduate Programs:
Comparing Correlates of U.S. NewsU.S. News Graduate Reputation Ratings Between
Five Academic Disciplines
26
Why study graduate program reputation ratings?
Prospective graduate students use graduate program ratings to inform their application and admissions decisions.
Administrators use graduate program ratings to inform resource allocation decisions.
(Ehrenberg and Hurst, 1996)
27
Existing studies on graduate rankings/ratings
Most of the studies have examined institutions’ graduate programs as a whole, via aggregating individual program ratings (Volkwein, 1986; Grunig, 1997).
Few studies have examined graduate program ratings at the department or school level.
Even fewer have looked at the U.S. NewsU.S. News graduate school ratings (most have examined the NRC ratings).
28
Research Questions
What variables relate to the U.S. NewsU.S. News peer assessment ratings of graduate programs in the professional school disciplines of business, education, engineering, law, and medicine?
Are there variables relating to prestige that are common across all of the disciplines in the study, and are there variables that are specific to certain disciplines?
How does the concept of prestige compare across professional school disciplines?
29
Population
Schools/Colleges appearing in the lists of “The Top Schools” in business, education, engineering, law, and medicine in the 2008 edition of U.S. News’ America’s Best America’s Best Graduate SchoolsGraduate Schools.
50 Schools of Business 52 Schools of Education 51 Schools of Engineering104 Schools of Law 51 Schools of Medicine
30
Variables / Data Sources
DEPENDENT VARIABLE – Peer assessment survey of deans, faculty, program directors
INDEPENDENT VARIABLES – Data from U.S. NewsU.S. News
--standardized admissions tests
--program acceptance rates
--full-time graduate enrollment in the school
--non-resident tuition
--student/faculty ratio
--undergraduate GPA
--variables specific to a discipline
31
Variables / Data Sources
Research activity was measured in terms of faculty publications.
Institute for Scientific Information Web of Science
Science Citation Index and Social Science Citation Index
Search on “Subject Category” for journals specific to a discipline.
32
Regression Models
A separate regression was estimated for each of the disciplines.
Many studies have suggested that two factors play the most significant role in explaining the variance in reputational ratings – size and admissions selectivity.
These two factors were entered into each regression model, along with other variables.
Regression Results, Schools of Business
• Standardized Betas of Significant Coefficients
• Variables Model 1 Model 2 Model 3• Full-time enrollment .624*** .407*** .267*• Non-resident tuition .330*** .228*• Student-faculty ratio ns• Avg GMAT score .388*** .253**• Pubs per faculty 2001-2005 ns• Starting salary of grads .596***• • Adjusted R-Square .736 .807 .878• ----------------------------------------------------------------------------------------------------------------• *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.• ns = non-significant when entered into model
33
Regression Results, Schools of Education
• Standardized Betas of Significant Coefficients
• Variables Model 1 Model 2 Model 3• Full-time enrollment .366** .354* .535**• Non-resident tuition .514***• Student-faculty ratio ns• Avg GRE score ns• Pubs per faculty 2001-2005 .421*• • Adjusted R-Square .368 .377 .474• ----------------------------------------------------------------------------------------------------------------• *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.• ns = non-significant when entered into model
34
Regression Results, Schools of Engineering
• Standardized Betas of Significant Coefficients
• Variables Model 1 Model 2 Model 3• Full-time enrollment .664*** .576*** .792***• Non-resident tuition .327**• Student-faculty ratio ns• Avg quantitative GRE score .443*** .226*• Pubs per faculty 2001-2005 .468***• • Adjusted R-Square .447 .619 .721• -----------------------------------------------------------------------------------------------------------------• *Significant at .05 level; ***Significant at .001 level.• ns = non-significant when entered into model
35
Regression Results, Schools of Law
• Standardized Betas of Significant Coefficients
• Variables Model 1 Model 2 Model 3• Full-time enrollment .213* .159* .163**• Non-resident tuition .508***• Student-faculty ratio – .207*** – .174***• Median LSAT score .712*** .530***• Pubs per faculty 2001-2005 .264***• Employment rate at graduation ns• • Adjusted R-Square .397 .795 .849• -----------------------------------------------------------------------------------------------------------------• *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.• ns = non-significant when entered into model
36
Regression Results, Schools of Medicine
• Standardized Betas of Significant Coefficients
• Variables Model 1 Model 2 Model 3• Full-time enrollment ns .224* .342***• Non-resident tuition ns• Faculty-student ratio ns .313**• Avg MCAT score .701*** .637***• Pubs per faculty 2001-2005 .374***• • Adjusted R-Square .016 .540 .653• ----------------------------------------------------------------------------------------------------------------• *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.• ns = non-significant when entered into model
37
38
Summary of ResultsThe SIZE variable (full-time enrollment) is
the only variable that remained significant in the final model for all 5 disciplines.
However, size has the greatest beta coefficient in only 2 of the 5 disciplines (education and engineering).
So for schools of education and engineering, the size of the school matters more to reputation than anything else!
Summary of Results• ADMISSIONS SELECTIVITY remains
significant in the final model for 4 of the 5 disciplines, and has the greatest beta coefficient for 2 of those 4 – Law schools and Med schools.
• So for Law schools and Med schools, the “quality” of the students matters more to the reputation of the school than anything else!
• Education is the one discipline for which selectivity does not affect reputation.
39
Summary of Results• FACULTY PRODUCTIVITY (pubs per
faculty) also remained significant in 4 of the 5 disciplines, and had the 2nd greatest beta coefficient in all 4.
• The 4 disciplines were: engineering, education, law, and medicine.
• Not surprising that faculty productivity is significant in explaining graduate reputation.
40
Summary of Results• The business schools may be the most
surprising all around --- not only is it the one discipline in which faculty productivity does not influence reputation, but the factor with the greatest influence on reputation is the starting salary of the graduates …..a factor determined by external (market) forces!!
41
Summary of Results
• TUITION did not remain significant in the final model for any of the 5 disciplines.
• Student-faculty ratio only remained significant in 2 of the 5 disciplines (Law and Med), and was one of the weaker predictors even for them.
42
Summary of Results
• So in 4 of the 5 disciplines, either SIZE or ADMISSIONS SELECTIVITY is the biggest determinant of a school’s reputation.
• These results confirm prior studies on graduate program reputation that analyzed the 1995 NRC ratings.
43
44
Summary of Results
Variable with the largest beta coefficient:
Business Starting salary of graduates
Education Enrollment size
Engineering Enrollment size
Law Admissions selectivity (LSAT)
Medicine Admissions selectivity (MCAT)
45
Implications
Like with the undergraduate ratings, the question remains as to how well the
U.S. News graduate ratingsU.S. News graduate ratings measure the concept of quality in education.
Is the magazine really determining“America’s BestBest Graduate Schools?”
US News Graduate Rankings
See two handouts:
• Highlights of MSU’s rankings for 2010 edition (rankings that are in the actual magazine – additional rankings are in the online edition)
• Summary of Big 10 schools – number of ranked programs over time
46
Other rankings beyond US NewsWORLD RANKINGS (entire university ranked)
The Times Higher Education Supplement World University Rankings
(MSU ranked # 203 for 2008 ranking)
Peer review of faculty worldwide (40%)
Employer review (10%)
Citations per faculty (20%)
Faculty / student ratio (20%)
International faculty (5%)
International students (5%) 47
Other rankings beyond US NewsWORLD RANKINGS (entire university ranked)
Shanghai Jiao Tong Academic Ranking of World Universities
(MSU ranked # 83 for 2008 ranking)
Faculty winning Nobel & Fields medals (20%)
Highly cited researchers (20%)
Articles published in Nature and Science (20%)
Articles in ISI Science and Soc Sci indices (20%)
Alumni winning Nobel & Fields medals (10%)
Above 5 weighted per FTE academic staff (10%)