international trend on university rankings and its impact on higher education

INTERNATIONAL TREND ON UNIVERSITY RANKINGS AND ITS IMPACT ON HIGHER EDUCATION

Angela Yung Chi Hou , 19 Dec, 2012 Fu Jen Catholic University

1

Presentation Outline

I. IntroductionII. Development of Rankings III. Limitations of RankingsIV. Impact of RankingsV. Outcomes of RankingVI. Use of RankingsVII. Conclusion

2

I. Introduction

1. Three major concerns2. Meeting challenges to higher

education in the 21st century3. Recipe for higher education4. World class university5. Global rankings and world class

university6. Rational of college ranking

3

1. Three Major Concerns for Development of Higher Education in Global Times

AccessibilityHigher education enrollment rate has been increasing in the past decade

AffordabilityFinance need is not an impediment for eligible students to attend a college

AccountabilityEnormous resources and talent are available in higher education institutionsHelp students be ready for college, and be equipped to graduate from college

4

2. Challenges to Higher Educationin 21st Century

Over expansion of higher education2305 post secondary higher education institutions in China 726 universities and 488 junior colleges in Japan 201 universities in Korea160 universities in Taiwan 15 in Hong Kong (9/6)12 in Macau(4/8)

Private universities outnumber public universities (except China) Declining birth rate

three are below 1.5 Declining financial governmental support

5

3. Recipe for Higher Education - quality assurance and global competitiveness

Set up Internal and External Quality Assurance System

Establishment of National Accrediting agency Establish Quality Culture on Campus

Compulsory auditEnhance international academic competitiveness

Launching several Excellence Programs Brain21 (Korea) Center for Excellence COE (Japan)5 year-50 Billion Program (Taiwan)China 985 project

Aiming at establishing world class universities 6

4. What does a world class university look like?

In terminologyworld class universities: top universities striving for “excellence”

quality must surpass the expectation of stakeholders

Philip Altbachexcellence in research, top professors, academic freedom and an atmosphere of intellectual excitement, governance, adequate facilities and funding

7

4. What does a world class university look like?

Jamil Salmi (World Bank) based on two rankings (Shanghai and QS)

a high concentration of talent (faculty and students)abundant resources to offer a rich learning environment and conduct advanced research favorable governance

features that encourage strategic vision, innovation and flexibility, and enable institutions to make decisions and manage resources without being encumbered by bureaucracy

8

5. Relevance between Global Rankings and World Class University

characteristics of world class universities are strongly correlated to most indicators used by global rankingsnations use global rankings as a basis of building world class universities despite methodological flaws

9

5. Relevance between Global Rankings and World Class University

top administrators at leading universities use global rankings to achieve the short term and long term strategic plans, not just to boycott them

Minnesota’s initiative become one of the top three research institutions in the worldTaiwan National University “Moving into the top 100” at its 80th anniversaryBaylor University one of the U.S. News Top 50 by 2012

10

6. Rational of College Ranking

Higher education expansion Resources allocation Accountability BenchmarkingMarketization in higher education

11

II. Development of Rankings

1. National College Ranking2. Global Ranking3. Types of Rankings4. Scoring Methods5. Five Global Rankings6. Criteria & Indicators of Rankings

12

1. National College Ranking

U.S. News & World ReportThe most influential college ranking – “American Best Colleges” published by U.S. News & world Report in 1983Maclean’s , The Times, CHE, etc.

13

2. Global Ranking

intense international competitionsglobal college rankings have drawn international attention

14

3. Types of Ranking

By region by country/continent/worldwide

By field / programsuch as Engineering/social science/humanities

By subjecti.e., Biology/ Math

College guidePrinceton review

15

4. Scoring Methods

The indicators were weighted at a certain ratio and the scores were aggregated to rank each college. The top one university received highest points while the scores for the remaining schools descended accordingly.

16

5. Five Global Rankings

Academic Ranking of World Universities Shanghai Jiao Tong University, 2003

World University RankingQS, 2004

Webometrics Rankings of World UniversitiesSpanish National Research Council, 2004

Performance ranking of scientific papers of world class universities

HEEACT, 2007World University Ranking

The Times Higher Education, 2010

17

6. Criteria & Indicators of Ranking

2011 ARWU Ranking2011 QS Ranking2011 Webometrics Ranking2011 THE Ranking2011 HEEACT Ranking

18

6. 2011 ARWU Ranking 分類指標 分項指標 權重 操作型定義

分類指標 分項指標 權重 操作型定義

教育品質 畢業校友(Alumni)

10%

獲得諾貝爾及菲爾茲獎校友人數。得分高低依畢業年度，每往前推十年，降低 10% 權重。如 2001 年以後每人以 1 人計算； 1991-2000 年每人以 0.9 人計算 ，依次遞減。

教師品質獲得重要獎項人數

(Award)20%

獲得諾貝爾及菲爾茲獎教師人數。得分高低如上。如 2001 以後獲獎者每人以 1 人計算； 1991-2000 年每人以 0.9 人計算，依次遞減。獲獎人同時署名兩單位，各計 .05 人。諾貝爾共同得獎者，以獎金比例分配權重。

論文高度被引用研究人數(HiCi)

20% 在 21 個科學領域中，論文名列 ISIHighlyCited.com高度引用之教師人數。

研究成果《自然》與《科學》兩期刊論

文發表篇數 (N&S)20% 2006 至 2010 年間，一所大學在《自然》與《科

學》兩期刊論文發表篇數，不包含評論與快訊。SCI 與 SSCI 論文數 (PUB) 20% 2010 年，一所大學發表論文被 SCI 及 SSCI 收錄的

數量。機構規模 機構規模（專任教師數）

(PCP)10% 以上五項總分 / 專任教師數

6. 2011 QS Ranking

評比內容 權重 分項指標

研究(Research)

60%

學術聲譽(Academic Reputation)

教師平均論文被引用數(Citations per Faculty)

就業力(Employability)

10%就業聲譽

(Employer Reputation)

教學(Teaching)

20%師生比

(Faculty/Student Ratio)

國際化(Internationalization)

10%

國際教師比例(International Faculty Ratio)

國際學生比例(International Student Ratio)

6. 2011 Webometrics Ranking

21

Indicators Definition Covergae Sources Weight

IMPACT Number of backlines Number of back domains

Current Majestic SEO and ahrefs

50%

Presence Number of webp ages

Current Google 20%

openness Number of papers (pdf, doc, docx, ppt)

2007-2010 Google scholar

15%

Excellence 10% most cited papers in their respective scientific fields

2003-2010 Scimago 15%

6. 2011 THE Ranking

評比內容 權重 分項指標 分項指標權重

教學－學習環境

(Teaching -

the learning environment

)

30%

教學聲譽 (Reputational survey-Teaching) 15%

教師平均指導博士畢業生數 (PhD awards per academic) 6%

教師平均教授大學生數 (Undergraduates admitted per academic) 4.5%

授予博、學士學位比 (PhD-to-bachelor’s ratio) 2.25%

教師平均分配院校收入 (Income per academic) 2.25%

研究－數量、收入與聲譽 30%

研究聲譽 (Reputational survey-Research) 18%

研究收入 (Research Income) 6%

教師與研究員平均論文篇數 (Papers per academic and research staff) 6%

論文引用－研究影響 30% 平均論文引用數 (Average citations per paper) 30%

國際化－職員與學生(International outlook

-staff and students)

7.5%

國際對本地生比 (Ratio of international to domestic students) 2.5%

國際對本地教職員比 (Ratio of international to domestic staff) 2.5%

國際合作論文比例 (Proportion of internationally coauthored research papers) 2.5%

業界收入－創新(Industry income -

innovation)

2.5% 研究人員自業界之平均研究收入(Research income from industry per academic staff)

2.5%

Criteria Indicator Weight

Researchproductivity

Number of articles in the last 11 years (2000-2010) 10%

20%Number of articles in the current year (2010) 10%

Researchimpact

Number of citations in the last 11 years (2000-2010) 10%

30%Number of citations in the last 2 years (2009-

2010) 10%

Average number of citations in the last 11 years

(2000-2010) 10%

Researchexcellence

h-index of the last 2 years (2009-2010) 20%

50%Number of Highly Cited Papers (2000-2010) 15%

Number of articles in high-impact journals in the current year (2010) 15%

6. 2011 HEEACT Ranking

23

III. Limitations of Rankings

1. Characteristics of 5 major Global Rankings

2. Methodological Limitations

24

1. Characteristics of 5 Major Global Rankings

ARWU QS Webometrics HEEACT THEEstablished

year 2003 2004 2004 2007 2009

Institution Academicinstitution

Massmedia/PrivateEducationconsulting firm

Governmental research unit

QA Agency Massmedia

Goal Academiccompetition

Profit making Academicsharing

Benchmarking Academic Competition

Number ofindicators

6 6 4 8 13

Indicatorcategory

Researchoutput/ learning input

Research output /Reputation survey

/learning input

Web size/research

output/reputation

Research output

Teaching/ResearchOutput

/survey Data sources Database Survey/

database/institution

database Database Survey/database/institution

Outcomes Presentation

Only Top 100 of500 institutions are shown innumerical orders

Top 400 are shown

in numerical orders

Top 1000 innumerical

order

Top 500 innumerical

order

Top 200 areShown innumerical

ordersTransparency Highly medium Medium Highly

medium Highly medium Medium

25

2. Methodological Limitations of GlobalRankings

Reductionism / Simplicity Research focus Unfair for humanities, arts and social science fields English dominationArbitrary selection of indicators and weightingsData quality

26

IV. Impact of Rankings

1. Outcomes of Rankings2. Popular Use by Stakeholders3. Two Major Reports4. Performance in Asia

27

1. Outcomes of Rankings

US and UK institutions are on the top Asia is on rise, particularly those with Excellence policy

28

2. Popular Use of Global Rankings byStakeholders

Studentsto decide where to study

Governmentsto know where to invest

Scientiststo know where to work

Institutionsto know where they stand and whom they can partner with

29

3. Two Major Reports

OECD survey in 2007 showed: over 50 % of respondents: rankings has a positive impact on the institution’s reputation (student recruitment, academic partnerships and collaborations and staff morale)majority of the institutions incorporated the outcomes of rankings into strategic planning processes at all levels 70 % wanted to be in the top 25 internationally

30

4. Two Major Reports

OECD survey in 2007 showed: an on–line UK study focused on English Universities ‘ attitudes toward rankings

rankings reflect the views of what properties a good university should develop that influenced the institutional and governmental polices a high level of agreement that the reputation of an institution might be affected by rankings

many institutions further down in the rankings do not care too much about global rankings

31

5. Average Number of top 500 Universities of Japan, China, South Korea and Taiwan by three rankings

Country ARWU(2004~2012)

HEEACT(2007~2011)

QS(2007~2012) Ave No

Japan 29.56 29.60 25.67 28.27

China 16.44 14.20 12.33 14.33

South Korea 9.00 9.20 11.50 9.90

Taiwan 6.22 5.40 8.50 6.71

Hong Kong 5.00 5.00 5.67 5.22

India 2.00 2.00 8.00 4.00

32

V. Ranking Outcomes

1. ARWU / QS/ HEEACT/ THE Ranking2. Number of papers and

internationalization in China, Korea, Taiwan and Japan

33

Top 20 Universities in the Four Rankings (2/2)

University Name HEEACT(2011)

ARWU(2012)

THE(2012)

QS(2012)

University of California - San Diego 15 15 --- ---The University of Tokyo 16 20 --- ---University of London - University College London 17 --- 17 4

Duke University 18 --- --- 20Yale University 19 11 11 7Imperial College London 20 --- 8 6California Institute of Technology --- 6 1 10Princeton University --- 7 5 9University of Chicago --- 9 9 8Cornell University --- 13 20 14University of Wisconsin - Madison --- 19 --- ---ETH Zürich - Swiss Federal Institute of Technology Zürich --- --- 15 13

McGill University --- --- --- 1834

Top 20 Universities in the Four Rankings (2/2)

University Name HEEACT(2011)

ARWU(2012)

THE(2012)

QS(2012)

University of California - San Diego 15 15 --- ---The University of Tokyo 16 20 --- ---University of London - University College London 17 --- 17 4

Duke University 18 --- --- 20Yale University 19 11 11 7Imperial College London 20 --- 8 6California Institute of Technology --- 6 1 10Princeton University --- 7 5 9University of Chicago --- 9 9 8Cornell University --- 13 20 14University of Wisconsin - Madison --- 19 --- ---ETH Zürich - Swiss Federal Institute of Technology Zürich --- --- 15 13

McGill University --- --- --- 1835

Figure 1: The number of the paper published on SCI and SSCI journals in China, Taiwan, Japan, South Korea

36

Figure 2: Number of papers in Nature & Science in seven yearsin China, Taiwan, Japan, South Korea

37

Figure 3: Number of international students in China, Taiwan , Japan, South Korea

38

VI. Use of Rankings

1. Findings in rank Mobility2. Rank Differences and moving UP in 4

Global Rankings3. Berlin Principles and Ranking Audit4. Future Development5. CHE Excellence Ranking and Research

Ranking6. College Navigator in Taiwan

39

1. Some Findings in Rank Mobility of Global Rankings

Hou, Yung-chi, & M. Robert (2011) . An Analysis of Positions Mobility in Global Rankings: Making Institutional Strategic Plans and Positioning for Building World Class Universities. Higher Education Research & Development (SSCI).

To explore the major factors of rank mobility in 4 major rankings.

40

2. Rank Differences and moving Up in 4 Global Rankings

Comparison among 4 Global Rankings by positions risingImplication of 4 Global Ranking on making institutional strategic plans

41

2. Comparison among 4 Global Rankings by Positions Rising

ARWUQS Webmetrics HEEACT

Cluster one 1-171-30

1-39 1-19

Cluster two 20-45Over 30

40-99 20-45

Cluster three Over 46X

Over 100 Over 46

total number of positionsmoving ups 218(500)

170 (400) 242 (500) 231(500)

Highest ranks moving up 94125 212 82

42

2. Flow Chart of Implication of 4 Global Ranking on Making Institutional Strategic Plans

Webometrics Ranking QS Rankings ARWU/Shanghai Ranking

HEEACT Ranking: Used to inspect the quality and quantity of FACUTLY publications annually

Short term(3-5 years) Mid-term 5-15 years Long-term(15~30years)

Technology/Internet International Reputation Academic Excellence

43

3. Berlin Principles and Ranking Audit

International Ranking Expert Group (IREG)founded in 2004 by the UNESCO European Centre for Higher Education (UNESCO-CEPES) in Bucharest and the Institute for Higher Education Policy in Washington, DC.

44

3. Berlin Principles and Ranking Audit

It is upon this initiative that IREG’s second meeting (Berlin, 18 to 20 May, 2006) has been convened to consider a set of principles of quality and good practice in HEI rankings—the Berlin Principles on Ranking of Higher Education Institutions.IREG-5 in Oct, 5-7, 2010 proposed “Ranking Audit”

45

3. Berlin Principles: What Rankings and League Tables should Consider

To ensure the quality of rankings: research methods, indicators, data quality, transparency, varying user’s interests, etc. 4 major principles

Purposes and Goals of RankingsDesign and Weighting of IndicatorsCollection and Processing of DataPresentation of Ranking Results

19 criteria of audit published by IREG

46

4. Future Development for Rankings

Field/subject based rankingVarying ranking providersMore interactive, multi-dimensional, personalized

CHE European Excellence Ranking College Navigator in Taiwan

Web-based Benefit student mobility Student survey

47

5. CHE Excellence Ranking and Research Ranking

Multi-dimensional global ranking that will be based on the CHE ranking approach.Results of the feasibility study will be available in mid-2011. One aspect will be the development of a concept to introduce a web-based tool for personalized rankings for particular target groups on a global scale.

48

49

5050

College Navigator in Taiwan Published in 2009

Goal lead to a match between the student and the institution or the program that they’re most interested

Selection of Institutions69 4-year colleges and universities evaluated by HEEACT from 2006 to 2010. 77 University of Technology and Technical College

50

Research Team

51

3 professors/ one international consultant / one full time staff/ one IT designer / 7 graduate students

6. College Navigator in Taiwan-Home

Click here to start

52

College Navigator in Taiwan- Step 1: Indicators

53

Step 1: Choose the indicators

College Navigator in Taiwan- Step 1: Indicators

54

Step 1: Choose the indicators

College Navigator in Taiwan- Step 2: Weighting

55

Step 2: Give each indicator

a weight

College Navigator in Taiwan- Step 3: Preference

56

Step 3: Decide the

Universities you want to

compare

Way 1: Narrow down your choices

Way 2:Choose Universities

College Navigator in Taiwan- Step 4: Result

57

The indicators you choose

Our system will analyze the rank of the universities according to the indicators and weights you decide

The performance of each university

College Navigator in Taiwan- Step 5: General l information

58

59

Hou, Angela Yung-chi, Morse, R., & Shao, Y. J. E. （ 2012 ） . Is There a Gap between Students’ Preference and University Presidents’ Concern over College Ranking Indicators? : A Case Study of “College Navigator in Taiwan”, Higher Education ( in press) (SSCI) (2010 Impact Factor 0.823). ISSN: 0018-1560

60

Table 6 Mean and STD by items Figure 2: Scatter Plot of level of Importance over Criteria and Indicators

61

Categories Mean

STD

95% Confidence Interval

Upper Lower

Institutional policy making 3.95 0.61 4.14 3.76

Staff and faculty recruitment 3.93 0.67 4.14 3.73

Research output 4.06 0.76 4.30 3.83

Resources allocation 3.82 0.69 4.04 3.61

Student services and learning environment 3.86 0.7

2 4.09 3.62

System operation 3.93 0.75 4.16 3.69

62

Focus group’ opinions Most popular indictors Institutions’ attitude Graduate rate Academic survey * (higher) Institutional policy making

( rank 2) Total amount of equipment per student Expenditure per student Resources allocation

(Rank 6) Expenditure per student Enrollment rate Student services and learning

environment (Rank 5) Number of international academic awards earned by students within last 3 years students within last 3 years

Faculty-student ratio* (higher) Student services and learning environment(Rank 5)

Total library holdings per student/Total grant from collaborations between of University and industry

Graduation rate* (lower)Student services and learning environment(Rank 5)

Number of national academic awards by students (higher)

Student services and learning environment(Rank 5)

Total NSC Research grants Total library holdings per student** (lower)

Resources allocation (Rank 6)

Proportion of faculty members with Ph.D. Total NSC grants per faculty** (lower) Research output (Rank 1)

Number of national academic awards earned by students within last 3 years Proportion of full-time faculty

Faculty Resource (Rank 3)Average piece of collaborations between of University and industry per full-time faculty Proportion of professors with Ph.D.

Table 8 Comparison of rank of importance of the indicators among focus students, general users and institutions

VII. Conclusion

1. Conclusion2. Final Questions3. Yes and No Answers4. How Ecosystem Influences Top Research

Universities

63

1. Conclusion

To achieve a good rankings is becoming more and more important. Global rankings are increasingly being used as a tool for building world class universities and pursuing academic excellence .

64

2. Final question raised by the presentation

To what extent can a world class university be replicated by using the factors highlighted in a ranking model? How can it be done?

The answer is both “yes,” it can be replicated, and “no,” it can’t be.

65

3. Yes and No Answers

Rankings can only provide very rough guidance and clues to institutions on which road to take to achieve academic excellence. Not taken into account, but very crucial:

a clear visioninstitutional featuresfavourable governancesufficient resources

66

4. Understanding How the Ecosystem Influences the Performance of Top Research Universities by Jamil Salmi

67

“THERE IS NO SINGLE ROAD TO EXCELLENCE”

by Jamil Salmi (2010) Coordinator,

Tertiary Higher Education, World Bank

68

Thank you for your attention!

Questions and Comments

69

References *Hou, Angela Yung-chi (2012). Mutual Recognition of Quality Assurance Decisions on Higher Education Institutions in Three Regions-A Lesson for Asia. Higher Education, 64:911-926. (SSCI) (2011 Impact Factor 1.016). ISSN: 0018-1560*Hou, Yung-chi, Morse, R., & Shao, Y. J. E. （ 2012 ） . Is There a Gap between Students’ Preference and University Presidents’ Concern over College Ranking Indicators? : A Case Study of “College Navigator in Taiwan”, Higher Education, 64:767–787. (2011 Impact Factor 1.016)Hou, Angela Yung-chi (2012 Quality in Cross-Border Higher Education and Challenges for the Internationalization of National Quality Assurance Agencies in the Asia-Pacific Region –Taiwan Experience. Studies in Higher Education. (online) (SSCI)*Hou, Yung-chi., Ince, M., & Chiang, C.L. (2012). A Reassessment of Asian Excellence Programs in Higher Education – the Taiwan Experience. Scientometrics. 92, 23-42 (SSCI). Print ISSN: 0138-9130. (2011 Impact Factor 1.966)*Hou, Yung-chi. （ 2011 ） . An Analysis of Positions Mobility in Global Rankings: Making Institutional Strategic Plans and Positioning for Building World Class Universities. Higher Education Research & Development (SSCI ） (online) ISSN: 0729-4360(2011 Impact Factor 0.901)*Hou, Yung-chi. （ 2011 ） . Impact of excellence programs on Taiwan higher education in terms of quality assurance and academic excellence, examining the conflicting role of Taiwan’s accrediting agencies. Asian Pacific Educational Review, 13, 77-88 (SSCI). ISSN: 1598-1037*Hou, Yung-chi. （ 2011 ） . Quality Assurance at a Distance : International Accreditation in Taiwan Higher Education, Higher Education, 61(2), 179–191 (SSCI)