Love Them or Hate Them: Living With Rankings

Professor Ellen HazelkornDirector, Higher Education Policy Research Unit (HEPRU),

Education Consultant, BH Associates, (Ireland).Global University Rankings and Their Impact, Belgrade

7 November 2017

Which Are The Best Universities?

• Are the best universities those that focus disproportionately on research or those that focus on student learning and helping graduates earn credentials for sustainable living and employment?

• Are the best universities those which pursue global reputation OR those that encourage civic engagement and responsibility to their communities and wider society?

• Are the best universities those which adopt indicators chosen by ranking organisations for their own purposes OR those which choose indicators which best align with the university’s mission and purpose?

Origins of Rankings, Advantages & Disadvantages

What the Research Tells Us

What Experience Tells Us

Summary & Conclusion

1. Origins of Rankings, Advantages & Disadvantages

Quality Is A Geopolitical Issue

• As globalisation accelerates, the range and number of providers grows,demand for talent rises, and market principles intrude further:

– Growing demand for methodologies and global architecture tounderpin and facilitate international education and global science;

– Quality no longer ”owned” by HEIs or evaluators/accreditors.

• Quality and excellence are key differentiators in national/global market;

– National geo-political positioning and pride;

– Beacon to attract/retain investment, business and talent;

– Institutional reputation and status;

– Performance assessment of scientific-scholarly research;

Concerns About Quality

1. Assessment required to show qualifications are high quality and internationally comparable and transferable.

– Vital in globalised world, in which students/graduates are mobile and employers recruit internationally.

2. Government or students (or other stakeholders) increasingly aware of getting value for money.

– For government, desire to get more for less by achieving what it regards as efficiencies, while for students it is association with salary, career and lifestyle.

3. Massification and surge in student demand and mobility has led to growth in number/range of educational programmes/providers, e.g. for-profit and transnational/cross-border HE.

– Concerns about standards, accountability, unethical practices and promises.

Quality Is Contested Concept

• Quality has traditionally been responsibility key responsibility/feature of academic autonomy/self-regulation;

• Reliance on peer review, internal procedures of quality assurance at programmatic/institutional level, external examining/evaluation.

• Bologna emphasized student learning outcomes; Lisbon Agreement emphasized pursuance of excellence.

• Internationalisation of HE has seen:

– Codification of practice and processes; transparency & accountability;

– Government-led initiatives, and demand for internationally; comparative systems and data;

– Global architecture, w/ international/supra-national organisations.

• Transformed from something institutionally-led to being driven and regulated by the state.

Geopolitics Of Rankings

• Global rankings are inevitable product of internationalised higher education market and world economy.

- Signify national pre-eminence no longer enough in global economy;

- Succeed in placing educational quality, performance and productivity within wider comparative and international framework.

• Ability vs. inability to compete at this level is critical and amplifing changes in the world order.

- Elevated indicators of investment as powerful drivers of international benchmarking; resource-intensive competition and government policy;

- Reflect and structure the world economy and global science;

- Illustrate and affirm disparities in wealth.

• Rankings less about student choice; more about geopolitical positioning.

Most Influential Global Rankings Today

• Academic Ranking of World Universities (ARWU) (Shanghai Jiao Tong University, China), 2003

• Webometrics (Spanish National Research Council, Spain), 2004

• National Taiwan University Rankings (formerly Performance Ranking of Scientific Papers for Research Universities, HEEACT), 2007

• Leiden Ranking (Centre for Science & Technology Studies, University of Leiden), 2008

• SCImago Journal and Country Rank (SJR) (Spain), 2009

• University Ranking by Academic Performance (URAP) (Informatics Institute of Middle East Technical University, Turkey), 2009

• QS World University Rankings (Quacquarelli Symonds, UK), 2010

• THE World University Ranking (Times Higher Education, UK), 2010

• U-Multirank (European Commission, Brussels), 2014

• Best Global Universities rankings (USNWR, US), 2014

What Global Rankings Measure

Global Rankings Measure

• Bio- and medical sciences Research

• Student and Faculty Characteristics (e.g. productivity, entry criteria, faculty/student ratio)

• Internationalization

• Reputation – amongst peers, employers, students

• Emphasis on elite universities and elite/high achieving students

Global Rankings Do Not Measure

• Teaching and Learning, incl. "added value",

• Arts, Humanities and Social Science Research

• Impact and Benefit of Research

• Regional or Civic Engagement

• Student Experience

• Ignore non-traditional student, e.g. mature/adult learners

Advantages of Rankings

• Simple, quick and easy way to measure/compare HE performance &“quality” , within wider comparative and international framework;

– Inform student choice and stakeholder opinion;

– Beacon to attract/retain mobile capital and talent;

– Performance assessment of scientific-scholarly research;

– Signal of what to expect upon graduation and from graduates;

– Value-for-money and return-on-(public) investment;

• Accountability tool, esp. where QA culture/practices weak or immature;

• Heighten attention to quality and drive-up performance:

– Accelerate modernisation agenda;

– Emphasize institutional strategic decision-making and datacollection/analysis.

Disadvantages of Rankings

• HEIs are complex organisations meeting diverse needs, but rankings measure/compare “whole institutions” using same set of indicators;

– Leads to simplistic comparisons: whereas, statistical differences are insignificant;

• Academic quality is complex and not easily reduced to quantification;

– Use of proxy variables can misrepresent and lead to unintended consequences;

– Difficulty obtaining meaningful indicators and (international) comparative data.

– Undermines mission diversity, and ignores diversity of student cohort (e.g. SES factors);

• Focus on small number of indicators encourages perverse behaviour –because it promotes a single model of university or quality/excellence.

Research vs. Reputation

Rankings Research Reputation

Academic Ranking of World Universities [ARWU] (Shanghai Jiao Tong, China)

100 50

Times Higher Education World University Ranking [THE] (UK)

93.5 33

Quacquarelli Symonds World Ranking [QS] (UK)

70 50

NB. Computation based on an assumption of a strong correlation between academic reputation and research/research related activities.

Ranking Wealth?

• Indicators measure wealth – whether garnered via institutional age, endowments, tuition or government investment;

– Gap is widening

• Wealthiest private colleges/universities have maximised their advantages over recent decades:

– Compensation for Faculty– Recruitment of most able/high-socio economic students– Money spent on facilities and student experience

• Corresponds with growing income and social/capital inequality

2. What The Research Tells Us

What Have We Learned

• Rankings are driver of higher education decision-making at the institutional and national level;

⎼ Highlights ambition and sets explicit strategic goal;

⎼ Identifies KPIs used to measure performance and reward success;

⎼ Rankings help identify under-performers and "reputational" disciplines.

• Students, high achievers and international, use rankings to inform choice;

• Other HEIs use rankings to identify potential partners or membership of international networks;

• Employers and other stakeholders use rankings for recruitment or publicity purposes;

• Governments policy is increasingly influenced by rankings.

Who Uses Rankings

Students, public opinion formers and government are biggest users of rankings – and most likely to be influenced ‘negatively’ by changes.

• Domestic undergraduate students

• Internationally mobile students and faculty

• Postgraduate students

• Government/Policymakers

• Academic partners and academic organisations

• Employers

• Sponsors, philanthropists and private investors

• Industrial partners

• Public and public opinion

• Ranking agencies/organisations

Responses to Rankings

1. Strategic positioning and decision making – strategic decision-making, branding and promotional activity;

2. Redefining activities and altering perceptions – refocus priorities and activities as universities emphasize particular indicators and measures;

3. Evolving responses – shifts from initial dissonance to efforts to align university with rankings;

4. Affective responses – impact on staff morale and increased anxiety;

5. Self-management – use of rankings to leverage change within the university;

6. Degrees of control – resisting, managing, exploiting, and “gaming” –efforts to influence rankings via liaisons/consultations and changes in how data presented.

(Locke in Altbach et al, 2016, 227-228; Espland & Sauder, 2016)

Rankings’ Role In Institutional Strategy

Institutional Reaction: Some Findings

• 83% surveyed HEIs unhappy with rank cf. 58% in 2006;

• 84% HEIs have formal internal mechanism to review institution’s rank, and in 40% of cases, this is led by Vice Chancellor, President or Rector;

• Overwhelming majority use rankings to inform strategic decisions, set targets or shape priorities, and inform decisions about int’l partnerships;

• Majority of HEIs believe rankings more helpful than hindrance to institutional reputation;

• 84% use rankings to monitor performance of peer institutions in their own country cf. over 76% in 2006;

• ~77% HEIs monitor peers worldwide in 2014 cf. ~50% in 2006.

Changes In Institutional Decisions And Academic Behaviour

• Over 50% HEIs (2014 ) have made strategic, organizational, managerial or academic decisions to improve position in rankings:

– Revising policy and resource allocation;

– Prioritising research areas;

– Changing recruitment and promotional criteria;

– Creating, closing or merging departments or programmes; and/or merging with another HEI, research institute, etc.

– Identifying preferential journals in which faculty should seek to be published;

– Research “stars” rewarded while teaching often seen as a “punishment”.

Source: Hazelkorn (2015) Rankings and the Reshaping of Higher Education. The Battle for World-class Excellence. Palgrave MacMillan.

Psychology of Ranking Positioning

More effective to label a university “Top 100” than “Number 89”, or even “Top

90”. “Top 100” advantageous over “Top 101” which could be interpreted as

“Top 200”. Helps define market-specific communication strategies.

Does Your Institution Monitor Its Position In Rankings?

Reasons For Monitoring Other Institutions

Process for Monitoring Rankings

Examples of HEI Actions Taken ~Weighting (2014)

Research Increase output, quality and citations

Recruit and Reward faculty for publications in highly-cited journals

Publish in English-language journals

Set individual targets for faculty and departments

Increase number/proportion of PhD Students

ARWU = 100%

THE-QS = 60%

NTU = 100%

THE = 93.25%

QS = 70%

Organization Merge with another institution

Develop/expand English-language facilities

Establish Institutional Research capability

Embed rankings indicators as a performance indicator

Form task group to review and report on rankings.

ARWU = 10%;

Research related indicators as above

Students Target recruitment of high-achieving students, esp. PhD students

Offer scholarships and other benefits

More international activities and exchange programmes

Open International Office and professionalise recruitment

THE = 9.25

QS = 5%

Faculty Recruit/head-hunt international high-achieving/HiCi scholars

Create new contract/tenure arrangements

Set market-based or performance/merit-based salaries

Reward high-achievers & Identify weak performers

Enable best researchers to concentrate on research

ARWU = 80%

THE-QS = 95%

NTU = 100%

THE = 97.5%

QS = 95%

Public Image/ Marketing

Reputational factors

Professionalise Admissions, Marketing and Public Relations

Ensure common brand used on all publications

Expand internationalisation alliances and membership of global networks

ARWU = 50%

THE-QS = 40%

QS = 50%

THE = 33%

Factors Influencing Student Choice

• Quality – and especially “perceptions” of quality;

– Includes institutional prestige or country profile and whether the qualification will be recognized by future employers;

– But students are not a homogeneous group;

– Ability, ambition and socio-economic status influence propensity for studying abroad, and country/institutional choice.

• Family history and family support:

– Where family/relatives or friends have gone previously;

– Financial and life support

• Choice of subject – international students more likely to study STEM compared with domestic students who choose AHSS

– Helps explains interest in international student recruitment.

International PhD Students Favour Science

Top 10 Factors Influencing Student Choice, 2010 and 2014

Student Choice & Rankings: Some Findings 1

• Rankings provide good source of information especially for int’l students

– Institutional rank transmits social and cultural capital which resonates with family, friends and potential employers;

• Top destinations for internationally mobile students include a large number of top-ranked higher educational institutions;

• Students worldwide are increasingly aware of quality differences in tertiary education systems as university league tables and other international university rankings are widely diffused;

• Data often refers to reputation rather than rankings – but its difficult to disassociate the two concepts.

– Thus, OECD talks of “perceptions of quality”.

Student Choice & Rankings: Some Findings 2

• 80% undergraduate and postgraduate (taught and research) students have a high interest in rankings, with no real difference between undergraduate and postgraduate students (i-graduate, 2014);

• High achieving and high socio-economic students are most likely to make choices based on non-financial factors, e.g. reputation and rankings;

• International students continue to rate reputation and position in rankings as key determinants in their choice of institution, programme and country;

• Strong correlation between rankings, perceptions of quality, institutional reputation and choice of destination, at the national and institutional level;

Students Most Influenced By Rankings

3. What Experience Tells Us

Griffith University, Australia

• Australia, at the national policy level, highly influenced by rankings –geopolitical positioning and demographic/economic issues;

– Governments have issued statements over the years, highlighting importance of rankings as international benchmark and strategic target

• Strategic target: to be “one of the most influential universities in Australia and Asia-Pacific Region” - global issues subsumed national objectives

• Rankings form part of strategic planning mix but not used to set targets or for formal reporting;

– Uses benchmarking data provided by rankings,

– Faculty have had to adopt to use of a wide range of indicators and benchmarking techniques, e.g. citation and h-impact.

• Improvement is burden – increasing expectations and slippage is problem.(Sheils in Altbach et al., 2016, 12-37)

University X, China

• Chinese government set ambition to establish world-class universities –211 Plan, 985 Project, 2011 Plan

• Rankings identified huge gap between current status and ambition.

– Initially, gap was infrastructure;

– Today, gap is professional qualification and ability of academics.

• Attention focused on targeted actions has improved rankings:

– Recruit more overseas PhD graduates, and in-service/professional development of domestic Chinese academics without a PhD;

– HR reforms, incl. tenure system; classification of academic positions;

– Increased competitiveness within/between academic staff.

• But tensions emerge:

– Teachers without PhD at grave disadvantage;

– Preferential treatment/positions given to returnees. (review article, unpublished)

• Young (1981) research-intensive university on rapid global rise, due to combination of factors:

– Political support and investment in higher education and R&D;

– Introduction of tenure track, alongside flexible labour laws;

– Recruitment of international “stars”;

– Collaboration with international corporates and universities.

• Actions aligned with attributes promoted by rankings:

– Staff-student ratio, 16:1

– 32% international students

– Research publications and citation impact

– Student selectivity index.

• Ranked 1st amongst THE under 50 years.

Nanyang Technological University Singapore

Universitat Rovira i Virgili, Spain

• Established 1991 –decision by Catalan Parliament to divide U Barcelona;

– Establish “knowledge pole” to contribute “decisively to the involvement of Catalonia and Spain in the cultural, social and economical development of the world”;

• Align teaching, research & innovation along 5 strategic areas: chemistry/chemical engineering; classical/prehistoric archaeology; oenology; tourism/leisure; nutrition/health;

• Leverage local expertise for global recognition.

– Research performance in key fields → good position in rankings (rather than vice versa)

University of Kentucky, USA

• State legislature set goal to reach Top-20 by 2020 according to USNWR.

• University identified the need to: recruit additional 6200 undergraduates, 750 graduate and professional students, 374 post-doctoral researchers, 625 faculty, award 3065 bachelor and 189 doctoral degrees, and raise research expenditure by $470m [EUR 345.5 million]

• University had to alter student entry criteria and become more selective to meet completion/employment level, including graduate salaries:

– As a land-grant university this meant refocusing its mission and becoming more prestigious and exclusive.

• Post-2008, state & university faced difficult economic/budgetary problems;

• By 2009-2010, university had failed to keep pace with its 2006 metrics, plus there was a major funding gap of over $420m [EUR 309.8 million].

• The strategy was abandoned. (DeYoung and Baas, 2012, 89; University of Kentucky, 2005; Lederman, 2005).

4. Summary & Conclusion

Emerging Tensions

• Rankings are often strongly criticised but they do tell us something about a university’s performance and academic quality/productivity;

– Shift from self-declaration to external verification;

• Schism between local, regional, national and global ambitions

– Widening access and participation and elite recruitment

• Rankings increasingly important for international partnerships and scholarships;

• Rankings privileging of bio-sciences vs. arts, humanities and

social sciences;

• Allocation of different values, resources & benefits within

the university, e.g. research vs. teaching; graduate vs.

undergraduate students

Changing Our Views About HE

• Massification and globalisation have altered relationship between higher education and the state;

– Society more diverse and many more stakeholders have a view about higher education;

– Questions asked about role of HE in/for society.

• World becoming visibly more competitive and multi-polar;

– Cross-national/jurisdictional comparisons inevitable by-productof globalization and will intensify in the future;

– Rankings appear to be independent - operating outside traditional structures.

Moving Beyond Rankings

• Emphasis on quality, performance and accountability should be seen within the context of

– Wider concerns about “good” governance, and

– National competitiveness at a time of accelerating globalisation and correspondingly higher education’s ability to attract mobile capital and talent.

• Future developments

• Growing range of simple, comparable and quantitative instruments

– Growing demand for high quality and publicly accessible databases

– More sophisticated methodologies

– Alternative rankings and alternatives to rankings

• Renegotiating the social contract and new forms of accountability.

Contact Information:

Higher Education Policy Research Unit (HEPRU), Dublin Institute of Technology, Ireland

BH Associates – Education Consultants, Ireland

ellen.hazelkorn@dit.ieinfo@bhassociates.eu

https://www.dit.ie/hepruhttps://www.bhassociates.eu