martin ince communications limited the world university rankings 2007

Martin Ince Communications LimitedMartin Ince Communications Limited

The World University The World University Rankings 2007Rankings 2007


The Challenges of University The Challenges of University RankingRanking

Presentation by Martin IncePresentation by Martin Ince

Contributing editor, THES, Contributing editor, THES,

UniversitUniversité Libre, Brusselsé Libre, Brussels

December 12, 2007December 12, 2007


The THESThe THES

Since 1971Since 1971 Weekly newspaper formerly Weekly newspaper formerly

associated with The Times [of associated with The Times [of London]London]

Group including TESGroup including TES Online at Online at www.thes.co.ukwww.thes.co.uk since 1994 since 1994


Why rank universities?Why rank universities?

Interest in ranking things and peopleInterest in ranking things and people- HospitalsHospitals- SchoolsSchools- Local authoritiesLocal authorities- Rich lists; Britain, world, Asians, Rich lists; Britain, world, Asians,

footballersfootballers- Universities: The TimesUniversities: The Times


Does it matter?Does it matter?

Some of these rankings are fun or Some of these rankings are fun or pornography, eg Rich Listspornography, eg Rich Lists

Others are seriousOthers are serious- Bad marks for school or hospitalBad marks for school or hospital- Likewise for a university departmentLikewise for a university department- Over 1000 “failing” public bodies in Over 1000 “failing” public bodies in

UK – schools, police forces etc UK – schools, police forces etc


National RankingsNational Rankings The TimesThe Times- produced by John O’Leary, former produced by John O’Leary, former

editor of THESeditor of THES- Institutions as well as subjectsInstitutions as well as subjectsCriteria for subjects include:Criteria for subjects include:- Teaching quality- Teaching quality- Research qualityResearch quality- Entry standardsEntry standards- EmployabilityEmployability


National rankings (2)National rankings (2)

Criteria for institutions includeCriteria for institutions include- Teaching standardsTeaching standards- Staff/student ratioStaff/student ratio- Library spendingLibrary spending- Facilities spending Facilities spending - Good degreesGood degrees- JobsJobs- Research Research


The US ComparisonThe US Comparison US News and World Report US News and World Report

“America’s Best Colleges”“America’s Best Colleges”- Mainly about how likely you are to Mainly about how likely you are to

graduategraduate- Also student experience eg class sizeAlso student experience eg class size- However, many other tables eg However, many other tables eg

liberal arts, business, engineering liberal arts, business, engineering collegescolleges

- Likewise McLean’s (Canada) et al Likewise McLean’s (Canada) et al


Richness of dataRichness of data

Main table in US News has 18 main Main table in US News has 18 main columnscolumns

Likewise Times Good University Likewise Times Good University GuideGuide

How can we do this globally?How can we do this globally?


Why world rankings?Why world rankings?

Long overdue: higher education has Long overdue: higher education has always been very internationalalways been very international

Unique position of the THES Unique position of the THES Universities becoming more globalUniversities becoming more global Knowledge the real factor in Knowledge the real factor in

international competitivenessinternational competitiveness Increasing desire for comparative Increasing desire for comparative

informationinformation


Why world rankings (2)?Why world rankings (2)?

GATSGATS EU and BolognaEU and Bologna 3 million students outside home country3 million students outside home country Forecast to be 5 million by 2010Forecast to be 5 million by 2010 BTA BTA UK as a major source and destinationUK as a major source and destination UK as major collaboratorUK as major collaborator UK universities opening in China and UK universities opening in China and

elsewhereelsewhere


And Tony says soAnd Tony says so

““In 10 years we will think nothing of In 10 years we will think nothing of students going off to university students going off to university anywhere in the world”anywhere in the world”

Tony Blair at the Labour Party Tony Blair at the Labour Party conference, September 2006conference, September 2006


In additionIn addition

Interest from governments – UK Interest from governments – UK

Treasury, Lambert Treasury, Lambert EU, GermanyEU, Germany Shanghai Jiao TongShanghai Jiao Tong OECD from 2010OECD from 2010


How to do it?How to do it?

Audience includes –Audience includes – Internationally mobile studentsInternationally mobile students Internationally mobile staffInternationally mobile staff Internationally mobile moneyInternationally mobile money- Focus on:Focus on:- TeachingTeaching- ResearchResearch- International orientationInternational orientation


Peer reviewPeer review

Peer review is the way academic Peer review is the way academic value is measured value is measured

We decided to make it the We decided to make it the centrepiece of this rankingcentrepiece of this ranking

It is the least understood aspect of It is the least understood aspect of our workour work

So here is the explanationSo here is the explanation


Peer review (2)Peer review (2)

Total 5,101 over three yearsTotal 5,101 over three years International spreadInternational spread Subject spreadSubject spread Active academicsActive academics


The peersThe peers

Gathered by QS listbuildingGathered by QS listbuilding 41 per cent in Europe41 per cent in Europe 30 per cent the Americas30 per cent the Americas 29 per cent Asia Pacific29 per cent Asia Pacific

• Aggregate no more than three yearsAggregate no more than three years


The questionThe question

Online surveyOnline survey The top 30 universities in the topics The top 30 universities in the topics

they know aboutthey know about- Arts and humanitiesArts and humanities- Social sciencesSocial sciences- ScienceScience- BiomedicineBiomedicine- Technology Technology


PlussesPlusses

SimpleSimple UnderstandableUnderstandable RobustRobust Self-correcting if large enough Self-correcting if large enough

samplesample Hard to cheatHard to cheat


MinusesMinuses

Biases includeBiases include- AgeAge- SizeSize- Name Name

- BeijingBeijing- LoughboroughLoughborough- Brussels, LeuvenBrussels, Leuven

Audience conservatismAudience conservatism


EmployersEmployers

Another group who know about university Another group who know about university qualityquality

Innovation in 2005Innovation in 2005 Mainly private sectorMainly private sector At 10 per cent of totalAt 10 per cent of total Therefore academics cut from 50 to 40 per Therefore academics cut from 50 to 40 per

centcent Will be improvedWill be improved 2007 sample 1,4712007 sample 1,471


Employers are:Employers are:

Americas 43 per centAmericas 43 per cent Europeans 32 per centEuropeans 32 per cent Asia Pacific 25 per centAsia Pacific 25 per cent

• Again, maximum three years Again, maximum three years aggregationaggregation


Quantitative measuresQuantitative measures

Aim to measure universities in terms Aim to measure universities in terms ofof

- Student commitmentStudent commitment- Research commitmentResearch commitment- International commitment and International commitment and

competitivenesscompetitiveness


How to do thisHow to do this

Extensive data Extensive data gathering exercisegathering exercise

By UK firm QSBy UK firm QS Mix of data sourcesMix of data sources NationalNational InstitutionalInstitutional Direct contactDirect contact


First quantitative criterion…First quantitative criterion…

Staff/student ratioStaff/student ratio Classic measure of commitment to Classic measure of commitment to

teachingteaching 20 per cent of final score20 per cent of final score


How international?How international?

Two criteria rated at 5 per cent eachTwo criteria rated at 5 per cent each Is this somewhere where people want to Is this somewhere where people want to

be?be? StaffStaff StudentsStudents- Again raises issuesAgain raises issues

- Visiting scholars?Visiting scholars?- EU cross-border students?EU cross-border students?- Doing full courses?Doing full courses?- Geography advantageGeography advantage


CitationsCitations

Like peer reviewLike peer review Classic measure of research qualityClassic measure of research quality Use Scopus dataUse Scopus data


Citations (2)Citations (2)

Citations per staff member to see Citations per staff member to see density of brain powerdensity of brain power

Not citations per paperNot citations per paper Well-understood biasWell-understood bias- against non-English publicationagainst non-English publicationagainst arts and humanitiesagainst arts and humanitiesagainst national-oriented topics against national-oriented topics This accounts for the final 20 per centThis accounts for the final 20 per cent


Comparison with Shanghai Jiao TongComparison with Shanghai Jiao Tong

Not a newspaperNot a newspaper Nobel + Fields prizes Nobel + Fields prizes

Nobel 6/9 in 2007Nobel 6/9 in 2007

Fields 4/6 in 21Fields 4/6 in 21stst century century These used twiceThese used twice Science and NatureScience and Nature Science and Social Science citationsScience and Social Science citations 500 rather than our 200500 rather than our 200


Comparison with Shanghai Jiao TongComparison with Shanghai Jiao Tong

But:But:

Ends up looking rather similar near Ends up looking rather similar near the top the top

Share 133/200Share 133/200

Next – OECDNext – OECD


Changes in 2007Changes in 2007

Scopus: better at AsiaScopus: better at Asia Also more transparentAlso more transparent Better at non-English sourcesBetter at non-English sources


Changes (2)Changes (2)

Z score:Z score:

Measure distance from centreMeasure distance from centre

Normalise top to 100Normalise top to 100

Effects outliers such as CalTech, LSEEffects outliers such as CalTech, LSE


Changes (3)Changes (3)

Harder to cheatHarder to cheat

Main effects in South AsiaMain effects in South Asia

FTEs more rigorously definedFTEs more rigorously defined


What did we find?What did we find?

HarvardHarvard The US – 57 in top 200The US – 57 in top 200 Yale Oxbridge logjam at 2Yale Oxbridge logjam at 2


But…But…

The top 200 includes universities in The top 200 includes universities in 28 states28 states

US, UK, Australia, NetherlandsUS, UK, Australia, Netherlands Korea, China, JapanKorea, China, Japan Continental EuropeContinental Europe Developing world: Unam, Brazil (2), Developing world: Unam, Brazil (2),

Cape Town 200=Cape Town 200=


International commitmentInternational commitment

US shows up badlyUS shows up badly HKUST top for international staff, LSE HKUST top for international staff, LSE

in sixth placein sixth place London School of Economics top for London School of Economics top for

international studentsinternational students Yale among few US with international Yale among few US with international

staffstaff


Peer reviewPeer review

Berkeley the winnerBerkeley the winner Cambridge, Oxford, Stanford, Cambridge, Oxford, Stanford,

Harvard popularHarvard popular Well-liked universities all over the Well-liked universities all over the

worldworld Little evidence of patriotism biasLittle evidence of patriotism bias US, UK, Australia, Japan, China, US, UK, Australia, Japan, China,

Canada,Singapore dominate the top Canada,Singapore dominate the top 2020


CitationsCitations

Medical faculty is a big plusMedical faculty is a big plus Or major biomedical incomeOr major biomedical income CalTech the winner by some CalTech the winner by some

distance, then Stanford, MIT, ENSdistance, then Stanford, MIT, ENS Big country effect is at work hereBig country effect is at work here


Staff/student ratioStaff/student ratio

Winner CalTech, USWinner CalTech, US US, French, Swiss, etc institutions all US, French, Swiss, etc institutions all

well placedwell placed Harvard shows badly here at 15thHarvard shows badly here at 15th Asian and European universities well-Asian and European universities well-

placedplaced Weak correlation with research – but Weak correlation with research – but

not zeronot zero


Take home messageTake home message

Small variations don’t matter muchSmall variations don’t matter much A position can be gained by many A position can be gained by many

combinations of weakness and combinations of weakness and strengthstrength

eg, many Asian institutions do well eg, many Asian institutions do well despite scoring zero on citationsdespite scoring zero on citations


What we didn’t findWhat we didn’t find

Data on 500 institutions with Data on 500 institutions with significant citationssignificant citations

Have to teach undergraduatesHave to teach undergraduates Have to teach in at least two areasHave to teach in at least two areas


Things that don’t workThings that don’t work

Library spendingLibrary spending Course costCourse cost CompletionCompletion Entry standardsEntry standards WealthWealth Alumni givingAlumni giving


ResponseResponse

More work than writing the thingMore work than writing the thing Last year about 30 newspaper Last year about 30 newspaper

articles in Mexico alonearticles in Mexico alone Interest from media, universities etc Interest from media, universities etc

across Europe and Asiaacross Europe and Asia Less from the USLess from the US Political response – Ireland, Malaysia, Political response – Ireland, Malaysia,

SwitzerlandSwitzerland


Types of responseTypes of response

Reject the whole ideaReject the whole idea Complain about their positionComplain about their position Think it is about rightThink it is about right Wonder how to do betterWonder how to do better Compare Shanghai…Compare Shanghai…


Belgian institutionsBelgian institutions

Leuven 61 (was 96 in 2006)Leuven 61 (was 96 in 2006) Louvain 123Louvain 123 Ghent 124Ghent 124 ULB 154ULB 154 Antwerp 187 (up from 252)Antwerp 187 (up from 252) VUB 229 (down from 133)VUB 229 (down from 133) Liege 262 (201 in 2006)Liege 262 (201 in 2006)


This means…This means…

Strong showing cf comparable Strong showing cf comparable countriescountries

Cf Netherlands Cf Netherlands 11 in top 200, 12 in 50011 in top 200, 12 in 500 Perhaps less internationalPerhaps less international


Good at….Good at….

Peer reviewPeer review Employer reviewEmployer review Not very internationalNot very international Somewhat less citedSomewhat less cited


How to do betterHow to do better

Publish more in the right placesPublish more in the right places Be more internationalBe more international Be better represented academically Be better represented academically

around the worldaround the world Have better employer linksHave better employer links Have enough staff to teach your Have enough staff to teach your

studentsstudents


The future The future

Important for staff, students, parents, Important for staff, students, parents, management, employersmanagement, employers

Important for governmentsImportant for governments Important for businessImportant for business Important globally, eg for the EUImportant globally, eg for the EU Management tool:Management tool:- How do we look?How do we look?- Who to talk to?Who to talk to?- What are our ambitions?What are our ambitions?


Future developmentsFuture developments New dataNew data- Any suggestions?Any suggestions?- Subject specific?Subject specific?- Management indicatorsManagement indicators

Refine existing data, eg from employersRefine existing data, eg from employersMore global reach, eg AfricaMore global reach, eg AfricaNew analysesNew analysesNew entrantsNew entrants

And most importantly….And most importantly….


Job creationJob creation

More people looking at it than doing More people looking at it than doing itit

Leiden, CEPESLeiden, CEPES IREGIREG All highly valuableAll highly valuable


The bookThe book

Published 2006 and again early 2008Published 2006 and again early 2008 Yours from www.topuniversities.comYours from www.topuniversities.com 500 institutions including articles on 500 institutions including articles on

the top 100 and shorter details on the top 100 and shorter details on the restthe rest

Data in groups after 200Data in groups after 200


……really the last slidereally the last slide

Thanks toThanks to

Gerard Kelly, editor of The THESGerard Kelly, editor of The THES

Nunzio Quacquarelli, QSNunzio Quacquarelli, QS

Ben Sowter, QSBen Sowter, QS

Staff of ScopusStaff of Scopus

martin ince communications limited the world university rankings 2007

Documents

times slide

world rankings

research slide

world university rankings

comparative information

thes universities

times good university

us comparison us news