How to increase one's ranking “efficiently”?

- Decision Pathway Sensitivity Analysis

Fang XuKent Business School, UKC

Centre for the Evaluation of Research Performance (CERP)

3rd Workshop on IWIPM, 1st July, 2010, UKC

IntroductionIntroduction Input data sensitivity is an important issue for any

ranking methods. Results reliabilityInfluence of decision making

Normally we wish that a ranking system has a good stability with respect the data perturbation so to have stable ranking results.

If “iceberg tip” type of data (like Nobel Prize numbers, top awards, top journals etc.) are given heavy weights in a ranking system, the results often are quite sensitive to the data perturbation – one extra award may just change the rankings so much.

An example of CAS International An example of CAS International Institution Ranking ResearchInstitution Ranking Research

Great change of scores can be found in the results due to data perturbation.

Final Results Change Rate

Institutes Data Perturbation (+/-) 5%

Institute 1 64.86%

Institute 2 56.02%

Institute 3 19.78%

Institute 4 22.41%

Institute 5 36.05%

Institute 6 10.16%

Institute 7 13.54%

Institute 8 17.63%

……. …….

One of the possible remedy…One of the possible remedy…

To use data from different part of the “iceberg” – from the base to middle to tip – important is the shape of the iceberg, not necessary the most visible part – tip.

E.g. in CAS study, indicators aimed to describe the middle part of the “iceberg” have been applied, such as top 1% publications, top 10% publications etc.

More details have been introduced by Prof. Steve Liu.

On the other hand…On the other hand… This sensitivity does raise another issue:

it may be possible to increase one’s rank rapidly by invest on these sensitive areas;

E.g. recruit a small number of certain type of personnel ( Nobel prize winners etc)

Influence to decision makers

In this talk, we show some initial analysis in this regard.

Decision pathway analysisDecision pathway analysis To simulate such decision scenarios and analyze

how one’s rank change;

From the comparison and analysis of the simulation results, it is possible to find out “efficient” way to increasing one’s rank for an organization. 

We will illustrate our idea using Academic Ranking for World Universities (ARWU) produced by Shanghai Jiaotong University and World Famous University (WFU) produced by Wuhan University.

MethodologyMethodology Data recovery

Raw data is necessary for simulation analysis;

Raw data can be recovered via a sample (e.g. Harvard University) using the methods stated;

However, Răzvan Florian show that accurate raw data is hardly recovered in some ranking schemes.

(Răzvan Florian, Ad Astra 5, 2006,

www.ad-astra.ro/journal)

Several scenarios are consideredSingle indicator perturbationDual indicators perturbationChain influence of multi-indicators perturbation (Decision

Pathway sensitivity analysis)

One more Award winner

Two more Award

winners

Two more HICI

researchers

One more HICI

researcherChang

e in Rank?

One more N&S Pubs

Two more N&S Pubs

Criteria and weights of ARWU Criteria and weights of ARWU rankingranking

……..Criteria Indicator Code

Weight

Quality of Education

Alumni of an institution winning Nobel Prizes and Fields Medals

Alumni 10%

Quality of Faculty

Staff of an institution winning Nobel Prizes and Fields Medals

Award 20%

Highly cited researchers in 21 broad subject categories

HiCi 20%

Research Output

Papers published in Nature and Science N&S 20%

Papers indexed in Science Citation Index-expanded and Social Science Citation Index

PUB 20%

Per Capita Performance

Per capita academic performance of an institution

PCP 10%

Total 100%

For institutions specialized in humanities and social sciences such as London School of Economics, N&S is not considered, and the weight of N&S is relocated to other indicators.

Data sensitivity of ARWU rankingData sensitivity of ARWU ranking --- --- Single indicator Single indicator perturbationperturbation

Influence of sample universities’ ranks in the case of single indicator perturbation in an interval.

Change Rate

HICI PUB

A Universit

y

B Universit

y

A Universit

y

B Universit

y

2.50% 2 2 11 7

5.00% 7 5 16 14

7.50% 14 7 25 22

10.00% 16 8 30 26

12.50% 28 13 36 32

Data sensitivity of ARWU rankingData sensitivity of ARWU ranking --- --- multi-multi-indicators indicators perturbationperturbation Influence of A university’s ranks in the case of multi

indicators perturbation in an interval.

Change In Final Scores

HICI

2.50% 5.00% 7.50% 10.00% 12.50%

PUB

2.50% 13 15 18 25 34

5.00% 16 23 25 33 40

7.50% 25 28 34 36 51

10.00% 33 34 38 48 59

12.50% 36 42 48 56 65

Data sensitivity of ARWU rankingData sensitivity of ARWU ranking------ Chain Influence of multi indicators Chain Influence of multi indicators perturbationperturbationScenarios

Decision Pathway DescriptionChange in Rank

AUniversity

B University

Scenario 0 0 Award, 0 HICI winner, 0 N&S Pubs - -

Scenario 1 0 Award,1 HICI winner 11 5

Scenario 2 0 Award, 1 HICI winner, 2 N&S Pubs (SSCI) 15 8

Scenario 3 1 Award & HICI winner 70 45

Scenario 4 1 Award & HICI winner, 2 N&S Pubs (SSCI) 72 60

Scenario 5 1 Award & HICI winner, 6 N&S Pubs (SSCI) 83 82

Scenario 6 2 Award & HICI winners 118 82

Scenario 7 2 Award & HICI winners, 8 N&S Pubs (SSCI) 130 110

Scenario 82 Award & HICI winners, 16 N&S (SSCI), 20 other SSCI Pubs 144 127

Criteria of World Famous University Criteria of World Famous University (WFU)(WFU)

Description Indicators Weights

Research productivity

Total publications 0.3

Total citations 0.2

Research impact

Highly cited papers 0.2

Number of subjects in the rank 0.05

Research innovation

patents 0.05

Hot papers 0.1Research development Percentage of highly cited papers 0.1

Data sensitivity of WFU rankingData sensitivity of WFU ranking------ Chain Influence of multi indicators Chain Influence of multi indicators perturbationperturbation

Scenarios

Decision Pathway DescriptionChange in Rank of C University

Scenario 11 more Highly Cited paper 11

Scenario 21 more Highly Cited paper, 1 more Hot paper 48

Scenario 31 more Highly Cited paper, 1 more Hot paper & 1 more patents 48

Scenario 4 2 more Highly Cited papers 66

Scenario 52 more Highly Cited papers, 2 more Hot papers 97

Scenario 62 more Highly Cited papers, 2 more Hot papers & 2 more patents 98

Conclusions & LimitationsConclusions & Limitations

Data sensitivity has been estimated of two ranking schemes, significant changes have been detected in the case of single indicator perturbation, and dual indicators perturbation.

Particularly, more attention has been paid to decision pathway sensitivity analysis, where three universities have been selected for an illustration.

Unavoidable errors exist due to difficulties in data recovery in some ranking schemes.

Thanks to the useful discussions with AWRU and WFU groups!

Thanks for your patience!