pef cwts leiden ranking info 20140915 -...

PROPOSAL

CWTS-Leiden University Leiden Ranking 2014 information

September 15th, 2014

www.cwts.nl | of 16


Centre for Science and

Technology Studies,

Leiden University

CWTS-Leiden University

Leiden Ranking 2014 information

For: Perspektywy Education Foundation

Julia Lysik

Tel. (+48 22) 628 58 62 w. 15

E-mail [email protected]

From: CWTS B.V.

Mark Neijssel

P.O. Box 905

2300 AX Leiden, the Netherlands

Tel. +31 71 527 3909

Fax +31 71 527 3911

E-mail [email protected]

www.cwts.nl | of 16



Technology Studies,

Leiden University

Table of contentsTable of contents ................................................................................................................................................................ 2

Introduction ........................................................................................................................................................................... 4

CWTS Leiden Ranking 2014 .......................................................................................................................................... 5 Methodology ............................................................................................................................... 5

Main fields of science ................................................................................................................. 8

Bibliometric indicators ................................................................................................................ 9

Results Lomonosov Moscow State University .......................................................................... 12

Supporting publications ........................................................................................................... 12

About CWTS B.V. ............................................................................................................................................................... 14 The benefits of CWTS B.V. ........................................................................................................ 14

Track-record of CWTS B.V. ........................................................................................................ 14

Questions ............................................................................................................................................................................. 16

www.cwts.nl | of 16



Technology Studies,

Leiden University

Introduction The Leiden Ranking is a global university ranking based exclusively on bibliometric data. In

this document, we introduce the Leiden Ranking. The Leiden Ranking 2014 edition, 2013

edition and the 2011/2012 edition are all available on the website www.leidenranking.com.

University rankings have quickly gained popularity, especially since the launch of Academic

Ranking of World Universities, also known as the Shanghai Ranking, in 2003, and these

rankings nowadays play a significant role in university decision making. The increased use of

university rankings has not been hampered by the methodological problems that were already

identified in an early stage. There are now many rankings in which universities are compared

on one or more dimensions of their performance. Many of these rankings have a national or

regional focus, or they consider only specific scientific disciplines. There is a small group of

global university rankings. The Leiden Ranking belongs to this group of rankings.

Global university rankings are used for a variety of purposes by different user groups. Three

ways of using university rankings seem to be dominant. First, governments, funding agencies,

and the media use university rankings as a source of strategic information on the global

competition among universities. Second, university managers use university rankings as a

marketing and decision support tool. And third, students and their parents use university

rankings as a selection instrument.

An important methodological problem of the most commonly used global university rankings

is their combination of multiple dimensions of university performance in a single aggregate

indicator. These dimensions, which often relate to very different aspects of university

performance (e.g., research performance and teaching performance), are combined in a quite

arbitrary fashion. This prevents a clear interpretation of the aggregate indicator. A second

related problem has to do with the fact that different universities may have quite different

missions. Two universities that each have an excellent performance on the dimension that is

most relevant to their mission may end up at very different positions in a ranking if the

different dimensions are weighted differently in the aggregate indicator. These

methodological problems can partly be solved by providing separate scores on the various

dimensions and refraining from aggregating these scores in a single number. A third problem

is more practical. Some rankings rely heavily on data supplied by the universities themselves,

for instance data on staff numbers or student/staff ratios. This dependence on the universities

makes these rankings vulnerable to manipulation. Also, because of the lack of internationally

standardized definitions, it is often unclear to what extent data obtained from universities can

be used to make valid comparisons across universities or countries.

www.cwts.nl | of 16



Technology Studies,

Leiden University

A solution to these fundamental methodological problems is to restrict a ranking to a single

dimension of university performance that can be measured in an accurate and reliable way.

This is the solution that the Leiden Ranking offers. The Leiden Ranking does not attempt to

measure all relevant dimensions of university performance. Instead, the ranking restricts itself

to the dimension of research performance based on international journals. Other dimensions

of university performance, in particular the dimension of teaching performance, are not

considered. The Leiden Ranking includes 750 major universities worldwide and is based on

bibliometric data from the Web of Science database. No data is employed that has been

supplied by the universities themselves. A sophisticated procedure for assigning publications

to universities is used to further improve the quality of the bibliometric data. The first edition

of the Leiden Ranking was produced in 2007.

CWTS Leiden Ranking 2014 The CWTS Leiden Ranking 2014 measures the scientific performance of 750 major universities

worldwide based on international scientific literature. Using a sophisticated set of

bibliometric indicators, the ranking aims to provide highly accurate measurements of the

scientific impact of universities and of universities’ involvement in scientific collaboration. The

CWTS Leiden Ranking 2014 is based on Web of Science indexed publications from the period

2009–2012.

Methodology

Data collection

The CWTS Leiden Ranking 2014 ranks the 750 universities in the world with the largest

contribution in international scientific journals in the period of 2009–2012. The ranking is

based on our CWTS Citation Index system (CI-system). The core of this system comprises of an

enhanced version of Thomson Reuters/Institute of Scientific Information’s (ISI) citation

indexes: Web-of-Science version of the Science Citation Index; Social Science Citation Index

and Arts & Humanities Citation Index. A combination of smart computer algorithms and

manual data cleaning ensures a better unification of the names and addresses of universities

and other organisations. Moreover, CWTS ensures better citation counts by taking great care

in proper linking of the citing- and cited publications.

In the following section, a summary of the data collection methodology of the CWTS Leiden

Ranking 2014 is provided. It should be emphasized that, in general, universities did not verify

and approve the publication data of their institution and that publications have been assigned

to universities on the basis of the institutional affiliations mentioned by the authors of the

www.cwts.nl | of 16



Technology Studies,

Leiden University

publications. However, the assignment of publications from these affiliations is by no means a

straightforward task. A university may be referred to by many different (non-English) name

variants and abbreviations. In addition, the definition and delimitation of universities as

separate

Identification of universities

The criteria that have been adopted to define universities for the Leiden Ranking are not very

formal. Typically, a university is characterized by a combination of education and research

tasks in conjunction with a doctorate-granting authority. However, these characteristics do not

mean that the universities are particularly homogeneous entities that allow for international

comparison on every aspect. The focus of the Leiden Ranking on scientific research certifies

that the institutions included in the Leiden Ranking have a high degree of research intensity in

common. Nevertheless, the ranking scores for each institution should be evaluated in the

context of its particular mission and responsibilities. These missions and responsibilities in

turn are strongly linked to the national and regional academic systems in which universities

operate. Academic systems - and the role of universities therein - differ substantially from one

another and are constantly changing. Inevitably, the outcomes of the Leiden Ranking reflect

these differences and changes.

The international variety in the organization of academic systems also poses difficulties in

terms of identifying the proper unit of analysis. In many countries, there are collegiate

universities, university systems, or federal universities. Again, instead of applying formal

criteria when possible we followed common practice based on the way these institutions are

perceived locally. Consequently, we treated the University of Cambridge and the University of

Oxford as entities but in the case of the University of London, we distinguished between the

constituent colleges. For the United States, university systems (e.g. University of California)

were split up into separate universities. The higher education sector in France, like in many

other countries, has gone through many reorganizations in recent years. Many French

institutions of higher education have been grouped together in Pôles de Recherche et

d'Enseignement Supérieur (PRES), or in consortia. In most cases, the Leiden Ranking still

distinguishes between the different constituent institutions but in particular cases of very tight

integration, consortia were treated as if they were a single university (e.g. Grenoble INP).

Publications are assigned to universities based on their most recent configuration. Changes in

the organizational structures of universities up to 2013 have been taken into account. For

example, in the Leiden Ranking 2014, the University of Lisbon which merged with the

Technical University of Lisbon in 2013 encompasses all publications assigned to the old

University of Lisbon as well as the publications previously assigned to the Technical University

of Lisbon.

www.cwts.nl | of 16



Technology Studies,

Leiden University

Affiliated institutions

A key challenge in the compilation of a university ranking is the handling of publications

originating from research institutes and hospitals associated with universities. Among

academic systems a wide variety exists in the types of relations maintained by universities

with these affiliated institutions. Usually, these relationships are shaped by local regulations

and practices and affect the comparability of universities on a global scale. As there is no easy

solution for this issue, it is important that producers of university rankings employ a

transparent methodology in their treatment of affiliated institutions.

CWTS distinguishes three different types of affiliated institutions:

1. component

2. joint research facility or organization

3. associated organization

In the case of components the affiliated institution is actually part of the university or so

tightly integrated with it or with one of its faculties that the two can be considered as a single

entity. The University Medical Centres in the Netherlands are examples of components. All

teaching and research tasks in the field of medicine that were traditionally the responsibility of

the universities have been delegated to these separate organizations that combine the medical

faculties and the university hospitals.

Joint research facilities or organizations are the same as components except for the fact that

they are administered by more than one organization. The Brighton & Sussex Medical School,

the joint medical faculty of the University of Brighton and the University of Sussex and,

Charité, the medical school for both the Humboldt University and Freie Universität Berlin are

both examples of this type of affiliated institution.

The third type of affiliated institution is the associated organization which is more loosely

connected to the university. This organization is an autonomous institution that collaborates

with one or more universities based on a joint purpose but at the same time has separate

missions and tasks. In many countries, hospitals that operate as teaching or university

hospitals fall into this category. Massachusetts General Hospital, one of the teaching hospitals

of Harvard Medical School, is an example of an associated organization.

The treatment of university hospitals in particular is of substantial consequence as medical

research has a strong presence in the Web of Science. The importance of associated

organizations is growing as universities present themselves more and more frequently as

network organizations. As a result, researchers formally employed by the university but

working at associated organizations may not always mention the university in publications. On

www.cwts.nl | of 16



Technology Studies,

Leiden University

the other hand, as universities become increasingly aware of the significance of their visibility

in research publications, they actively exert pressure on researchers to mention their

affiliation with the university in their publications.

In the Leiden Ranking 2014, publications from affiliated institutions of the first two types are

considered as output from the university. A different procedure has been followed for

publications from associated organizations. A distinction is made between publications from

associated organizations that also mention the university and publications from associated

organizations that do not contain such a university affiliation. In the latter case, publications

are not counted as publications originating from the university. In the event that a publication

contains affiliations from a particular university as well as affiliations from its associated

organization(s), both type of affiliations are credited to the contribution of that particular

university to the publication in the fractional counting method.

The 750 universities: selection and counting method

The 750 universities that appear in the Leiden Ranking have been selected based on their

contribution to articles and review articles published in international scientific journals in

the period of 2009–2012. The contribution of a university to an article is calculated based on

the number of affiliations mentioned in the article. If an article mentions three different

affiliations of which two belong to a particular university, then the contribution of that

university to the article is counted as two thirds. Only publications in core journals are

included. The equivalent of more than 1,000 papers was required for a university to be ranked

among the 750 universities with the largest scientific output.

Data quality

It is important to highlight that the assignment of publications to universities is not free of

errors. There are generally two types of errors: 'false positives', which are publications that

have been assigned to a university when they do not in fact belong to that university, and

'false negatives', which are publications that have not been assigned to a university when they

should in fact have been. Considerably more false negatives than false positives should be

expected, especially since the 5% least frequently occurring addresses in the database may not

have been manually checked. This can be considered a reasonable upper bound for errors,

since the majority of these addresses are probably non-university addresses.

Main fields of science

The CWTS Leiden Ranking 2014 provides statistics not only at the level of science as a whole

but also at the level of the following seven broad fields of science:

• Cognitive and health sciences

www.cwts.nl | of 16



Technology Studies,

Leiden University

• Earth and environmental sciences

• Life sciences

• Mathematics, computer science, and engineering

• Medical sciences

• Natural sciences

• Social sciences

The above fields have been defined using a unique bottom-up approach. Traditionally, fields

are defined as sets of closely related journals. This approach is problematic especially in the

case of multidisciplinary journals such as Nature, PLoS ONE, PNAS, and Science, which do not

belong to one particular field. The seven broad fields of science listed above have been

defined at the level of individual publications rather than at the journal level. Using a

computer algorithm, each publication in the Web of Science database has been assigned to

one of these seven fields. This has been done based on a large-scale analysis of hundreds of

millions of citation relations between publications.

To get a more precise idea of the topics covered by each of the seven main fields, a VOSviewer

term map has been created for each field. The term maps are shown at the Leiden Ranking

website: www.leidenranking.com in the section “Methodology-Main fields”. The maps can be

explored using the VOSviewer software (readily available and free of charge via

www.vosviewer.com. The software requires Java version 6 or higher to be installed on your

system).

Bibliometric indicators

The CWTS Leiden Ranking 2014 is based on publications in Thomson Reuters' Web of Science

database (Science Citation Index Expanded, Social Sciences Citation Index, and Arts &

Humanities Citation Index) in the period 2009–2012. Book publications, publications in

conference proceedings, and publications in journals not indexed in the Web of Science

database are not included. Within the Web of Science database, only publications in

international scientific journals are included. In addition, only publications of the Web of

Science document types article and review are considered.

Impact indicators

The Leiden Ranking offers the following indicators of the scientific impact of a university:

• MCS (mean citation score). The average number of citations of the publications of a

university.

• MNCS (mean normalized citation score). The average number of citations of the

publications of a university, normalized for field differences and publication year. An

www.cwts.nl | of 16



Technology Studies,

Leiden University

MNCS value of two for instance means that the publications of a university have been

cited twice above world average.

• PP(top 10%) (proportion of top 10% publications). The proportion of the publications

of a university that, compared with other publications in the same field and in the

same year, belong to the top 10% most frequently cited.

Citations are counted until the end of 2013 in the above indicators. Author self-citations are

excluded. Both the MNCS indicator and the PP(top 10%) indicator correct for differences in

citation practices between scientific fields. 828 fields are distinguished. These fields are

defined at the level of individual publications. Using a computer algorithm, each publication in

the Web of Science database has been assigned to a field based on its citation relations with

other publications. Because the PP(top 10%) indicator is more stable than the MNCS indicator,

the PP(top 10%) indicator is regarded as the most important impact indicator of the Leiden

Ranking.

Collaboration indicators

The following indicators of scientific collaboration are provided in the Leiden Ranking:

• PP(collab) (proportion of interinstitutional collaborative publications). The proportion

of the publications of a university that have been co-authored with one or more other

organizations.

• PP(int collab) (proportion of international collaborative publications). The proportion

of the publications of a university that have been co-authored by two or more

countries.

• PP(UI collab) (proportion of collaborative publications with industry). The proportion of

the publications of a university that have been co-authored with one or more

industrial partners. For more details, see University-Industry Research Connections

2013.

• PP(<100 km) (proportion of short distance collaborative publications). The proportion

of the publications of a university with a geographical collaboration distance of less

than 100 km, where the geographical collaboration distance of a publication equals

the largest geographical distance between two addresses mentioned in the

publication's address list.

• PP(>1000 km) (proportion of long distance collaborative publications). The proportion

of the publications of a university with a geographical collaboration distance of more

than 1000 km.

Core journals

A journal is considered a core journal if it meets the following two conditions:

www.cwts.nl | of 16



Technology Studies,

Leiden University

• The journal publishes in English and has an international scope, as reflected by the

countries in which researchers publishing in the journal and citing to the journal are

located.

• The journal has a sufficiently large number of references to other core journals in the

Web of Science database, indicating that in terms of citation traffic the journal is well-

connected to these other journals. Many journals in the humanities do not meet this

condition. The same applies to trade journals and popular magazines.

In the calculation of the Leiden Ranking indicators, only publications in core journals are

included. The MNCS and PP(top 10%) indicators become significantly more accurate by

excluding publications in non-core journals. About 16% of the publications in the Web of

Science database are excluded because they have appeared in non-core journals. A list of core

and non-core journals is available via the Leiden Ranking website.

Size-dependent vs. size-independent indicators

The Leiden Ranking by default reports size-independent indicators. These indicators provide

average statistics per publication, such as a university's average number of citations per

publication. The advantage of size-independent indicators is that they enable comparisons

between smaller and larger universities. As an alternative to size-independent indicators, the

Leiden Ranking can also report size-dependent indicators, which provide overall statistics of

the publications of a university. An example is the total (rather than the average) number of

citations of the publications of a university. Size-dependent indicators are strongly influenced

by the size of a university (i.e., a university's total publication output) and therefore tend to be

less useful for comparison purposes.

Counting method

The impact indicators included in the Leiden Ranking can be calculated using either a full

counting method or a fractional counting method. The full counting method gives equal

weight to all publications of a university. The fractional counting method gives less weight to

collaborative publications than to non-collaborative ones. For instance, if the address list of a

publication contains five addresses and two of these addresses belong to a particular

university, then the publication has a weight of 2 / 5 = 0.4 in the calculation of the indicators

for this university. The fractional counting method leads to a more proper field normalization

of impact indicators and to fairer comparisons between universities active in different fields.

Fractional counting is therefore regarded as the preferred counting method in the Leiden

Ranking. Collaboration indicators are always calculated using the full counting method.

Stability intervals

A stability interval indicates a range of values of an indicator that are likely to be observed

when the underlying set of publications changes. For instance, the MNCS indicator may be

www.cwts.nl | of 16



Technology Studies,

Leiden University

equal to 1.50 for a particular university, with a stability interval from 1.40 to 1.65. This means

that the true value of the MNCS indicator equals 1.50 for this university, but that changes in

the set of publications of the university may relatively easily lead to MNCS values in the range

from 1.40 to 1.65. The Leiden Ranking employs 95% stability intervals constructed using a

statistical technique known as bootstrapping.

Results Lomonosov Moscow State University

The results for Lomonosov Moscow State University (LMSU) can readily be viewed on-line at

www.leidenranking.com.

In the default setting of the Leiden Ranking Lomonosov Moscow State University ranks 697th

with the following statistics:

P = 2888

PPtop10% = 4,8%

Ranked on the other important normalised impact indicator MNCS, LMSU ranks 725th with the

following statistics:

P = 2888

MNCS = 0,61

Ranked on output (size), LMSU ranks 305th with a P = 2888

Please find in the table below the collaboration statistics for LMSU:

Supporting publications

More information on the Leiden Ranking methodology can be found in a number of

publications by CWTS researchers. An extensive discussion of the Leiden Ranking is offered by

Waltman et al. (2012). This publication relates to the 2011/2012 edition of the Leiden

Ranking. Although not entirely up to date anymore, the publication still provides a lot of

relevant information on the Leiden Ranking. The bottom-up approach taken in the Leiden

Ranking to define scientific fields is described in detail by Waltman and Van Eck (2012). The

methodology adopted in the Leiden Ranking for identifying core journals is outlined by

Waltman and Van Eck (2013a, 2013b).

University PP_collab PP_int collab PP_industry collab PP_short_dist_collab PP_long_dist_collab

LMSU 83,2% 64,6% 3,6% 14,5% 65,6%

www.cwts.nl | of 16



Technology Studies,

Leiden University

• Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E.C.M., Tijssen, R.J.W., Van Eck,

N.J., Van Leeuwen, T.N., Van Raan, A.F.J., Visser, M.S., & Wouters, P. (2012). The

Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of

the American Society for Information Science and Technology, 63(12), 2419-2432.

• Waltman, L., & Van Eck, N.J. (2012). A new methodology for constructing a

publication-level classification system of science. Journal of the American Society for

Information Science and Technology, 63(12), 2378-2392.

• Waltman, L., & Van Eck, N.J. (2013a). Source normalized indicators of citation impact:

An overview of different approaches and an empirical comparison. Scientometrics,

96(3), 699-716.

• Waltman, L., & Van Eck, N.J. (2013b). A systematic empirical comparison of different

approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4),

833–849.

www.cwts.nl | of 16



Technology Studies,

Leiden University

About CWTS B.V. CWTS B.V. (Centre for Science and Technology Studies) is an independent contract

research organisation that provides high-quality research performance and

evaluation studies. Leiden University’s Centre for Science and Technology Studies

forms the core of the company. This prominent institute has been an internationally-

recognised leader in this specialised field for more than twenty years. The strong

working relationship with this institute ensures that CWTS B.V. can always utilise

state-of-the-art bibliometric techniques and indicators.

CWTS B.V.’s reports are based on highly advanced bibliometrics, mapping and network

analyses. These reports provide clients with a well-founded basis for making key strategic

decisions with respect to improving their research performance. This gives them real added

value because it significantly expands and improves their funding opportunities. The reports

are well respected and accepted within the academic world due to CWTS B.V.’s sound scholarly

credentials and professionalism.

The benefits of CWTS B.V.

1. Strong expertise and vast experience in quantitative research evaluation spanning

more than twenty years.

2. World-class data infrastructure providing high-quality data for research evaluation.

3. Standardised processes and protocols for state-of-the-art publication data collection.

4. Leading role in validating and implementing bibliometric mapping and network

analyses thanks to the close affiliation with CWTS Leiden University.

5. Extensive track record with over 500 projects for more than 250 clients in the last

decade.

6. Long-standing relationships with prestigious universities ensuring validity, recognition

and acceptance of CWTS BV.’s reports throughout the academic world.

Track-record of CWTS B.V.

In the past 20 years, CWTS B.V. has built-up ample experience in the field of science &

technology indicators and research performance- and benchmark studies. Part of our staff can

be considered as the founding fathers of this field, shaping it by contributing novel

indicators (.e.g. SNIP, PPtop10%) and methodologies. In the past years, we have performed

over 500 studies. These projects varied in size ranging from a straight-forward researcher

profile to very sophisticated analyses based powerful methodologies such as network

analysis, text mining and visualization techniques. The latter are typically used for research

www.cwts.nl | of 16



Technology Studies,

Leiden University

strategies and funding decisions. Our analyses and reports are used by the following

spectrum of clients: Universities, Academic Hospitals, Research Institutes, Funding Bodies,

Government/European Union, Industry and Network Organizations. A sample of our each

client-network is listed below.

• Leiden University

• University of Amsterdam

• University of Freiburg

• Heidelberg University

• University College London

• Lund University

• ETH

• EPFL

• NWO

• STW

• Department of Health UK

• EPSRC

• VINNOVA

• BMBF

• FWF Austrian Science Fund

• Elsevier

Furthermore, CWTS contributes to the field by ensuring well trained personnel. Twice per year,

the course 'Measuring Science and Research Performance' is organized by us. This course

enables policymakers, librarians, researcher, etc to interpret bibliometric reports, and put the

indicators which are presented in the correct framework.

www.cwts.nl | of 16



Technology Studies,

Leiden University

Questions What does the Leiden Ranking measure (formulate carefully)?

How are the Leiden Ranking impact statistics corrected/normalised for differences in citation

cultures between various disciplines?

What are the advantages of the way CWTS has defined/constructed the 7 broad scientific

disciplines?

Why is there a focus on a core set of journals?

pef cwts leiden ranking info 20140915 -...

Documents