methodological aspects of a new generation of relational charts … · 2009. 7. 3. · cluster 1...

Structure Introduction Methods Normalised citation indicators Applications Conclusions

methodological aspects of a new generationof relational charts for comparative

assessment of citation impact

wolfgang glänzel1,2 & koenraad debackere2

1katholieke universiteit leuven, steunpunt o&o indicatoren, dept. msi, leuven(belgium)

2hungarian academy of sciences, issru, budapest (hungary)

leuven 2008

Glänzel & Debackere New generation of relational charts

1 Structure of presentation

2 IntroductionObjectives – Problems – SolutionsData sources and data processing

3 MethodsProfile analysis of research institutionsField-depth for subject normalised indicatorsChoosing the citation window

4 Normalised citation indicatorsMeasuring high impactThe indicator toolbox

5 Applications

6 Conclusions

Objectives – Problems – Solutions

Objectives

Main objective of the study

Adjusting and refining bibliometric indicators for the use at themeso level

Challenges

institute/group-specific specialisation (different profilesmight cause biases)

diversification (might result in deviations from theassumed field standard)

Objectives

Main objective of the study

Adjusting and refining bibliometric indicators for the use at themeso level

Challenges

institute/group-specific specialisation (different profilesmight cause biases)

diversification (might result in deviations from theassumed field standard)

Problems

Avoid “comparing apples with pears” (e.g. medical schooland business school)

Development of methodology for intra- and inter-classcomparison whenever possible

Application of consistent subject normalisation andstandardisation of citation windows.

We use the methods developed in Budapest and Leuven andproceed along the following logical scheme.

Solutions

1 Classification of institutions by their publication profiles.We use multidisciplinary and specialised clusters to checknormalisation for adequacy and robustness.

2 Validation of the choice of the standard citation window.

3 Measuring the high-end of research by means ofsubject-normalised indicators.

Solutions

Data sources and data processing

Data sources

Bibliographic data were taken from theWeb of Sciencedatabase of Thomson-Scientific (Philadelphia, PA, USA), andprocessed to bibliometric indicators.

Only papers of the document types article, letter, note andreview indexed in the 1999-2001 volumes have been selected.

Papers were assigned to fifteen major fields in the sciences,social sciences and humanities according to theLeuven/Budapest scheme.

Profile analysis of research institutions

Profile analysis

The eight clusters resulting from the second optimum solution

Cluster Code

Cluster 1 (Biology) BIOCluster 2 (Agriculture) AGRCluster 3 (Multidisciplinary) MDSCluster 4 (Geo & Space Science) GSSCluster 5 (Technical & Natural Sciences) TNSCluster 6 (Chemistry) CHECluster 7 (General & Research Medicine) GRMCluster 8 (Specialised Medicine) SPM

Profile analysis

Example for the deviating field structure of different clusters:

Profile analysis

Different (sub)field standards of citation impact

Field-depth for subject normalised indicators

SOOI subfields

Plot of NMCR based on subfields vs. ISI Subject Categories for 676European institutions

SOOI major fields

Plot of NMCR based on major fields vs. subfields for 676 Europeaninstitutions

Multidisciplinary vs. specialised clusters

NMCR based on subfields (left) and major fields (right) vs. ISI SubjectCategories

Choosing the citation window

Citation window

Plot of NMCR based on 5-year citation window vs. 3-year window for 676European institutions

Measuring high impact

Characteristic Scores and Scales – Definition

Characteristic scores and scales provide subject-sensitive,self-adjusting thresholds for citation distributions.

This method can be summarised as iteratively truncatingsamples at their mean value and recalculating the meanof the truncated sample until the procedure is stopped orthe sample is empty.

⇒ bk = E(X |X > bk−1) with b0 := 0.

Characteristic Scores and Scales – Properties

For Paretian distributions we have

bk ≈α

α−1· bk−1 + b1 ,

where α is the parameter of the underlying Paretodistribution.

The choice of α= 2 results in b2 = 3 · b1 and b3 = 7 · b1 forthe thresholds k = 2 and k = 3, respectively.

The citation impact of each individual paper is comparedwith the three-fold or seven-fold of the correspondingsubject standard, according as k = 2 or k = 3 is chosen.

bk ≈α

α−1· bk−1 + b1 ,

bk ≈α

α−1· bk−1 + b1 ,

The indicator toolbox

Observed citation impact

Observed citation ratesMean Observed Citation Rate (MOCR)

MOCR =

n∑i=1

ci

n

Expected citation impact

Expected citation ratesMean Expected Observed Citation Rate (MECR)

MECR =

n∑i=1

xi

n

Field Expected Citation Rate (FECR)

FECR =

n∑i=1

fi

n

Expected citation impact

Expected citation ratesMean Expected Observed Citation Rate (MECR)

MECR =

n∑i=1

xi

n

Field Expected Citation Rate (FECR)

FECR =

n∑i=1

fi

n

Relative citation impact

Relative citation rates“Publication Strategy” (MECR/FECR)

Normalised Mean Citation Rate (NMCR)

NMCR =MOCRFECR

=

n∑i=1

ci

n∑i=1

fi

Relative Citation Rate (RCR)

RCR =MOCRMECR

=

n∑i=1

ci

n∑i=1

xi

NMCR =MOCRFECR

=

n∑i=1

ci

n∑i=1

fi

RCR =MOCRMECR

=

n∑i=1

ci

n∑i=1

xi

NMCR =MOCRFECR

=

n∑i=1

ci

n∑i=1

fi

RCR =MOCRMECR

=

n∑i=1

ci

n∑i=1

xi

High-impact indicators

High-impact Activity is the ratio of the unit’s share ofhighly cited papers in all papers and the correspondingworld standard.

High-impact Atractivity is the ratio of the unit’s share ofcitations attracted by its highly cited papers in all citationsreceived by the unit under study and the correspondingworld standard.

Indicators based on characteristic scores and scales aresub-field normalised by definition, and can therefore be usedas direct supple-ment to the previous set of relative indicators.

Chart prototype

Prototype of a subject-normalised relational chart plottingNMCR vs. MECR/FECR

The effect of subject normalisation

Measuring high-impact activity and attractivity

Relational chart of high-impact activity and attractivity for24 selected universities

Conclusions and discussion

Normalised indicators are highly necessary if one doesnot want to compare “apples” and “pears”.

The level of 60 subfields proved an acceptable choice andprovided stable and consistent results.

The normalised indicators can be used for both intra- andinter-cluster comparative analysis as well as for domainstudies at this level of aggregation.

In addition to the relative citation indicators, the high-endof institutional research is best reflected by highly citedpapers. Two thresholds are suggested, depending on theunderlying publication output.

The study has also shown that the use of a three-yearcitation window suffices for building both relative andhigh-impact citation indicators.

In addition to the relative citation indicators, the high-endof institutional research is best reflected by highly citedpapers. Two thresholds are suggested, depending on theunderlying publication output.

The study has also shown that the use of a three-yearcitation window suffices for building both relative andhigh-impact citation indicators.

Thank you very much

Structure of presentationIntroductionObjectives -- Problems -- SolutionsData sources and data processing

MethodsProfile analysis of research institutionsField-depth for subject normalised indicatorsChoosing the citation window

Normalised citation indicatorsMeasuring high impactThe indicator toolbox

ApplicationsConclusions

methodological aspects of a new generation of relational charts … · 2009. 7. 3. · cluster 1...

Documents