new developments in electronic publishing and bibliometrics

New developments in electronic publishing and bibliometrics

Henk F. Moed

CWTS, Leiden University, Netherlands

Elsevier, Amsterdam, Netherlands

Contents

1 Beyond journal impact factor and H-index

2 How useful are university rankings?

3 Why combine indicators and peer review?

4 Can indicators be manipulated?

5 Does Open Access lead to higher impact?

6 More downloads more citations?

Contents



3 Why combine indicators and peer review




Journal impact measures are no good predictors of an individual paper’s

actual citation impact

Partly based on International Mathematical Union’s Report ‘Citation Statistics’ (2008)

Length boys vs. adults

0

5

10

15

20

25

30

35

0 15 35 55 75 95 115 135 155 175 195 215Length (cm)

% P

erso

ns

Boys (Meanlength=95 cm)Players (Meanlength=185 cm)

Citations to P-AMS vs. T-AMS

0

10

20

30

40

50

60

70

80

0 1 2 3 4 5 6 7 8Nr Cites

% P

aper

s

PAMS (JIF=0.43)

TAMS (JIF=0.85)

Normal vs. skewed distributions

0

5

10

15

20

25

30

35

0 15 35 55 75 95 115 135 155 175 195 215Length (cm)

% P

erso

ns

Boys (Meanlength=95 cm)Players (Meanlength=185 cm)

0

10

20

30

40

50

60

70

80

0 1 2 3 4 5 6 7 8Nr Cites

% P

aper

s

PAMS (JIF=0.43)

TAMS (JIF=0.85)

What is the probability that .......

a randomly selected boy is at least as tall as a randomly selected player?

Almost zero

62 %a randomly selected PAMS paper is cited at least as often as a randomly selected TAMS paper?

Probabilities still substantial for high JIF journals

‘Free’ citations

Thomson/JCR Journal Impact Factor

Citations to all docs

# Citable docs

Citable vs. non-citable docs

Citable documents “non-citable” documents

Articles Letters

Reviews Editorials

Discussion papers

The problem of “free” citations - 1

Cites

Docs + + + + +

+ + + + +

The problem of “free” citations - 2

Cites

Docs + +

+ + + + +

“Free” Citations

All three publication lists have a Hirsch Index of 5

30 P110 P2 8 P3 6 P4 5 P5 1 P6 0 P7

30 P110 P2 8 P3 6 P4 5 P5 4 P6 4 P7 4 P8 4 P9

100 P1 70 P2 8 P3 6 P4 5 P5 1 P6 0 P7

H=? H=? H=?5 5 5

123456789

Author 2Author 1 Author 3

Different bibliometric distributions

have the same H-Index

Indicators are becoming more informative

Feature Example

Embody ways to put numbers in context

Field-normalized citation measures

Take into account “who” is citing

Citations weighted with impact of citing source

Take into account relationship citing-cited author

Impact outside the own niche; multi-disciplinarity; bridging paradigms

Contents







University ranking positions are primarily marketing tools,

not research management tools

Research assessment methodologies must take into account… [EC AUBR Expert Group]

1. Inclusive definition of research / output

2. Different types of research and its impacts

3. Differences among research fields

4. Type and mission of institution

5. Proper units of assessment

6. Policy context, purpose and user needs

7. The European dimension

8. Need to be valid, fair and practically feasible

Types of outputs (SSH)

Impacts Publication/text Non-publication

Scientific-scholarly

Journal paper; book chapter; monograph

Research data file; video of experiment

Educational Teaching course book; syllabus

Skilled researchers

Economic Patent Product; process; device; design; image

Cultural Newspaper article; Interviews; events; Performances; exhibits

Top-down institutional analysis

Select an institution’s papers using author addresses (incl. verification)

Categorize articles intoresearch fields

Calculate indicators

Compare with benchmarks

Bottom-up institutional analysis (CWTS)

Compile a list of researchers

Compile a list of publications per

researcher (incl. verification)

Aggregate researchers into groups, departments, fields, etc.

Calculate indicators;

compare with benchmarks

Secondary analyses of ‘ranking’ data are informative

Contents







Case study: A national Research Council

• Proposals evaluated by committees covering a discipline

• Reports from external referees

• Committee members can be applicants

Affinity applicants – Committee

0 Applicants are/were not member of any Committee

1 Co-applicant is/was member of a Committee, but not of the one evaluating

2 First applicant is/was member of a Committee, but not of the one evaluating

3 Co-applicant is member of the Committee(s) evaluating the proposal

4 First applicant is member of the Committee(s) evaluating the proposal

For 15 % of applications an applicant is a member of the evaluating Committee (Affinity=3, 4)

0

10

20

30

40

50

60

70

% A

PP

LIC

AT

ION

S

AFFINITY APPLICANTS-COMMITTEE

Projects 63.2 10.2 11.5 5.9 9.1

0 1 2 3 4

Probability to be granted increases with increasing affinity applicants-Committee

30

40

50

60

70

80

% G

RA

NT

ED

AP

PL

ICA

TO

NS

AFFINITY APPLICANTS-COMMITTEE

Projects 37.0 46.9 60.1 62.6 74.0

0 1 2 3 4

Logistic regression analysis: Affinity Applicant-Committee has a significant effect

upon the probability to be granted

MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 Publ Impact applicant 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 Affinity applicant-Committee 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638

The future of research assessment exercises lies in the intelligent

combination of metrics and peer review

Contents







Effects of editorial self-citations upon journal impact factors

[Reedijk & Moed, J. Doc., 2008]

• Editorial self-citations: A journal editor cites in his editorials papers published in his own journal

• Focus on ‘consequences’ rather than ‘motives’

Case: ISI/JCR Impact Factor of a Gerontology Journal (published in the journal itself)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2000 2001 2002 2003 2004

IMPACT FACTOR YEAR

CIT

ES

PE

R 'C

ITA

BL

E' D

OC

Decomposition of the IF of a Gerontology journal

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2000 2001 2002 2003 2004

IMPACT FACTOR YEAR

CIT

ES

PE

R 'C

ITA

BL

E' D

OC

Editorial self citations

Free citations

One can identify and correct for the following types of

strategic editorial behavior

• Publish ‘non-citable’ items• Publish more reviews• Publish ‘top’ papers in January• Publish ‘topical’ papers (with high short term

impact) • Cite your journal in your own editorials• Excessive journal self-citing

Contents







Deposited in OA rep

(o)

Not depositedin OA rep.

(no)

Journal articles

Average Impact (CPPo)

Average Impact

(CPPno)

?><=

Three effects [Kurtz et al., 2005]

Open Access ArXiv increases access

Early View Articles appear earlier in ArXiv than in Publisher’s Website

Self-Selection (Quality bias)

Better authors use ArXiv

Authors deposit their best papers in arXiv

0

96 97 98 99 00 01 02 03 04 05

Publication Years

OA

Imp

ac

t a

dv

an

tag

e

Quality Bias: Better authors use ArXiv

EarlyView Effec

t

ArXiv papers appear earlier

ArXiv, Cond Mat Phys[Moed, JASIST 2007]

100

Age distribution of citations to Arxiv and non-ArXiv papers

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84

Months after Publication Date

Cit

es

pe

r P

ap

er

in ArXiv-CM

Not in ArXiv-CM

3 per. Mov. Avg. (inArXiv-CM)3 per. Mov. Avg. (Notin ArXiv-CM)

Move curve by 6 months to the right

Early view effect: Citations to papers deposited in ArXiv-CM start about 6 months earlier

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84

Months after Publication Date

Cit

es

pe

r P

ap

er

in ArXiv 6 monthstranslated

Not in ArXiv-CM

3 per. Mov. Avg. (inArXiv 6 monthstranslated)3 per. Mov. Avg. (Notin ArXiv-CM)

More research questions

• Early view effect also visible in a non-OA environment?

• Citation impact measured in biased sample?

Contents







Downloads vs. Citations

More downloads more citations

or

More citations more downloads?

Relation between citations and internet hits for 153 papers in volume 318 of the BMJ (1999)

Figure 1 from: Perneger, TV. BMJ. 2004, 329 (7465): 546–547. Relation between online “hit counts” and subsequent citations: prospective study of research papers in the BMJ

Analogy Model

Formal use (e.g., SCI) Informal use (e.g.,SD)

(Collections of) publishing authors

(Collections of) users

Citing a document Retrieving the full text of a document

Article User session

Author’s institutional affiliation

User’s account name

Number of times cited Number of times retrieved as full text

Age distribution downloads vs. citations[Tetrahedron Lett, ScienceDirect; Moed, JASIST, 2005]

0

4

8

12

16

20

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

AGE (MONTHS)

%

SD USESCITATIONSDownloads

Citations%

Age (months)

Ageing downloads vs. citations: Two factor vs. single factor model

0.01

0.1

1

10

100

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88

Age (months)

Us

es

DownloadsObserved

DownloadsComputed

DownloadsSingularPoints

CitationsObserved

CitationsComputed

%

Age (months)

Downloads

Citations

Citations lead to downloads[Moed, J. Am Soc Inf Sci Techn, 2005]

1

10

100

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

AGE PAPER A (MONTHS)

DO

WN

LO

AD

S

A

B (B cites A)

C (C cites A and B)

PaperA published

Paper B published;it cites A

Download of A increases

Paper C published;it cites A and B

Rank correlations between downloads and citations

Variables Spearman

R

Downloads vs. citations 0.22

‘Later’ (>3 months) downloads vs. citations

0.33

‘Initial’ (<3 months) downloads vs. citations

0.11

Conclusions

• Positive correlation between downloads and citations partly due to the effect of citations upon downloads

• ‘Initial’ downloads and citations hardly correlate, and relate to distinct phases in processing relevant scientific information

• ‘Later’ downloads and citations show statistically similar properties of ageing and frequency distribution

Downloads and citations relate to distinct phases in

scientific information processing

.... but (many) more cases must be studied

Thank you for your attention!

References• Eysenbach, G. (2006). Citation Advantage of Open Access Articles. PLOS Biology, 4, 692–698.• Garfield, E. (1972). Citation Analysis as a tool in journal evaluation. Science, 178, 471–479.• Garfield, E. (1979). Citation Indexing. New York: Wiley.• Yassine Gargouri, Chawki Hajjem, Vincent Lariviere, Yves Gingras, Les Carr, Tim Brody, Stevan Harnad

(2010). Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research arXiv:1001.0361v2 [cs.CY]

• Harnad, S., & Brody, T. (2004). Comparing the impact of open access (OA) vs. Non-OA articles in the same journals. D-Lib Magazine, 10, Nr 6.

• Kurtz, M.J., Eichhorn, G., Accomazzi, A., Grant, C., Demleitner, M., Henneken, E., & Murray, S.S. (2005). The effect of use and access on citations. Information Processing & Management, 41, 1395–1402.

• Moed, H.F. (2005). Citation Analysis in Research Evaluation. Dordrecht (Netherlands): Springer. ISBN 1-4020-3713-9, 346 pp.

• Moed, H.F., Glänzel, W., and Schmoch, U. (2004) (eds.). Handbook of Quantitative Science and Technology Research. The Use of Publication and Patent Statistics in Studies of S&T Systems. Dordrecht (the Netherlands): Kluwer Academic Publishers, 800 pp.

• Moed, H.F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology 56, 1088-1097.

• Moed, H.F. (2007). The effect of “Open Access” upon citation impact: An analysis of ArXiv’s Condensed Matter Section. Journal of the American Society for Information Science and Technology 58, 2047-2054.

• Moed, H.F. (2009). New developments in the user of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis (Warszawa) 17, 13-18.

• Reedijk, J., Moed, H.F. (2008). Is the impact of journal impact factors decreasing? Journal of Documentation 64, 183-192.

http://arxiv.org/find/cs/1/au:+Gargouri_Y/0/1/0/all/0/1

http://arxiv.org/find/cs/1/au:+Hajjem_C/0/1/0/all/0/1

http://arxiv.org/find/cs/1/au:+Lariviere_V/0/1/0/all/0/1

http://arxiv.org/find/cs/1/au:+Gingras_Y/0/1/0/all/0/1

http://arxiv.org/find/cs/1/au:+Carr_L/0/1/0/all/0/1

http://arxiv.org/find/cs/1/au:+Brody_T/0/1/0/all/0/1

http://arxiv.org/find/cs/1/au:+Harnad_S/0/1/0/all/0/1

http://arxiv.org/abs/1001.0361v2

ISI/JCR Journal Impact Factor of journal J for year T

Citations in year T to items published in J in years T-1 and T-2

÷Number of “citable” items published in J in

years T-1 and T-2

new developments in electronic publishing and bibliometrics

Documents

higher impact

length boys

journal impact measures

tams jif

university rankings

pams jif

peer review4can indicators

hindex2how useful