on standards in science metrics and classifications

On Standards in Science Metrics and Classifications

Henk F. MoedElsevier, Amsterdam, Netherlands

Workshop on Science Metrics, Classifications, and Mapping Standards, August 11-12, 2011

School of Library and Information Science, Indiana University, Bloomington, USA

Workshop on "Bibliometric Standards", River Forest, Illinois , USA (11/06/1995)

Subject TopicMetrics (mathematical- statistical aspects)

Macro indicators; journal impact factors; relative citation rates;

Classifications Document type; subject classifications; years; author and institutional names

Terminology, concepts (theoretical aspects)

Quality; impact; visibility; productivity; activity index;

Research methodology Models; normative principles; ‘Good’ analysis practices

Workshop on "Bibliometric Standards", River Forest, Illinois , USA (11/06/1995)

1. Standards can not be set by committee but must evolve through an on-going debate

2. Perhaps, the Scientometric community needs a refereed forum more dedicated to methodological issues

3. Often technical controversies reflect theoretical issues

Standardisation is (also) ........

• Providing frameworks in which various approaches can be positioned and compared with one another

• Example 1: Research assessment indicators

• Example 2: Subject classifications

Example 1: The Multi-Dimensional Research

Assessment Matrix

Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi‐dimensional Research Assessment Matrix (Part)

Unit of assessment

Purpose Output dimensions

Bibliometric indicators

Other indicators

Individual Allocate resources

Research productivity

Publications Peer review

Research group

Improve performance

Quality, scholarly impact

Journal citation impact

Patents, licences, spin offs

Department Increase multi-discipl. research

Innovation and social benefit

Actual citation impact

Invitations for conferences

Institution Increase regional engagement

Sustainabi-lity & Scale

Internat. co-authorship

External research income

Research field

Promotion, hiring

Research infrastruct.

citation ‘prestige’

PhD com-pletion rates


Unit of assessment



Other indicators




Research group

Improve performance












Research field

Promotion, hiring




Read column-

wise


Unit of assessment



Other indicators




Research group

Improve performance












Research field

Promotion, hiring





Unit of assessment



Other indicators




Research group

Improve performance












Research field

Promotion& hiring




MD-RAM: Example 1

Individual

Hiring/promotion

Productivity &

impact

Individual

Hiring/promotion

Productivity &

impactPhD date, place,

supervisor;

Invitations for

conferences

PhD date, place,

supervisor;

Invitations for

conferences

Publications in

international jrnls;

Actual citation

impact

Publications in


Actual citation

impact


Unit of assessment



Other indicators




Research group

Improve performance












Research field

Promotion, hiring




MD-RAM: Example 2

Research group

Allocate resources

Res. productivity &

impact

Research group

Allocate resources

Res. productivity &

impactCompetitive

research income;

Ratio research

active/total staff

Competitive

research income;

Ratio research

active/total staff

Publications in


Actual citation

impact

Publications in


Actual citation

impact

Positioning bibliometric products

Unit of assessment

Purpose Aspect Product

Research group

Improve performance;Allocate resources

Productivity/ impact A

Department; institution

Allocate resources; Stimulate multi-disciplinary research

Research infrastructure and potential

B

Institution Stimulate multi-disciplinary research;Improve performance;

Social benefits; sustainability;Prod./impact

C

Group, Dept, Institution

Improve performance& regional engagement;Allocate resources;

Productivity/ impact; social impact; infrastr.

D

Example 2: Journal subject classification

systems

Journal subject classifications based on .....

• Semantic words in journal titles• Journal-to-journal citations• Journal co-citation analysis• Journal bibliographical coupling• Journal co-usage analysis• Semantic words from article titles and abstracts• Thesaurus subject terms in disciplinary databases

Three citation relationships

A

B

A

C

A

B

B

C

A cites B

B is a cited reference in A

B and C are co-cited by A

A and B are bibliographically coupled via C

Analogy Model

CITATIONS USAGE

(Collections of) publishing authors

(Collections of) users

Citing a document Retrieving the full text of a document

Article User session

Author’s institutional affiliation

User’s account name

Number of times cited Number of times retrieved as full text

Forming subject categories from journal titles

Journal title Subject category

Journal of Modern Optics Optics

Journal of Orthopaedic Science

Orthopedics

Legal and Criminological Psychology

Law; Criminology; Psychology

Journal of Social Work Social work

Subject categories: Examples

“Linguistics” Nr

Jrnls“Agriculture” Nr

JrnlsBI- & MULTILINGUALISM 12 AGRICULTURAL ECONOMICS 18

DIALECTOLOGY 5 AGRICULTURE 138

DISCOURSE STUDIES 10 AGRONOMY 26

LANGUAGE 226 AQUACULTURE 12

LINGUISTICS 217 CROP SCIENCE 18

LINGUISTICS, APPLIED 17 DAIRY SCIENCE 10

PHILOLOGY 38 FISHERIES 23

PHONETICS 13 FOOD S&T 147

PRAGMATICS 7 HORTICULTURE 19

SEMANTICS 7 PEST SCIENCE & PESTICIDE 14

SEMIOTICS 3 POULTRY SCIENCE 9

Proposal

• Use the classification based on journal title words as a benchmark....

• ... as it represents a classification that everyone can grasp intuitively....

• .... and illustrate the special features and added value of the new approach

Thank you for your attention

What is measured by citations and references? - 1

Principal author(s)

References conceived as Citations measure

Garfield Salton

Descriptors of document content

Garfield Manifestations of scholarly information flows

Utility (quantity of information use)

Small Garfield

Elements in a symbol making process

Highly cited items as concept symbols

Merton Registrations of intellectual property and peer recognition of the knowledge clain

Intellectual influence

Cole & Cole Zuckerman

Socially defined quality

Gilbert Persuasion tools Authoritativeness

Zuckerman Referencing motives and their consequences are analytically distinct

Citations are proxies of more direct measurements


Principal author(s)


Martin & Irvine

Manifestations of peer recognition, information flows and political pressures

Impact (actual influence)

Cozzens References are at the intersect of the reward, rhetorical and communication system but rhetorics comes first

Recognition, persuasiveness and awareness each generate a certain portion of variation in citation counts

White Inter-textual relationships mainly reflect straightforward acknowledgement of related documents

Co-citation maps provide an aerial view measuring a historical consensus as to impor-tant authors and works


Principal author(s)


van Raan References are partly particularistic but in large ensembles biases cancel out

The upper part of the distribution of a ‘thermodynamic’ ensemble of many citers measures ‘top’ research

Wouters The reference is the product of the scientist

The citation is the product of the indexer. Validity of citations cannot be grounded merely in reference behaviour

on standards in science metrics and classifications

Documents