November 17, 2008

John MacCollEuropean Director, RLG

Partnership, OCLC ResearchPresentation to RLG UK

PartnershipUniversity of Leeds, 18

September 2009

Libraries in Research Assessment: Implications from a Study of Five Countries

Research questionsResearch questions

• What are the key characteristics of ‘high-intervention’ and ‘low-intervention’ research assessment regimes?

• What are the advantages and disadvantages of the systems of research assessment in the countries studied?

• What are the main activities undertaken by universities in data-gathering in support of research assessment?

• What are the roles which libraries play in data-gathering within their institutions in support of research assessment?

• To what extent are libraries developing systems which achieve their institutional commitment to research assessment but also go further, taking account of possible future changing circumstances, or of the wider need for institutional self-assessment?

• In regimes in which institutional research self-assessment has a significant role, what part do libraries play?

• What roles do libraries play in relation to bibliometrics?• What evidence is there of library management of access to non-standard research

outputs?• Where might universities – and libraries – rationalise their efforts through collaborative

activity, within countries and internationally? • What evidence of good or best practice can be identified within research libraries in

support of national or institutional agendas?

?

Number of HEIs per country (Webometrics)Number of HEIs per country (Webometrics)

UK – 236Netherlands – 154Ireland – 50Denmark – 137Australia – 82(USA – 3328)

Number of universities in Top 200Number of universities in Top 200

THE Top 200 Universities 2004 2008

UK 30 29

Netherlands 5 11

Ireland 1 2

Denmark 3 3

Australia 14 9

USA 62 58

Number of universities in Top 50Number of universities in Top 50

THE Top 50 Universities 2004 2008

UK 8 8

Netherlands 0 0

Ireland 0 1

Denmark 0 1

Australia 6 6

USA 20 20

Gross domestic expenditure on R&D as % of GDP (OECD)Gross domestic expenditure on R&D as % of GDP (OECD)

UK expenditure on QR - £1.6b p.a.?

Transitional timeTransitional time

Regime characteristicsRegime characteristics

Link to funding UK

Self-assessment-based NL

Unit-led NL

Includes publication venue rankings DK, AU

Bibliometrics-led DK

National research administration system NL

?

Publication venue ranking systemsPublication venue ranking systems

• Danish Bibliometric Research Indicator (BRI) System• Modelled on Norwegian: points allocated for publication

in different ‘publication channels’• Eg 5 points for a monograph in Level 2; 8 points for a

monograph in Level 1• But: all Danish publishers were classed as Level 1!

Same tendency within Australian journals.• A Level 1 journal article attracts 3 points; a Level 2 only 1• A Level 1 anthology attracts 2 points; a Level 2 only 0.5• PhDs – 5 points• Has advantage of not being tied to ISI or Scopus• So can much better accommodate humanities• But libraries feel obliged to buy at least all of the top-ranked

journals and publications

Library roles in bibliometric regimesLibrary roles in bibliometric regimes

Researcher fractionalisationResearcher fractionalisation

NetherlandsNetherlands

• Even though Dutch system is ‘non-competitive’, administrators complain about the excessive burden. Mainly because Dutch system is so fine-grained (down to research group level).

• So they are trying to make the internal assessments more light-touch• Some complain that lack of link to funding means that top research talent is not

attracted to the Netherlands• Government gradually building more research themes to encourage

specialisation (in competitive grants awards), so inhibiting academic freedom• Some faculties ‘bribe’ academics to publish in high-impact journals! So the

absence of a link to funding on a national level does not remove game-playing. Indeed, it may lead to the other side of the dual support regime behaving even more competitively.

• ‘Tenure is achievable relatively easily, so there is a need for a stable funding regime to support the existing structure of universities.’ Even if academic jobs are safe, who wants to work in a department with no funding?

• Dutch university system quite decentralised. Many faculty libraries are still the main libraries, and the university library therefore takes little role in research assessment.

Choose: stability, or excellence?Choose: stability, or excellence?

IrelandIreland

• Irony in Ireland is that it has a quite cohesive, well-integrated infrastructure for research management, but no national system of assessment.

?

The accidental success of the IRThe accidental success of the IR

AustraliaAustralia

• ‘There is a general sense of Australia’s distance from world research centres’

• So – partly about national status. Or – perhaps more, about researchers’ fear of being overlooked because of being in Australia

• Creative artists are happy at recognition. And in Denmark (journals) and UK (peer review) too, there is a sense that a national system recognises them, and therefore that the status quo – even without an assessment system – is prejudiced against them

Negative impact on scholarshipNegative impact on scholarship

The ‘central library’The ‘central library’

Could we say this about the UK?Could we say this about the UK?

Fascinating peer-review idea! What does it mean?Fascinating peer-review idea! What does it mean?

?

Reluctance to involve librariesReluctance to involve libraries

Library leaders must engage institutionallyLibrary leaders must engage institutionally

Disciplinary divisionsDisciplinary divisions

Discipline sensitivityDiscipline sensitivity

Disciplinary differences (a RIM theme)Disciplinary differences (a RIM theme)

ValuesValues

?

Key findingsKey findings

• Libraries with repositories have had most involvement in their institutions’ assessment submissions

• Australia furthest ahead there: largely because of government funding for repositories

• The freedom to research what you want is fast disappearing everywhere – and in countries like the UK and Australia, it has been reducing for years. However, governments don’t want to lose it altogether. What is the right balance?

• In most countries, in most institutions, the library senior management role is operational rather than strategic (UK, Australia, Denmark)

• In the Netherlands, ironically, libraries are closely involved at the strategic level – but not in relation to research assessment

Directions for research libraries?Directions for research libraries?

1. Help make the existing processes more efficient. It’s in the library’s interest for its institution to do well in research assessment exercises. Individual institutional libraries want to learn about best practice in order to create and embed new workflows as cost-effectively as possible.

2. Aim to provide more data to the mix that might help the system to become fairer. Are there ways in which the library community can arrive at its own metrics (eg libcitations; MESUR)?

3. Develop neutral role into a strategic asset. The library should be a source of interdisciplinary, international advice to its institution, and to national and international bodies, as we work through a transition towards fairer assessment methods in an increasingly competitive environment.

?

Thank YouThank YouJohn MacCollmaccollj@oclc.orgOCLC Research

St Andrews University Library (Gareth JM Saunders)