The Usage Factor

How can we enhance the relevance of usage as

an indicator of relative value? Richard Gedye, Sales Director, Oxford Journals

Measure for Measure – or Much Ado about NothingUKSG Seminar

14 June 2007

5 years ago, COUNTER set out to make usage statistics…

Consistent

Credible

Compatible

How successful has COUNTER been?

Standard format usage reports now adopted by over 60 publishers and hosts

Formal auditing process starts in 2007

Yes, but………

The challenge…….

COUNTER statistics provide a reliable comparison of amount of use between journals

But they don’t provide a meaningful usage-based measure of relative quality or value

All other things being equal, a journal publishing 2000 articles a year will generate significantly more downloads than one publishing 50.

If I was guided solely by usage statistics, I would cancel all my subscriptions to humanities journals, which tend to publish far fewer issues per year than the monster science titles……

Terry Bucknell, Electronic Resources Manager at Liverpool University

Addressing the challenge…….

ISI's Impact Factor compensates for the fact that larger journals will tend to be cited more than smaller ones

Can we do something similar for usage?

In other words, should we seek to develop a “Usage Factor” as an additional measure of journal quality/value?

For example…..

Usage Factor =

Total usage over period ‘x’ of articles published during period ‘y’

Total articles published during period ‘y’

Basis for Usage Factor

A possible basis for the calculation of Usage Factor could be:-

COUNTER Journal Report 1: Number of Successful Full-text Article Requests by Month and Journal

Reality check……

Is there a demand for it among:- Publishers Authors Librarians

What are the practical issues that would need to be addressed?

How much would it cost to develop and maintain?

Some initial evidence……

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

Some initial evidence……

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

Some initial evidence……

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

“Note that the question explored the measurement of utility rather than `quality’ but this is nonetheless a surprising finding and it may indicate that download metrics would have considerable credibility amongst the author community.

“Alternatives to the traditional impact factor, based on article downloads and modeled using the same time windows as are used to construct impact factors might offer a very interesting and worthwhile direction for future research and development: they would certainly be of great appeal to librarians and many publishers.”

The Research…….

UK Serials Group and COUNTER have commissioned research which examines:- the various ways in which journal quality is

currently assessed the degree to which any additional usage-based

metrics might prove valuable to each stakeholder community

practical ways in which such metrics might be derived and constructed to provide the maximum utility for all, within defined resource constraints

Research….. Phase 1 October 2006 – January

2007 COUNTER Director Peter Shepherd carried

out a series of in-depth interviews with:- 7 authors 9 librarians 13 publishers

Phase 2 March 2007 Broader web based survey of:-

155 librarians 1400 academic authors

May 2007 Final report delivered to UKSG

Phase 1 Headline Results - Users

Would Journal Usage Factors be helpful to you in assessing the value, status and relevance of a journal?

Librarians – YES 100% Authors - YES 100%

Phase 1 Perceptions

Usage Factor Advantages Issues to address Who should do the calculations What are the optimal variables to

measure

Phase 1 Usage factor advantages

A useful counterweight to Impact Factors

Especially helpful for journals and fields not covered by ISI

Especially helpful for journals with high undergraduate or practitioner use

Especially helpful for journals publishing relatively few articles

Data available potentially sooner than with Impact Factors

Phase 1 Usage factor advantages

“Authors select journals that will give their articles prestige and reach. Impact Factor is a widely used surrogate for the former, while perceived circulation and readership reflect the latter. But usage is becoming more important as a measure of reach”

Carol Tenopir

“Many of the publications in which I publish and in which I would like to publish do not have IFs and the current system almost requires serious authors to publish in journals that have IFs.”

Author

Phase 1 Usage factor advantages

“UF would definitely benefit publishers.

First, it would be a useful balance for IFs, which are not always accurate reflections of the value of a journal.

Second, UF is a simple metric that would make usage more understandable to editors and authors as a measure of value. There is currently much talk of usage and a lot of data, which the non-librarians find confusing.”

Publisher

Phase 1 Issues to address

Only COUNTER statistics should be used and not all publishers are yet COUNTER compliant

COUNTER data needs first to be made more robust

Would another global measure, such as usage half-life per journal or per discipline, not be of greater value?

“Our journals do well in the IF rankings and we fear that introducing usage into the picture would adversely affect our journal rankings. Ranking by usage will tend to favour lower quality journals. We are at the high end of the market. We have ‘invested a lot’ in IFs as a measure of quality.”

Phase 1 Issues to address “There is a danger that the development of UF will

further stimulate publishers to inflate their usage by every means.”

How would print usage, still significant for many journals, be taken into account?

“We remain to be convinced that it would be a meaningful measure. Elsevier, for example, has a huge sales force devoted to selling online licenses and increasing usage of their journals. Societies do not have such resources and feel that IF is a more level playing field on which to judge journal quality.”

To evaluate a title in total you will need to integrate the information from all hosting sites. In addition to the publisher site this could include aggregators, PubMed Central, locally loading libraries, etc.

Phase 1 Who should do the calculations?

“It would be difficult for librarians to consolidate global usage statistics. Could not publishers do this?”

Librarian

“As publishers will end up paying for this they should have some control over the process”

Publisher

Phase 1 Who should do the calculations?

“I am against having a large role for any central organization, as it is likely to be a considerable additional cost to the industry.

“The role of such an organization should be restricted to compiling the Usage Factors, which have already been calculated by the publishers and independently audited. This would minimise the cost.”

Librarian

Phase 1 Who should do the calculations?

Publishers are, on the whole, unwilling to provide their usage data to a third party for consolidation calculation of Usage Factors.

The majority appear to be willing to calculate UFs for their own journals and to have this process audited.

This is generally perceived as a natural extension of the work already being done for COUNTER.

While it may have implications for systems, these are not seen as being problematic.

Some publishers already consolidate their usage data from several sources and also maintain figures on the total number of articles published in each of their journals.

Phase 1 Headline Results - Publishers Would you be prepared, in

principle, to calculate and publish global Usage Factors for your journals, according to an agreed international standard? Would you be prepared for such a process to be independently audited?

YES/QUALIFIED YES…. 84% PERHAPS 8% NO 8%

Phase 1 What to measure

Many views on how UF should be calculated

In particular, how to define total usage specified usage period total number of articles published online

Tests with real usage data will be required to refine the definitions for these terms.

Phase 1 What to measure

SPECIFIED USAGE PERIOD There was overwhelming agreement that

the specified usage period be one calendar year.

TOTAL NUMBER OF ARTICLES PUBLISHED ONLINE Need to agree a methodology for counting

items in journals not covered by ISI

Is the ISI definition of “source items” too narrow when measuring usage?

Phase 1 What to measure

SPECIFIED PUBLICATION PERIOD There was a diversity of responses to this

question, with no clear consensus on any time period among authors, librarians or publishers.

Insufficient data to support the selection of

any one specific option

Tests using real data will be required before any firm conclusions can be drawn.

Research….. Phase 1 October 2006 – January

2007 COUNTER Director Peter Shepherd carried

out a series of in-depth interviews with:- 7 authors 9 librarians 13 publishers

Phase 2 March 2007 Broader web based survey of:-

155 librarians 1400 academic authors

May 2007 Final report delivered to UKSG

Phase 2 Author results - support for a new, usage based measure

32

38

17

9

4

0

5

10

15

20

25

30

35

40

Yes, definitely Yes I'm not sure No Definitely not

Per

cent

Would you welcome the development of new quantitative measures to helpassess the value of scholarly journals based up verifiable data which describesthe number of times articles from those journals have been downloaded?

Phase 2 Librarian results: new journals

Ranking without Usage Factor Ranking with Usage Factor

1. Feedback from library users 1. Feedback from library users

2. Price 2. Usage Factor

3. Reputation/status of publisher 3.Price

4. Impact Factor 4. Impact Factor

5. Reputation/status of publisher

Phase 2 Librarian results: existing journals

Ranking without Usage Factor

Ranking with Usage Factor

1. Feedback from library users 1. Feedback from library users

2. Usage 2. Usage

3. Price 3. Usage Factor

4. Cost per Download 4. Price

5. Impact Factor 5. Cost per Download

6. Reputation/status of publisher

6. Impact Factor

7. Reputation/status of Publisher

“I would view Usage Factor as an aid for collection rather than cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation.”

Recommendations…….

“…in order to fully assess the value of a Usage Factor, I would need more information on the calculation, its origins and maybe some worked examples where sample data is applied to the equation.”

Recommendations…. That the UF concept be developed

further, with a view to testing it as a practical, implementable measure of journal quality, value and status:-

Test each of the individual elements in the UF equation using real publisher usage data

Compare UF journal rankings with IF journal rankings

Refine and investigate further the different workflow/organizational scenarios for the definition, calculation and dissemination of UFs

Recommendations….

If satisfactory results are obtained from the above investigations and tests, it is likely that the system would have to be tested for one or two years to check the large scale validity of the outcomes before a comprehensive list of journal UFs is published.

Next steps…..

On May 18, 2007 the UKSG Committee accepted these recommendations.

It asked the Project Working Group to

draw up a set of practical proposals for putting the recommendations into practice.

We will meet to get started on June 27

Reality check….. “Currently journal publishers are under

a lot of pressure to demonstrate the value they provide. By participating in this process, publishers will influence it, helping to develop useful measures in which they can have confidence.”

“This is going to happen in any event, so it is best that UF is developed and implemented by a trusted organization in which publishers are represented.”

Usage Factor Project

More information at:- http://www.uksg.org/usagefactors

Or contact:-Richard Gedye richard.gedye@oxfordjournals.org