New measures for new servicesIndicators for quality, cost, and impact

of electronic library services

Roswitha PollMünster

LIDA2007LIDA2007

The new electronic library services

• catalogue databases• portals• online ordering and delivery• personalized services• online reference• digitized collections• electronic publishing• online self-paced training• Internet access via the library

LIDA2007LIDA2007

Reasons for evaluation of quality and impact

accountability

special funding needed: evidence of positive effectsresults-based budgetingjustifying investment into change

resource management

rising expenses for new servicesincreasing demand for new servicesnew allocation of resources necessary

LIDA2007LIDA2007

promotion of the library‘s role

communication of benefits by using library services

What library standards want to achieve

to spread the knowledge about existing tested methods and practices

to standardize

with the aim of comparing and aggregating results

terminology and definitionsterminology and definitions

methods of data collectionmethods of data collection

methods of analyzing the datamethods of analyzing the data

LIDA2007LIDA2007

ISO Technical Information and committee 46: documentation

Subcommittee 8: Quality, statistics and performance evaluation

WG 2: International library statistics

ISO 2789 (2006) International library statistics4th edition

WG 4: Performance indicators for libraries

ISO DIS 11620 (2006) Library performance indicators2nd edition

WG 7: Quality measures for National Libraries

ISO TR 281180 Performance indicators for National Librariesin preparation

LIDA2007LIDA2007

Stages of an ISO standard

Approved New Work Item

Working Draft

Committee Draft (CD)

Draft International Standard (DIS)

Final Draft International Standard (FDIS)

Standard

votingrequired

LIDA2007LIDA2007

Problems of standardizing data for electronic services

quick changes in media and information ways changes not always predictable no clear boundaries of documents and

localities data must be collected from many sources

(vendors, suppliers, institutional servers)

State of technical

development varies in libraries

LIDA2007LIDA2007

In terms of• collection size• number and type of users• use of services• staff• space• equipment• training

LIDA2007LIDA2007

Assessingquantity

(ISO 2789)

problemof countingproblem

of counting

can be counted manually

can be counted manually

OPACOPACOPACOPAC

web-web-sitesite

web-web-sitesite

electronic electronic collectioncollectionelectronic electronic collectioncollection

Internet Internet accessaccess via the via the librarylibrary

Internet Internet accessaccess via the via the librarylibrary

electronic document delivery

electronic document delivery

online online referencereference

online online referencereference

user training on electronic

services

user training on electronic

services

Electronic library services: counting use

LIDA2007LIDA2007

Electronic collection

Electronic collection

databasesdatabases electronicserials

electronicserials

digital documents

digital documents

full textfull textabstract

andindexing

abstractand

indexing otherother electronicpatents

electronicpatents

networkedAV-docu-

ments

networkedAV-docu-

mentsotherothereBookseBooks

ISO 2789

computerfiles

computerfiles

LIDA2007LIDA2007

Forms of useForms of use

sessionssessions rejectedsessionsrejectedsessions

contents or

recordsdownload

contents or

recordsdownload

searchessearches InternetsessionsInternet

sessionsvirtualvisits

virtualvisits

sessiontime

sessiontime

inside the libraryinside the libraryoutside the library,

but inside the institution/population

outside the library,but inside the

institution/population

from outsidethe institution/

population

from outsidethe institution/

population

ISO 2789

LIDA2007LIDA2007

ISO 2789: Definitions of electronic use

LIDA2007LIDA2007

searchspecific intellectual query

sessionsuccessful request of a database or the online catalogue

rejected sessionunsuccessful request exceeding the simultaneoususer limit

session time

period between log-inand implicit or explicitlog-off

content unit downloaded

content unit beingsuccessfully requested

record downloaded

descriptive record being successfully requested

Internet access by a user from inside the library

Internet session

a user's request on the website from

outside the library premises

a user's request on the website from

outside the library premises

web browservisit

web browservisit

IP visitIP visit home pagevisit

home pagevisit

ISO 2789: virtual visits

LIDA2007LIDA2007

Counting electronic usage in German academic Counting electronic usage in German academic libraries: Percentage of libraries delivering datalibraries: Percentage of libraries delivering data

2005 2006

OPAC sessions 45.3 % 39.8 %

Database sessions

28.0 % 31.1 %

E-journal sessions

25.8 % 32.0 %

Digital document sessions

21.8 % 17.0 %

Downloads from databases

7.1 % 9.2 %

Downloads from e-journals

9.3 % 13.1 %

Virtual visits 36.0 % 34.5 % LIDA2007LIDA2007

In terms of

• user-orientation• reliability• currency• usability/accessability

• competence• responsiveness• speed• cost-effectiveness• perceived quality

LIDA2007LIDA2007

Assessingquality

(ISO 11620)

ISO 11620: Library performance indicators

resources, access, infrastructure

use

efficiency

potentials and development

New structure: Balanced scorecard

LIDA2007LIDA2007

Indicators for "hybrid" services

ISO 11620

resources, access,

infrastructure

• percentage of required titles in the collection• speed of interlibrary lending

use • library visits per capita (including virtual visits)• percentage of population reached• user satisfaction

efficiency • cost per library visit• cost per user• correct answer fill rate

potentials,development

• attendances at training lessons per staff member

LIDA2007LIDA2007

Indicators for electronic services

ISO 11620

resources, access,

infrastructure

• percentage of rejected sessions• public access workstations per capita

use

• number of content units downloaded per capita• percentage of information requests submitted electronically• workstation use rate

efficiency • cost per database session• cost per content unit downloaded

potentials,development

• percentage of acquisitions expenditure spent on the electronic collection• percentage of library staff providing and developing electronic services

LIDA2007LIDA2007

Possible performance indicator for website quality

Direct access from the homepage

Measured by• number of clicks neccessary to find relevant information• adequacy of terms used on the homepage Example: renewalsDirectly on the homepage: user account, my accountUnder broader terms: circulation, loan service

LIDA2007LIDA2007

Performance indicators for costs ISO 11620

costs includecosts include• total recurrent expendituretotal recurrent expenditurecost per loan

cost per database session

costs includecosts include• contractual costs of a databasecontractual costs of a database

cost per content unit downloaded

costs includecosts include

• contractual costs of an electronic contractual costs of an electronic resourceresource

LIDA2007LIDA2007

Cost issues

Changing from print to electronic collections

What changes for the library?

decrease of traditional services (loans, visits, decrease of traditional services (loans, visits, copying)copying)

increase of electronic useincrease of electronic use

In use

in costs

less costs for storage, bindingless costs for storage, binding

higher systems and staff costshigher systems and staff costs

better qualified staff neededbetter qualified staff needed

lower cost per uselower cost per use

overall:overall:increase in useincrease in use

LIDA2007LIDA2007

Cost issues

Changing from print to electronic collections

What changes for the users?

access from the working-place or homeaccess from the working-place or home

access at onceaccess at once

24-hours access24-hours access

searching options, linkssearching options, links

In use

in costs

time saved !time saved !

no ways, no fetching of materials,no ways, no fetching of materials,

no waiting in queuesno waiting in queues

additional additional valuevalue

academics spend half a day academics spend half a day per week on seeking and per week on seeking and

procuring informationprocuring information

LIDA2007LIDA2007

Cost issues

Journals: cost per use

Print onlysubscription

costs

usage cost per use

E-onlysubscription costs

usage cost per use

257.737 € 13.795 18.86 € 223.647 € 64.491 3.47 €

Users accessed online Users accessed online versions 10 times versions 10 times more than print more than print versionsversions

Users accessed online Users accessed online versions 10 times versions 10 times more than print more than print versionsversions

Only subscription costs are considered!Only subscription costs are considered!

Example 1: Medical library MüsterExample 1: Medical library Müster

LIDA2007LIDA2007

Journals: cost per use

Cost typeElectronic Journals

Print Journals (current)

Print Journals (bound)

Space $ 5,000 $40,000 $205,000

Systems $ 10,000 $ 2,500 $ 2,400

Supplies & Services $ (2,000) $ 600 $ 8,000

Staff        $125,000 $46,000 $ 42,000

TOTAL OPERATIONAL COSTS $138,000 $90,000 $258,000

Operational costs of print and electronic journals

Montgomery, Carol Hansen, Donald W. King: Comparing libraryMontgomery, Carol Hansen, Donald W. King: Comparing library and user related costs of print and electronic journal collections.and user related costs of print and electronic journal collections.

D-Lib Magazine October 2002D-Lib Magazine October 2002Drexel University PhiladelphiaDrexel University Philadelphia

Example 2: Drexel UniversityExample 2: Drexel University

LIDA2007LIDA2007

Staff costs by function (Drexel)

E-journals

Print (current)

Print (bound)

    Director/Administration $ 22,000 0 0

    Communications (among staff) $ 7,000 $ 2,500 $ 800

    Collection development $ 18,000 $ 6,000 0

    Acquisitions $ 8,000 $ 4,000 0

    Physical handling: bindery, labeling, reshelving 0 $10,000 $22,000

    Record creation and maintenance $ 16,000 $10,000 $ 6,000

    Public relations $ 11,000 $ 1,000 $ 600

    Reference $ 37,000 $ 7,000 $ 4,000

    Teaching (users) $ 6,000 $ 1,000 $ 600

    Other 0 $ 4,000 $ 6,000

        $125,000 $46,000 $42,000 LIDA2007LIDA2007

Cost per use by journal type (Drexel)

recorded use

subscription cost per use

operational cost per use

total cost per use

print current

15.000 $ 2.50 $ 6.00 $ 8.50

print bound

9.000 ---- $ 30.00 $ 30.00

total print 24.000 $ 2.50 $ 15.00 $ 17.50

total E-journals

335.000 $ 1.40 $ 0.45 $ 1.85

LIDA2007LIDA2007

Journals:cost per use

Problems in measuring cost per use

package deals for journalspackage deals for journals

consortia with "additional access"consortia with "additional access"

publishers insist on "print + electronic"publishers insist on "print + electronic"

For subscription pricesFor subscription prices

For measuring useFor measuring use

reshelving counts may under-estimate use reshelving counts may under-estimate use

E-journal use data may differ between publishersE-journal use data may differ between publishers

For assessing operational costsFor assessing operational costs

assessing staff time will be time-consumingassessing staff time will be time-consuming

long-term preservation costs for E-media still long-term preservation costs for E-media still questionablequestionable

High acceptance and use

High acceptance and use

of E-media will evidently result in lower

of E-media will evidently result in lower

cost per usecost per use

LIDA2007LIDA2007

In terms of

knowledge

information literacy

academic or professional success

social inclusion

individual well-beingchanges in skills, competences, attitudes, behaviour

Assessingimpact/outcome

LIDA2007LIDA2007

Methods for assessing impact

Use statistics as measures of impact

electronic media• frequency of use

• forms of use (sessions, searches, downloads)

• range of resources used

change of use in traditional services

change in remote use

number of new users• that had not used the library

• that had not used the same service in traditional form

percentage of the primary user group using E-services

Users are familiar with the use of

electronic resources and

services

LIDA2007LIDA2007

Qualitative methods

Surveys (postal, face-to-face, telephone, online)

Interviews

Focus groups, discussion groups

Self assessment of users

Data and „stories“ must be brought in a systematic

order in order to make trends visible

The "anecdotal evidence" supports

the statistics

Methods for assessing impact

LIDA2007LIDA2007

Quantitative methods

Tests of user skills before and after training

Observation by trained observers

Data mining

Analysis of bibliographies in publications of users

Comparing library use to academic success

Problem:Influences on individuals are manifold. The library‘s influence not always clear

Methods for assessing impact

LIDA2007LIDA2007

Example: Example:

Changing the journal collection to electronic formChanging the journal collection to electronic form

Statistical resultsStatistical results Survey to facultySurvey to faculty

Assessing the outcome of electronic services

online versionswere accessed10 times more

usage of an online article 5.4 times

cheaper high decline in print

usage

remote use increased

physical libraryvisits decreased

• In 1998 a number ofIn 1998 a number of academics were sceptical academics were sceptical as to E-journalsas to E-journals

• In 2002 it was exceptional forIn 2002 it was exceptional for faculty staff not to use the faculty staff not to use the electronic version electronic version

LIDA2007LIDA2007

change of change of attitudes attitudes

and and behaviourbehaviour

Projects of assessing outcome

eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information serviceshttp://www.evalued.uce.uk/index.htm

IMLS (Institute of Museum and Library Services): outcome-based evaluation of projectshttp://www.imls.gov/index.htm

ARL New Measures Initiative: mehrere Projekte - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services)http://www.arl.org/stats/newmeas/index.html

SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing listhttp://www.sconul.ac.uk/activities/performance/impact.html

LIDA2007LIDA2007

Standardization of impact measures?

projects still on the wayIFLA Section Statistics and Evaluation has instituted a working groupBibliography of projects and literature:

http://www.ulb.uni-muenster.de/outcome

LIDA2007LIDA2007

Experience Experience is is the mother the mother of of sciencescience

Traditional ProverbTraditional Proverb

LIDA2007LIDA2007