1

Fourth Northumbria

Performance Measures for 21st Century LibrariesPeter Brophy, Director

Centre for Research in Library and Information Management (CERLIM), The Manchester Metropolitan University, U.K.

The purpose of this paper is to review EuropeanCommission funded performance measurement studiesand in particular to report briefly on the outcomes andlessons of the EQUINOX project. Based on this experi-ence and other research in CERLIM, the paper thengoes on to examine some models that may be useful indeveloping our understanding of libraries in the 21stcentury. Finally some examples of the types of libraryperformance indicators, which might be applied in thefuture, are suggested.

“Librarianship is a curious profession in which weselect materials we don’t know will be wanted, whichwe can only imperfectly assess, against criteria whichcannot be precisely defined, for people we’ve usuallynever met and if anything important happens as aresult we shall probably never know, often because theuser doesn’t realise it himself.” (Charlton, quoted byRevill,1985)

European Commission PerformanceMeasurement ProjectsAlthough the European Commission (EC) had showninterest in the development of performance measure-ment for libraries as early as the late 1980s, the first sig-nificant EC-funded study was one led by De MontfortUniversity in the UK under the title PROLIB-PI in 1994-95 (Ward et al., 1995).The study analysed previousresearch into performance measurement for libraries,including work by Nancy Van House, Charles McClureand others, and drew up a consolidated list of perform-ance indicators and the datasets needed to calculatethem. It was particularly useful because it helped par-ticipants in the four subsequent major EC-fundedlibrary performance measurement projects – EQLIPSE,DECIMAL, DECIDE and MINSTREL – to focus on devel-oping from a reasonably common, and reasonablyadvanced, starting point.The four projects were part ofa Programme entitled, rather inelegantly, Telematics forLibraries, which had an overall aim of advancingEurope’s competitive position by the innovatory use ofinformation and communication technologies (ICTs) –the EC maintains a web site of information on the pro-gramme (European Commission, 2001a). A major focusof each project was therefore on the development ofsoftware tools which could assist library managers inapplying performance measurement. However, becauseEC-funding is for ‘pre-competitive’ development (i.e. itshould not give one developer a competitive advantageover others), it has subsequently become apparent that

the effort needed to bring such software beyond thepre-prototype stage to ‘shrink-wrapped’ product is toolarge for independent commercial exploitation in whatis a small and specialised market.

The results of the four projects were subsequentlyused in a ‘Concerted Action’ called CAMILE.TheseActions are funded by the EC in order to raise aware-ness of issues and provide a forum for discussion.CAMILE delivered a series of expert workshops acrossEurope, and was successful in bringing the whole issueof performance measurement to a much wider libraryaudience than hitherto.

In the next Call for Proposals from the EC just oneproject was selected for funding.This was EQUINOX(the name is not in fact an acronym since it has nolonger form, but was chosen to indicate continuitywith EQLIPSE – which stood for Evaluation andQuality in Libraries: System for Europe). EQUINOX,unlike its predecessor projects, focused attention onthe development of electronic library services.Again, itwas part of the Telematics for Libraries Programme, sosoftware development was also to the fore.There hasbeen more interest in commercialising this product,although negotiations are still ongoing.The projectmade progress in four areas:

- the EQUINOX software product itself (illustrated inFigs 1 and 2);

Fig. 1: The EQUINOX Software System, illustratingthe hierarchy of mission, aims, objectives, perform-ance indicators and datasets

- the publication of an XML DTD designed to resolveinteroperability issues between heterogeneouslibrary & information delivery systems and decisionsupport systems like EQUINOX;

- the promotion of quality management approaches.Unlike EQLIPSE, EQUINOX promoted a generalisedapproach which was not tied to the internationalquality management standard ISO9000;

- the development of international consensus on a setof performance indicators for the electronic library.In this aspect the project aligned itself closely withISO activity in the field.

Only the last of these is considered in this paper.

The EQUINOX Performance Indicator SetAs part of the project, EQUINOX convened a numberof meetings and project staff held both formal andinformal discussions with librarians from all sectorsand from across Europe.We also maintained liaisonwith the work being undertaken by McClure andBertot in the United States.A draft set of performanceindicators was produced in early 2000, and this wasrefined in further discussions. By the formal end of theproject, in November 2000, the set had been reducedto 14 indicators.These are shown in Fig. 3.

1. Percentage of the population reached by elec-tronic library services

2. Number of sessions on each electronic libraryservice per member of the target population

3. Number of remote sessions on electronic libraryservices per member of the population to beserved

4. Number of documents and entries (records)viewed per session for each electronic libraryservice

5. Cost per session for each electronic libraryservice

6. Cost per document or entry (record) viewed foreach electronic library service

7. Percentage of information requests submittedelectronically

8. Library computer workstation use rate

9. Number of library computer workstation hoursavailable per member of the population to beserved

10. Rejected sessions as a percentage of totalattempted sessions

11. Percentage of total acquisitions expenditurespent on acquisition of electronic library services

12. Number of attendances at formal electroniclibrary service training lessons per member ofthe population to be served

13. Library staff developing, managing and providingELS and user training as a percentage of totallibrary staff

14. User satisfaction with electronic library services

The later stages of EQUINOX involved 45 librariesacross Europe in the testing of the software tool and ofthe performance indicator set. Not all libraries wereable to gather datasets for all indicators, although alltested some and their experiences with the softwaretool were uniformly positive.The full results of thiswork have been published in the project’s formalreports, which are available from the web site(EQUINOX, 2001).

Among the key issues identified from these trialswere:

- there is an ongoing tension between demands forperformance indicators which describe the situationas it now is, and for those which describe newlyemerging services. Because we were aligned withISO practice, there was pressure to develop indica-tors which were applicable in the majority of exist-

2

Fourth Northumbria

Fig. 2: The EQUINOX Software System, illustratingdata display capabilities

Fig. 3: The EQUINOX Performance Indicator set

ing libraries, rather than to move towards entirelynew indicators. For this reason, the EQUINOX indi-cators are designed to be the ‘equivalents’ of estab-lished indicators for traditional services.

- from its inception, EQUINOX had taken the viewthat data collection should wherever possible beautomated; indeed this was the motivation for theXML DTD development. However, as had beenfound in EQLIPSE, library systems are capable ofproviding only a small fraction of the required data.Much of the most interesting and valuable data can-not be collected in this way – user satisfaction isone example.The balance between using what isavailable – an example might be number of log-ins –against expensive manual data gathering – inter-viewing users to find out why they log in (and whythey don’t) – is not an easy one to achieve.

- the perennial problem of just what should be count-ed occupied a considerable amount of our time.Thisissue had been rehearsed in CERLIM’s earlier workfor the UK eLib Programme, and it is worth quotingfrom that report:

“use of electronic services could be measured by ref-erence to connect time, number of sessions, num-ber of concurrent sessions, number of hits, cost,number of active users or a variety of other factors.Care has to be taken with each of these possibili-ties, since it is possible that each could be affectedby irrelevant and indeed uncontrollable variables.For example, connect time may well depend on net-work response times outside the control of the indi-vidual library, while number of hits could be drasti-cally curtailed if a cache came into use.” (Brophyand Wynne, 1998)

As in that earlier work, consensus was reached on theuse of sessions as the basic unit, but of course thisremains a controversial issue, not least because of dif-fering definitions of session boundaries.

- there are considerable difficulties with interpreta-tion of data and indicators. For example, while therewas consensus that an indicator such as “Number oflibrary computer workstation hours available permember of the population to be served” was need-ed, there was less agreement on its interpretation.For a university such as Carnegie Mellon inPittsburgh, for example, which has recentlyannounced that the campus is now fully wireless,such an indicator is all but meaningless.

- most importantly, these discussions and debatesraised the most fundamental issue of all – just whatis a library in the networked age? What should itsrole be? How is it related to other informationproviders, aggregators and intermediaries? When weattempt to measure its performance, how do werelate its contribution to the broader environment –of learning, of commerce, of leisure?

What is a library in the 21st century?The networked environment is not simply anotherissue for librarians to address. It changes fundamentallythe whole business in which they are involved. Just afew of the impacts are:

- Information plethora: we have moved from a situa-tion of information poverty, where people had tosearch hard, and needed expert help, to find theinformation they needed, to one of multiple sourcescompeting for attention coupled with informationoverload. Far more than in the past, the role of theinformation intermediary will be to filter and select– to provide less rather than more.

- Unstable information objects: a large proportion ofthe world’s information is now inherently unstable.Unlike print on paper, which has a reasonable lifeand in any case is held in multiple copies in distrib-uted storage, electronic objects often exist in onlyone copy, and that copy is changed on a whim.There is no certainty that if I give you a URL theobject you discover will be the one I viewed –assuming there’s anything there at all. More andmore objects will be created dynamically ondemand, so that no two copies will be identical.Thissituation is entirely different from that which drovethe development of traditional libraries.

- Quality of content is patchy and often unknown: itis far more difficult than in the past to judge the reli-ability and authority of any information object. Nowthat self-publication is so easy, the mechanismswhich provided quality assurance of information arebreaking down. Libraries have held a clearly under-stood place in the information chain in the past, butif that chain is shattering, what will their future rolebe?

- Services are technically heterogeneous: there arethousands of different information services availablefor access by libraries and their users, but they arenot like books, which could be shelved, convenient-ly (even if a special ‘Oversize’ section were needed)in straightforward sequences.They use a wide vari-ety of access protocols, hold data in different for-mats, respond differently to the same request and

3

Fourth Northumbria

the results they return are frustratingly difficult tointegrate into a single service.This is a huge chal-lenge for anyone wishing to use a wide variety ofsources to deliver information services.

- Metadata is haphazard and semantically con-fused: in a world of almost endless informationresources, description is all-important and offers thekey to effective selection of resources.Yet only aminority of information objects have useful meta-data, and even where this is structured it is oftensemantically confused or ambiguous. Libraries prid-ed themselves on the quality of their catalogues –how do we reproduce this quality in the networkedinformation world?

- Users think they don’t need intermediaries: thereis a growing body of evidence that students rarelyuse library interfaces in their search for information,preferring to go straight to an Internet searchengine (in the UK the preference seems to be forGoogle – at least this is a preliminary finding ofCERLIM’s EDNER project, described later).Theywant some information on their topic, not a com-prehensive list; the best is unnecessary if a ‘goodenough’ resource can be found quickly and easily.How do services which pride themselves on thequality of their product market themselves toachieve survival in this environment?

- Client access is pervasive and will soon be univer-sal: although the library has an important role inproviding access to IT hardware and software atpresent, this can only be a transitional phase.Internet access via TV sets and mobile devicesshows the likely trend. Before very long virtuallyeveryone in developed countries – and this is not todeny the very real, but different, issue in less devel-oped regions – will have Internet access at home orat work or on the move. In time – and the timeperiod will not be all that long – Internet accesswill be as universal as telephony.

These are merely examples of the issues that weface, but they indicate the scale of the challenge.A steptowards meeting that challenge will be to rethink ourconcepts of what libraries are for and what they do.

Five models of the LibraryMODEL A: THE TRADITIONAL LIBRARY

This is the library with which we are familiar. Itfeatures:

- A physical building which, through its architectureand interior design, is a statement about the valuesof the institution or community that owns and spon-sors the library service.

- Bookstock: carefully selected titles which are organ-ised and presented to meet the likely needs ofusers.

- Journals and newspapers, enabling users to keep upto date with their areas of interest.

- Inter-library loans, extending the reach of the librarybeyond its own walls and stock.

- Reference services, designed to provide answers tothe questions users pose.

- Friendly and knowledgeable staff, who are theapproachable face of the library.

- A welcoming atmosphere, perhaps taking a leaf outof the bookshops’ book by including a modern cof-fee bar and comfortable seating.

It is important that we do not lose sight of theadvantages of this model of the library for the future.People will still need physical places, they will almostcertainly want to use some kind of individual, physicalinformation objects for some purposes and they willvalue the expertise of trained and knowledgeable staffwho can help them with their enquiries.

MODEL B: THE MEMORY INSTITUTION

In Europe there is an increasing emphasis on thelibrary as a ‘memory institution’, a body – along withmuseums, archives, galleries etc. – with a responsibilityfor maintaining humankind’s recorded memory.AsDempsey et al. (1999) have written:

“Archives, libraries and museums are memory insti-tutions ….Their collections contain the memory ofpeoples, communities, institutions and individuals,the scientific and cultural heritage, and the productsthroughout time of our imagination, craft and learn-ing.They join us to our ancestors and are our legacyto future generations.They are used by the child, thescholar, and the citizen, by the business person, thetourist and the learner.These in turn are creatingthe heritage of the future. Memory institutions con-tribute directly and indirectly to prosperity throughsupport for learning, commerce, tourism, and per-sonal fulfilment.”

In this description we have another model, whichhas a resonance with established library activity, butexpressed in a different way and starting to pose someinteresting challenges.While national, major academicand other libraries have taken on the role of preserv-ing (mainly textual) heritage in the past, the common-alities with the broader ‘memory institution’ arena havenot been explicit. Emphases on conservation andpreservation, on bibliographic description will remainimportant.To these may be added the museum sector’sexpertise in interpretation.

4

Fourth Northumbria

The challenge of this model, of course, lies in itsextension to the world of digital objects. So much ofmankind’s memory is being lost that a new emphasison concerted efforts to preserve, conserve, interpretand make available the historical record is vital.Thiswill take individual libraries away from their sole con-cern with “their” holdings into community-wide effortsto find collaborative solutions.As this happens, the roleof the library will be redefined.

MODEL C: THE LEARNING CENTRE

Governments throughout the world are giving ever-increasing emphasis to education and lifelong learning,in recognition that a skilled and educated workforce isa prerequisite for economic well-being and prosperity.Libraries have recognised that, since informationresources are crucial to education, they have a contri-bution to make. It is unfortunate, however, that theirrhetoric is sometimes divorced from the reality of therole that they can realistically play. In the UK, for exam-ple, there has been talk of the public library as a ‘streetcorner university’. Such sound-bites display a view ofthe library which is not only unrealistic but potentiallydamaging to the co-operative relationships which offera real way forward. Libraries certainly have a role toplay in education, but the model of the library as learn-ing centre, in a networked world, needs much morerigorous analysis than has been undertaken so far.

The issue may be illustrated by reference to a reportprepared for the Formative Evaluation of theDistributed National Electronic Resource (EDNER), amajor UK project that CERLIM is leading (EDNER,2001). EDNER is funded at approximately $1 millionover 3 years and is examining in depth the develop-ment of national-level higher education informationservices in the UK. Our partners in this work are edu-cational researchers in the Centre for Studies inAdvanced Learning Technology (CSALT) at LancasterUniversity. In an early contribution to the project(Goodyear, 2001), the CSALT team noted, in examiningdevelopment projects designed to extend the use ofnetworked information resources in learning andteaching, that little attention appeared to have beenpaid to pedagogical issues. I would suggest that that isa finding that might be applied more widely tolibraries’ efforts to become learning centres. CSALT’smodel suggests that attention needs to be:

– paid to pedagogical philosophy, including:

• the high level pedagogy – the broad approachbeing taken, such as a belief in a Problem BasedLearning approach.

• pedagogical strategy – the planned actions, whichenable tutor and learner to share and developagreement on what is to be done at a general level.

• pedagogical tactics – the detailed level, such ashow learners are to be encouraged to participate,how tutors will deal with problems and so on.

– associated with an educational setting in whichtutors set tasks, students undertake actions (andnote that the actions are often not the same as thetasks set) and there are outcomes – which aresometimes, but not always, learning outcomes.Allof this happens within an educational context,which includes but is not limited to assessmentregimes, student tutoring systems and othersupport, and so on.

Added to these issues we need to explore how theconcept of the library as learning centre relates to cur-rent developments in the area of virtual learning envi-ronments (VLEs) and managed learning environments(MLEs). Clearly there is a great deal of work to bedone.

MODEL D: THE LIBRARY AS COMMUNITY RESOURCE

The fourth model focuses on the role of the librarywithin its own community. In part this is traditionalterritory, but a number of developments suggest that itcould become more critical.‘Community librarianship’has a long track record and has made important contri-butions.This role may be more important in a net-worked world in two ways: partly by offering a humanpresence when so much else is virtual; partly by help-ing create community ‘glue’ in both physical and virtu-al communities.

One of the limitations of the library’s role in thisarea in the past has been a reluctance to go beyond‘information provision’ to the offer of ‘advice’.Therehave been cogent reasons for this reluctance, not leastfears over liability if the advice is mistaken, but it doesnot fit with community needs. Citizens, particularlythose who are disadvantaged, do not want to bereferred to a book or even a web page, but want helpin understanding the possible answers to theirproblems and queries and in making informed choices.Librarians may need to form alliances with socialworkers, counsellors and others to provide the kindof integrated service that is needed.

In the networked world, citizens can also becomepublishers. CERLIM has recently been awarded a majornew European Commission project entitled CulturalObjects in Networked Environments (COINE) whichwill seek to create systems that enable local communi-ties to create their own structured information spaces,which might be termed MyWorlds (COINE, 2001).Thisoffers a role for expert institutions, at the local level,which can not only point citizens towards information-resources but also help them capture, record and sharetheir own stories. Such interactive systems that pro-

5

Fourth Northumbria

mote involvement may help libraries to secure theirfuture as community resources.

MODEL E: THE INVISIBLE INTERMEDIARY

The final model is perhaps the most important, for itsuggests that libraries could forge for themselves a cen-tral role in the information society.As has already beennoted, the vastness of the ‘information universe’, andits lack of comprehensive organisation, makes informa-tion overload and what might be termed ‘informationconfusion’ serious problems. Organisations are neededwhich can select from the information universe onbehalf of their users – creating their own ‘informationpopulation’ by discarding the dross and the irrelevant –and present meaningful descriptions of that informa-tion to enable selection and retrieval.

The universe of potential users needs to be treatedin the same way: users are ‘registered’ with the serviceand detailed descriptions are developed. In this casethe descriptions will include preferences data, individ-ual interests and so on. In the future the library mayneed to pay as much attention to maintaining this typeof metadata as to that related to information objects –many commercial e-providers already do so.

The library, in this model, acts as an intermediarybetween user and information, but may be largely invis-ible – it facilitates meaningful interactions and‘smoothes the way’ for the user.The model is depictedin Fig. 4, and has been further described in TheLibrary in the Twenty-First Century (Brophy, 2001).

An important issue for libraries is that they are notalone in seeking to fulfil this role. Each library is, tosome extent, in competition with others and there aremany other players in this marketplace. But libraries do

have advantages: they are already trusted as intermedi-aries, they have vast experience in organising anddescribing information and they have developed expe-rience of the real issues in the interoperation ofheterogeneous information systems in complexenvironments.

New Performance Indicators for New LibrariesIn this final section I want to make some suggestionsas to the kind of performance indicators that might beappropriate to the models outlined above.These are nomore than indicative suggestions, and much work isneeded to define a suitable set, but they illustrate howfar we may need to go beyond the performance indica-tor sets currently available. Briefly:

FOR MODEL A: THE PHYSICAL LIBRARY

We already have a wide range of indicators.As this con-ference has illustrated, effort now needs to be concen-trated on assessing outcomes and impacts, and on qual-ity assessments, which go beyond simple user satisfac-tion statements. LIBQUAL+ is an excellent example ofsuch work (Cook et al., 2001).

FOR MODEL B: THE MEMORY INSTITUTION

Here we need to assess how well the library is fulfill-ing the role of preserving and encouraging use ofhumankind’s memory.We might want to measure:

– The proportion of archival objects, which arearchivally secure – conserved in appropriate ways.

– The quality of the descriptive metadata available.

– The number of scholars served per archival item.

– The number of interpretative events (akin to muse-um exhibitions) per archival item.

FOR MODEL C: THE LEARNING CENTRE

In this case we might want to concentrate on:

– The proportion of ‘events’ designed by or run bythe library for which explicit learning objectivesexist. (This is akin to the kind of performance indi-cator that is common in educational institutions,though these days 100% would be required.)

– The proportion of learning outcomes that weresuccessful.

– The proportion of information objects cited in dis-sertations or essays that were library-supplied.

FOR MODEL D: THE COMMUNITY RESOURCE

Here the concerns might include:

– Market penetration, especially among targetedgroups, measured by reference to activeparticipation.

– “MyWorlds” created per member during the last12 months.

– The user view of the service against its rivals.

6

Fourth Northumbria

Fig. 4: the Library as Invisible Intermediary

FOR MODEL E: THE INVISIBLE INTERMEDIARY

Possible performance indicators might include:

– Personalisation

– The proportion of users for whom personalmetadata has been updated in the previous 3, 6,9, 12 months.

– Reach

– A Conspectus-style measure of the extent ofsources to which access may be provided.

– Information description

– The proportion of sources delivered which had‘high level’ descriptive metadata.

– Quality of content delivered

– ‘Quality’ results returned per query.

– Interoperability

– The ability to handle the latest versions of allstandards (e.g. full Z39.50).

– Delivery mechanism of choice

– The proportion of cited references in users’papers, which were accessed via library mecha-nisms.

ConclusionIn this paper, I have reported on the state of researchand development in library performance measurementin Europe. I have also suggested that we need torethink the underlying models of libraries in order tofind new and relevant performance indicators forlibraries operating in the networked world. I have sug-gested some examples of the type of indicator whichmay be appropriate.

In concluding, however, I would also like to drawattention to a problem I alluded to earlier but whichwill loom ever larger, namely the interpretation ofthose indicators.The issue is best illustrated by twodiagrams. In Fig. 5 there is a depiction of our traditionalinterpretation of library ‘goodness’ – we assume thatthe more books issued per user or the more docu-ments delivered, the better the service.This is illustrat-ed by the straight line. But in the networked environ-

ment, Fig. 6 may be more apposite.The library whichdelivers no documents per user is, we would probablyall agree, poor. But so also is the library that deliversvery large numbers (indicated by M in Fig. 6) – it issimply swamping its users and overloading them withinformation.The ‘good’ library will be somewhere inthe middle – indicated by N – for it will have selectedthe items it presents carefully and filtered out the irrel-evant. Operating in such environments will requirevery careful attention to the real messages that per-formance indicators bring.

7

Fourth Northumbria

Fig. 5: ‘Traditional Library’ goodness

Fig. 6: ‘New Library’ goodness

ReferencesBrophy, P. The Library in the Twenty-First Century.

Library Association Publishing, 2001.

Brophy, P. and Wynne, P.M. (1998) ManagementInformation Systems and PerformanceMeasurement for the Electronic Library. FinalReport. eLib/LITC, 1998:<http://www.ukoln.ac.uk/models/studies/> (Visited8th September 2001)

COINE (2001) <http://www.coine.org> (Visited8th September 2001)

Cook, C. et al. (2001) LIBQUAL+: Service quality assess-ment in research libraries. IFLA Journal, 27(4),264-8.

Dempsey, L. et al. (1999) Scientific, industrial, and cul-tural heritage: a shared approach. The New Reviewof Information and Library Research, 5, 3-29.Anearlier version is available at <http://www.aradne.ac.uk/issue22/dempsey/> (Visited 8th September2001)

EDNER (2001) <http://www.cerlim.ac.uk/projects/edner.htm> (Visited 8th September 2001)

EQUINOX (2001) <http://equinox.dcu.ie/> (Visited8th September 2001)

European Commission (2001a) Telematics forLibraries. <http://www.cordis.lu/libraries/> (Visited8th September 2001)

European Commission (2001b) CAMILE<http://www.cordis.lu/libraries/en projects/camile.html> (Visited 8th September 2001)

Goodyear, P. (2001) Pedagogical frameworks for theDNER: EDNER Deliverable DC1. (Not yet published)

Revill, D. (1985) The measurement of performance. In:Cowley, J. ed. The management of polytechniclibraries. Gower. pp. 104-34 (the quotation byCharlton is found on p. 132.)

Ward, S. et al. (1995) Library performance indicatorsand library management tools. (PROLIB-PI)Luxembourg; European Commission DGXIII-E3[EUR 16483 EN].

8

Fourth Northumbria