Demonstrating excellence:a European Perspective

American Library Association26th June 2010

Summary

• Context– Pressures to prove excellence

• Responses and experience– Quality– Impact– Value

• Where next?

Context

Trends - Library concept

• “Storehouse to service to educational integration” (Lancour, 1951)

• “Digitisation increases tempo” but reduces explicit mediation

• “Library as place” to “Learning space”• Reinvention, rebranding, repositioning of

academic libraries

Environmental Trends

• European “harmonisation”– Bologna declaration– New nations and national library systems

• Economics– Competition (local and international)– Consumerism– Downturn

Bologna process

• Driven by common European issues– Growth, diversification, skills shortages, graduate skills,

life long learning

• European Higher Education Area– 29 countries initially; now 49– Common degree frameworks

• “Institutions should routinely monitor, review and improve the effectiveness of the support services available to their students” Bologna QA guidelines

Cross pressures on assessment 1

• Quality assurance systems• Customer driven quality

– “The student experience”

• Deeper customer understanding– Digital and other generational issues

• Impact– Teaching and learning– Research library contribution and advocacy

Cross pressures on assessment 2

• Statistical aggregation and development– Beyond national boundaries– Evidence-based management

• Value– Costs and cost analysis– Cost comparison and benchmarking– Return on investment & contingent valuation– “Real” worth and value

RIN Report on academic library challenges

• “ … there is a strong feeling among senior librarians that they have failed effectively to communicate the value of their services … there is an increasing risk that much of what libraries actually do may be invisible in a virtual environment. … we believe it is important that libraries should be able to show … that they provide services with demonstrable links to success in achieving institutional goals. Return on investment is thus an increasingly important issue. Libraries need to be more proactive in seeking to understand user behaviour and workflows; and in rigorously demonstrating the value of their activities … . The focus of performance indicators up to now has tended to be on inputs and outputs … rather than addressing the much harder issues relating to impact and value. … we believe it is essential that more work is done to analyse the relationships between library activities … and learning and research outcomes … .”

Comparison

SPEC Kit 303 in the UK and Ireland

Sample & characteristics

• 77 libraries (43% of SCONUL membership but 60% of University institutions)

• Majority engaged with PM from late ‘80s onwards

• User surveys were first assessment activities in most cases

• Rationale was internal and user driven

PM Activities in use

Range of 3-19 of listed methods; median of 10; average of 10.6

• Statistics (96%)• Suggestions (91%)• Data mining (72%)• Outcome evaluation (67%)• Benchmarking (63%)• KPIs (63%)

Least used, but …

• Value/ROI assessment• Impact assessment• Balanced scorecard• Physical orientation studies• Mystery shopper studies

Culture of assessment?

• Results used to improve library (75%)• Evaluation for service quality (69%)• Assessment is a library priority (67%)

• Staff development is adequate (13%)• Staff have necessary skills (26%)• Staff accept responsibility (34%)

Basic comparisons (ARL/SCONUL)SPEC Kit 303

• 73 of 123 (60%)• 99% active• 91% customer driven• 29% accreditation driven• Majority survey first• Improvement 76%• No particular training 29%• Preference for interface

testing

UK & Ireland

• 77 of 129 (60%)• 100% active• 84% customer driven• 9% accreditation driven• Majority survey first• Improvement 75%• No particular training 51%• Preference for surveys

Responses

Quality & Deeper Understanding

The Student Experience

• National Student Survey– UK comparative & competitive data for guidance– Single library question

• LibQUAL+– Benchmarking non-competitive data for

improvement and international comparison

• SCONUL and other survey tools• “Student first” initiatives

National Student Survey

LibQUAL+ Consortia outside North America

• Belgium• Canada• European Business Schools (EBSLG)• France• Hong Kong (JULAC)• Ireland• Japan• NHS England• Norway• UK & Ireland (SCONUL)

Overall SCONUL Sample to 2009

• Full variety of institutions• 50% of institutions* (53% of RAE top 50)• >53% of HE students potentially sampled

(>850,000)

*Based on a selection of Universities UK membership of 131

LibQUAL+ SCONUL results

Use for national policy influence

• SCONUL LibQUAL+ results contributed to national enquiries on libraries and learning in higher education, with particular reference to learning spaces, and the competitiveness of UK University libraries in comparison to global peers

Research Information Network

• Studies of research, researchers and information resource use

• Impact of collaboration• Response of librarians to difficult economic

times• The future academic library, with SCONUL and

others• The value of the academic library to

researchers, with RLUK

Deeper understanding

• Space use– SCONUL Working Group

on Space Planning– Snapshot survey figure:% responses to the statement

‘There is an increased incidence of students ‘booking’ study spaces by leaving possessions on desks during the exam period’

Responses

Value & Impact

Impact initiative

• July 2003-December 2005• Two cycles• 22 UK Universities• Impact of a specific innovation• Majority focussed on information literacy or

e-resource use

SCONUL’s ‘Performance Portal’

• The VAMP programme for value and impact measures, plus …

• A Wiki of library performance measurement containing a number of ‘approaches’, each (hopefully) with:– A definition– A method or methods– Some experience of their use in libraries (or links to

this)– The opportunity to discuss use

The Ontology of Performance

• ‘Frameworks’• ‘Impact’• ‘Quality’• ‘Statistics’• ‘Value’

Value measurement in libraries

“focusing on cost without being able to demonstrate [service] value and quality … leaves the initiative to people whose chief concern is cost-control or profit: the funders and the vendors”

Whitehall, T (1995)

University influences

• Research, Teaching & Reductionism– ‘Mode 1’ Research & impact ‘transcendental’– ‘Mode 2’ Research & impact ‘instrumental’– Value, Price & ‘Mandarinisation’ of research and its

support– Learning as a set of discreet assessed modules

• All of this may damage the idea of Libraries as ‘transcendent’, collective and connective services

Guardian editorial on Michael Sandel’s Reith Lectures, 2009

“the credit crunch has exposed myriad mirages, demonstrating how the market can get things badly wrong when it comes to valuing things … when bureaucracies price things which should not be priced, they start trading them off against other objectives, instead of appreciating their absolute obligations.”

Libraries and economic value: a review of recent studies

A natural history of value initiatives:

• Activity based costing for output efficiency• Perceived value based on labour saving• Balanced scorecard pressure for ’hard’ value measurement

“demonstrate value by linking to the organisation’s value statements”

Missingham, 2005

UK cost analyses

• UK HE Transparency Initiative (TRAC)– University cost breakdown to identify and

separate teaching and research costs• Open University

– Business reporting– Process costing and continuous improvement– Service planning– Benchmarking (for international exercise)– ‘to generate real accountability’

Financial Benchmarking

• International Groups• Commercial

– LISU– Tribal Group

• National– BIX – the Library Index

Contingent valuations and ROI

• British Library– £4.40 for every £1 spent

• Reviews and commentary– Aabo, 2009– Missingham, 2005– White, 2007

Intangible assets for academic libraries.

• Requires intellectual capital reporting model– Using ROI, contingent valuation etc

• Dimensions– Human capital– Structural capital– Relational capital

Kostagiolas & Asonitis, 2009

Where next?

• Comprehensive and holistic measurement• “True worth”

– Value and impact– Real understanding

• Transcendent valuation• The narrative of worth

Valuation to values

“Whenever valuation takes place … values must enter in .. in evaluation an indispensable recourse to underlying values is involved”

“values cannot be deduced from .. data or logic … they have to be chosen”

Values-based value

• Competing values• Establishing the link between values and value

– Reawakening of the broader common good of libraries

• University of York– Values “jam”– User perceptions of value as opposed to quality– Value(s) scorecard

Competing values

The Clan Culture

Customer Services?

Collaborate

The Adhocracy Culture

Digital Library Development?

Create

The Hierarchy Culture

Content Services?

Control

The Market Culture

Academic Liaison?

Compete

Staff values debate using ideascale

User values (Academic Engineer)• Library as a “real tangible physical expression of knowledge”• “Intellectual heart, a collection of knowledge made without

fear or favour”• Exaltation of solitary study - deeper understanding by

“conquering the stuff alone”• Organisation of knowledge reflected in how things are laid

out; browsing and walking through physical objects• Browsing; overview of knowledge by the way it is structured;

‘to steer thinking”; density tells you what’s important• “A real physical existing thing where I can see the celebration

of scholarship”

Web references

• BIX see: http://www.bix-bibliotheksindex.de/index.php?id=115• Bologna see: http://ec.europa.eu/education/higher-education/doc1290_en.htm• CIBER see: http://www.ucl.ac.uk/infostudies/research/ciber/ • LIBER see: http://www.libereurope.eu/node/457• Impact initiative (Poll & Payne, 2006) see: http://eprints.bbk.ac.uk/373/• National Student Survey see: http://unistats.direct.gov.uk/• Research Information Network see: http://www.rin.ac.uk/• SCONUL Performance Portal see: http://vamp.diglib.shrivenham.cranfield.ac.uk/• Performance measurement in the UK & Ireland (Stanley & Killick, 2009):

http://www.arl.org/bm~doc/LibraryperformanceUKIreland.pdf

J. Stephen Town

jst504@york.ac.uk