www.arl.orgassociation of research libraries assessing library performance: new measures, methods,...

www.arl.orgAssociation of Research Libraries

Assessing Library Performance:

New Measures, Methods, and Models

24th IATUL Conference • 2-5 June 2003 • Ankara, Turkey

Julia C. Blixrud, ARL Assistant Executive Director, External Relations


Measures that Matter

• Input Output Outcome Impact• Consistent with organizational mission, goals

and objectives• Integration with program review• Balance customer, stakeholder, and employee

interests and needs• Establish accountability• Collection and use of reliable and valid data


Forces and Challenges

• Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution

• Increasing pressure to maximize use of resources through benchmarking resulting in:– Cost savings– Reallocation


ARL New Measures Projects

• Demonstration project for service effectiveness measures (LibQUAL+™)

• Project to define usage measures for electronic information resources (includes institutional outcomes)

• Identification of measures that demonstrate a library’s contribution to student learning outcomes

• Investigation of role libraries play in support of the research process

• Identification of cost-drivers and development of cost-benefit studies (ILL/DocDel)


User Surveys

• Obtain direct responses to a series of questions from the community surveyed

• Identify user issues, concerns and needs• Measure library performance from the user

perspective, including satisfaction• Acquire quantifiable data that can be statistically

analyzed and generalizeable for the larger population

• Improve or change services• Increase library visibility and marketing• Contribute to broader institutional

assessment/accreditation


LibQUAL+™ Project Goals

• Establishment of a library service quality assessment program at ARL

• Development of web-based tools for assessing library service quality

• Development of mechanisms and protocols for evaluating libraries

• Identification of best practices in providing library service


Survey Instrument

3 scales: Minimum Desired Perceived


Dimensions of Library Service Quality

Affect of Service

Empathy

Information Access

Personal Control

Responsiveness

Symbol

Utilitarian space

Assurance

Scope

Timeliness

Convenience

Library as Place

LibraryServiceQuality

Refuge

Reliability

Ease of Navigation

Convenience

Modern Equipment


Percentages of Acquisitions Dollars Devoted to Electronic Resources

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

1993 1994 1995 1996 1997 1998 1999 2000 2001


ARL E-Metrics Project

Three phases:– Initial Phase (May-October 2000): What do we

know? Inventory of current practices at ARL libraries as to statistics, measures, processes, and activities that pertain to networked resources and services.

– Second Phase (November 2000-June 2001): What can we collect? Identified and field tested an initial draft set of statistics and measures

– Final phase (July 2001-December 2001): What difference does this make? Build linkages to: educational outcomes/impact, research, technical infrastructure


Recommended Statistics & Measures

• Patron Accessible Electronic Resources (R1-3)• Use of Networked Resources & Services (U1-5)• Expenditures for Networked Resources & Related

Infrastructure (C1-3)• Library Digitization Activities (D1-3)• Performance Measures (P1-3)


Characteristics of Each Recommended Measure

• Definition• Rationale• Unit of Measure• Data source• Frequency• Process• Related Issues


Focus on Electronic Resources

• Project COUNTER

• ICOLC

• Measuring Impact (Franklin/Plum)


Cost-Effectiveness

• ILL/DD study

• Technical Services Cost Study

• Activity Based Costing

• Balanced Scorecard

• Baldrige Quality Award


Outcomes

• Institutional

• Learning

• Research


Learning Outcomes

• Develop strategy to involve library in campus assessment activities to demonstrate the value of the library to the learning community– Move from content view (books, subject knowledge) to

competency view (what students are able to do)– Understand learning outcomes of academic degree

programs– Develop curriculum segments through which the library

achieves outcomes

• Information Literacy Competency Standards for Higher Education approved by the Association of College and Research Libraries in January 2000


Standardized Assessment of Information Literacy Skills

• Base Based on outcomes as defined by ACRL information literacy competency standards d on ACRL standards

• Contains items not specific to a particular institution or library

• Is easily administered• Has been proven valid and reliable• Assesses at institutional level• Provides for both external and internal

benchmarking


Research Outcomes(Surrogates)

• R&D expenditures

• PhDs awarded/year

• Post-doctoral appointments

• Research awards per year

• Academy memberships

• Faculty quality rankings


New Measure Challenges

• Resources (i.e., time and money)• Buy-in (commitment to a culture of assessment• Access to individuals to evaluate• Expertise to conduct evaluation• Project management experience• Appropriate benchmarks• Conceptual clarity• Measurement & design requirements• Instrument validity and reliability


Julia C. Blixrud

Assistant Executive Director, External Relations

Association of Research Libraries

21 Dupont Circle, Ste 800

Washington, DC 20036

[email protected]

202-296-2296 ext. 133

202-872-0884 (fax)

202-251-4678 (cell)

www.arl.orgassociation of research libraries assessing library performance: new measures, methods,...

Documents

arl libraries

library service slide

valid data slide

external relations slide

usage measures

library visibility

measure library performance

arl development of web