charles r. mcclure, ph.d. ([email protected]) wonsik “jeff” shim, ph.d

Coalition for Networked Information, San Antonio, November 30, 2001

Information Use Management and Policy Institute at Florida State University

1

Charles R. McClure, Ph.D. ([email protected])Wonsik “Jeff” Shim, Ph.DJohn Carlo Bertot, Ph.D.

Information Institute/School of Information StudiesFlorida State University (www.ii.fsu.edu)

ARL E-Metrics Project Data Collection Manual Institutional Outcomes Proposal Training and Dissemination Concluding Activities Issues and Challenges

The E-metrics Project: Update and Project Completion



2

ARL E-Metrics Project

Project Co-ChairsRush Miller, Univ. of PittsburghSherrie Schmidt, Arizona State Univ.

Project Logistics (ARL)Duane WebsterMartha Kyrillidou

Project Directors (Florida State Univ.) Charles R. McClure, Wonsik "Jeff" Shim,

and John Carlo Bertot



3

Project Participants (24 ARL Libraries)

Alberta Arizona State AuburnChicago Connecticut Cornell Illinois-Chicago Manitoba MarylandU. Mass Nebraska Notre DamePennsylvania Penn State PittsburghPurdue Texas A&M S. CaliforniaVirginia Tech W. Ontario Wisconsin Yale LC NYPL



4

ARL E-Metrics Project Phase One (May-October 2000)

Knowledge Inventory Methods: Surveys, Site Visits, Document Analysis, Phase Two (November 2000-June 2001)

Developing Measures and Testing Methods: Surveys, Site Visits, Focus Group

Interviews, Document Analysis Phase Three (July 2001-December 2001)

Developing Manuals and Tools



5

Need for network statistics and measures To demonstrate the use of digital collections

in order to make a case for continued collection development and support;

To enable libraries to compete for resources with other organizations and/or depts by documenting the range, extent, and impact of library-provided networked resources;

To provide a decision-making framework for library staff, managers, and administrators to determine resource allocation strategies and meet other management needs;



6

Need for network statistics and measures (continued) To allow libraries to compare themselves to

others re: collection and service development, costs, provision of services, and use;

To enable libraries to measure and track internal changes to library library operations as well as uses and users of library resources and services; and

To provide a means through which to measure aspects of effectiveness, efficiency, and quality of library services and resources in the networked environment.



7

Findings from Phase I Libraries are collecting some data, often statistics

related to patron accessible resources and cost of electronic databases.

For use statistics, libraries depend almost solely on vendor reports.

Libraries have little information about the users of networked services.

Data most frequently used for immediate decision makings: licensing contracts, budget requests

Less frequently used to paint a bigger picture of information usage patterns.



8

Findings from Phase I (continued)

Librarians point to problems associated with data collection, especially vendor statistics.

Lack of consistent, comparable and detailed data, problems with interpreting and summarizing data, lack of technology and personnel support, inability to link data to peer comparison, quality of service, and outcomes.

Little organizational infrastructure to support data collection, analysis, and reporting.



9

Development of Statistics and Measures for Field Testing

November 2000 - June 2001 Participants had significant and on-going input into

the development of the proposed statistics and measures.

Data collection instruction manual was also developed to aid field testing.

Field testing conducted during May 2001. A separate field testing of vendor statistics

conducted during June 2001. Manual published October 2001.



10

Measures and Statistics for Research Library Networked Services: Procedures and Issues (Washington DC: ARL)

Complete report and manual at:

http://www.arl.org/stats/newmeas/ emetrics/index.html



11Statistics/Measures Described in the Manual

Patron Accessible Electronic Resources R1 Number of electronic full-text journals R2 Number of electronic reference sources R3 Number of electronic booksUse of Networked Resources and Services U1 Number of electronic reference transactions U2* Number of logins (sessions) to electronic databases U3* Number of queries (searches) in electronic databases U4* Items requested in electronic database U5 Virtual visits to library’s website and catalog (* Tested in the vendor statistics field testing)Expenditures for Networked Resources and Infrastructure C1 Cost of electronic full-text journals C2 Cost of electronic reference sources C3 Cost of electronic books C4 Library expenditures for bib. utilities, networks, and consortia C5 External expenditures for bib. utilities, networks, and consortia



12

Library Digitization Activities

D1 Size of library digital collectionD2 Use of library digital collectionD3 Cost of digital collection construction and management

Performance Measures

P1 % of electronic reference transactions of total reference P2 % of virtual library visits of all library visits P3 % of electronic books to all monographs

Total of 19 Statistics and Measures Described with data collection procedures.

Statistics/Measures Described in the Manual



13

Institutional Outcomes

Definition:

A clearly identified result or end product that occurs as a consequence of individual or combined activities from units at the Institution. These outcomes are a preferred or desired state and clarify specific expectations of what should be products from the institution.



14

Institutional Outcomes: Next Steps

Refine the existing model Integrate model into library planning

and evaluation Encourage libraries to design and

develop the outcomes assessment process at their institution

Submit research proposal on outcomes assessment and the role of academic libraries in that assessment process.



15

Develop Training Modules

Re-work the statistics and measures manual to be a “stand alone” product

Develop 2-3 instructional modules that support the manual to help train the trainers

- Importance of evaluation, statistics and measurement - Explanation and how to use the statistics and measures Training workshop for member libraries

to be held January 19, ALA New Orleans



16

Improving Organizational Resources to Support Evaluation Efforts

Need for organizational infrastructure of staff knowledge, staff time, resources, and organization for evaluation

On-going program of evaluation and assessment -- not a one-shot

Continue working directly with vendors and ICOLC

Organizing and coordinating efforts among vendors, ARL, ICOLC, NCLIS, and others.



17

Issues and Challenges

Incomplete, non-comparable data from multiple sources in the networked environment make it difficult to collect and use data. But we must start now. The statistics and measures proposed in the E-Metrics project will evolve.

Library community needs to work with

database vendors, standards organizations on data definitions and reporting procedures. International collaboration should also continue.



18

Issues and Challenges (continued)

The degree to which libraries will be able to collect these data and report them is linked to the amount of resources they can commit.

Libraries have different operating environments, different data needs/expectations. They need to think about which statistics and measures would be best to use, strategically and politically, in their own settings.



19

Issues and Challenges (continued)

Conducting research for 24 ARL libraries with differing situations and objectives.

Coordinating and extending next steps for the New Measures initiative -- priorities and strategies?

Linking assessment approaches to work together and not as stand alone strategies.



20

Inputs Outputs

Outcomes

Service Quality

Invested Resources

Measured through library budget, personnel, otherCaptured through library statistics

Question:What resources are we putting into the services?

Invested Resources Results

Measured through library MIS, organizational charts, otherCaptured through library statistics

Question:What services do we get out of the invested resources?

Anticipated service provider impacts of servicesUser perceived impacts of library services

Question:What difference did the services make?

Anticipated service provider expectations of the quality of library servicesUser expectations (anticipated and perceived) of library servicesGaps between user and service provider expectations and perceptions

Question:How good are the services?

Putting the PiecesTogether



21

Remember: Multiple Approaches are Available to Assess Networked Services

The inputs-outputs approach (as used in this project) for statistics and performance measures

- Quantitative - Qualitative LibQual Service Quality Quality Standards Educational and Institutional Outcomes And others...



22

Importance of the E-metrics Project

19 new statistics and measures have been developed and field tested to describe use, users, and uses of networked resources

ARL members have a better understanding of the resources and organizational concerns needed to conduct effective evaluation efforts

We have better knowledge of the role of academic libraries in Institutional Outcomes

The project provides a springboard for additional research in measurement.



23

Comments and Questions?

Charles R. McClure Francis Eppes Professor, and Director, Information Institute Florida State University

[email protected] http://slis-two.lis.fsu.edu/~cmcclure/

http://www.arl.org/stats/newmeas/emetrics/index.html

charles r. mcclure, ph.d. ([email protected]) wonsik “jeff” shim, ph.d

Documents