two programs, one service

04/12/23 E. Edwards & S. Palacios-Wilhelm 1

Two Programs, One Service:Recursive Assessment in the Gelman Library System's IM Reference ServiceAmerican Library Association Annual Conference2008 MERS Virtual Poster Session

Sarah Palacios-WilhelmReference & Instruction LibrarianEckles Library

Elizabeth EdwardsReference/Technology LibrarianGelman Library


Two Programs, One Service: 2008 MERS Virtual Poster Session

Abstract

This virtual poster session presents the recursive implementation process utilized by the George Washington University’s Gelman Library System in the context of its successful IM reference service. In this presentation, librarians from two campus libraries discuss the advantages of using pilot testing in a recursive cycle to inform testing, implementation, and evaluation of services, software, training, and assessment models. This process has resulted in a climate that encourages innovation and decreases anxiety amongst service providers. Specific examples of recursive implementation are explored, as well as challenges faced throughout the process. The presentation closes with a list of lessons learned that will inform institutions that wish to follow a similar model.



Background

In 2005, the Gelman Library System pilot tested the delivery of reference services via IM. This pilot test began at Eckles Library, which serves a primarily residential and freshman population. Since its inception, this service has expanded to include Gelman Library, which serves the entire University population.

During the 2006-2007 school year the IM reference service fielded 1,458 questions, and was nominated for a campus-wide Service Excellence Award. The success of this service is in no small part a result of the innovative testing and evaluation model that integrates the two campuses in a recursive cycle.



The George Washington University

Located in the heart of the Foggy Bottom neighborhood of Washington, D.C.

Comprised of nine schools: Columbian College of Arts & Sciences School of Medicine & Health Sciences Law School School of Engineering & Applied Science Graduate School of Education & Human Development School of Business Elliott School of International Affairs School of Public Health & Health Services College of Professional Studies

High research activity according to Carnegie ratings. 17,494 students enrolled full-time in Fall 2007. 7,092 degrees conferred in 2007, including 264 Doctorates.



The Gelman Library System

Gelman Library System

Gelman Library Eckles Library Virginia Campus Library



Gelman Library

Main library for urban campus 30,000 visitors per week Reference desk staffed by 17

librarians with assistance from student workers

IM service established in Fall 2006

Eckles Library

Small library serving residential campus

7,000 visitors per week Single service point staffed by 2

librarians, 3 staff, and 7 students IM service piloted in 2005,

established in 2005



Models of Implementation

The “Normal” Implementation Process

The Recursive Implementation Process



The "Normal" Implementation Process



The "Normal" Implementation Process

Each phase has a defined beginning and end.

Each phase occurs in order.

Pilot programs lead directly to implementation.

Test groups are organized as necessary, but do not persist after the implementation process is complete.



The "Normal" Implementation Process: An Example

A problem is identified: patrons are eating in the library, which is in violation of library policy.

A solution is proposed: create a space where patrons can eat without violating library policy.

Pilot testing: a student lounge is opened, and patrons attempting to bring food into the library are directed to this new space.

Implementation: the student lounge is used for a special event, drawing attention to the new space.

Evaluation: the problem of patrons eating in the library has not decreased. A new solution must be devised.




In the recursive implementation process, testing, implementation, evaluation, and reflection are cyclical, with each step returning to the pilot setting.

Problems can be identified at any location, and solutions are proposed based on previous experience, understanding of patron needs, and possible resources.

Pilot testing occurs with a selected population, informed by feedback from ongoing brainstorming and evaluation from both locations.

Based on success in the pilot phase, implementation expands to include both locations, with the next phase of brainstorming or pilot testing beginning at the smaller location.

Evaluation is an ongoing process, incorporating both formal and informal methods of assessment.



The Recursive Model in Practice

Pilot Testing

Service Hours

Staffing Models

Training

Assessment

Platform Testing



The Recursive Model in Practice: Pilot Testing

Experience with virtual reference collaboratively staffed with other local academic libraries resulted in hesitation about new forms of electronic reference.

Librarians at Eckles, in an attempt to provide enhanced service for their small, tech-savvy population, recognized the potential of IM for service delivery and experimented with IM reference with limited hours.

Following the positive reception at Eckles, the IM service expanded to Gelman with limited hours, with ongoing evaluation occurring at both locations.



The Recursive Model in Practice: Service Hours

After the initial pilot, Eckles expanded service hours.

Eckles' statistics revealed peak hours, resulting in an expansion of Gelman's service hours.

Demand for Eckles’ service decreased as a result of Gelman's expansion.

Statistics allow both campuses to anticipate service needs and inform discussions of staffing models.

020

406080

100120

140160180

8am 9am 10am 11am Noon 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm

Total IM Transactions by Time AY 2006-2007 Total: 1458

Gelman

Eckles



The Recursive Model in Practice: Staffing Models

Eckles' service was initially staffed by one librarian, with additional support from student workers.

Gelman's service was initially staffed by librarians from individual desks.

Gelman’s distributed model proved problematic, so service moved to librarian staffing at the reference desk, with assistance from student workers.

Eckles' service moved to triage staffing model using student workers, with librarians available for questions.

Gelman is exploring new staffing models in the context of moving to a single service point and incorporating new departments into the IM reference service.



The Recursive Model in Practice: Training

Technology-adept student workers at Eckles complete reference service training modules developed for new Gelman librarians.

Gelman librarians receive training on new technologies first piloted at Eckles.

The exchange of training modules draws on the expertise and improves the skills of service providers at both campuses.

GelmanInfo IM Transactions by Question TypeAY 2006 - 2007

Total: 1,458

55%

41%

4%

Reference

Directional

Other

EcklesInfo IM Transactions by Question TypeAY 2006 - 2007

Total: 1,458

64%

32%

4%

Reference

Directional

Other



The Recursive Model in Practice: Assessment

Usage statistics from both locations are used to inform how the services are tested, implemented, and expanded.

Question-type statistics from both locations are used to inform staffing models and training.

Ongoing qualitative transcript analysis is used to inform service definition, training, and staffing models.

"Secret Shopper" evaluation used in student worker training at Eckles identified larger training needs, which will inform future training strategies that will be used at Gelman.



The Recursive Model in Practice: Platform Testing

Eckles adopted Trillian and Pidgin/GAIM.

Gelman adopted Trillian.

Eckles abandoned Pidgin/GAIM for Trillian.

Eckles tested and adopted Trillian Pro.

Gelman adopted Trillian Pro.

Platform testing remains an ongoing project, with Meebo and other widgets under consideration for implementation.



Challenges of the Recursive Model

Scalability

Service Providers

Patrons

Trying new things is always hard



Challenges of the Recursive Model: Scalability

Organizational challenges differ between locations.

Recursive cycle does not always flow evenly.

The larger the organization or the greater the bureaucracy, the more staff time and energy will be required to encourage and implement innovation.



Challenges of the Recursive Model: Service Providers

Service providers vary in technology skills, attitudes towards service, and interpretations of service requirements.

Training models vary based on staff needs, which will affect the service provided at each location.

Staffing models directly affect the quality of service provided.

Technology in different contexts may affect providers’ ease of use.

Familiarity with appropriate resources will affect the service provided.



Challenges of the Recursive Model: Patrons

Even the most representative pilot group will be different than the full population of both sites.

Patrons may be less patient with the new service or changes in service.

The service may not be able to meet the expectations of patrons used to more personalized service in the pilot location.

Patron usage may shift away from the pilot location as services expand, which may result in a smaller population for future testing.



Challenges of the Recursive Model

Trying new things is always hard.



Lessons Learned from the Recursive Model

Effective Pilot Groups

Identifying Pilot Groups

What Not to Do

Reflections




Effective Pilot Groups

Create a safer environment for testing, which inspires innovation.

Include both providers and patrons.

Facilitate a two-way conversation.

Provide both positive and negative feedback.

Allow your audience to engage in the testing process.

Work collaboratively: with you rather than for you.




Identifying Pilot Groups

Look for groups with a population that changes regularly: Freshmen or graduating seniors Student workers

Groups with a small population: Special or departmental libraries Branch libraries Cohort-based programs




What Not to Do

Don’t use your staff as a pilot group. Needs are not the same as those of patrons. Expectations are not the same as those of patrons. Feedback will reflect service provider concerns and biases rather than

genuine patron experience.

Don’t forget about staff input. While not representative of patron experience, the service provider’s

perspective is critical to any new service.




Reflections

Innovation encourages cycles of evaluation and improvement in other service areas.

Challenges or failures will not derail other library services when done in parallel.

Established framework for comparison and improvement builds an expectation of ongoing evaluation.

Stagnation is prevented because something interesting is always happening.




You Can Do It!

The process has been challenging, but has proved to be both successful and rewarding.

Creating an environment that facilitates innovation also sparks excitement among service providers, ensuring that library

service remains dynamic and relevant for the patron.



Contact Information

Sarah Palacios-WilhelmReference & Instruction Librarian

Eckles [email protected]

AIM/Google Talk/ Yahoo!: sarahpwilhelm

Elizabeth EdwardsReference/Technology Librarian

Gelman [email protected]

AIM: leepforward

AIM/Google Talk/Yahoo!: GelmanInfo and EcklesInfo



Presentation prepared for:

2008 American Library Association Annual Conference

MARS Management of Electronic Resources and Services (MERS) Committee

Virtual Poster Session: Evaluation Virtual Reference Services