waves of numbers: how to use and interpret benchmarking tools presentation given at mla-san diego,...

Waves of Numbers: How to Use and Interpret Benchmarking Tools

Presentation given at MLA-San Diego, May, 2003Revised as a PowerPoint Show for instructional purposes

byRosalind. F. Dudden, AHIPHealth Sciences Librarian

Gerald Tucker Memorial Medical LibraryNational Jewish Medical and Research Center

Denver, CO

MLA Benchmarking Network 2May, 2003

Waves of NumbersThis slide show will move forward at a steady pace.

Text will fade in and out.

I have moved my words into the slide to mimic my speech.

I hope it works out for you.

Let’s start the show!

You can go faster using the right arrow, repeat with the left arrow.

The show lasts 12 minutes.


Waves of NumbersIn today’s world, one is bombarded with numbers,

waves of them coming at you.

I thought this title was appropriate since the Benchmarking Network has just provided you with over 46 million (46,089,864) new numbers.

You can use these numbers to defend or improve your library service.

My subtitle mentions use and interpretation of these numbers and I am attempting to cover both.

It is much easier to show people how to use the tools than it is to interpret them.


Background

From December 2001 to March 2002, MLA members were encouraged to provide data on library services and resources using a Web intake form.

The data was edited.

Tables representing the aggregate data of 344 hospital libraries were developed and made available to members on MLANET in September, 2002

In March 2003, an interactive site became available to survey participants on MLANET. (For a fee to all members)


MLA Benchmarking Network

Outcomes Team, 00-03 Rosalind Dudden, Chair

Statistical Consultation Jeff Magouirk,

Dept of Bio-statistics, National Jewish Medical and Research Center, Denver, CO

Jim Mills, Jim Mills Engineering Services, Denver, CO

Data Collection and Report Programming MLA Headquarters team, lead by Kate Corcoran

Benchmarking Implementation Task Force, 00-03 Debbie Rand, Chair

Benchmarking Network Task Force, 96-99 Bernie Todd Smith, Chair

Let’s give some credit to all these hardworking MLA volunteers, Right up front!


MLA Benchmarking NetworkSeveral people have asked:

Why do you want to know this? or What is your question?… My response to Why:

To have data to present if I am asked by the administration to prove that the library operations are similar to other libraries of comparable size.

orTo improve services through process improvement



My response to “What is your question?” is that everyone has a different question and we need to be prepared to answer it.

One question is: What do librarians do? For instance, you could ask

“On average, how many monographs do hospital libraries circulate?”


What do we do? On average that is…

Problem: Very large/very small libraries distort the average.

•Number of Responses

•Mean •Median •Minimum •Maximum

•Quartile 1

•Quartile 2

•Quartile 3

•Quartile 4

To find an average, one could divide the data by the size of the library and then find the average, or mean, of each group.


Annual Circulation

qualified answers

mean median third quartile

All 242 1,596 839 1,918

! But I have 4 FTEs in my library and I circulate 2,835!

! But I work in a 1 person library and I circulate 471!

Problem: Very large/very small libraries distort the average.

The answer for all 242 libraries that responded:

The outcomes team developed a system of tables to display the data to take this variation into account.


Vocabulary - ParametersParameter

A numerical value describing a characteristic of a population

Hospital parametersPhysicians (pa05)Hospital FTE (pa06)Patient Discharges (pa07)Outpatient Visits (pa08)Staffed Beds (pa09)Patient Admissions (pa10)

Library parametersTotal Library FTE (a01)Total Expenditures (a21)

12 parameters of size are used.

First we identified parameters of size for a hospital, training programs, and the hospital library itself.


Distribution of a ParameterNumber of libraries in each parameter range for Hospital FTEs:

number in range

% inrange

Range 1 : Hospital FTE (pa06) : 0 to < 500 10 3.0% Range 2 : Hospital FTE (pa06) : 500 to < 850 30 9.0% Range 3 : Hospital FTE (pa06) : 850 to < 1350 63 19.0% Range 4 : Hospital FTE (pa06) : 1350 to < 1825 65 19.6% Range 5 : Hospital FTE (pa06) : 1825 to < 2600 63 19.0% Range 6 : Hospital FTE (pa06) : 2600 to < 4250 62 18.7% Range 7 : Hospital FTE (pa06) : 4250 to < 8000 30 9.0% Range 8 : Hospital FTE (pa06) : 8000 and up 9 2.7%

Number reporting = 332

Ranges 1 and 8: Extreme Outliers, top and bottom - 2.5%Ranges 2 and 7: More Outliers , top and bottom - 10 %Ranges 3 to 6: Remaining 75% divide into Quatriles

The parameters were distributed over 8 ranges to account for the outliers but not eliminate them. It is an acceptable statistical method to eliminate outliers but these very large and very small hospitals need a chance to benefit from the statistics. The middle ranges need more comparable statistics. No outliers were eliminated from the parameters.


Vocabulary - MeasuresExamples of the 73 measures of library activity collected, as developed by the Benchmarking Content Team.

For each measure, this data is reported:

NumberMeanMedianThird QuartileMaximum Minimum

Reference Questions (ps01)Mediated Searches (ps02)Educational Program Sessions (ps04)Monographs Circulated (ps06)Item Received from Outside Sources (Borrows) (ps08)Item Sent to Outside Sources (Lends) (ps09)Expenditures - Monographs (a14)Expenditures - Print Serials (a15)Print Monographs (ts01)Current Print Serials Titles (ts02)Print Serials Titles/Electronic Full-Text Access (ts03)Serials Titles/Electronic Full-Text Access Only (ts04)

Measures Developed:

The third quartile or 75% mark, was recommended for use in TQM or QI programs.


Measures - by - Parameters

You put these two concepts together: number of ILL borrows (ps08) by number of hospital FTEs (pa06)

You produce a table that looks like the next slide.

Note that the each of the 4 middle parameters have about 60 libraries.

The total, or “All,” line shows 311, which means that 311 libraries out of 344 participants answered BOTH questions, how many FTEs and how many ILL borrows.


Number of ILL borrows (ps08) by Number of hospital FTEs (pa06)

Hospital FTEs qualified answers


maximum minimum

Range 1: 0 to 499 10 389 244 467 1,500 72

Range 2: 500 to 849 27 905 808 1,048 2,647 45

Range 3: 850 to 1,349 59 1,010 800 1,217 4,200 96

Range 4: 1,350 to 1,824 62 1,282 1,100 1,573 4,400 216

Range 5: 1,825 to 2,599 61 1,726 1,314 2,000 6,640 156

Range 6: 2,600 to 4,249 58 2,206 1,617 2,987 7,026 300

Range 7: 4,250 to 7,999 26 3,038 2,836 4,155 6,800 353

Range 8: 8,000 or more 8 6,298 7,031 9,461 10,697 401

All 311 1,704 1,200 2,002 10,697 45


How many tables are there?12 Parameters

Multiplied by

73 Measures

876 Tables on the MLANET Members Only Website

Available to all MLA members! Each table has 6 columns and 9 rows 12 x 73 x 6 x 9 = 46,089,864 numbers

Which table is going to prove your case, or inspire you to improve you system?


Demonstration

To demonstrate the Benchmarking tools on the MLA Website, what follows are some slides and screen shots from the Website.First we need a question.And we need to know how the answer is going to be used.You can often find an answer to a question, but to interpret the answer, one usually needs to know why it is being asked.


Question:

How does the number of Interlibrary Loan Borrows relate to the number of : Physicians Hospital FTEs Training Positions?

Which is the best parameter to use when describing my library to the administration?Which parameter, if it changes, will impact the library workload?


Benchmarking Access• Get out your membership card • Use Internet Explorer as a browser• Go to URL: http://MLANET.org

Click on Members Only

Enter UserID

Enter Password


Benchmarking Access

Choose Benchmarking Network Note other statistical surveys:

MLA/Hay Group Study Salary Survey


Benchmarking Access

Interactive Site

Aggregate Tables - First demonstration

Choose -->


Benchmarking AccessAggregate Data

Collected December 2001 to March 2002

Composite Hospital Library Tables Index to the tables

Explanation of Statistics UsedIn-depth explanation of the tables

MethodologyDescription to use when asked how the figures were gathered

Worksheet of 2001 Benchmarking QuestionsTo review to see what data was gathered

ChooseResources available to you:


Choose a measure

• Administrative Services Measures (Financial), by

• Administrative Services Measures (Staffing and Other), by

• Public Services Measures, by

• Technical Services Measures, by

• Special Services Measures, by

Composite Hospital Library Tables

This page is divided into five parts;We assume that ILL questions are under “Public Service Measures” and scroll down to that area.We get a list that looks like the next slide.


Choose a measure

This screen shot shows that the tables on the page selected will show the number of borrows.


Worksheet of 2001 Benchmarking Data

Public Service MeasuresResource Use•PS07. Interlibrary loaning/borrowing: indicate whether or not

•your library is an official LOANSOME DOC provider. •PS08. Indicate the approximate NUMBER of items

•OF ALL TYPES your library borrows or receives •from outside sources annually (include ILL AND •commercial document delivery services.

•PS09. Indicate the approximate NUMBER of items •OF ALL TYPES your library lends or sends •to outside sources annually.

You could also have found out which section the question was in by looking at the Data Worksheet.


Choose a parameterTo find the data on these three parameters you need to link

into each parameter separately. These are the twelve parameters, 6 for hospital size, 4 for

teaching hospitals, and two for library size.


To Review the Question:

How does the number of Interlibrary Loan Borrows relate to the number of : Hospital FTEs Physicians Training Positions?

Assumptions Hospital FTEs

1500 Physicians

600 Training Positions

20 Borrowed items

1,245


To Review: Choose a measure

1. Administrative Services Measures (Financial), by

2. Administrative Services Measures (Staffing and Other), by

3. Public Services Measures, by

4. Technical Services Measures, by

5. Special Services Measures, by


To Review: Choose a parameter

Number of Physicians in the Hospital (pa05) Number of Hospital Full Time Equivalent (FTE) Employees (pa06) Number of Patient Discharges Annually (pa07) Number of Outpatient Visits Annually (pa08) Number of Staffed Beds (pa09) Number of Patient Admissions Annually (pa10) Number of Postgraduate Training Positions (pa11) Number of Medical School Clerkships (pa12) Number of Nursing Student Positions (pa14) Number of Allied Health Student Positions (pa16) Number of Total Library FTEs (a01) Total Annual Expenditures for the Library (a21)


Interlibrary Loan Borrows by Hospital FTEs

The average for this hospital (Hospital FTEs = 1500) is 1,282.


Interlibrary Loan Borrows by Hospital FTEs

Range Hospital FTEs qualified answers


max min

1 0 to 499 10 389 244 467 1,500 72

2 500 to 849 27 905 808 1,048 2,647 45

3 850 to 1,349 59 1,010 800 1,217 4,200 96

4 1,350 to 1,824 62 1,282 1,100 1,573 4,400 216

5 1,825 to 2,599. 61 1,726 1,314 2,000 6,640 156

6 2,600 to 4,249 58 2,206 1,617 2,987 7,026 300

7 4,250 to 7,999 26 3,038 2,836 4,155 6,800 353

8 8,000 or more 8 6,298 7,031 9,461 10,697 401

All 311 1,704 1,200 2,002 10,697 45

311 libraries answered both questions.

The middle 4 ranges have 58 libraries and over.

The merged data of 58 libraries gives the answers a high confidence level. The librarian could use the mean or median.

When the data is spread as widely as this, from 216 to 4,400, generally the median is a more representative number.

In her range of Hospital FTEs, 1500, the median number is 1,100 borrows.


Collecting the DataRemember the librarian wants to compare three tables. She can do this visually, or print them out, or she can copy them

and put them in a spread sheet.The next slides show how to export the data to Excel.


Moving data to a SpreadsheetUsing Internet Explorer-- Highlight the table and Copy


Moving data to a SpreadsheetOpen up an Excel Spreadsheet-- Put your cursor in the first cell-- Paste

After you paste the three tables into Excel, you can move the data around to produce comparative tables.


Working in a SpreadsheetInterlibrary Borrowing

010002000300040005000600070008000

Range

1

Range

2

Range

3

Range

4

Range

5

Range

6

Range

7

Range

8

Nu

mb

er

of

Bo

rro

ws Number of

physicians

Number of hospitalFTEs

Number ofpostgraduatetraining positions


Median for Each Range

Number of physicians

N=57

Number of hospital FTEs

N=62

Number of training positions

N=42

Range 1 322 244 1,162

Range 2 900 808 861

Range 3 993 800 1,224

Range 4 1,200 1,100 1,392

Range 5 1,100 1,314 1,622

Range 6 1,280 1,617 1,536

Range 7 2,436 2,836 3,218

Range 8 5,515 7,031 6,136

all 1,168 1,200 1,439

This is the data that produced that chart on the previous slide.


Interpretation - The difficult part! - I may be wrong!

Question (review): How does the number of Interlibrary Loan Borrows

relate to the number of Hospital FTEs, Physicians, Training Positions?

Interpretation: Given that the training program figure was not only

higher but closer to the library’s own figure of 1245, any personnel cuts or additions in that program would most likely impact the workload more than the other two parameters.


Benchmarking AccessPart 2 - Interactive Site

Go to URL: http://MLANET.org


Benchmarking Access

Choose -->


Benchmarking Access

This takes you to a page where, if you are a participant, you can login.

Any MLA member can buy access to the site.


Step One: Demographic Selection

Benchmarking Network Report Selection

You can chose to match your fiscal year to the results.

You can choose to use only your state or region.

But if you are from Utah, only one person participated and that person was you.

But if you are from the Midcontinental region, 29 libraries participated.



Step Two: Select One or More Criteria Similar to Your Institution

You can chose to look as system data . 99 hospital classified themselves as systems.

If being, or not being, a teaching hospital is important to your operation, this is the only place to choose this option.

It was not possible to choose this parameter on the tables.



Teaching Non-Teaching

225 102

An example of output from the interactive tables.

By choosing teaching hospital, you are automatically limiting your retrieval to less than 225 hospitals.



Step Two: Select One or More Criteria Similar to Your Institution

Physicians

Range 5:

500 to 699

Hospital FTEs

Range 4:

1,350 to 1,824

Training positions

Range 3:

15 to 29

By choosing the parameter for training positions, we are limiting ourselves to teaching hospitals, so we chose “I don’t care.”

We scroll down and choose the three parameters we want.

Here we have chosen range 5 for number of physicians.

We would then scroll down to choose range 4 for number of hospital FTEs and range 3 for number of training positions.



Step Three: Data Category Selection

Here you select your set of measures.

All the measures for each section are displayed on the results page.

The results page is too long to display in a slide show.


Benchmarking Network Report Results

• Based on your selections, there were 3 institutions that matched your criteria.

• If fewer than five institutions have met your criteria, they are not identified here, for confidentiality reasons.

• To increase the number of institutions matching specific criteria you choose, go back to the selection page and use fewer criteria.



Total Number of items received from outside sources annually (borrows) ps08

Mean (average)

Median Third Quartile (75%)

Maximum

904 1245 1128 •1245

With the three hospitals, one of them must be you, so you are comparing yourself to only two other hospitals.

With only three results and the median and maximum the same, one can hypothesize that two libraries are reporting 1245 and one fewer than 904.

So you go back to the selection page.



Physicians

Range 5:

500 to 699

Hospital FTEs

Range 4:

1,350 to 1,824

We chose all teaching hospitals instead of a parameter for number of training positions.

We scroll down and choose the other two parameters we want.

Here we have chosen range 5 for number of physicians.

We would then scroll done choose range 4 for number of hospital FTEs.



Based on your selections, there were 9 institutions that matched your criteria , including your own (marked by an asterisk). They are: 1 Pomona Valley Hosp. Medical Ctr. Pomona, CA2 Presbyterian Intercommunity Hospital Whittier, CA3 St. Vincent Hospital Worcester, MA4 Durham Regional Hospital Durham, NC5 Medical Center at Princeton Princeton, NJ6 Mercy Fitzgerald Hospital Darby, PA7 The Westerly Hospital Westerly, RI8 Primary Children's Medical Center Salt Lake City, UT9 Waukesha Memorial Hospital Waukesha, WI

This time you get nine libraries from around the country. They are similar in size but do not necessarily have the same number of training positions.

From this list and others that you produce, you could select benchmarking partners to do process improve projects, as described in the Fact Sheet available on the MLA Benchmarking Website.




Teaching Hospitals; Physicians - Range 5; Hospital FTEs - Range 4

Mean (average)


Max

1325 1245 1267 •3210

N=9

In these results, even with the large maximum, the mean and median are not that far apart so here the median again shows that your number of borrows compares of others. It is by chance the same number.

Nine teaching hospitals with the same range of Hospital FTEs and Physicians have data comparable with your activity.

Unfortunately, the chart does not work out as a visual.




All Hospitals; Physicians - Range 5; Hospital FTEs - Range 4

Mean (average)


Max

1294 1260 1318 •3210

N=12

If you go back to the interactive site and try again, here are some different results.

This time you choose all hospitals to see what the result will be.

This time, 12 hospitals with the same range of Hospital FTEs and Physicians have data comparable with yours activity, but slightly higher.


Interpretation - The difficult part! - I may be wrong!

Given 9 teaching hospitals with the same range of numbers of hospital FTEs and physicians, the number of borrows was similar to ours.

Given 12 hospitals, teaching or not, the indicators are higher.

Still use the Training Positions as an indicator


Conclusion

Have your question in mind all the time.

Know what questions were asked.

Take you time surveying the aggregate tables that answer your question.

Use the interactive site to refine your parameters of size.

Use the interactive site to identify Benchmarking Partners for process analysis.


Assessment and Benchmarking Special Interest Group

First Meeting

Mon, May 5, 2003

Co-Conveners:Linda Garr Markwell [email protected]

Lyn Dennison [email protected]

1. Provide a forum for exchange of knowledge about assessment, benchmarking and outcome measures.

2. Promote the exchange of ideas and experience in implementing assessment, benchmarking, and outcome measures.

3. Collaborate with Sections to encourage MLA annual programming relevant to the interests of this SIG and promote professional development for librarians in this area.

waves of numbers: how to use and interpret benchmarking tools presentation given at mla-san diego,...

Documents

mla benchmarking network7may

mla benchmarking network5may

mla benchmarking network2may

mla benchmarking network6may

mla benchmarking network3may

mla benchmarking network4may

mla benchmarking network8may

mla members