Putting “Value” in EvaluationBuilding Relevant, Dynamic Statistical Analysis

Presented by Joshua Johnson @ #ULA2013See Josh Blog: www.joshinglibrarian.infoSee Josh Tweet: @JoshinLibrarian

Don’t We Already Use Statistical Analysis?

We’ve been doing this for years;

Librarians are well known as gatherers of statistics and other data. However, we do not always use the information that we gather, preferring instead to point to the numbers themselves as evidence of our work.

Blake & Schleper, “From Data to Decisions…”

Quotable Research/Literature Review

What is the worth of the library in the networked-computer age? How do shifts in use patterns reflect changes in customers’ valuations of library services, and how would customers prefer that library resources be added or reallocated? What benefits are conferred on different types of library customers by their variant uses of … libraries? And, how can those benefits be measured?

Holt & Elliott, “Measuring Outcomes…”

Quotable Research/Literature Review

Librarians may be good at counting, but as a whole the profession is not trained to evaluate and analyze statistics. Librarians offer numbers often as proof of the value of their work with little thought as to whether those numbers really establish anything of value…Library Managers need to think critically about which statistics are useful.

Tim Spindler, “Statistical Analysis Models: Applications for Libraries”

Quotable Research/Literature Review

It doesn’t really matter whether you can quantify your results. What matters is that you rigorously assemble evidence—quantitative or qualitative—to track your progress.

If the evidence is primarily qualitative, think like a trial lawyer assembling the combined body of evidence. If the evidence is primarily quantitative, then think of yourself as a laboratory scientist assembling and assessing the data.

Jim Collins, Good to Great and the Social Sectors

Quotable Research/Literature Review

Examine Current Methods of Analysis

• Are you basing reference staffing decisions on circulation data?• Are you comfortable extrapolating future trends from your current data sets?• Does the data you collect match the services you currently provide?• Is the process by which you collect and synthesize your data easily transferred from one staff member to another in the event of a promotion or retirement?

WHERE DO I START?

Okay, I’ll bite. I’m rethinking my library’s statistical analysis anyway;

Evaluate Your Mission Statement

• Does your mission statement accurately describe what your library does?• What promises are you making in your mission statement?• Which of your services, collections, etc. are related to each promise? • Are you considering adding to your services and unsure if you can handle the change without adding staff?

Evaluate Your Mission Statement

For example, part of the mission statement of the American Library Association:

The American Library Association was created to provide leadership for the development, promotion, and improvement of library and information services and the profession of librarianship in order to enhance learning and ensure access to information for all.

Evaluate Your Mission Statement

Let’s highlight a few of the promises that reasonable people might assume based on the quote:

• ALA members will be better at promotion.• ALA members will be better librarians.• Library patrons served by ALA members will be better served than those served by non-ALA members.• Library patrons served by ALA members will receive better access to information than those served by non-ALA members.

Brainstorm Related Key Indicators

• What factors, practices, personnel influence your ability to make good on these promises?• What physical or monetary limitations could impact your organization’s ability to fulfill promises?• How might you be able to assess the organization’s perceived value or measure your effectiveness at providing services or staffing?• Would the organization be better able to fulfill some promises if it changed the promises it made or cut some services in favor of others?

In this case, the indicators represent daily, measurable tasks performed by each set of employees. I consider the lists of indicators “in process.”

What Do I Mean by Indicators?

TAKE THE PROCESS FOR A SPIN

Now that we’ve talked about the process,

Parsing the Mission Statement• Form groups and select a “scribe.”• Examine the sample mission statement as a group and write as many promises made as possible in the time allotted.• Was it difficult? Was it worth it?

Brainstorming Key Indicators• Stay in the same groups.• Take your sample mission statements and the list of promises your group made in the last exercise and switch them with another group.• Read the mission statement as necessary, but focus on the promises made by the statement.

Brainstorming Key Indicators• Make a list of indicators (services, practices, etc.) that relate to these promises.

• As your group brainstorms, keep in mind:• What factors, practices, personnel influence the organization’s ability to make good on these promises?• What physical or monetary limitations could impact the organization’s ability to fulfill promises?• In what ways might you be able to assess the organization’s perceived value or measure your effectiveness at providing services or staffing?• Would the organization be better able to fulfill some promises if it changed the promises it made, or cut some services in favor of others?

Process Evaluation• How difficult was this process?• How productive was it?• What other documents could be used in addition to mission statements?• What kinds of ideas did it spark for examining your own workplace processes?• How could we streamline the brainstorming process?

What reference indicators might reasonably be added, based on your experience in libraries?

Process Evaluation

WHERE DO I GO FROM HERE?

That was interesting, and perhaps useful, but

Leverage Current Statistical Analyses

• Reuse the data and statistical analysis you already have. You almost certainly keep some form of raw statistical record-keeping or statistical analysis. There’s no need to completely reinvent the wheel.• Track additional data as needed - make sure it is relevant.

Leverage Your Current Statistical Analyses

• Nearly all of the information used in this project was pulled directly from statistics already kept by the library system.

• Excel, for example, will pull information from other Excel files and update it automatically; this makes pulling data for your own projects a breeze.

“FIND & REPLACE” & MACROS SIMPLIFY REPETITIVE TASKS.

Leverage Current Statistical Analyses

• Take advantage of time/labor saving computer programs. Spreadsheets often allow you to use macros and a wide range of other helps.

• The example below contains instructions for copying/updating monthly statistics using the “find & replace” function available in many spreadsheets. Each type of information can be updated in a slightly different way.

Leverage Your Current Statistical Analyses• Be logical - Be certain the picture you paint with statistics is as objective as possible; some statistics are misleading or unethical.• Compare apples to apples - This can be more difficult than you think. Statistics make more sense when you find ways to compare commonalities.

Methods Should Allow Adapdability• Think about how you lay out your information; why organize it over and over?• Will your methods be easily passed to your future successor?

• What libraries “do” in 5-10 years will change, so expect it.

• When planning, leave room to grow, even if it is just a blank space in the sheet.

• Review promises and indicators for relevance and value to ensure the most valuable evaluation of your organization.

Evaluate Methods to Ensure Adaptability

Brainstorming Helps for Key IndicatorsHere are some examples to get you going; they are not exhaustive. Many come from Blake & Schleper (2004).

Quantitative• Circulation statistics (most commonly "check-outs," but also in-house uses, etc.)• Website analytics (hits, unique visitors, duration, etc.)• Subject/date analysis (examining average publication dates in a given subject or collection)• Cost-per-use analysis• ILL requests

Brainstorming Helps for Key IndicatorsQualitative•Comparison of collections to peer organizations•User input

• Questionnaires• Interviews

• Compare holdings to collection development policies

• Comparison of overall services to peer institutions• Print collections• Electronic holdings• eBooks/vendors• Study space• Training/community outreach• Staffing levels

Brainstorming Helps for Key IndicatorsQualitative• Physical examination of collection (wear & tear or dust)

• Anecdotes (patrons' or colleagues' opinions)

QUESTIONS?Are there any

Spindler, T (2009). Statistical analysis models: Applications for libraries. Library Publications. Paper 11. http://docs.rwu.edu/librarypub/11

References & Suggested Resources

Blake, J C & Schleper, S P. (2004). From data to decisions: Using surveys and statistics to make collection management decisions. Library Collections, Acquisitions, & Technical Services 28(4).

Holt, G & Elliott, D. (2003). Measuring outcomes: Applying cost-benefit analysis to middle-sized and smaller public libraries. Library Trends. 51(3). 424-440 <link to article>

Collins, J (2005). Good to great and the social sectors: Why business thinking is not the answer. Boulder, CO: J. Collins. Print.

Presented @ #ULA2013By Joshua JohnsonJosh’s Blog: www.joshinglibrarian.infoJosh’s Tweets: @JoshinLibrarian