analyzing data, getting results: making it all make sense
DESCRIPTION
Riley, Jenn. "Analyzing Data, Getting Results: Making it All Make Sense." Statewide California Electronic Library Consortium (SCELC) Research Day, March 5, 2013.TRANSCRIPT
Analyzing Data, Getting ResultsMaking it All Make SenseJenn RileyUniversity of North Carolina at Chapel Hill
SCEL
C Re
sear
ch D
ay3/
5/13
2
Evidence-driven decisions are a
powerful guide for library operations.
SCEL
C Re
sear
ch D
ay“The plural of anecdote is not data.”
After a quote with the opposite meaning, by Raymond Wolfiger.
3/5/
13
3Sometimes attributed to Frank Kotsonis.
SCEL
C Re
sear
ch D
ay
“There are three kinds of lies – lies, damned lies, and statistics.”
Mark Twain, perhaps after Benjamin Disraeli.
3/5/
13
4
SCEL
C Re
sear
ch D
ay
Using data for planning library operations
3/5/
13
5
Existence/hours of service points
Materials to buy/license/accept/digitize/keep/preser
ve
Designing web sites and other online
resources
Effectiveness of/satisfaction with procedures/services
Evaluating a pilot service or project
Projecting future expenditures
SCEL
C Re
sear
ch D
ay
Both cost and value are keyALCTS Heads of Technical Services in Large Research Libraries Interest Group, Task Force on Cost/Value Assessment of Bibliographic Control (2010)
Proposes definitions of value for cataloging:
3/5/
13
6
Discovery successUseDisplay understandingData interoperability
Support for FRBR user tasksThroughput/timelinessSupport administrative goals
SCEL
C Re
sear
ch D
ay
Example studies
•By Joyce Chapman, then at North Carolina State University• Benefits of manually enhanced metadata
for images• Comparing effort to utility for specific
EAD elements
3/5/
13
7
See Chapman, Joyce. “Metrics & Management: Cost & value of metadata workflows.” SAA 2011. http://www.academia.edu/1708422/Return_on_Investment_Metadata_metrics_and_management
SCEL
C Re
sear
ch D
ay
Some common analyses
3/5/
13
8
Cost per unit produced
Change over time
Error/problem rate
Predicting impact of a change
Identifying unmet needs
SCEL
C Re
sear
ch D
ay
Back to library scenarios
3/5/
13
9
SCEL
C Re
sear
ch D
ay
Existence/hours of service points• Who is using what and when?• How can we most effectively staff them?• Costs• Staff time• Facilities management costs
• Benefits• Number and type of visitors, and how they use it• Service transactions completed• Specific services used at the location
• Other data to collect• Usage by time of day
• Calculate cost per transaction
3/5/
13
10
SCEL
C Re
sear
ch D
ay
Materials to buy/license/accept/digitize/keep/preserve• Should we acquire, make more accessible, or keep this?• Costs• Initial purchase/license• Ongoing license/maintenance• Staff for
cataloging/processing/digitizing/ingesting/preserving• Software• Hardware/storage
• Benefits• Current and predicted future use• Opportunity for transformative use
3/5/
13
11
SCEL
C Re
sear
ch D
ay
Evaluating a pilot service or project• Is the cost/benefit ratio appropriate?• What is the raw cost?• But it’s not all about cost/benefit:• Is the pilot achieving its aims?• Does this [whatever] do what we thought it
would?• What collateral effects will it have?• Were the assumptions we made correct?
• Data collection will be varied for this task
3/5/
13
12
SCEL
C Re
sear
ch D
ay
Designing web sites and other online resources• A/B testing• User-centered design• Satisfaction surveys with previous
iterations, similar sites, or prototypes•Web stats for previous iterations or similar
sites• Task-based usability testing• Don’t forget the cost of sustaining it once
you have it up!
3/5/
13
13
SCEL
C Re
sear
ch D
ay
Effectiveness of/satisfaction with procedures/services• What parts of our current service are users most and
least happy about?• What are the ineffieciences in our procedure for
[whatever]?• Some data collection ideas• User surveys• Ratio of potential to actual users• Ratio of returning to non-returning users• Error/failure rates• Time from request to delivery• Time tracking during staff activity
3/5/
13
14
SCEL
C Re
sear
ch D
ay
Projecting future expenditures• Equipment• Define its lifecycle• Amortize purchase cost• Add in maintenance costs• Compare to use as context
• Staff• Educated guess at raises, turnover, benefit costs changes
• Consider:• Inflation• Past trends• Upcoming sea changes
3/5/
13
15
SCEL
C Re
sear
ch D
ay
Strategies for getting data that can be
analyzed
3/5/
13
16
SCEL
C Re
sear
ch D
ay
Tracking use
• Circulation• COUNTER/SUSHI• Physical visitors•Web hits• Social media engagement• Attendance at events/sessions
3/5/
13
17
SCEL
C Re
sear
ch D
ay
Tracking time
• Can be effective when collected as a representative snapshot• Options for data collection• Clipboard next to a clock• Spreadsheet• Free time tracking apps•Make it as simple as possible
3/5/
13
18
SCEL
C Re
sear
ch D
ay
Calculating costs
• Staff time• 2080 hours per year is full time• Standard benefit percentages
•Materials (including software)• Initial purchase• Maintenance contracts for big-ticket items• Amortize big costs over time in service• Overhead• Universities typically have standard rates
3/5/
13
19
SCEL
C Re
sear
ch D
ay
Calculating error rates
•Both objective and subjective criteria• Typically best when done as a sample•Consider both automated and manual
means to locate errors for study
3/5/
13
20
SCEL
C Re
sear
ch D
ay
Categorization
• Putting things into like groups• Compare size of groups to one another• Compare effort spent on one group to another• Compare priority/value of one group to
another• Can be done at time of data collection, or
afterwards• Good idea to have some sense of
categories at the beginning of the study
3/5/
13
21
SCEL
C Re
sear
ch D
ay
Calculating benefit
• Change in knowledge or status• Over time• After an interaction• Survey – ask about knowledge level before
and after• Pre- and post-tests• Indirect measures• Number of people reached• Use
3/5/
13
22
SCEL
C Re
sear
ch D
ay
Additional data analysis strategies
3/5/
13
23
SCEL
C Re
sear
ch D
ay
Mechanics
• Code qualitative data to make it processable•Make sure you pick a representative and
consistent sample• Extrapolate based on known data when
you need to• ALWAYS do a sanity check• Spreadsheets are your friend
3/5/
13
24
SCEL
C Re
sear
ch D
ay
More advice
• Context is key• Don’t be paralyzed by a perceived need for
perfection• Know your basic analysis plans before you
collect/identify data• Utilize pilot projects to generate data where
there is none• Use the right tool for the job• Document your assumptions• It’s OK to use “napkin math”
3/5/
13
25
SCEL
C Re
sear
ch D
ay
Get in the habit of collecting data.
It will make your next decision easier.
3/5/
13
26