discovery or displacement?: a large scale longitudinal study of the effect of discovery systems on...
DESCRIPTION
Levine-Clark, Michael, John McDonald, and Jason Price, "Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage," Charleston Conference, November 7, 2013. Plenary session for Charleston Conference 2013. Authors: Michael Levine-Clark, John McDonald, Jason Price. In this first large scale study of the effect of discovery systems on electronic resource usage, the authors present initial findings on how these systems alter online journal usage by academic library researchers. The study examines usage of content hosted by four major academic journal publishers at 24 libraries that have implemented one of the major discovery systems, EBSCO's EDS, Ex Libris' Primo, OCLC's Worldcat Local, or SerialsSolutions’ Summon. A statistically rigorous comparison of COUNTER-compliant journal usage at each library from the 12 months before and after implementation will determine the degree to which usage rises or falls after discovery tool implementation and address rumors that discovery tools differ in their impact on electronic resource usage.TRANSCRIPT
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online
Journal Usage
Charleston Conference November 7, 2013
Michael Levine-‐Clark, University of Denver John McDonald, University of Southern California
Jason Price, SCELC ConsorJum
“…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/
h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html
Vendor marke5ng
Does implementation of a discovery service impact journal usage?
Web-‐scale discovery services
• Single source for finding informaJon – Books – ArJcles – Local content
• Metadata and/or full text
• Content is pre-‐indexed and/or pre-‐harvested
• Single fast search
ILS
HathiTrust
MLA Bibliography
InsJtuJonal Repository
Publisher Metadata
Discovery Service
An assump5on
• At any given insJtuJon, given a relaJvely stable user base, the total search effort will remain roughly the same. – X students will have Y assignments and Z hours per day to search
– X faculty will publish Y papers and have Z hours per day to search
Discovery services § Will take up an increasing amount of a finite Jme for searching
§ Will draw users from other (more or less efficient) search tools
§ Will alter the overall producJvity of searches (users will find more or less)
§ Will alter the overall efficiency of users (users will access more or less full-‐text)
Prior studies
• Some studies have indicated substanJal increases in usage a]er Discovery implementaJon – DescripJve staJsJcs only – Single insJtuJon studies only
• Some publishers report decreased usage of content – Anecdotal, may affect some and not others
Data collec5on • List of libraries with discovery services
> Searched on lib-‐web-‐cats • Surveyed Libraries
> Discovery service Implemented > ImplementaJon Date (month/year) > Search box locaJon > MarkeJng effort
• 149 Libraries Gave Approval > 24 libraries selected for this phase > 6 for each of the 4 major discovery services
Library demographics • 20 US, 1 each from UK, AUS, NZ, CA • 10 ARL Libraries included • WorldCat book holdings
> Average: 1,114,193 > Median: 1,044,153 > High: 2,665,796 > Low: 298,365
• ImplementaJon dates: > 2010 (3), 2011 (19), 2012 (2)
Dataset
• 24 Libraries
• 4 Discovery services
• 6 Publishers
• 9,206 Journals
• 159,278 ObservaJons
• 141,048 Usable ObservaJons
Methodology Compared COUNTER JR1 total full text arJcle views for the 12 months before vs 12 months a]er implementaJon date
June
2010
Start
Implem
entaJo
n May 201
1
May 201
2 End
Year 1 Year 2
Included implementaJon month in Year 1 to ensure that both periods included an enJre academic year
Collec5ons notes
o Excluded journals that did not have 24 months of COUNTER reporJng
o Limited ability to control for changes in aggregator, backfile access, or expanded holdings
o Outliers removed from analysis
General trends
• VariaJon by insJtuJon within each discovery service
• VariaJon by publisher within each discovery service
• Some publishers saw overall net increase, while some experienced a decrease in usage
Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects
Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level
Start with an exploratory analysis and end with a comprehensive model
ANOVA - Analyzing the data
Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8)
After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other
Average variability due to the factor F-ratio = --------------------------------------------------- Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Overall Average = 8.5
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Overall Average = 8.5
Does usage change vary across publishers?
Publisher (sorted by Mean Change)
Overall Average = 8.9
Does usage change vary across discovery services?
Overall Average = 8.9
Does the affect of discovery service differ across publishers?
Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)
Do the discovery service means differ in the 2 way model?
Publishers (distinguished by color)
4.5
12.3
15.0
3.7
Publishers (distinguished by color)
4.5
12.3
15.0
3.7
Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?
23.8
-3.9
6.8-9.5
Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?
23.8
-3.9
6.8-9.5
Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)
Publishers (distinguished by color)
Full Model – including disc srvc, publisher, and library
ANOVA Model including all three factors
Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions?
Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes
Differential library effect by publisher -- Yes
Interpreta5ons & Conclusions > Analyzing usage is a complex task > No discovery service increased or decreased usage across all libraries and/or all publishers
> Discovery service and publisher as variables on their own were significant predictors of usage change
> InteracJon of Discovery service & Publisher was significant
> Some control needed for no discovery service and for size of insJtuJon.
A plethora of pending possible pursuits • Design & test for effects of:
– Aggregator full text availability – InsJtuJon Size / Enrollment Profile – Publisher Size – Journal Subject – Overall usage trends (Requires Disc Srvc ‘control’) – ConfiguraJon opJons in Discovery services
• Follow-‐up presenta5on at UKSG (April 2014) – Including Control group & AddiJonal libraries – Add AddiJonal variables & further analysis
