niso webinar: keyword search = "improve discovery systems"

NISO Webinar: Keyword Search =

"Improve Discovery Systems"

November 12, 2014Speakers:

John McDonald, Associate Dean, Collections, University of Southern California LibrariesJason Price, Program Manager, Statewide California Electronic Library Consortium

(SCELC)

Steve Guttman, Senior Director of Product Management, ProQuest

Marshall Breeding, Library Consultant, librarytechnology.org

http://www.niso.org/news/events/2014/webinars/improve_discovery/


Differential Discovery: The Effect of Discovery

Services on Journal Usage

NISO WebinarNovember 12, 2014

Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California

Jason Price, SCELC Consortium

…our customers insist that usage of our content decreased after implementation of discovery service “X”.

A publisher told us . . .

https://www.flickr.com/photos/kongping/7192138660/in/gallery-flickr-72157645846953449/

Librarians speculate . . .…of course discovery vendors direct their users to their own aggregated content.

http://www.flickr.com/photos/hypotekyfidler/15012731920

Goals of our research

• Determine whether discovery services impact usage

• Help librarians and publishers understand how their choices impact use

• Help librarians, publishers, and vendors improve the discovery experience for end users

What did we measure?

• Whether there is an effect

• NOT why that effect exists

Caveat 1: Publisher-hosted journals are only part of the picture

eBooks, pBooks, newspaper articles, aggregator journal content, etc.

publisher journal content

The six publishers in this study

Not to scale!

Caveat 2: More usage may not be better!

• Decreased usage might be a sign of greater efficiency– Relevant articles found faster = fewer articles to examine

OR– Fewer articles examined because other relevant content

types found

Does implementation of a discovery service impact usage of publisher-

hosted journal content?

Did Journal Usage Change? (and if so, to what extent?)

• 4 discovery services– 6 libraries in each group

• A control group– 9 libraries that did not implement a discovery

service in this time period• 6 major journal publishers

– 9,206 Journals in the study– 163,545 Observations (Library + Journal)

Participating libraries• 157 asked for permission, 155 granted permission

– 124 from the US, 33 from other English-speaking countries• Has your library used a different discovery service in

the past?– Only libraries answering “No” were selected

• Is your discovery service featured on your homepage?– All participants answered, “Yes, with a search box”

• To what extent did your library market the discovery service at its release?– 4 said “None”, one from each vendor– 12 said “A limited extent”, 2 WCL, 2 EDS, 4 Summon, 4 Primo– 8 said “A significant extent, 4 WCL, 2 EDS, 1 Summon, 1

Primo

Methodology

Compared COUNTER JR1 total full text article views for the

12 months before vs 12 months after implementation date

June

201

0St

art

Impl

emen

tatio

nM

ay 2

011

May

201

2En

d

Year 1 Year 2

Included implementation month in Year 1 to ensure that both periods included an entire academic year

Tota

l Stu

dent

FT

E

Journals by Library & Service

EDS Primo Summon WorldCat Control

Tota

l # o

f Jo

urna

ls b

y In

stitu

tion

Average Journal Usage by Library

Testable Effects

• Discovery Service– Implemented by multiple libraries– Used to find content from all publishers

• Publisher– Accessible in all discovery services– Accessible across all libraries

• Library– Uses content from multiple publishers– Uses only one discovery service

Nested ANOVA Model Results

How does usage change differ across discovery services?

ABB

C

D

Letters indicate statistically significant differences (Tukey multiple comparisons, p < .05)

Does usage change vary across libraries?

Institution (sorted by Mean Change)

Does usage change differ across libraries within discovery services?

How does usage change differ across publishers?

Summary of Results• Discovery Service

– Every service increased usage compared to control – Some services increased usage more than others

• Library– The degree of usage change differed among libraries

using the same discovery service• Publisher

– Usage change differed across publishers:• 1 of 6 publishers saw a significant decrease• 2 of 6 publishers saw significant increases• 3 of 6 publishers saw no detectable change

Conclusions to be avoided…

Our research does not indicate that:a) one service increases usage more than another for

every library or publisher effects vary across libraries and publishers

b) one service is better than another libraries or their users may benefit from increased usage of

other content instead higher usage may indicate lower efficiency

Next Steps• Design & test for effects of:

–Aggregator full text availability–Linking configuration options in discovery services

• Expand pool of libraries• Explore the why?• Other possibilities

–Journal Subject?–Journal age (archive vs current)?–eBook usage?

1. What is the best service: Summon, Primo, EDS or Worldcat? • Our research can’t prove one is better than another, and

usage is only one reason to install a discovery system

2. Why was there so much variation in the effect of implementation across publishers at my institution? Why did some increase and some decrease?

Potential Librarian Concerns (1)

3. What about configuration differences?• This undoubtedly contributes to some variance, so be

careful to review and maintain your configurations and preferences.

4. What about other resource types (ebooks, print books, etc)? • They may be affected, we haven’t studied these.

5. Was it missing metadata that caused the differences?• Maybe, we can’t know for sure, so the publishers and

vendors need to come up with best practices for metadata exchange to ensure everyone is on an equal playing field (encourage participation in NISO ODI)

Potential Librarian Concerns (2)

Potential Publisher Issues (1)1. Are users being directed to Aggregated full text

before publisher hosted full text? • Maybe, but it might be the Library’s configuration that is to

blame.• This could be counterproductive for aggregators.

2. Does implementation of abstract-based discovery reduce the ranking of publisher content that is not indexed (or available full text) in a library’s aggregator databases?• Maybe, but it its up to the Library to decide if that is a

desired effect or not.

Potential Publisher Issues (2)3. Why was there so much variation in the effect of

implementation on change in usage of our content across institutions using the same discovery service? • Because usage can affect the publisher bottom line, this is a

key question. Some publishers are finding it of value to invest in research and engagement of customers at the extremities of change after implementation.

4. How can publishers know what is being done with the metadata they send? • Discovery vendors need to be more proactive in proving

the positive benefits of providing more robust metadata

1. How do we prove we’re content neutral?• Develop best practices that indicate to libraries their

configuration and linking choices.• Allow for independent studies• Make the case for the risks related to bias?

2. Our systems make other resource types (ebooks, print books, etc) more discoverable, does anyone value that? • More research needs to be done to assess effectiveness for

other content types

3. Does increased usage necessarily mean a discovery system is better? • Definitely not! Depends on what a library values.

Potential Discovery Vendor Concerns

Differential Discovery: The Effect of Discovery

Services on Journal Usage

NISO WebinarNovember 12, 2014

Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California

Jason Price, SCELC Consortium

A SINGLE SEARCH BOX IS NOT (nearly) ENOUGH

Transforming Discovery with ProQuest

Steve GuttmanSr. Dir Product Mgmt, ProQuest

Steve GuttmanSr. Director of Product Management,Research & DiscoveryProQuest

Former:MarkLogicMicrosoftAdobe

It’s “Discovery” not “Search”

Christian Reusch, Flickr creative commons

https://www.flickr.com/photos/kriis__xx/

https://www.flickr.com/photos/kriis__xx/

https://creativecommons.org/licenses/by/2.0/legalcode

Discovery is more than a Single Search Box

What does it mean to “go beyond” a single search box?

04/13/2023 35


04/13/2023 36

context


04/13/2023 37

context

suggestions


04/13/2023 38

context

suggestions

direction


04/13/2023 39

context

suggestions

direction

ease of use

Discovery needs to “channel” the Library

04/13/2023 40Nico Kaiser, Flickr creative commons


3 Pillars of Discovery

1.Help users “discover”

2.Institutional enrichment

3.Bring patron insights to libraries

Help Users “Discover”

Chris Harrison, Flickr creative commons

https://www.flickr.com/photos/cdharrison/

https://www.flickr.com/photos/cdharrison/


Discovery encourages exploration

• Auto-suggest

• Topic Explorer pane

• Related search suggestions

• Automated query expansion

• Fluid previews

• Dynamic content spotlighting

• Live Help with integrated chat

• Faceted search

• Responsive Design

Discovery encourages exploration

• Auto-suggest

• Topic Explorer pane

• Related topic suggestions

• Automated query expansion

• Fluid previews

• Dynamic content spotlighting

• Live Help with integrated chat

• Faceted search

• Responsive Design

User Experience Matters!

User Experience Matters!

Summon design tenet:Never take the user away from their results

No modal dialogs!

Reveal info when needed

Institutional Enrichment = Making the library part of Discovery

Rich Grundy, Flickr creative commons


What is Institutional Enrichment?

Suggested Librarian

Libguides, Research Guides

Database Recommender

Bring patron insights to libraries

James Royal-Lawson, Flickr creative commons

https://www.flickr.com/photos/beantin/


Bring patron insights to libraries

Features that make you more efficientFeatures that help libraries understand users and content better





4

4.Integration

DefinitionConceptualization, Funding

DiscoverySearch, Recommen-dations,

Networking

Organization

ConsumingReading, Annotating

Sharing, Collaboration

AuthoringSynthesis, Collaborative authoring

(Pre)pubSubmission, OA prepublication,

hosting

Researcher Workflow

DefinitionConceptualization, Funding

DiscoverySearch, Recommen-dations,

Networking

Organization

ConsumingReading, Annotating

Sharing, Collaboration

AuthoringSynthesis, Collaborative authoring

(Pre)pubSubmission, OA prepublication,

hosting

ResearchCompanion

Pivot

Summon360 Linkebrary

COS

Flow

Flowebrary

Flow

RefWorks,Flow

?

Researcher Workflow

3rd party ecosystem integration

MS WordGoogle Docs

(OA) publishersPeer review systems (Aries)

figshare

Social mediaDropbox

ORCIDVIVO

Transforming Discovery

ACCESS

DISCOVER

USE & COLLABORATE

RESE

ARCH

ERS

LIBR

ARIA

NS

Alan Levine, Flickr creative commons

https://www.flickr.com/photos/cogdog/


04/13/2023 60

LIBRARY RESOURCE DISCOVERY:

Marshall BreedingIndependent Consultant,Founder and Publisher, Library Technology Guideshttp://www.librarytechnology.org/http://twitter.com/mbreeding

November 12, 2014NISO Webinar: Keyword Search="Improve Discovery Systems"

Next Steps

http://www.librarytechnology.org/

http://twitter.com/mbreeding

Talking points

Current state of the industry NISO Open Discovery Initiative Discovery White Paper for NISO

Emergence of Discovery

Online catalogs and specialized research databases

Metasearch Local discovery interfaces Index-based discovery …?

Issues with index-based discovery Discovery services populated through

private agreements Uneven participation Library uncertainty regarding

performance and capability Interest in increased transparency

Update on the NISOOpen Discovery Initiative

Balance of Constituents

Libraries

Publishers

Service Providers

66

Marshall Breeding, Vanderbilt UniversityJamene Brooks-Kieffer, Kansas State University Laura Morse, Harvard UniversityKen Varnum, University of Michigan

Sara Brownmiller, University of OregonLucy Harrison, College Center for Library Automation (D2D liaison/observer)Michele Newberry

Lettie Conrad, SAGE PublicationsRoger Schonfeld, ITHAKA/JSTOR/PorticoJeff Lang, Thomson Reuters

Linda Beebe, American Psychological AssocAaron Wood, Alexander Street Press

Jenny Walker, Ex Libris GroupJohn Law, Serials SolutionsMichael Gorrell, EBSCO Information Services

David Lindahl, University of Rochester (XC)Jeff Penka, OCLC (D2D liaison/observer)

ODI deliverables

Standard vocabulary NISO Recommended Practice:

Data format & transfer Communicating content rights Levels of indexing, content availability Linking to content Usage statistics Evaluate compliance

Inform and Promote Adoption

67

ODI Timeline

Milestone Target Date Status

Appointment of working group Dec 2011

Approval of charge and initial work plan Mar 2012

Completion of information gathering Jan 2013

Completion of initial draft Jun 2013

Completion of final draft Sep 2013

Public Review Period commences Sep 2013

NISO Publishes Recommended Practice June 2014

68

ODI Recommended Practices Metadata elements for content providers

to contribute to discovery service providers

Content providers disclose extent to which they participate with each discovery service

Discovery Service providers disclose what content is represented in index

Discovery services disclose any bias in search results or relevancy relative to business relationships

Discovery services provide use statistics

ODI Standing Committee

Fulfilling recommendation of the ODI that NISO charge an ongoing committee to promote ODI best practices and related issues.

Discussions may include but are not limited to: brainstorming on ways to publicize and educate

the community on ODI answering any support questions checking on status of vendor support liaising with other standards efforts as applicable determining when is an appropriate time to

consider updating ODI

ODI Standing Committee Roster

Laura Morse – Harvard University

Lettie Conrad – SAGE Aaron Wood – Ingram

Content Elise Sassone – Springer Jason Price – SCELC Jill O’Neill – NFAIS Julie Zhu – IEEE

Marshall Breeding – Independent Consultant

John McCullough – OCLC Michael McFarland –

Credo Rachel Kessler – Ex Libris Scott Bernier – EBSCO Steven Guttman –

ProQuest Ken Varnum – University of

Michigan Library

Current State of Resource Discovery Four commercial index-based discovery

services Summon, EDS, WorldCat Discover Service,

Primo Many commercial and open source

discovery interfaces Library Portal products: BiblioCMS, Arena,

Iguana, etc Increasing penetration of commercial

products in academic libraries

Web-scale Index-based Discovery

Search:

Digital Collections

Web Site Content

Institutional

Repositories

…E-Journals

Reference Sources

Search Results

Pre-built harvesting and indexing

Conso

lidate

d In

dex

ILS Data

Aggregated Content packages

Usage-generate

dData

Customer

Profile

Open Access

Public Library Information Portal

Search:

Digital Collections

Web Site ContentCommunit

yInformatio

n

…Customer-providedcontent

Reference Sources

Search Results

Pre-built harvesting and indexing

Conso

lidate

d In

dex

ILS Data

Aggregated Content packages

Archives

Usage-generate

dData

Customer

Profile

NISO Discover White Paper

Advise Discovery to Delivery Topic Committee on possible areas of future interest or activity

Overview of the current state of library resource discovery

Recommendations for next stages of ODI API ecosystem: extend and interoperate Discovery beyond the library Importance of Linked Data on future models of

discovery Extend keyword relevancy to leverage Linked

Data

White Paper Status

Draft submitted to D2D Topic Committee Comments expected through November Review and Finalize Informal concepts: not yet a NISO

product Still working to fill in gaps or missing

topics

Open Access / Open Source

Open source tools exist for discovery Interfaces: VuFind Blacklight

No open access discovery indexes High threshold of expense and difficulty to build

index Platform costs Software development Publisher relations Billions of content items to index and maintain

Opportunities to lower barriers to entry?

Issues and concerns

No open access discovery index Uneven participation despite ODI (too

early?) Inherent limitations of library-centered

interfaces Patrons ignore library-provided tools Interest in discoverability and delivery

via global internet infrastructure Exploration of enhanced discovery

through linked data mechanisms

Discovery White Paper Topics

Integration of Discovery with Resource Management

Varying degrees of association Tightly Coupled: high level of difficulty to

mix discovery and management platforms Alma / Primo Intota / Summon OCLC WorldShare / WorldCat

Loosely Coupled: Kuali OLE / Any discovery Any Management product/ EBSCO Discovery

Service

Integration expectations

Should there be better defined mechanisms for integration

Disclosure of integration support Recommended practices?

Linked data

Not yet a fully operational method for library-oriented content Increasing representation of bibliographic

resources BIBFRAME stands to make great impact

Universe of scholarly resources not well represented

Will current expectations for content providers to make metadata or full text available for discovery expand to exposure as open linked data?

OLD Projects Proliferate

VIAF: Virtual International Authority File Europeana (

http://pro.europeana.eu/linked-open-data ) Library of Congress: subjects and

authorities. (http://id.loc.gov/) National Library of Sweden: LIBRIS

Swedish Union Catalogue Zepheira: BIBFRAME, LibHub … Many others

http://pro.europeana.eu/linked-open-data

http://id.loc.gov/

Tools and Technologies for Linked Data

Easily implemented components and toolkits for keyword-based discovery SOLR Elastic Search

Identify or package similar tools for linked data

Hybrid models

Can index-based search tools be improved through Linked Data Browse to related resources Add additional hierarchies of structure to

search results

Gaps in Discovery Remain

Coverage remains uneven Weak coverage in non-roman languages Systematic coverage of open access

materials Difficulty with known-item and precision

search Some new systems lack traditional catalogs

Inconsistent linking mechanisms

Opportunities for Future Enhancements

Improved ecosystem of APIs Increased use of social and usage data to

improve relevancy and retrieval Non-textual search tools Expansion of discovery scope:

Research data Special collections / Archives

More sophisticated performance statistics Altmetrics

Possible new topics for ODI

Address topics marked out of scope by initial ODI workgroup More conducive to A&I resources Relevancy Data exchange protocols

Initial phase described rather than prescribed transfer mechanisms

High threshold of difficulty remains for new services Interoperability with library resource management

systems Interoperability with university learning

management systems

Discovery beyond Library Interfaces

Improved performance of library content through Google Scholar Same expectations for transparency?

Better exposure of library-oriented content Schema.org or other microdata formats

Better exposure of scholarly resources Open access & Proprietary

Embedded tools in other campus interfaces

The future of Resource Discovery More comprehensive discovery indexes Stronger technologies for search and

retrieval Discovery beyond library-provided

interfaces Linked Data to supplement discovery

indexes

Questions and discussion

NISO Webinar • November 12, 2014

Questions?All questions will be posted with presenter answers on the NISO website following the webinar:


NISO Webinar: Keyword Search = "Improve Discovery Systems"


Thank you for joining us today. Please take a moment to fill out the brief online survey.

We look forward to hearing from you!

THANK YOU

niso webinar: keyword search = "improve discovery systems"

Education

journal usage change

libraries library

service increased usage

service increases usage

higher usage

decreased usage

ebook usage

different discovery