user-centered evaluation of a discovery layer system with google scholar
DESCRIPTION
Presentation at the Human-Computer Interaction International (HCII) 2013 ConferenceTRANSCRIPT
![Page 1: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/1.jpg)
User-Centered Evaluation of a Discovery Layer System with Google Scholar
Tao ZhangDigital User Experience SpecialistPurdue University [email protected]
Thursday, July 25, 13
![Page 2: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/2.jpg)
Library Website is still Important
Breadth of scholarly resourcesCredible, highly relevantGuaranteed full text access
Thursday, July 25, 13
![Page 3: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/3.jpg)
User Expectation
“The problem with the databases is that you can’t content yourself with just searching one place, like you can with Google. It is hard for the students to differentiate between the different databases.”
Akselbo et al., The Hybrid Library: From the Users’ Perspective, 2006
Thursday, July 25, 13
![Page 4: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/4.jpg)
Thursday, July 25, 13
![Page 5: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/5.jpg)
Thursday, July 25, 13
![Page 6: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/6.jpg)
Discovery Layer System - Ex Libris Primo®• Uni"ed search interface for (almost) all library resources
• Catalogs, databases, e-journals, digital collections, and repositories
• Increased discoverability
• Easier, faster, and more efficient search interface
• Faceted browsing/"ltering of search results (e.g., by format, date, subject, topic, etc.)
• Record grouping (e.g., multiple editions or versions of the same material)
• Consistent “Get it” option
Thursday, July 25, 13
![Page 7: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/7.jpg)
Thursday, July 25, 13
![Page 8: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/8.jpg)
Study Goals• Compare users’ subjective assessment of search
results from a discovery layer system (Ex Libris Primo) and Google Scholar
• Evaluate the discovery layer system for usability issues
Thursday, July 25, 13
![Page 9: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/9.jpg)
Method• 12 graduate students (mean age = 26.3, SD = 3.2)
• Search tasks on Primo and Google Scholar
• Given topic for "rst three tasks: classroom assessment techniques, food addition, natural language processing
• Participant own keywords for fourth search task
• Rate "rst ten results from 1 (highly irrelevant) to 7 (highly relevant)
• Within-subject design, balanced order of search systems, randomized tasks
• After search tasks: System Usability Scale (SUS) ratings, preference ratings, open questions
Thursday, July 25, 13
![Page 10: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/10.jpg)
Relevancy Ratings
Search 1: Article 9 and 10 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"Ra,ng"
Ar,cle"2"Ra,ng"
Ar,cle"3"Ra,ng"
Ar,cle"4"Ra,ng"
Ar,cle"5"Ra,ng"
Ar,cle"6"Ra,ng"
Ar,cle"7"Ra,ng"
Ar,cle"8"Ra,ng"
Ar,cle"9"Ra,ng"
Ar,cle"10"Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
![Page 11: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/11.jpg)
Relevancy Ratings
Search 2: Article 10 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"Ra,ng"
Ar,cle"2"Ra,ng"
Ar,cle"3"Ra,ng"
Ar,cle"4"Ra,ng"
Ar,cle"5"Ra,ng"
Ar,cle"6"Ra,ng"
Ar,cle"7"Ra,ng"
Ar,cle"8"Ra,ng"
Ar,cle"9"Ra,ng"
Ar,cle"10"Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
![Page 12: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/12.jpg)
Relevancy Ratings
Search 3: Article 3 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
9"
Ar-cle"1"Ra-ng"
Ar-cle"2"Ra-ng"
Ar-cle"3"Ra-ng"
Ar-cle"4"Ra-ng"
Ar-cle"5"Ra-ng"
Ar-cle"6"Ra-ng"
Ar-cle"7"Ra-ng"
Ar-cle"8"Ra-ng"
Ar-cle"9"Ra-ng"
Ar-cle"10"Ra-ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
![Page 13: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/13.jpg)
Relevancy Ratings
Search 4: Article 3 signi!cantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"Ra,ng"
Ar,cle"2"Ra,ng"
Ar,cle"3"Ra,ng"
Ar,cle"4"Ra,ng"
Ar,cle"5"Ra,ng"
Ar,cle"6"Ra,ng"
Ar,cle"7"Ra,ng"
Ar,cle"8"Ra,ng"
Ar,cle"9"Ra,ng"
Ar,cle"10"Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13
![Page 14: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/14.jpg)
Results• No signi!cant difference of relevancy ratings between
given and participant chosen searches
• Usability ratings and preference of Google Scholar signi!cantly higher than Primo
• SUS: 4.27 (SD = 0.29) vs. 3.59 (SD = 0.43)
• Preference: 8.33 (SD = 0.65) vs. 7.08 (SD = 1.38)
• Usability correlated to preference ratings (r = 0.59, p < 0.01)
Thursday, July 25, 13
![Page 15: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/15.jpg)
Usability Issues of Primo - Work$owSearch Results List
Identify Article
Link Resolver
Article Page
Thursday, July 25, 13
![Page 16: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/16.jpg)
Usability Issues of Primo
!
Thursday, July 25, 13
![Page 17: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/17.jpg)
Usability Issues - Consistency
Thursday, July 25, 13
![Page 18: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/18.jpg)
ConclusionPrimo’s search relevancy may be on par with Google Scholar, but usability issues have affected users’ perceived usability and preference.
User interface of discovery layer systems needs to address work$ow and consistency issues.
Thursday, July 25, 13
![Page 19: User-Centered Evaluation of a Discovery Layer System with Google Scholar](https://reader034.vdocuments.net/reader034/viewer/2022042813/5457c2f1b1af9fc15d8b4861/html5/thumbnails/19.jpg)
What’s More• “Blind” test on relevancy ratings
• Test large sample of search keywords
• Server log analysis on keywords, facets, session length, etc.
• Better (Google Scholar-like?) link resolver
• UI improvement
Thursday, July 25, 13