[ieee 2013 ieee 4th international conference on cognitive infocommunications (coginfocom) -...

Users' retrieval skills in library environment: ways to measure and improve them

Klára Koltay*, Mária Borbély+, Erzsébet Dani+, László Balázs*, Mária Csernoch+, Gyöngyi Bujdosó+ *Debrecen University University and National Library

+Department of Library and Information Science Debrecen University Debrecen, Hungary

[email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract—The paper examines the ways library catalogues are used in a university environment. The quantitative and qualitative analyses of users’ retrieval skills are based on the search logs of a web-based online catalogue. The analysis reveals the problematic points of user behavior that cause a relatively high number of unsuccessful searches. It suggests that 3D visualization of authority data could become an easy to use and quick tool of surfing among concept hierarchies thus carrying out the task that browse searches failed to do. It could help to solve problems more traditional tools failed, the pre-selection of the concept closest to the user’s needs and to lead to increased search efficiency.

Keywords—bibliographic databases; search techniques; retrieval skills; university library; subject searches; 3D visualization

I. INTRODUCTION The research field of cognitive infocommunications aims at

getting a systematic understanding of various aspects of cognitive processes primarily described in terms of interactions between human and artificial cognitive systems. [1.] The present paper focuses on investigating the human side of a case of inter-cognitive communication and is interested in how a central concept of human cognitive processes and human social interaction, the concept of Theory of Mind works in an inter-cognitive communication situation. How the ability of humans to access the mind of the others translates when investigating the interactions between humans and artificial information systems. It is important to see how good the users can become in accessing either intuitively or through conscious learning the services, the potentials, the innate rules of the systems they interact with, how well they can predict the systems responses and optimize their strategies to get adequate responses. As human environment becomes more and more entwined with artificial cognitive systems it is important to understand how humans can get better in assessing their artificial partners and how the systems themselves can help the process. The present study picks a relatively simple and widely used system with traditionally established rules and practices, the library catalogue to study this aspect in real life settings for a relatively large number of cases.

It is often experienced that library patrons even in higher education environment cannot fully and efficiently exploit all the information provided by bibliographic databases when doing their research. Their performance do not significantly gets better even after they take part in brief introductory courses to library databases or when they encounter enhanced “single point of entry”, “Google-like interfaces”. [14.] The phenomenon is all the more worth noting as the majority of users are of the age groups that are generally referred to as the bit-generations. In the present study we gather and analyze data on how patrons of a university library use their catalogue in real life with the aim of identifying underlying patterns, problematic points and misleading practices. Based on the analysis the article also intends to show possible ways and assisting tools to improve retrieving capabilities of both catalogues and their users

II. PREVIOUS RESEARCH The bit-generations (Y to Z) are the real consumers of new

informational technologies and applications. Ours is the age of secondary orality in the history of media, where the hegemony of written communication yields to the simultaneous presence of sound and image. The dawning of the new (Neumann) galaxy is resulting in a radical shift in communicational, learning, information-search, and information-processing habits. The “children of the digital age” (or internet generation), Don Tapscott maintains, are the propelling force of the new society. [8.] The receiving (or reading) brain deciphers the lines like pictures. The priority of thinking in images is taking the upper hand more and more in the world of digital natives. The emergence of 3D is speeding up this transformation even more.

The phenomenon that Katherine Hayles calls “hyper attention” must also be taken into consideration. Hayles discourses upon hyper attention versus deep attention in her study Hyper and Deep Attention: The Generational Divide in Cognitive Modes; upon how the former manifests itself among members of Generation M (“media generation” is the term she uses). Her notion of hyper attention is defined as follows: “switching focus rapidly among different tasks,” preference for “multiple information streams,” high stimulation threshold,

The publication was supported by the TÁMOP-4.2.2.C-11/1/KONV-2012-0001 project. The project has been supported by the European Union, co-financed by the European Social Fund.

319

CogInfoCom 2013 • 4th IEEE International Conference on Cognitive Infocommunications • December 2–5, 2013 , Budapest, Hungary

978-1-4799-1546-0/13/$31.00 ©2013 IEEE

“low tolerance for boredom.” Attention of this nature is very useful in situations that require rapid shifts among environments as well as fast reactions [4.] [13.]

Hayles discusses hyper attention apropos of Generation Z after all (“Generation M,” then, in her terms) diagnosing the learning problems of a generation which is soon to enter higher education. The scope of investigation is broadened by the findings of a BBC research project published in 2010. The project studied the information-search habits of Generations X, Y, and Z (our Z is, in fact, their “Google Generation”), from the point of view of internet-use, multi-tasking, information search strategies, and working memory. It was found that Z’s (the Google Generation) are the fastest in information search but use the fewest possible sources. Y’s are slower in search, but use many more sources, avail themselves to multi-tasking methods, and produce better results as far as working memory is concerned [6.] This finding correlates with hyper attention, which is gaining ground with Generation Y more and more: fast reaction, absence of deep attention, being superficial. The spreading of hyper attention has a very important role to play in the context of our study since using bibliographic databases requires conformation to a smaller and more controlled world of information than the universe of the internet. They may yield very adequate results based on their specialized search aids, more adequate than a browser can provide, but their users must have a basic picture of how they work.

2004 American surveys already indicated that university students typically rely on the internet, using search engines when searching for information, only four percent using library information sources. Several studies have been devoted to search habits: iProspect (2004): Search Engine User Atitudes Survey [5.]; Gunn, H. & Hepburn, G. (2003): Seeking Information for School Purposes on the Internet [3.]; Raven, M. (2012): Bridging the Gap: Understanding the Differing Research Expectations of First Year Students and Professors [7.]; Eagleton, Maja B. & Dobler, E. (2007): Reading the Web: Strategies for Internet Inguiry [2]. Each study reports (among other findings) that information search takes place with the help of Google (66% - Gunn & Hepburn).

The reports reveal that the new generation demands new search-technology representations and a better guidance how to bridge the ease of internet browsers and the more sophisticated techniques a more controlled environment requires.

There is research devoted to the methodological part of collecting reliable and easily analyzable data on user behavior. [10.] [11.] and on possible ways of visualization of library data [12.].

III. METHODS AND RESULTS

Our research was conducted in the University and National Library of the University of Debrecen (Hungary), and was based on the search logs of the library’s web catalogue between March 9, 2013 and April 17, 2013. The logs contain all the Common Command Language (CCL) queries sent to the catalogue and subsequent actions within this interval. The analysis was carried out on two levels: a quantitative analyses aims at identifying which services of the catalogue are the most

commonly used. A qualitative analyses of the user activity looks for data on how well formed and successful the searches were. Identifying chains of activity performed by particular users made it also visible what tactics they use to amend their unsuccessful searches.

The subject searches were one of the special focus points of the research as they should be one of the most typical query types when the user is not looking for a known specific title. The Debrecen catalogue is suitable for a subject based research as it provides ample subject information and tools to help subject searches.

A. A breif description of the database and the user interface

The catalogue database in the research is a union catalogue of the university and some other research libraries in Debrecen, which makes it cover a wide variety of topics. It contains 860 thousand bibliographic records of books, periodicals, sound recordings, videos and graphics and 850 thousands authority records, which contain the name, subject and title headings used in the catalogue and give possible variant forms to assist searches. 13 thousand records contain links to information on the web. Most of the bibliographic records contain two types of subject access points. They contain Universal Decimal Classification (UDC) numbers and subject headings. The subject headings belong either to the Library of Congress Subject Heading system and are present in both in English and Hungarian or to the English language Medical Subjects Headings of the National Medical Library (US).

The database can be accessed with various interfaces. In the research we used the log files of the IKR’s standard WEB OPAC interface (http://webpac.lib.unideb.hu/WebPac/) , which provides remote access to the database and is the supported interface on the public terminals within the library. Library staff with the exception of readers services librarians assisting patrons use other OPAC interfaces. Thus we assume that most of the searches are performed by the students and staff of the university and the general public. Other main types of usage are those of robots and individuals searches that are forwarded from the catalogue of the Hungarian Document Delivery Database.1 These special usages were not included in the study.

The WEB OPAC offers 22 types of keyword index searches and browse options for names, title, subjects and UDC strings. For entering keyword searches there is a simple search form with the possibility to enter one aspect searches. Advanced searches can be formulated using a combination of up to three primary indices chosen out of the 22 and limited by up to four aspects (publication date, language, document type and holding location).

The WEBPAC gives assistance for users with how to enter search terms in various ways. In the case of keyword searches there exists an autocomplete service for names, title, subject,

1 The National Document Delivery System Database (ODR, http://www.odrportal.hu) is a national union catalogue with some location information, which refers its users for detailed item information to local databases with the help of record identification number searches.

320

K. Koltay et al. • Users' retrieval skills in library environment: ways to measure and improve them

publication place and publisher. Browse searches as such give much help in finding relevant search terms: the entered search term is subject to an automatic right truncation, and the resulting alphabetic list of matching terms gives the searcher suggestions of which terms may be relevant. It is here that the references towards the preferred terms appear. The subject lists may help to pick the right aspect of a topic with is subdivision lists. The alphabetic result list displays the content of authority records, which contains subject hierarchies in the case of subject authority records and variant forms in the case of names, which again might give further search suggestions for users. A third available tool presents itself when we already have a displayed search result. The name and subject headings of the displayed records appear as links and can be used as starting points for further searches with the chosen heading.

B. Quantitative analyses Within the examined 40 days period between March 9,

2013 and April 17, 2013 the log files register 50442 searches altogether. 35578 searches of them can be considered user searches while the others are enquiries arriving from the ODR database to discover local item information (3165) and a relatively large number of searches initiated by robots (12523).

Our analysis focuses on the 35578 searches initiated by patrons 97% of which are keyword searches. The relatively small number of scan searches shows that the majority of patrons have not discovered the advantages of searching by lists of controlled access points and the cross-references displayed in the lists. They miss one of the tools offered for them by the catalogue, which would be a great help in formulating proper searches.

1) Browse searches

The majority of the 824 browse searches are author and title scans with not too many subject ones. The UDC scan searches can be considered only a few unsuccessful trials out of curiosity where 4 out of 6 search terms are formally irrelevant giving names or subject terms instead of UDC strings.

It is more surprising that even author scans show a 13,6% of ill formulated searches, where titles or subjects are entered instead of names.

Altogether 6.6 % of the scan searches are irrelevant in the above sense.

2) Keyword searches

97% of all searches are inquiries by keywords. Of all the primary and secondary indices the title, author and subject searches are the most popular with document type and holding location and in some case with language limits.

It is interesting to note that “all keywords” searches are relatively rare. (1.4%)

12,2 % of the keyword searches combine more than one indices and 1% uses more than 5 indices at the same time. The most frequent combined searches uses one primary index combined with document type and holding location limits.

When more than one primary indices are combined in a search the number of the author and subject combination (440) is double of that of the author and title combination (179).

When joining indices, it is almost always the “and” operator that is chosen. “Or” appears 61 while “not” only 3 times.

Truncation is used relatively rarely, only in 482 cases (1.3%). As Hungarian is an agglutinative language it would be justified to use it more widely especially in title searches where we meet real life expressions and sentences instead of formally structured headings. On the other hand, the presence of autocomplete makes it less important for users to be aware of this factor.

When examining the search terms, we discover that altogether 4.4% of keyword searches are irrelevant in a way that the entered search terms do not match the chosen indices. The phenomenon naturally occurs in the case of rarely used specialized indices (ISSN, music number etc.), but it is more surprising that in 11.1% of the author searches title and/or subject terms are entered. It happens despite the fact that when a patron starts typing his or her search term the autocomplete function gives a list of the relevant options.

The length of search terms varies a lot. The majority of them are between 1 and 3 words, but 13 % are above 4 words. There are 54 occurences of search expressions with 16 words and a few with even more.

The overall effectiveness of the searches is characterized by the fact that 24 % of the searches returned 0 result.

Data suggesting how search results are managed by patrons can be the number of times the default settings are changed.

Search result lines are sorted by author as a default. It is relatively rare that this setting is changed. In 0.2% of the cases the settings is changed to a title sort, while in 1% to sort by date.

Another default that can be changed to make navigation in relatively larger result sets easier is the number of records displayed at a time. The default is 12 and it can be changed to 36, 60, 160, which happened in 12.3% of the searches.

A special type of searches is the one that can be initiated when a result is already displayed in long format. Name and subject searches can be initiated with the names and subjects present in the displayed record. When two or more bibliographic records are linked together, which is most typical in the case of periodicals and compared to the record number of the whole database is relatively rare, patrons can navigate along the links. Names and subjects searches based on displayed records were initiated 1305 times, while record links were followed 73 times.

The above quantitative findings already suggest the problematic points of user behavior. They are not always

321


conscious when choosing their indices, do not match the chosen index and the search terms. Approximately in 10 % of the search cases they show unawareness of the nature of keyword searching and try to be on the safe side by entering too many words and even irrelevant data: author, title or publication data mixed in the same search field. They seem to be unaware of the services of the WEBPAC that would help them in their searches: do not seem to have discovered the advantages of scan searches and they seem not to pay attention to the suggestions of the autocomplete function, which tries to insert some of the advantages of scan searches into the keyword search windows.

C. Qualitative analyses In the present phase of the research 1000 of the all the 3611

subject searches formed the bases of the quantitative analyses. The overall success rate (52% of searches have 0 result ) in the analyzed set is the same as that of the total pool of subject searches. (It can be compared to the 24% failure rate of all keyword search types.)

As a first step we examine the individual search terms used in the subject queries and we aim at identifying the typical reasons of failure. Though the log files do not explicitly identify the actual sessions of user activity, it seems possible to group sets of searches as trials of a user to formulate an optimal search string. Our grouping is based on the time stamps of activities in the log file, and the similarity of concepts the search terms tried to express. We do not attempt to get an overview of all the activities of the users’ whole sessions. The extent of the grouping is only that of the searches trying to capture an individual concept. It is only 42 % of the examined 1000 keyword searches that stand alone. The others can be grouped into approximately 316 attempts to capture a search term. The related searches make it possible to identify typical research strategies and establish user groups defined by their typical user behavior patterns.

The high number of “stand alone” searches may mean that our users are good at choosing the best possible search strategy and find what they want in one step. It is surprising, on the other hand, how many times they reveal a group of users who give up at the very first failure. In the case of searches with 0 result or with an unmanageable high number of hits 35 to 40 % of patrons do not give a second chance for themselves.

When analysing the related search groups the following typical erroneous patterns can be established.

• The concept is not formulated with one or a few keywords but in sentence fragments (“development of credit”, “history of the banks”, “20th century flats”, “church history of the Bach era”, “history of Debrecen local government” etc.). These terms can especially prove inefficient in agglutinating languages. Still when they are used in the internet or in large fulltext databases they will yield results. In bibliographic databases it is most likely that the search will fail. After a few failed trials the users either give up or chose a very broad term (e.g. “bank”, “equipment”, “church”, “local government”, “Debrecen” in the above examples), which will not match their original

concepts in mind, on the one hand, and give a large number of mostly irrelevant titles.

• It is very common that when trying to formulate their searches in alternative ways, the users alter their original expressions not paying attention to the fact whether they leave the scope of their original concept. Sometimes they even use the adjective part of the original expression alone. (When the expressions “human resources” and the misspelt “human capital” failed “resources” was used instead with the result of a lot of titles about mines, energy industry etc. “Labor economics”, “education – economic aspects” , “capital” or other broader but conceptually related terms would have been much better choices. Another telling example of “Alternative medicine” followed by “alternative” as a “better” choice.)

• It is quite frequent that the roles connected to a book or other documents are misinterpreted. It happens quite often that full titles and/or author names are given in subject searches, or performers, publishers may appear as subject, as well. (The failure of search terms like “Cambridge language exams” and “Cambridge grammar books” was followed by a subject search with “Cambridge”, though Cambridge in the publisher position would have been the right choice.) In a number of cases the wrong choice of the index is corrected in the next trials but too often the searchers do not realize that the cause of the failure lies in the mismatch between the index and the search term. (note the following chain of connected searches where the aim of the user was to find a novel with the title “Ajtó” by Szabó Magda. 1. FIND SUBJ "ajtó"; 2. FIND SUBJ "az ajtó"; 3. FIND SUBJ "Ajtó"; 4. FIND SUBJ "szabó magda". What he got in the end was 30 books on Szabó Magda.)

• The above “Ajtó” example can be the representative of the fact that some of the users do not realize what keyword searches mean. When a term fails to return a result they add some more words (an article perhaps) or capitalize the first letter of titles or names “to make their term more precise” and try again. The tendency is more evident in the case of title searches but it is present in the case of subject queries, as well.

• The way the limiting indices are used suggests that users either do not understand their role and meaning correctly or they set them on a trail and error bases. A subject search “Pride and prejudice” limited by document type “periodical” would assume the existence of a journal devoted to the novel “Pride and prejudice” which is not very likely the user had in mind. Was he trying to find articles on the novel or just experimenting with the settings?

Beside identifying typical search patterns the research attempted to define relatively homogeneous user categories based on the characteristics of their search terms, the number of trials, and the efficiency of their searches. Classifying our

322


searchers into user types gives us a clearer picture of how bibliographic databases are used in university library environment, how efficient user can be when they work in online environment, how well they can use the available online tools and what further services can give more assistance for them.

• Professional users (T1) correctly use the technical terms of the discipline in question in their subject searches. They consciously choose the correct form the terms which is supposedly based on their prior usage of the subject index and other tools (autocomplete, references etc.). Their ratio in the present research set is 20 %.

• Conscious subject searchers (T2) may be more hesitant when they initiate their searches but they realize how they can navigate among the possible synonyms, broader and narrower terms, and handle the various inflected forms. After an optimal number of trials with the well chosen terms they close the searches with some kind of definite result. Their ratio is 21 %.

• Specializers (T3) tend to use very special technical terms in their searches without prior usage of the database subject index. Their ratio in the present research set is 9 %.

• Internet addicts (T4) do not differentiate between internet browsers and bibliographic database interfaces. Their searches in the latter are characterized by typical internet search methods: they pick their search terms randomly, in various grammatical forms, sometimes very narrow concepts appear and abbreviations are used, which result in unsuccessful search transactions. Even when trying to amend their searches, they cannot think of using keywords checked against subject indices. Their ratio in the present research set is 22 %.

• Topic searchers (T5) start their searches without prior checking of subject indices. They are not aware of the presence of controlled vocabularies, just randomly surf among the various terms, synonyms, broader and narrower terms. We may feel them to be uncertain in the concepts of their theme. They, however, do not lose their interest after several unsuccessful searches and keep on with their loose trials. Their ratio in the present research set is 26 %.

• Overanxious (T6) users tend to use unnecessarily complex compound searches with elements that make the otherwise correct search expressions uncertain. Their ratio is 2%.

It is interesting to compare our present findings with the results of a previous study, which examined the search technique of 250 users in 2008 and established similar user categories [9.] In the past five years the ratio of Internet addicts has significantly grown from 5% to 22%, and the professionals decreased from 35% to 20%. The number of conscious subject

searches is doubled, while the number of specialisers and overanxious users decreased by 50 %.

IV. CONCLUSION The findings of our research reveal that the present

generation of users is not efficient enough when using bibliographic databases. They are not aware of the fact that their internet browsing techniques cannot be fully adapted when using bibliographic databases, they are not patient enough to discover the special features of a catalogue even if there are services built in the search interfaces to help them. It is in line with the studies characterizing the bit generations with keywords like “simultaneous presence”, “thinking in images”, “hyper attention”, absence of deep attention”. To better serve their information needs bibliographic databases must follow the patterns they use in their cognitive processes. It does not mean that catalogues and bibliographic services have to resemble internet browsers with their one box search tools and “do whatever first comes to your mind” search technique, as it is not a very effective approach. They have to develop tools that help the users to assess their logic, that reveal their more controlled nature quickly and possibly in pictures. They have to make use of their controlled vocabularies to assist searches. They do not have to forget about labor intensive subject heading systems, on the contrary they have to make them more visible. Visualization, i.e., 3D-display of the subject-heading systems could become an easy to use and quick tool of surfing among concept hierarchies thus carrying out the task that browse searches failed to do. They could help the pre-selection of the concept closest to the user’s needs and lead to increased search efficiency through the mobilization of hyper attention.

REFERENCES .

[1] Baranyai, P., Csapó, A. (2012), Definition and Synergies of Cognitive Infocommunications. In.: Acta Polytechnica Hungarica, Vol. 9. No.1. pp. 67-83.

[2] Eagleton, Maya B. @ Dobler, E (2007).: Reading the Web: Strategies for Internet Inquiry. Guilford Press, 2007 http://books.google.hu/books?id=4ErD68K__eMC&printsec=frontcover&hl=hu&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false (accessed 2013.11.06.)

[3] Gunn H.& Hepburn G. (2003): Seeking Information for School Purposes on the Internet. In: Candian Journal of Learning and Technology, Vol. 29. No. 1. http://cjlt.csj.ualberta.ca/index.php/cjlt/article/view/35/32 (accessed 2013.11.06.)

[4] Hayles, N. Katherine, Hyper and Deep Attention: The Generational Divide in Cognitive Modes, In: Profession 2007, pp. 187–199 , DOI: 10.1632/prof.2007.2007.1.

[5] iProspect (2004): Search Engine User Attitudes Survey; http://c.ymcdn.com/sites/www.sempo.org/resource/resmgr/Docs/iProspectSurveyComplete.pdf (accessed 2013.11.06.)

[6] Nicholas, D. et al (2011): Google Generation II: Web behaviour experiments with the BBC. In: Aslib Proceedings.2011. 1. p. 28-45.

[7] Raven, M. (2012): Bridging the Gap: Inderstanding the Differing Research Expectations of First-Year Students and Professor. In: Evidence Based Library and Information Practice, Vol 7, No 3 (2012). http://ejournals.library.ualberta.ca/index.php/EBLIP/article/view/17172 (accessed 2013.11.06.)

323


[8] Tapscott, D. (2009): Grown up Digital: How the Net Generation is Changing your World. USA, McGrowHill, 2009

[9] Eszenyiné Borbély M (2010).: A használói elégedettséget befolyásoló tényezők egyetemi könyvtári környezetben. In: Koltai Zsuzsa (szerk.) A "Kulturális valóságismeret és EKF 2010. 35 éves a pécsi kultúraközvetítő képzés" című tudományos konferencia és emlékülés szerkesztett anyaga. Pécs, Magyarország, Pécs: Pécsi Tudományegyetem, pp. 474-486.

[10] Csernoch, M., Bujdosó, Gy., Borbély, M., Dani, E., Koltay, K. and Balázs, L..: LibSearchNet: Analyses of Library Log Files to Identify search Flows In: 4th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 2013, (in press)

[11] Bujdosó,Gy., Csernoch, M., Borbély, M., Dani, E., Koltay, K. and Balázs, L.:LibSearchNet: Library Log File Initiatives –as a Part of Semantic Library Interface Development for the VirCA 3D Virtual

Collaboration Area. In:4th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 2013, (in press)

[12] Virágos, M. and Gilányi A.: Library Treasures in a Virtual World . In: 4th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 2013, (in press)

[13] Dani, E. (2012): E-létezés és hiperfigyelem In.: Könyv és nevelés 15 : 4 (2012) submitted

[14] Asher, Andrew D., Duke, Lynda M., Wilson, Suzanne (2013) Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources in: College and Research Libraries Vol. 74. No. 74. pp. 464-488. http://crl.acrl.org/content/74/5/464.full.pdf+html (accessed 2013.11.06.)

324


[ieee 2013 ieee 4th international conference on cognitive infocommunications (coginfocom) -...

Documents