helping students during online searches: an evaluation

= v v

Helping Students During Online Searches: An Evaluation

by Linda Stewart

An evaluation of student online searches explored the value of staff assistance to trained end

users searching BRS Menus. Interactions between 19

student volunteers and library staff “coaches” were analyzed

via tape recordings, a questionnaire, and postsearch

interviews. The results, and a discussion of their

implications, are presented here.

Linda Stewart is Online Coordinator,

Albert R. Mann Library,

Cornell University, Ithaca, NY.

E arly in the evolution of end-user searching programs, librarians felt the need to provide assistance

by offering pre-search counseling’ or by mediating during the search.* Reasons ranged from the desire to save money to the hope of educating users. Some librarians, such as Pavelsek, feel such ac- tivities are unnecessary today:

The availability of menu-driven, user- friendly systems and the greater computer sophistication of most college students are two important factors con- tributing to the reduction, if not the entire elimination, of the need for end- user instruction.3

It is true that user assistance programs are labor intensive and thus vulnerable to budget cuts. On the other hand, the proliferation of self-service database systems suggests that, if user assistance is effective, it should be even more wide- spread today than it was a decade ago.

Considerable research has been car- ried out on the usefulness of mediation in the online search processmost of it pertaining to intermediary-end user com- munication. Belkin and Vickery reviewed pre-1985 literature in this area: and Belkin devised a method of discourse analysis for audiotaped online searches conducted by an intermediary with the user present.5 Saracevic and co-workers designed a major study involving discourse analysis of videotaped searches6 and made a partial report of results including stages in interaction, styles in interaction, and the role of user feedback in modifying search strategies.7

In November 1989, librarians at the Albert R. Mann Library at Cornell Uni- versity undertook a study to determine the value of the Library’s end-user coaching program. The Mann Library study differed most importantly from that of Saracevic in that trained end users, as opposed to search intermediaries, actu-

ally performed the searches. Moreover, as the goal was specifically to evaluate Mann’s own end-user program, the research design was targeted to, and limited by, that program.

The End User Coaching Program Since 1983, Cornell undergraduate,

graduate, and postgraduate students have been taught to search online bibliographic databases at Mann Library. The program, known as U-Search, offers group instruction sessions and individualized consul- tation to prepare students to do their own searches on the BRS Menus system. (DIALOG has recently been added, but was not available during the time of the study.) The students conduct their online searches with the assistance of a library staff member. This staff member, or “coach,” provides in-depth advice and individual attention. Students are charged $8 per half hour of time spent online. About half of the students, usually those enrolled in social sciences departments that have allocated “computer literacy” funds for this purpose, have their searches paid for by their departments.

The U-Search program supplements other online search services offered at Mann. Librarians perform traditional searches on a cost-recovery basis as well as answering brief online search questions free of charge. In addition, 22 CD- ROM bibliographic databases may be used at no charge to the user.

This inquiry was designed to evaluate the coaching component of U-Search. Throughout the history of the program, various staff members have served as coaches-professional librarians with experience as search intermediaries, graduate students with brief training in BRS Menus, and students employed in the library-based microcomputer center with microcomputer experience but only brief

The Journal of Academic Librarianship, vol. 18, no. 6, p. 347-351

01993 by the Journal of Academic Librarianship. All rights reserved.

training in BRS Menus. In addition to determining whether or not a coach was necessary to online search success, the investigator hoped to determine in a general way what sort of background and training was most appropriate for the position.

Methodology The study, funded by a Sears-Roe-

buck Mini-Grant for Instructional Inquiry, involved examining BRS Menus searches performed by Cornell students as part of the U-Search program. The aim, based on available funding, was to analyze about 20 U-Searches. Library staff recruited students by soliciting volunteers from courses where students were aheady being given U-Search training. Students were told that they would receive a free half- hour U-Search in return for allowing the search interaction with the coach to be audiotaped and for returning the following day for a one-hour interview, which would also be taped. Since the students were already involved in learning BRS Menus it was assumed, correctly, that they would have selected topics of interest prior to the search.

Volunteers were drawn from an ento- mology class emphasizing the social and economic costs of pesticides, a food science course on microbiology, and three library “open workshops,” held each se- mester, which any Cornell student may elect to attend. Of the 19 volunteers, 8 were undergraduates and 11 were graduate or postgraduate students. This group had a larger proportion of graduate students and life science students than normally took part in U-Search.

The study consisted of two stages. In the first stage students were tape-recorded as they performed half-hour computer searches, assisted by their coaches. The coaches were four professional librarians who performed computer searches as part of their job responsibilities. This half of the study was used to examine students’ retention of class instruction and the level of assistance needed from the coaches.

For the second stage, students returned the day after their U-Searches for an interview with the project’s investigator. The interview used the “stimulated recall” technique, in which portions of the taped searches were played back to the students, and they were encouraged to describe their reactions to the search process and the help they received. This technique was developed by Bloom, who taped college classes and played the tapes

to students to facilitate their recollec- tion of their thought processes during the classes.8 It was hoped that audio- tapes would enhance the memories of Cornell students as well. This portion of the study was used to collect student perceptions of successful and unsuccessful techniques used by coaches.

Of the 19 taped searches and interviews, 16 were transcribed and analyzed to examine the nature and amount of help provided to the students by their coaches.

Analysis of Taped Searches Three questions were considered when

analyzing the taped searches: What did the coach contribute to achieving the program goals? Was the coach necessary? If so, what level of staff training and experience was most appropriate for a coach? Analysis of the taped searches was primarily used to examine the be- havior of the coach.

It is a common misconception by patrons that online searching involves simply sitting down at the keyboard and typing in a few keywords. On the con- trary, the taped searches showed that online searching and the coach-student interaction during the searches were highly complex. In the 16 analyzed searches, the median number of questions asked by students during the half-hour was 3 1, with a range of 14 to 43. They would probably have asked many more if the coaches had not anticipated some questions during the search: coaches aver- aged 200 to 300 comments during each session.

To see exactly what was being com- municated during the search, the investigator analyzed these comments in some detail, defining a comment as a subject- verb pair or implied subject-verb pair. For one of the study objectives-determining the necessity of the coach-it was of particular interest to ascertain how many comments provided information that could be covered using other means, such as the lecture or handouts, and how many required an individual interaction with the searcher.

It was found that the coaches’ comments tended to be highly specific and tailored to individual search situations. Analysis of the comments resulted in several hundred categories that were distilled into nine broad areas: database advice, topic negotiation, vocabulary advice, prompting, reviewing results, time factors, procedures, social interaction, and other library-related comments. (Inter-

estingly, the investigator later found many similarities to the eight categories described by Saracevic et al.)9

Database Advice. These comments included selection of the initial database, changes of database during the search, and advice on specific database features. For example: A student arrived for a search on parental selection of day care, having been advised by the librarian at the reference desk to search the Family Resources database. The coach agreed with this choice, but suggested that the student also search ERIC, Sociofile, and PsycLIT later on compact disk. The student mentioned wanting to search her topic in the marketing literature, looking for articles on decision theory. She was referred to an appropriate business database by the coach, who also suggested a database covering popular magazines.

Topic Negotiation. Even though searches were discussed at the reference desk before the appointments, coaches felt the need to review search strategy before going online. They also assisted students in modifying unsuccessful strategies during the search. For example: A student was looking for articles relating to biodiversity in the United States. The student had prepared a search strategy to retrieve all records containing any one of the words biodiversity, diversity, bio- mass, or species and United States. The coach suggested eliminating the word species from the list on the grounds that the combination of species with United States would retrieve thousands of articles. The student was rightly concerned with losing some useful articles but agreed to the modification. During the search, when many more articles were still retrieved than the student had time to print, the coach helped him narrow the search further-first, by restricting the topic to certain categories of organisms only, then by limiting results to a more specific set of concepts. At this point in the search, the student was receptive to suggestions for narrowing the search because he was able to try various combinations of search terms and see immediate results.

Vocabulary Advice. Coaches also gave students assistance with online vocabulary-using truncation to search different forms of a word, specifying synonyms and scientific names, and selecting terms from a thesaurus.

Prompting. Coaches prompted students in the use of the correct commands and in mechanical manipulation of the mi-

348 the Journal of Academic Librarianship, January 1993

crocomputer, diskettes, and the printer. Printing alone required knowledge of the print command, the number of the set to be printed, selection of the descriptive elements of each record to be printed, and the number of the documents needed. These entries had to be made in the correct order, using the correct syntax. Most students needed prompting through their first few attempts. Eventually, they be- gan to remember the sequence on their own. Coaches also guided and then prompted students through new techniques, such as saving a search and re- executing it in another database. These techniques were considered too advanced to be covered in the classroom session.

Reviewing Results. The student and the coach looked together through a sample of titles or complete records to determine if appropriate materials were being retrieved. This often resulted in further clarification of the topic and attempts to improve results while the student was still connected to the database.

Time Factors. Students often needed to know how much time remained. This category also included advice on how many records could be printed or what length (format) of records would be appropriate to print in the remaining time.

Procedures. Coaches also provided information on a broad range of systems procedures that included log-on procedures, general descriptions of the system, and typing conventions (e.g., word wrap, lack of need for capital letters, etc.)

Social Interaction. Coaches engaged in general social conversation with students, discussing their classes or fields of study, or offering encouraging comments on their search progress.

Library-related/Other. Finally, coaches gave information to students on other library services, on how to locate jour- nals in the library, and on general library-specific features.

The frequency of the coaches’ comments were tabulated using the average percentage of each type occurring during a search session. To test the reliabil- ity of the coding scheme, two librarians who had not served as coaches independently coded the first two search transcripts. Since the codes assigned by the two coders were virtually identical, one librarian coded the remainder of the transcripts. Table 1 presents a breakdown of all comments by category.

The taped searches demonstrated that a vast amount of individualized assis-

Table 1 Breakdown of Coaching Comments by Category

Category Breakdown

Prompting Procedures Topic negotiation Reviewing results Database advice Vocabulary advice Time factors Library-related/other Social interaction

21.8% 19.6% 18.3% 12.5%

9.0% 8.0% 6.3% 2.6% 1.8%

tance was provided during the coached searches. Three categories-topic negotiation, vocabulary assistance, and reviewing references, which comprised 38.8 percent of the total comments made during the searches-involved commu- nicating information unique to each student and search topic. The principles of searching can be covered during an introductory lecture; the application of these principles to specific situations cannot. Developing search skills involves ob- servation and practice, both of which were provided to students during the tutored searches. Parts of other categories-database information, prompting, and procedure, which comprised 50.4 percent of all comments-can be covered by the introductory lecture and by improving the handouts. However, if students forget the material from the lecture or need more assistance than the handouts provide, their searches may founder.

Anal sis of Simulated Reca I Sessions 9 The goals of U-Search are: to give

students a sense of confidence in their searching ability, to clear up misunderstood concepts, to provide guided practice in interacting with the system, and to provide the bibliographic references that were the original purpose of the search. The follow-up interviews were used to determine whether these goals were met. Since we hoped to discover trends among students, interviews were relatively free-form. The investigator played back portions of the tape-recorded searches and asked each student to comment freely concerning the emotional tone of the search, specific concepts and techniques of searching, the appropri- ateness of the help they received, and the results of the search. Suggestions to improve the program were also solicit- ed.

Since the questions asked were de- pendent on each search and since each search was different, questions varied for each interview. However, the inter- viewer used a prompt sheet of about 30 questions to make sure the appropriate areas were covered. Sample questions related to the coaches, comments included: Did that answer your question? Was there anything else. you would have liked to ask? Questions related to online operations included: What did you ex- pect to happen at that point? Did that strategy work? How does that seem to you now?

Student responses were analyzed using the following method: each interest- ing or revealing response was copied onto a notecard; all notecards were then grouped into the four broad themes explored below. Participants also completed a written questionnaire that included questions on basic demographics and on their search skills and experiences. Be- cause of the small number of respon- dents, the investigator did not attempt a formal statistical analysis, but looked for simple numerical trends. Questionnaire data are integrated with interview responses in the discussion that follows.

Student confidence. Feelings of confidence about the search are important to success. If students dislike the online search experience, they may adopt alternate means of retrieving information that are less effective but less stressful. High levels of anxiety or discomfort may detract from the learning process. The questionnaire provided a list of 18 adjectives describing emotional reactions to the session. Students were asked to circle all adjectives that applied to their feelings during the search. The adjectives circled most often were:

l Interested (17) l Pleased (14)

the Journal of Academic Librarianship, January 1993 349

l curious (13) l Confident (8) l Confused (6) l Grateful (6) l Enthusiastic (6)

The adjectives circled not at all or only once were embarrassed, bored, an- noyed, indifferent, depressed, and ex- hausted; and intermediate numbers of students claimed to be anxious, impa- tient, relieved, absorbed, and frustrated.

The adjectives checked most often have an overall positive tone, yet fewer than half of the students described them- selves as “confident,” and almost two thirds of the students checked at least one “negative” emotion. Previous use of CD-ROM self-service databases and graduate or postdoctoral status seemed to relate very slightly to checking only positive adjectives. More women (8 out of 9) than men (4 out of 10) reported negative emotions during the search, but there was no difference between the sexes where confidence was concerned. Un- dergraduates and previous users of CD- ROM databases were slightly more likely to report feeling confident.

In the follow-up interviews, most students expressed feelings of comfort and confidence with the coached online searching. Some related this to their experience with computers: others saw the search process and results as more important determinants of confidence. Spe- cific confidence factors mentioned included:

putting their thoughts down on paper or using the search planning form,

knowing a library staff member would be present,

attending the lecture prior to the search, feeling that the concepts of searching were easy to understand,

having already found many useful ci- tations in a printed index. Judging from the interviews, student

confidence could be enhanced by encouraging them to make adequate pre- search preparation. For students uncomfortable with computers, offering them a staff-assisted hands-on session with an electronic database when time is not an issue (e.g., offering a guided CD-ROM search session) was also viewed as a way to build confidence.

Understanding the concepts and techniques of searching. In the U-Search program, learning can take place at several junctures:

through exposure to computer searching prior to the U-Search session,

during the pre-search lecture, during the interview at the reference desk, during the actual search, and during the evaluative follow-up interview. Using information from students’ self-

reports, the investigator’s analyses of the search transcripts, and the follow-up interviews, it was possible to evaluate students’ understanding of search concepts and their ability to use appropriate search vocabulary, Boolean logic, field limita- tion, and other useful techniques.

“Student comments showed a clear need for some staff

presence during the first search performed by an individual, but opinions varied on whether the

coach should remain in the search office during

subsequent searches.”

Some students were confused about the resources and system they were using-the contents of various databases, the difference between BRS and the online catalog, and the relationship between databases available through U-Search and on CD-ROM. Many students, however, arrived at their search sessions having made at least a tentative database selection-basing their decisions on information provided in the lecture, the list distributed in the lecture, the advice of faculty members, or previous database experience. Some students were unaware of significant indexes in their fields.

Students were expected to be able to name alternate terms and synonyms, to specify hierarchical relationships, and to list examples of concepts. The investigator’s analyses revealed that most students seemed to have learned the basic principles of selecting vocabulary, although there were some knowledge gaps for individual students. Their most significant deficiency was knowledge about thesaurus use. Since thesauri are often database-specific, thesaurus use was covered by the lecture in only a general way. Apparently, more specific advice was needed during the reference interview or from the coach during the search, since almost half of the students did not

consult a thesaurus either before or during their search. Almost all students un- derstood the concept of truncation when it was explained during the lecture, although some needed reminding while planning the search.

Almost half of the students under- stood Boolean logic before attending the lecture, due to previous exposure in sta- tistics or computer programming classes or while performing CD-ROM database searches, and by the time of the follow- up interview, all but one student knew how to use Boolean operators. Students seemed reasonably aware of the nature of fields in records and how to narrow a search through limiting by field. They did not normally remember the syntax for field limiting until it was explained by the coach. Often, when students were faced with a large number of search results, it was the coach’s suggestion to limit by title and subject descriptor.

Techniques added to the students’ rep- ertoire by the coach included: stripping hyphens from search terms, splitting up a search statement to avoid reaching the character limit, displaying previously cre- ated sets, creating a tailored display for selecting fields to be printed, saving a search and re-executing it in another database. Problems resolved by the coach included: observing that the number of records retrieved in a search was unre- alistically high and must have been caused by line noise, discovering typographical errors, explaining that the system does not understand causation between concepts, and replacing paper and print car- tridges.

Student reactions to coaching. All students made some positive comments about the amount of help they received. Reasons for needing help included the limited amount of online time, the long (in some cases) interval between attending the lecture and performing the search, the unpredictability of system responses, and the notion that since students normally have to pay for U-Search, they should receive help.

Help was often viewed as a step to- ward independence. Students appreciated coaching that involved explaining a technique thoroughly the first time the need for it occurred, then waiting a few mo- ments before prompting the second time to see if the student remembered what to do. Students preferred controlling the pace by their own questions.

350 the Journal of Academic Librarianship, January 1993

The prospect of searching without staff assistance drew mixed reactions. Some students felt they did not need a staff member in the room at all times, but would appreciate someone available in the vicinity to answer questions. Sev- eral felt that if they used the system more often, they would become profi- cient enough to perform their searches solo.

Although all students commented positively on the help, slightly more than half had negative comments as well, with some finding it inadequate and others excessive. In general, they seemed to think that too much assistance was pref- erable to too little. Mixed reactions sug- gest that searchers should feel free to reject suggestions they do not find ad- vantageous and to hurry the process along if they are concerned about the time, and that coaches should be sensitive to both hesitation and impatience on the part of students.

Search success. Most students were pleased with their online search results. Several either had performed or planned to perform additional searches. Some students, however, criticized the diffi- culty or inadequacy of the BRS search software or the time limits and prices imposed by the library.

Discussion The goal of this evaluation was to

discover whether an online search coach was necessary to successful student searches, what experience and training were most appropriate for coaches, and whether the coach contributed to achieving the goals of the educational program. Student comments showed a clear need for some staff presence during the first search performed by an individual, but opinions varied on whether the coach should remain in the search office during subsequent searches.

Since about 40 percent of the 200- 300 comments made by coaches to each student involved providing information unique to the student’s search (topic negotiation, vocabulary assistance, results reviewing), coaches should have experience or training in both the technical aspects of online searching and in instructing others.

The coaches were expected to pro- mote confidence, clear up misunderstood concepts, provide guided practice in interacting with the system, and help en- sure that searchers were satisfied with the results of their searches. Interviews with the students led to some general

conclusions about how to increase student online search success.

Since students linked confidence to advance preparation for the search and to prior understanding of search techniques, some pre-search contact is ad- visable. Preparing for the search in advance can be stressed to the students as a way of increasing their own comfort as well as of improving the quality of their results. Almost all suggestions for improving the program concerned promoting greater independence in the searcher. The instruction sheets provided could give more examples or provide more information on the content and special features of different databases. Students suggested that the coach point out commands on the handouts during the fist search, so that users would be confident about performing the second one independently. Others suggested the inclusion of hands-on practice during training.

Some features that were explained frequently enough during the searches to be added to pre-search assistance included:

saving and executing searches, strategies for searching the same topics on multiple databases, interpreting the handouts and printed search aids, and using the database descriptions list at the reference desk. Students varied in their understand-

ing of search techniques, as well as in the type of exposure (lecture, reference interview, search) that promoted learning. Given the differing backgrounds and learning styles of students, we felt it was important to maintain the multi-experience character of the program.

The Future Since most students required exten-

sive assistance during their first search, it has been decided to retain the coaching program, and indeed to increase the amount of time scheduled for the first- time search from one half-hour to one hour. However, only one half-hour of this initial session will actually be spent online, enabling coaches to explain material more comprehensively before the search. During the search, students will be encouraged by their coach to ask questions, reject the coach’s suggestions if they do not agree with them (or try both their own ideas and those of their coaches in order to compare results), and tell the coach if the process needs to be hastened or slowed down.

Because the number of available commercial databases is growing, and because libraries now provide database access to patrons at remote locations such as their homes, offices and dormitories, it is not practical to coach students through every search. The next issue to address is the development of support systems for remote users. An innovative approach has been taken by one commercial ven- dor, BIOSIS, which offers an online con- sultant for its Life Science Network during the hours of highest use. Users type “SOS” at any time during a search and a professional search specialist responds on the screen. BIOSIS specialists can see, on their own terminals, records of searches input by users, allowing them to guide users through trouble spots.

This approach may not be practical for libraries, except in the case of databases that are locally mounted and very heavily used. Call-in telephone services, however, could be developed, or in some cases expanded. On-screen tutorials could be produced for locally mounted databases. Such services must be heavily pub- licized and readily available at the moment of need. Then users must be encouraged to seek assistance.

References ‘Richard V. Janke, “Online After Six: End User SearchingComesofAge,“Online8(November 1984): 15-29. 2Linda Friend, “Independence at the Terminal: Training Student End Users to Do Online Literature Searching,” Journal of Academic Librarianship 11 (July 1985): 136-141. ‘Mary Jean Pavelsek, “A Case against Instructing Users of Computerized Retrieval Systems,“College & Research LibrariesNews 52 (May 1991): 297-299,301. 4Nicholas J. Belkin and Alina Vickery, Interaction in Information Systems: A Review of Research from Document Retrieval to Knowledge-basedsystems (London: TheBritish Library, 1985). 5Nicholas J. Belkin, “Discourse Analysis of Human Information Interaction for Specification of Human-computerInformation Interaction ,” Canadian Journal of Information Science 12 (No. 314, 1987): 31-42. ‘Tefko Saracevic, “Modeling and Measuring User-Intermediary-Computer Interaction in Online Searching: Design of a Study,” Proceedings of the American Society for Information Science 26 ( 1989): 75 -80, 7Tefko Saracevic et al., “Nature of Interaction between Users and Intermediates in Online Searching: A Qualitative Analysis,” Pro- ceedings of the American Society for Information Science 27 (1990): 47-54. ‘B.S. Bloom, “Thought Processes in Lectures andDiscussions,“JournalofGeneralEducation 7 (April 1953): 160-169. %racevic et al., “Nature of Interaction.” .

the Journal of Academic Librarianship, January 1993 351

helping students during online searches: an evaluation

Documents