structure of domain novice users' queries to a history database

Structure of Domain Novice Users’ Queries to a History Database

Charles Cole, John E. Leide, Emeka Nwakamma, Jamshid Beheshti, and Andrew Large Graduate School of Library and Information Studies, McGill University, 3459 McTavish Street, Montreal, Quebec, Canada H3A 1 Y1. Emails: {charles.cole, johnleide, chukwuemeka. nwakanrna, jamshid.beheshti, andrew.large} @mcgill.ca

This paper presents an information need identification system for interactive information retrieval (IR) for undergraduates researching a history topic, called the INllReye System. The overall purpose of the INllReye System is to facilitate domain novice user identification of their information need while they are online interacting with the information store. Here, we give preliminary results from a study that narrows undergraduates’ initial topic statements to information need statements. Students may use a faulty accessing point in their queries because before information need identification they base their queries on broad topic terms. We first categorize the type of query terms used by users of an historical database provider, to create a taxonomy of query terms. Next, we use a case study of a history student who visually represents the narrows his essay topic in a series of steps. We conclude that our query taxonomy must include levels of topic specificity because while general topic-based queries are inappropriate as query terms, more specific topic-based queries may be closer to the domain novice user’s real information need.

Introduction The problem domain novices such as undergraduates

have formulating an adequate query to an information retrieval (IR) system is well known. That is, the domain novice doesn’t know what information he or she needs to complete a task, so finds it difficult to ask for it in the query to an IR system. Taylor (1968) analyzed th~s difficulty by dividing information need into four levels. The real need, Q1, is unknown to the domain novice; it is “inexpressible” and does not exist “in the remembered experience of the inquirer” (Taylor, 1962, p. 392; 1968, p. 182). 4 2 is the conscious information need, which is eventually expressible if the user receives help from mformation objects to fill in the gaps.

LIS literature has focused on the gap between 43, the user’s formal conception of hisher information need in hislher own mind, and 44, the compromised need. In

ASIST 2003 Contributed Paper

place of the real need, the domain novice expresses his or her need, either to the IR system or librarian, in a sort of code consisting of his or her estimation of the technical or controlled vocabulary words used by librarians andlor the discipline for which the information is being sought. There is a tendency for the user to formulate the query either too narrowly or broadly (Ranganathan, 1962). The user, for example, will ask to see books on birds rather than books on blue jays, the actual information need (Donovan, 1991).

The gap between Q1 and Q2, the main concern of this paper, is due to the problems domain novice users of IR systems have in identifying their information need. The undergraduate is forced by the process of undergraduate essay writing to engage in independent information seeking before this identification takes place. This is due to time and subject constraints. The time constraint is that the undergraduate cannot wait and let information need identification happen on its own; he or she must somehow get the process going. The subject constraint is that in first year undergraduate courses the instructor must stick to relatively broad outlines of a topic during lecture, depending on students’ reading and course preparation to fill in the details necessary for information need identification to take place. The domain novice, and any other user of an IR system, eventually identifies their information need by interacting with information objects, traditionally off-line, or away from the IR system itself (Bates, 1989), then comes to the system with the information need formulated into a query.

The general problem domain novice users such as undergraduates have using IR systems, which is the general problem addressed in our overall research program, is that such users must use the IR system before information need identification has taken place; therefore, they do not effectively query the system for information they need, and cannot judge the IR system output in the results list for relevance to that need. In traditionally designed IR systems, the user has limited avenues of interacting with information objects online, but Internet and interactive technology have expanded the possibility of this interaction by bringing these information objects online into the domain novice user’s information search session.

27 1

The INllReye System Our overall research project is to develop an interactive,

front-end add-on information need identification system for Internet search engines for undergraduates in an exploration phase of researching their essay. Undergraduates are a convenient group with which to start designing such systems as they are readily available as research subjects, but we envisage creating similar applications for a wide variety of user groups.

In Figure 1, we provide an outline of the information need identification system, called the Information Need Identification Information Retrieval eye System (INIIReye System). The INIIReye System is fully described in Leide, Cole, Large, Beheshti, & Brooks (submitted for publication) and will only be briefly described here. The secondary purpose of the INIIReye System is to organize the domain novice’s search of a database, and to enable the domain novice to keep track of where helshe has been and where helshe is going. This navigation function of the INIIReye System is fully described in Leide, Large, Beheshti, Brooks, & Cole (in press) and will only be briefly described here.

The primary purpose of the INIIReye System is to cognitively prepare the domain novice for the process of identifying his or her information need, then to organize the domain novice’s interaction with the information store, contained in the search engine’s database, for information need identification to take place. The domain novice can then select information sources based on their

real information need.

Traditionally, library subject cataloging rules, which bring together surrogates of information items in the library catalog, and classification schemes, which bring together like or similar items on the library shelf, promote user browsing, recognition and information need identification. As with many other IR systems, the INIIReye System uses subject cataloging and classification schemes to act as facilitating access points to the information store for the domain novice user. The difference is that it also collocates the user’s cognition or thinking about his or her topic, bringing together these thoughts into a specifiable information need.

The INIIReye System is made up of 4 separate schemes: the classification and subject cataloging collocation schemes and the associative index and the navigation schemes. The classification and subject cataloging collocation schemes are system or topichnformation object determined, while the associative index and the hyperlink (navigation) schemes are user determined. The associative index scheme is the orchestrating scheme. All four schemes essentially bring like items together in different ways, thus creating the prior conditions necessary for information need identification to occur in the domain novice user.

F D804.3 a Q1 42 4 3 Q4

FIG. 1. The INIIReye System for a domain novice user in the early stages of his or her ISP. (a) Associative index scheme (user chart). (b) Classification scheme. (c) Subject cataloging collocation scheme. (d) Associative index scheme (selector at top; task analyzer at

bottom). (e) Hyperlink scheme (navigation scheme). ( f ) Instantiated associative index, providing the user with an identified information need schema for seeing in his or her mind’s eye INIIReye System output as meaningful (or not) to the task at hand.

ASIST 2003 Contributed Paper 272

The INIIReye System is designed to facilitate the bringing together of the domain novice user’s dunlung about a topic by linking it with the subject cataloging and classification schemes, which are system representations of the user’s topic area. The purpose of doing so is to enable the domain novice user:

1. by readying the domain novice, to create the necessary prior conditions for domain novices to identify their information need; to facilitate the identification of his or her own information need; and to facilitate recognition in subsequent stimulus information, in the IR system’s display output and elsewhere, as being a relevant instance (or not) of an aspect of the user’s information need.

2.

3.

The Navigation Scheme The navigation scheme is sketched into the entire

NIIReye System in Figure l(e). We will briefly detail the navigation part of the INIIReye System here. The navigation scheme takes the domain novice user through a voyage of information discover and information need identification. To ready the domain novice to identify hislher information need through interaction with the classification and subject cataloging schemes, and other information objects in the information store, we put the domain novice in a simulated environment where he or she is a navigator in a ship exploring a Topic Sea, as shown in Figure 2. The navigator has a chart, and determines where helshe is by interacting with lighthouses containing the classification and subject cataloging scheme, as shown in Figure 3.

FIG.. 2. The undergraduate begins the navigation scheme by clicking on the large lighthouse at the entrance to the Topic Sea. The scheme asks the student to list four research questions-answers; each is assigned a numbered triangle. The system distributes the triangles to the four small lighthouses around

the Topic Sea.


FIG. 3. Inside ship: Ship’s Navigator (user’s agent) is looking at chart to locate position. Right side of figure is a Lighthouse (IR system’s agent). Navigator’s chart is used to keep track of voyage through the user’s topic

space.

How The Lighthouse Navigation Scheme Works

To illustrate the functioning of the navigation scheme, we take the example of an undergraduate interviewed for a study of topic-thesis formation while seeking information for a 3rd year genocide hstory course at Concordia University, Montreal, reported in Cole (2001). The student was asked to supply four research questions he would like answered by the information sources, and then he was asked to guess the answers. The four questionslanswers in effect determine the Topic Sea for the student, and the points on the sea she will investigate when stopping at each of the four small lighthouses along the way to the destination (the castle in Figure 2). The student supplied the following research questions- answers:

Student’s Research Question 1: How could somethg like the Holocaust happen? Student’s Answer: Nazis segregated Jews into different groups; said committed crimes, so Jewish population in dark. At a certain point, however, too many Jews taken away.

Student’s Research Question 2: What was daily life like in Warsaw ghetto? Jews treated how? Guards - some helpful?

Student’s Answer: Lack of cleanliness, food, own rules, morals. Demoralization, to control Jews and increase Nazi power.

Student’s Research Question 3: How were the relations between own people (Jews)? Student’s Answer: A few breakdowns - societal conduct [selfish] [anger] [religion breaks down]. [New leaders: secret subgroups to keep control].

Student’s Research Question 4: Psychological aspects: their thoughts about themselves and group changes? Student’s Answer: Spirituality suffers [religion] [life] [frustrated with God].

These four questionslanswers are determined at the first, big lighthouse at the entrance to the Topic Sea in Figure 2. Each of the four questionslanswers is then dispersed through the topic sea in four little lighthouses. In each of the four, small lighthouses, the student is met by a subject cataloging collocation scheme, which at this point in our research program is a syndetic map. A syndetic map is a common accessing device which visualizes a portion of a subject catalog or thesaurus. In Figure 4, we have derived the syndetic map for the Holocaust using the Library of Congress Subject Headings (LCSH). LCSH represents a hierarchy of


broader, related and narrower subject headings connected to the user’s selected subject heading. In the syndetic map shown in Figure 4, we start with the subject heading:

“Holocaust, Jewish (1939-1945),” which we put into the central triangle.

Genocide Jews-Persecutions World War, 1939-1945-Atrocities

Holocaust, Jewish (1939- 1945)

Babi Yar Massacre, Gentiles in the Holocaust

Theology) Underground

FIG. 4. The syndetic map for the central subject term “Holocaust,” translated into LCSH controlled vocabulary “Holocaust, Jewish (1939-1945).” Broader terms are shown above the central triangle, the related term “World War,

1939-1945-Jews’’ is shown beside the triangle, and narrower terms below the triangle.

General Problem Addressed represented in circles, juxtaposed beside a system From our research program, an obvious general generated syndetic map visualizing broader, related and

narrower subject terms. If the domain novice is to learn the structure of hisher topic from the syndetic map, helshe must make a sort of leap from hisher cognitive map of that same topic.

problem we must confront is the disconnect between the domain novice undergraduate’s cognitive map of hisher essay topic and the system generated syndetic map. We graphically represent this disconnect in Figure 5, which shows a user’s cognitive map of hlsher topic, here

Figure 5: A juxtaposition of a student’s cognitive map

of his topic space on the left and the system representation

of the same topic area in a syndetic map.


c‘

275

Specific Problem Addressed In this paper, the specific problem we address is the

problem domain novice users have deriving search terms from system generated representations of controlled vocabulary such as a syndetic map. Our concern for this

Student A

particular aspect or issue of our overall research program was stimulated by the juxtaposition of two cognitive maps from two undergraduates in a study reported in Cole and Leide (2003). In Figure 6, we show these two student cognitive maps side-by-side.

Escalation C Student B Student B

FIG. 6 . Two students’ cognitive maps of their history essay topics taken from a study reported in Cole and Leide (2003). Student A, on the left, put a topic concept as hidher central accessing point, while Student B,

on the right, put a person concept as hidher central accessing point.

In this previous study, the students were asked to draw circles to represent the subject terms for their undergraduate research essay topic. They were instructed to use large and small circles drawn close together or far from each other to indicate importance of concepts and degree of associations of concepts respectively. The students essays were marked by the course instructor. Student A received an A for the essay, while Student B received a B minus. (There were other evaluation mechanisms in this history course so these were not the final marks received by either student for the course.) Because of the similarity of concepts used by Student A and Student B, we wondered from thls anecdotal evidence whether the students’ central accessing points-a topic, “Vietnam Policy” for Student A; a person, “Johnson,” for Student B-affected their ability to find information in the history database used in the study (America: History and Life and Historical Abstracts).

It is our hypothesis that a topic accessing point such as “Vietnam Policy” is a better accessing point for the domain novice user than a person accessing point such as “Johnson” because a topic such as Vietnam Policy has an lnherent structure that is easily comprehended and represented by system or discipline accessing tools such as LCSH (or in textbooks or encyclopedias, books and articles on the subject). A “person” accessing point, on


the other hand, is not a good accessing point for domain novice users because it either does not have an inherent structure or that structure is not easily comprehended or represented to the novice user.

The Specific Problem investigated in the study reported in this paper is the structural problem (query type and level of specificity) of the accessing query of a domain novice IR system user.

The Study The study is divided into two phases. In the first phase,

we analyzed query logs from ABC-Clio, the owner and distributor of the scholarly history databases America: History and Life and Historical Abstracts, to determine types of queries by broad category. In the second phase, the case study reported here is part of a larger study investigating domain novice accessing issues. The purpose of the larger study is to build linlung devices between domain conceptualizations of their essay topic and system generated representations of the topic space such as syndetic maps. The case study is an undergraduate history student registered in a McGill University hstory course in the winter term 2003. McGill University is one of North America’s oldest universities, offering a full range of undergraduate and graduate programs.

276

Methodology In the first phase of the study, the methodology used

was to analyze the query logs for one month chosen at random, lent to us by the owners and distributors of America: History and Life and Historical Abstracts, ABC- Clio Inc. The logs analyzed contained the first 1,000 queries sent to ABC-Clio’s databases for the month of June 2002. ABC-Clio Inc. uncompresses the first 1,000 queries (out of approximately 50-200,000 queries received per month) to gauge system functioning. We assume sufficient randomness to this 1,000 query sample for the present purposes. (We are presently completing cluster analysis of the logs, to be reported at a later time.) The 1,000 queries were analyzed for the type of query used by the person making the query. A preliminary taxonomy of query types was created. The 1,000 queries were then put into a taxonomy consisting of five categories of queries, shown in Table 1.

Personality/ Time Names Number = I 62 queries 29 queries

In the second phase of the study, a case study, taken from a larger study of 60 undergraduates, was investigated to explore the notion of undergraduate or domain novice accessing points for a history course research essay. In this case, after signing a ethics consent form, an undergraduate volunteer was asked questions about his essay topic, and then asked to visualize his essay topic using various visual means, including circles and connecting arrows. The interview was videotaped for later analysis.

Results For phase 1 of the study, Table 1 indicates the five

categories of queries used in the 1,000 sample queries from the ABC-Clio logs. The 1,000 queries were assigned to one of these five categories (about one quarter of the queries were left unclassified).

Space Event Society Unclassified

154 queries 48 queries 420 queries 267 queries

For phase 2 of the study, the case study interview had five parts.

Part 1: The student was asked to state his topic and list out the concepts, terms or keywords in his topic. He was then asked to label them using capital letters.

A. Reconstruction B. McKenzie King Government C. Welfare State CANADA D. Social Programs (1939-1945) E. Department of Reconstruction

The student then drew a curved line linking A. and E. together, linked the five topics to the word Canada, which was written to the right of the five concepts.

Part 2 and Part 3: The student was asked to visualize the concepts fiom Part 1 in any manner he wished, Figure 7 (left side). The student was then asked to visualize the same concepts using circles and lines or arrows, using circle sizes to indicate importance of the concept in the essay, and distance between circles to indicate closeness of association of concepts, Figure 7 (right side).

C D FIG. 7. On left, the student’s own visualization of his essay topic; on right, student’s drawing using circles and lines.


Part 4: The student was asked to list four questions he wanted the information sources to answer, resulting in:

1.

2. 3. 4.

How was reconstruction developed by the King government? Who were the main people involved with h s ? What was the public’s reaction to this measure? With what type of economic policies were put into place with this?

The student was asked to rank order the above questions in order of importance, resulting in:

1.

2.

3. 4.

With what type of economic policies were put into place with ths? How was reconstruction developed by the King government? What was the public’s reaction to this measure? Who were the main people involved with this?

Part 5: The student was asked to choose from (a) the list of concepts, (b) the student’s own visualization of the concepts, and (c) the visualization using circles and lines. The student selected (b), his own visualization. The student was asked to map the 4 questions from Part 4 onto the selected visualization, as shown in Figure 8.

C D

FIG. 8. Student maps four questions he wants information sources to answer onto selected visualization

Part 6: The student was asked to list his concepts, terms or keywords for his essay in light of the interview, resulting in:

1. Post-war reconstruction policies 2. King’s economic polices 3. Canadian economic policies (1939-45) 4. Social security (1939-45)

Discussion and Conclusion The study is a preliminary investigation only, giving the

researchers a first opportunity to explore issues related to domain novice access points. In the study’s first phase, we analyzed the first 1,000 queries from ABC-Clio’s query log for June 2002. We then assigned category labels to create a query taxonomy. Fully one quarter of the queries were not amenable to our taxonomy.

We are in a very preliminary phase of investigating the type of queries sent to ABC-Clio’s history databases. A subsequent research phase will associate query type with information acquisition and processing by domain novice users of IR systems, divided into decoding and encoding phases of their research process. It is possible that domain novices using IR systems in an exploration phase of their research are primarily decoding information received from an IR system. This is in contrast to domain experts who may be primarily encoding information received from an IR system. For example, in Cole (2000), domain experts in history were found to use “names” as accessing points to information sources, which may be indicative of an encoding type of information acquisition, while the anecdotal evidence referred to earlier in this paper, where one undergraduate receiving a low mark from the course instructor had a “name” focused concept map while another student had an “event” focused concept map and received a high mark, may indicate domain novices engage in a decoding type of information acquisition, requiring a decoding type of accessing query (e.g., a topic query rather than a “name” query).

In the study’s second phase, we interviewed a single history student to investigate how students mentally visualize their history essay topic. We then asked the student to think of his essay in more specific terms than topic by asking him to list-out four questions he wanted the information sources to answer. (We make the assumption that asking him to do this had the effect of making him think in a more specific way than topic, and that these specific questions are closer to his real information need.)

We will make preliminary observations about the case study, which we are now applying to a larger study involving 60 McGill history students. We divided the student interview into 6 parts. In Part 1, the student listed out concepts, terms or keywords that constituted his topic. These were extremely general, topic level concepts. In Part 4, the student was asked to list four research questions. In Part 6 , when we asked the student to list-out the concepts, terms or keywords that now constituted his topic, he listed much more specific concepts that were in fact closer to what we as information scientists consider representations of an undergraduate’s information need. By facilitating student cognition in Parts 2-5, the student


was able to achieve focus, if we use Kuhlthau’s (1991) terminology. The student’s last comment to us before leaving was in fact: “I’ve got a better focus on it.”

Here, we make general conclusions only regarding promising avenues of research inquiry for the development of our INIIReye System. The purpose of the INIIReye System is to overcome the disconnect between user and system by facilitating the linking of a domain novice user’s preliminary conceptions of hisher essay topic to system representations of the topic space (e.g., syndetic maps). The study in 2 phases reported here indicates that the disconnect may be due to an “inappropriate” type of query to the system. Secondly, the study indicates that the specificity of the query may be inappropriate but can be made more specific with the help of an IR system whch includes as its mandate preparing and facilitating user identification of their real information need.

We would like to end t h ~ s paper by indicating a next step in our study. The case study reported here shows that topic-based queries can be overly broad (for efficacious accessing of the database). A rigorous analysis of the ABC-Clio query logs, from the point of view of query type and query specificity, may allow us to assign specificity levels to user queries.

ACKNOWLEDGMENT Funding for the research supplied by the Social

Sciences and Humanities Research Council of Canada (SSHRC).

REFERENCES Bates, M. (1989). The design of browsing and benypicking

techniques for the online search interface. Online Review, I3(5), 407-424.


Cole, C. (2000). Inducing expertise in history doctoral students via information retrieval design. Library Quarterly, 70, 86-1 09.

Cole, C. (2001). Intelligent information retrieval: Part IV. Testing the timing of two information retrieval devices in a naturalistic setting. Information Processing & Management,

Cole, C., & Leide, J. E. (2003). Using the user’s mental model to guide integration of information space into information need. Journal of the American Society for information Science & Technology, 54(1), 39-46.

Donovan, J. M. (1 99 1). Patron expectations about collocation: measuring the difference between the psychologically real and the really real. Cataloging & Classification Quarterly, 13(2),

Kuhlthau, C. C. (1991). Inside the search process: information seeking from the user’s perspective. Journal of the American Society for Information Science, 42(5), 361-371.

Leide, J. E., Large, A., Beheshti, J., Brooks, M., & Cole, C. (in press). Visualization schemes for domain novices exploring a topic space: the navigation classification scheme. Information Processing 8~ Management.

Leide, J. E., Cole, C., Large, A., Beheshti, J., & Brooks, M. (submitted for publication). Putting it together online: information need identification for the domain novice user.

Ranganathan, S. R. (1962). Elements of library classification. Bombay: Asia Publishing House.

Taylor, R. S. (1962). The process of asking questions. American Documentation, 13, 391-396.

Taylor, R. S. (1 968). Question-negotiation and information seeking in libraries. RQ, 29(3), 178-194.

37, 163-182.

23-43.

279

structure of domain novice users' queries to a history database

Documents