Download - Modeling a Domain Ontology for Cultural Heritage Resources ...cpattuelli.files.wordpress.com/2010/04/21453_ftp.pdfModeling a Domain Ontology for Cultural Heritage Resources: A User-Centered

Modeling a Domain Ontology for Cultural HeritageResources: A User-Centered Approach

M. Cristina PattuelliSchool of Information and Library Science, Pratt Institute, 144 West 14th Street, New York, NY 10011-7301.E-mail: [email protected]

The use of primary source materials is recognized askey to supporting history and social studies education.The extensive digitization of library, museum, and othercultural heritage collections represents an importantteaching resource. Yet, searching and selecting digitalprimary sources appropriate for classroom use can bedifficult and time-consuming.This study investigates thedesign requirements and the potential usefulness of adomain-specific ontology to facilitate access to, and useof, a collection of digital primary source materials devel-oped by the Library of the University of North Carolinaat Chapel Hill. During a three-phase study, an ontologymodel was designed and evaluated with the involvementof social studies teachers. The findings revealed thatthe design of the ontology was appropriate to support theinformation needs of the teachers and was perceived as apotentially useful tool to enhance collection access. Theprimary contribution of this study is the introduction of anapproach to ontology development that is user-centeredand designed to facilitate access to digital cultural her-itage materials. Such an approach should be consideredon a case-by-case basis in relation to the size of the ontol-ogy being built, the nature of the knowledge domain, andthe type of end users targeted.

Introduction

While a great deal of attention has been paid to ontologydevelopment in recent years, especially in the context of thesemantic Web, building ontologies remains a challenge inregard to both design and implementation. Ontologies rep-resent agreed domain semantics. Modeling a domain is acritical component of the ontology building process. Devel-opers are responsible for making choices that will impactthe effectiveness and ultimately the usefulness of the tool.A possible target for domain ontology development is thevast and impressive collection of cultural heritage materialsthat libraries, archives, and museums have spent the past fewdecades digitizing. These individual collections are a rich

Received April 18, 2010; revised August 20, 2010; accepted September 27,2010

© 2010 ASIS&T • Published online 16 November 2010 in Wiley OnlineLibrary (wileyonlinelibrary.com). DOI: 10.1002/asi.21453

source of instructional material for history teachers and arefundamental to a strong, inquiry-based social studies edu-cation. Nevertheless, they remain largely underused by theeducation community, partially because of the challenge ofaccessing and selecting digital primary sources appropriatefor classroom instruction. Although still scarce if comparedwith the wealth of digital content available, new discov-ery tools and support services are emerging to facilitatediscovery and use of such resources. Among those tools,ontologies are gaining increasing attention for their potentialto improve functionality and, more specifically, to facili-tate subject access to digital content (Greenberg, & Méndez,2009; Kruk & McDaniel, 2009).

Is it possible to design an ontology with any assurancethat it will be useful to the end users it is intended to support?With this research question in mind, this study investigatesa user-centered approach to the development of an ontol-ogy for the domain of history. To test this hypothesis, theresearcher chose a small collection of primary source mate-rials digitized by the University Library of the University ofNorth Carolina at Chapel Hill (UNC-CH) for research andinstructional purposes. The study also focuses on only one ofthe intended audiences for the collection—middle and highschool social studies teachers. It is suggested that improvingour understanding of how educators search, select, and incor-porate digital cultural heritage materials into their teachingpractice can assist in the development of domain-specificontologies designed to facilitate educational access to dig-ital cultural heritage resources. The primary aim of this studyis to confirm whether or not an ontology designed from a user-centered perspective is ultimately more useful in meeting theneeds of a digital collection’s intended audience.

Literature Review

Digital primary source materials are increasingly val-ued as educational resources that support effective strategiesfor inquiry-based teaching and learning (Coventry & Bass,2003). Academic libraries and cultural institutions are begin-ning to recognize the need to be responsive and to actively

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 62(2):314–342, 2011

participate in this new learning environment (McLean &Lynch, 2004). However, the use of digital libraries for educa-tional purposes represents a serious challenge that developersof digital libraries have just begun to recognize and address(Sumner, Khoo, Recker, & Marlino, 2003). The wide rangeof uses that cultural digital collections may enable and sup-port, and the variety of user communities that may benefitfrom these collections, make it important to rethink how thesematerials are discovered, presented, and used. As Borgmanet al. (2005) point out, tools and services to facilitate schooleducators in their teaching and learning practices are still veryscarce if compared with the wealth of digital content avail-able to them. Tailoring digital collections to their intendedcommunities of users—“customization by community” inLynch’s words (2003, p. 196)—can be the key to bridgingthe distance between users and resources.

The limitations of traditional knowledge representationand organization methods are seen as a major impediment tothe discovery of and access to digital collections. In general,libraries have treated digital materials as they have treated thephysical information objects under their stewardship, that is,by cataloging them with descriptive and semantic standardsdesigned for traditional search and retrieval methods. Theshortcoming of this approach has been particularly confound-ing for collections of digital cultural heritage materials, thecontent of which poses unique challenges to its representation(Baca, Harpring, Lanzi, McRae, & Baird Whiteside, 2006).The limitations of subject access in online library catalogshave been extensively analyzed and described in past stud-ies (Sridhar, 2004; Fischer, 2005). One of the major factorsreducing the effectiveness of subject searches is the difficul-ties users have in formulating subject queries that match theLibrary of Congress Subject Headings used by most aca-demic libraries (Larson, 1999). The necessity to developalternative subject access tools has long been recognized(Lancaster, Connell, Bishop, & McCowan, 1991). The inte-gration of library catalogs with other discovery tools thatwould complement and possibly augment or enhance currentsubject access has been at the center of recent discussions onthe future of cataloging (Calhoun, 2006; Coyle & Hillmann,2007, Library of Congress, 2008). The usefulness of seman-tically enriched and domain-specific metadata to improvedigital cultural heritage resource discovery and retrieval hasalso been explored, leading to a growing interest in applyingsemantic Web technologies to cultural heritage collections ofmaterials (Benjamins et al., 2004; Hyvönen, 2009).

Ontologies are seen as a possible solution for improvingdiscovery and access to digital content and as such havebecome an emerging area of study in the field of digitallibrary research (Greenberg & Méndez, 2009). The potentialof semantic technologies, ontologies in primis, for improvingdigital library functionality and facilitating interoperability,sharing, and reuse of knowledge across repositories is thefocus of a series of research projects. The notion of a semanticdigital library is beginning to emerge (Sure & Studer, 2005;Kruk & McDaniel, 2009). Fast and Campbell (2001) discussthe usefulness of ontologies for providing more granular

access to digital content and enabling content aggregationbased on the needs of the users. Within the field of education,the roles that ontologies could play have also generated agrowing body of research, especially in the area of e-learning(Naeve, Lytras, Nejdl, Balacheff, & Harding, 2006). The needfor services and tools that aid teachers’discovery, access, anduse of digital resources for educational purposes beyond thesupport of traditional metadata standards has been a matterof concern for several years.

As the literature suggests, ontologies and ontology-drivenmetadata have the potential to significantly enhance infor-mation systems, including digital libraries and educationalapplications. However, ontologies are a rather novel tool inthe context of the Web and digital libraries. Despite the largecorpus of ongoing research on ontologies, primarily as the keyinfrastructure to enable the semantic Web, real-scale appli-cations are limited. The number of ontologies developed in awide range of domains is rapidly increasing, but few have leftthe research labs (Hepp, 2007). The technical and economicchallenges associated with the development and deploymentof a domain ontology are far from trivial. DeRidder (2007),who discussed various applications of ontologies in digitallibraries, stresses the need to carefully weigh issues of feasi-bility, scalability, and usefulness in relation to cost and effortrequired to create ontologies for digital libraries. The rangeof benefits that ontologies may offer has to be considered inlight of the technical challenges that the development of suchtools poses. The cost benefits of ontology development arenot yet known (Bontas & Mochol, 2006), especially for man-ually constructed ontologies that are typically labor-intensiveand error-prone (Ding & Foo, 2002).

Building ontologies for information systems “remains anarcane art form” (Guarino & Welty, 2002, p. 61). One ofthe greatest challenges in constructing an ontology is thescarcity of standard methodologies. As Ceusters, Smith, andGoldberg (2005) point out, there are no standards compara-ble to ISO standards for ontology development. Ontologicalengineering, the discipline applied to ontology developmentand use, is relatively young, especially compared with moreestablished fields such as software engineering or knowl-edge engineering (Fernandez-Lopez, 1999). As a result,ontological engineering does not yet have a solid set ofmethodologies based on proven principles. In recent years, anincreasing number of studies have focused on methodolog-ical issues (Bouaud, Bachimont, Charlet, & Zweigenbaum,1995; Gruber, 1993; Mizoguchi, Vanwelkenhuysen, & Ikeda,1995; Noy & Hafner, 1997; Noy & McGuinness, 2001;Uschold, 1996; Uschold & Gruninger, 1996). Several ofthese studies have attempted to define principles thatwould systematically guide the development of ontologies.Others, such as Fernandez-Lopez and Corcho, Fernandez-Lopez, and Gomez-Perez (2003), simply offer extensiveoverviews of methodologies for ontology construction, whilestill others, like Beck and Pinto (2002) and Pinto andMartins (2004), analyze and compare the most representa-tive methodologies, techniques, and guidelines. It has beenargued that a unified ontology engineering process that is

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2011 315DOI: 10.1002/asi

flexible and extensible enough to enable developers to adoptand combine different components appropriate for differ-ent application scenarios is needed (Simperl & Tempich,2006). One of the most comprehensive and well-establishedmethodologies used for constructing ontologies is Methon-tology, which offers a general framework that defines designcriteria, practices, activities, and tools for ontology engi-neering. Methontology was developed by the Laboratoryof Artificial Intelligence of the Polytechnic University ofMadrid, and it is based on a real-world ontology-buildingapplication in the chemical domain inspired by knowledgeengineering techniques (Fernandez-Lopez, Gomez-Perez, &Juristo, 1997).

The conventional model of ontology design and construc-tion is system-oriented. Typically, knowledge engineers, whomake their own interpretations of the knowledge to be for-malized, build ontologies. Domain experts may be involvedin the process, but end users are typically excluded from theentire lifecycle of ontology development. Although the user-centered approach is broadly adopted in system developmentas a strategy to improve system usefulness and usability andto overcome the limitations of traditional system-centereddesign (Gould, Boies, & Lewis, 1991; Mao, Vredenburg,Smith, & Carey, 2005), ontology development methods thatfactor in end users are not common. Only a handful ofstudies have addressed this approach. Holsapple and Joshi(2002) conducted one such study. They present a collabo-rative approach to ontology-building based on the Delphimethod, where the ontology is a result of “a joint effortof people’s experiences and points of view” (p. 44). Thisapproach is shared by Mizoguchi (2003), who argues that“an ontology should be shared by many people in nature. Ifit is not shared by a community, it loses its utility. As a con-dition to reach such a goal, an ontology should be designedcollaboratively, with a happy agreement on its developmentin a community” (p. 373). Consulting with members of theintended audience is central to developing a domain modelthat best reflects the mental model of the community. Forexample, one of the keys to the success of the gene ontol-ogy (GO) lies in its construction from within the biologycommunity rather than from external knowledge engineers(Bada et al., 2004).

Study Overview

This study investigates methodological issues concerningthe design of an ontology in the domain of history, specif-ically a segment of U.S. history. Based on the assumptionthat a domain ontology can better serve its function when itis designed with the end users in mind, the study proposesa process for constructing an ontology that involves the par-ticipation of the intended community of end users duringthe design stage of the process. Toward this end, a proto-type ontology model was designed for a collection of digitalprimary sources and later evaluated with the participation ofsocial science teachers, who represent one set of potentialusers of the chosen collection.

A collection of digital primary materials from the UNC-CH university library provided the knowledge domain forthe ontology and served as the testbed for the study.The collection, entitled “Tobacco Bag Stringing” (TBS),1

describes the lives of individuals and families, in the tobacco-growing regions of North Carolina and Virginia, who areinvolved in the cottage industry of tobacco bag stringing. It isbased on a report prepared by the Virginia-Carolina ServiceCorporation in their effort to lobby against Roosevelt’s FairLabor Standards Act and the minimum wage it required(Stutz, 1939). The contents of the report included pho-tographs and rich descriptions of 147 men, women, andfamilies impacted by the New Deal legislation, along withan assortment of letters and other supporting documents. Itoffers a powerful and unique look at life during the GreatDepression in the South and has enormous educational poten-tial. The report was digitized; its component parts separatedand cataloged separately so they could be used and reusedin different instructional contexts. The university librarianshad created the TBS collection primarily, although not exclu-sively, to facilitate teachers’ access and use of the libraryprimary sources in the context of a broader project intendedto repurpose their digital content for educational purposes.Once fully developed, the TBS ontology would serve as anindexing tool to provide semantically enhanced annotationof the digital content of the collection.

The TBS collection represented an ideal subject for thestudy. First, the digitized primary source material covers animportant historical period, the Great Depression and theNew Deal era, both of which are well-suited for a varietyof instructional objectives. Further, it reflects a defined andthus manageable segment of the knowledge domain to bemodeled. Finally, the UNC-CH university library staff wasinterested in and supportive of exploring innovative meth-ods of making their digital collections more accessible to theeducation community.

For the purposes of the study, an initial prototypeontology model was designed. The design process wascarried out using Methontology, the ontology engineer-ing methodology discussed earlier. Methontology uses aniterative process based on evolving prototypes. The pro-cess consists of a series of activities required for con-structing and maintaining ontologies that encompasses theentire lifecycle of the ontology. These activities are asfollows: specification, knowledge acquisition, conceptualiza-tion, formalization, integration, implementation, evaluation,documentation, and maintenance. Notably absent from theprocess, however, is the perspective of the end users. To fac-tor in the user perspective and translate it, whenever possible,into a set of design requirements, the study introduced twonew steps or subactivities to the ontology engineering pro-cess intended to be performed during the design phase. Thetwo methodological additions are illustrated in Figure 1 andinclude

• a preliminary user study conducted in the form of personalin-depth interviews and

316 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2011DOI: 10.1002/asi

• a user-centered evaluation of the ontology model performedbefore the formalization stage.

The first addition to the methodology is represented bya user study conducted through a series of individual inter-views with representatives of the intended audience of theontology-enhanced collection. The purpose of this activitywas to collect background information about the needs, work-ing practices, knowledge level, and expectations of the endusers to inform the design of the ontology. The second addi-tion is the evaluation of the ontology at the stage of modelprototype. The evaluation is performed from the perspectiveof end users and is conducted to review and strengthen themodel before the formalization activity.

Research Questions

Based on the assumption that the TBS ontology can betterserve its target audience if the perspective of the intended endusers in terms of work practices, searching needs, and levelof knowledge is considered during the design process, thestudy addresses two specific research questions:

Is the ontology model appropriate to capture and representteachers’ searching needs?

Is the ontology perceived to be useful by the teachers in theirseeking process?

Study Design

To address the research questions, the study comprisedthree phases as follows.

Phase 1 is a set of interviews intended to investigate socialstudies teachers’ instructional practices, information needs,tasks and expectations and that was conducted to gather back-ground information to help determine whether an ontologywould be helpful and what kind of ontology would be pre-ferred. This phase involved semistructured interviews withsix middle and high school social studies teachers. The inter-views were aimed at gathering information on the teachers’perspective on the content domain they address in their teach-ing and their preferences in terms of access and use of thedigital primary sources.

Phase 2 is an ontology model that was designed to rep-resent a subdomain of North Carolina History and reflectsthe subject area of the TBS collection of digital objects.The design of the model was informed by the backgroundinformation gathered in phase 1. The development processincluded the following activities: specification, knowledgeacquisition, and conceptualization. The outcome of this phasewas a model prototype that was further tested by end users inthe third phase.

Phase 3 is a user-centered evaluation of the model proto-type that was conducted to assess the quality of the design ofthe ontology and its usefulness as perceived by the intendedcommunity of end users.

Phase 1: Study of the End Users

Methodology

In phase 1, six in-depth interviews were conducted withmiddle and high school social studies teachers aimed atgathering information on the teachers’ perspective on thecontent domain they address in their teaching and their pref-erences in terms of access and use of the digital primarysources. An extensive review of the literature helped to framethe interview questions by indicating key issues in the con-text of history education and the use of primary sources inclassroom instruction that should be addressed. The goal ofthe interviews was to contextualize the teachers’ experienceusing or attempting to use digital primary materials in theclassroom to help determine whether an ontology would beuseful and, if so, what kind of ontology would be preferred.

Participants

Six social studies teachers from high schools and mid-dle schools throughout Orange, Durham, and Wake Countiesin North Carolina participated in the study. Only participantswho had used digital primary resources were invited to partic-ipate. The researcher explored issues surrounding the searchfor, and use of, digital primary source materials in the class-room. Subjects were recruited by direct solicitation throughan e-mail sent to the list of participants in the UNC Library’sDocumenting theAmerican South (DocSouth) Summer Insti-tute and the list of the cohort of students enrolled in theUNC Master of Education for Experienced Teachers programobtained from the School of Education. The pool of partici-pants was evenly distributed by sex and their ages ranged from25 to 50+ years. Likewise, their teaching experience varied,from 1 to 29 years. All had additional advanced degrees inhistory and education. They taught in both suburban and ruralschools, medium to large in size. The student body popula-tion of each school differed slightly in economic status butthe majority were middle class and predominantly White.Teacher participants were all trained in the use of technol-ogy for instruction. All but one had Internet access in theirclassrooms.

Procedures

The interview protocol included 10 questions about howteachers use primary sources in the classroom, how theysearch and find primary sources, and the broader con-text of their teaching of U.S. history. The semistructuredformat of the interviews allowed specific questions and top-ics to be addressed while offering participants an opportunityto give additional feedback and elaborate further on anyaspect of their experience they considered relevant to thestudy. Through the use of open-ended questions, participantswere invited to provide examples related to their profes-sional experience. Interviews were audio-recorded and latertranscribed.


Findings

Although limited in scope, the responses from the six par-ticipants offered a clear view of the type of resources socialstudies teachers need, what they are able to find and not find,and what they would like to find. Their comments revealedhow primary sources were integrated into the classroomand described the strategies they adopted to apply currentpedagogical principles to cope with institutional constraints.

The main themes that emerged concerned the teachers’general use of primary sources, the role of educational stan-dards and standardized testing in delivering instruction, andthe search strategies, habits, problems, and expectationsof the teachers. A summary of the interview findings ispresented below.

• Teachers generally have a positive attitude about technology,and they use digital resources and value the benefits of primarysources for teaching history according to current pedagogicalprinciples.

• They need to save time and quickly find what they need for theclass. They find the search services they typically use unsat-isfactory, including commercial search engines and librarycatalogs. Commercial search engines return overwhelminglists of results and little functionality to narrow them downto select reputable resources. Special digital collections areoften hard to use.

• Curriculum standards represent the context for planning,instruction, and sources, from which guidance for the dis-cipline content and the pedagogical approach to the subjectmatter are derived.

• Performance-award programs and related high-stakes testssignificantly limit the time they can dedicate to foster inquiry-based learning and even to fully address the standard courseof study.

• Visual materials are a more powerful means to engagestudents than textual materials.

• Teachers consider topicality a primary aspect to search andselect results relevant to them.

• For teaching history in ways that promote critical thinking, itis important to compare and contrast resources, ideas, people,and institutions.

• It is critical for teachers to contextualize the content theypresent. They find it useful to compare and contrast throughtime and space.

• A good strategy to keep students engaged is to make connec-tions between students’ lives and the lives of people from thepast. Students relate better to materials that show or describeindividuals their own age.

This study revealed that teachers operate in a landscapewhere factors such as instructional practices, personal teach-ing styles, pedagogical principles, curriculum standards, andinformation sources shape and constrain the way they search,select, and use digital library content for instruction. Suchcontextual information helped to identify instructional sce-narios and tasks or activities in which an ontology wouldhelp the seeking process and add value to the search system.For example, teacher participants indicated that comparingand contrasting resources and relating historical content tostudents’ lives were important strategies to teach students

interpretative skills. Moreover, feedback from the interviewsencouraged the use of the teaching standards as an integra-tive source for knowledge acquisition. One of the main tenetsthat emerged is that people, time, space, and domain-specificconcepts are central dimensions in teaching history. Teach-ers’ input was then utilized to inform the ontology designprocess, especially regarding decisions on how to model thedomain knowledge of the collection.

Phase 2: Design of the Ontology ModelPrototype

Methodology

The development of the TBS ontology followed Methon-tology as the general methodology framework. As discussedearlier, two additions were introduced to the framework.The first was represented by the interviews with teachers thathelped to guide modeling decisions based on the teachers’expressed information needs and work practices. The sec-ond addition was the evaluation of the ontology prototypemodel performed in phase 3. What follows is the descriptionof the activities involved in the design of the TBS ontol-ogy model. Because this study focuses on the design processwith the aim of modeling a prototype to be later evaluated,only the activities specifically focused on the design com-ponent of the development process are addressed, includingspecification, knowledge acquisition, and conceptualization.

Specification. The specification phase includes the defini-tion of the purpose and the scope of the ontology.As discussedearlier, the general purpose of the TBS ontology is to facilitateaccess to and ultimately use of the TBS collection. One waythe ontology can improve access is by semantically enhanc-ing the annotation of the digital content of the objects. Tothis end, the primary purpose of the TBS ontology is to serveas an indexing tool to support semantic markup of the TBScollection.

The scope of the ontology primarily reflects the knowl-edge domain or subject matter covered by the TBS collection.The TBS ontology is domain-specific, as domain ontologiesare viewed as suitable sources of semantics for describingWeb resource content. In ontological engineering, one of themethods for defining the scope of the ontology is based onthe use of competency questions that should be formulatedin relation to the ontology application scenario (Gruninger &Fox, 1995). Competency questions created in the initial stagesof the ontology-building process are utilized as benchmarksto test the efficacy of the ontology in providing satisfyinganswers. Competency questions are typically formal and areintended to verify whether the ontology contains a necessaryand sufficient set of axioms to represent and solve compe-tency questions (Gruninger & Fox, 1994). However, they canalso be informal and intended to be answered by the ontol-ogy on the basis of motivating scenarios (Pinto & Martins,2004). Formulated at the beginning of the development pro-cess, they do not need to be exhaustive but they are drawn to


help reflect on whether the ontology contains enough infor-mation to answer these types of questions and whether theanswers require a particular level of detail or representation ofa particular area (Noy & McGuinness, 2001). In the contextof this study, informal competency questions and motivatingscenarios were derived from teachers’ real-world examplesdescribed during the in-depth interviews. They were alsoinformed by the North Carolina Standard Course of Study.These questions served to frame the subject domain in theearly stages of development of the TBS ontology. They werelater adopted as task questions within the ontology evaluationin phase 3 of the study.

Competency Questions

Imagine that you need to prepare a class on the GreatDepression and you would like to find some primarysources that illustrate the following aspects of the historicalperiod:

• Assess the impact of the Great Depression on the day-to-daylife of a low-income family in Virginia in the 1930s.

• Analyze the effects of the New Deal policies on the Depres-sion Era life of the women in North Carolina.

• Identify the role of tobacco in the economic development ofNorth Carolina in the 1930s.

• Describe the differing impact of the Depression on variousminority groups.

• What was life like for children of your age in different areasof North Carolina during the Great Depression?

Knowledge acquisition. Knowledge acquisition was per-formed manually because of the small size of the ontologyprototype and the amount of relevant nontextual contentpresent in the collection. Knowledge acquisition was car-ried out through document and text analysis of the targetcollection and through knowledge elicitation from the inter-views with teachers discussed earlier. The textual content ofthe digital collection provided the main set of conceptsof the domain. Background knowledge was derived from theVirginia-Carolina Service Corporation’s report—an exten-sive report on tobacco stringing operations in North Carolinaand Virginia written between 1938 and 1939. This docu-ment contains descriptions and photographs of the familiesthat relied on tobacco stringing labor in Wilkes Countyand Reidsville, North Carolina, and Richmond and SouthRichmond, Virginia, and it provided the basis for the TBScollection.2 Several knowledge organization systems wereconsulted in developing the TBS ontology including Word-Net and the Getty Thesaurus of Geographic Names (TGN).3

Lexical resources like WordNet are helpful in capturingthe nuances of the language, providing both generalityand consistency (Guarino, 1997). For this study, WordNetproved to be a useful reference tool for suggesting con-cepts to be manually incorporated and to help frame theontology structure. The TGN also provided a useful ref-erence tool for standardized place names in the geospatialsegment of the ontology. Reference materials relevant tothe field of history were utilized for gathering background

and domain-specific terminology, including the Encyclope-dia of Southern Culture (Wilson & Ferris, 1989), whichfocuses on the American South. Terms and concepts werealso derived from the history textbook American History:The Modern Era Since 1865 (Ritchie, 2001), history Websites (e.g., The New Deal Network4), and lesson plans fromeducational Web sites (e.g., LEARN NC5) as suggested byinterviewees and faculty at the UNC School of Education.Reuse of existing knowledge was limited because of a lackof suitable sources. Although the number of ontologies inelectronic form and the services to facilitate the discoveryof Web-based ontologies is continuously growing, ontolo-gies available in the domain of U.S. history and culturalheritage in general are very few (Nagypal, Deswarte, &Oosthoek, 2005). One of the most mature initiatives is rep-resented by CIDOC Conceptual Reference Model (CRM),a core ontology expressing upper-level concepts commonacross cultural heritage documentation (Crofts, Doerr, Gill,Stead, & Stiff, 2010). Developed within the museum com-munity, CIDOC CRM has the broader goal of enabling asemantically enriched information exchange between muse-ums, libraries, and archives (Gill, 2004).Although not a directsource of knowledge for the TBS ontology, CIDOC CRMwas viewed as an appropriate overarching framework wherea full-fledged TBS ontology could be linked and related toother germane domain-specific ontologies.

The knowledge elicited during the user study in phase 1was an integral part of the knowledge acquisition develop-ment phase. Although interviewing teachers was not pri-marily intended for knowledge elicitation, their descriptionof their information searching and seeking practices andteaching scenarios were an invaluable source of knowl-edge for the development of the ontology. They offeredinput on areas of knowledge domain and perspectives theyfind important when delivering instruction. For example,teachers indicated clusters of domain concepts and themesthat they consider essential or useful in teaching U.S. his-tory and the Great Depression (e.g., what was it like toattend school in North Carolina during the Great Depres-sion). Teachers’ descriptions of their instructional methodsand practices offered suggestions on candidate propertiesto be associated with concepts and on potentially usefulways to link the concepts. For example, they indicated theimportance of contextualizing content and of comparing andcontrasting resources as part of their pedagogy.Also, teachersgave specific guidance on the desirable level of granularityfor specific areas of knowledge (e.g., for geospatial con-cepts, they indicated “county” as the most specific levelfor which they would search). One of the main tenets thatemerged from the user study was that people, time, space, anddomain-specific concepts are key elements in the teachingof history. These dimensions served to shape the concep-tual structure of the ontology and were translated into acore of upper-level categories that provided the foundationfor the ontology knowledge framework. The outcome of theknowledge acquisition process, including the knowledge elic-itation phase, was a list of terms and candidate concepts


and their relationships. Once the terms were identified, thenext step was to address the conceptualization underlyingthe terms.

Conceptualization. Conceptualization is considered to beone of the most critical activities in ontology design. Infact, the quality of a codified ontology depends on the qual-ity of its underlying conceptual model (Guizzardi, Halpin,Guizzardi & Halpin, 2008). From a pragmatist view ofsemantics, conceptualization reflects human interests andpurposes. In other words, a given domain of interest canbe described and classified differently according to differentpurposes (Hjorland, 2007). Tasks influence what knowledgeis represented and how it is represented. As with any othermodel, ontologies represent specific aspects of a domainof interest and the ontology designer selects those aspectsrelevant to the tasks to be performed (Devedžic, 2002).The approach to domain conceptualization adopted in thisstudy takes into account end-user factors and was constructedas a system of meaning that includes how this domain contentis interpreted and used by teachers and what types of tasksthe ontology tool needs to support.

The subject domain of cultural heritage in general hasscarcely been explored in ontology research. Modeling adomain of history, and most disciplines in the humanities,poses several challenges. First, history does not rely on a tra-dition of structured terminology and rigorous classification,as do most scientific disciplines. Fields such as medicine,biology, and engineering, but also business and e-commerce,can rely on rather accurate and stable vocabularies. Someof these domains, notably biology, have been systematicallyclassified and their semantics are now specified in a numberof well-known ontologies (e.g., Galen for medicine,6 GO7 forbioinformatics, and EngMath8 for engineering).Also, if com-pared with hard science domains, history presents a highlyinterpretative nature, populated with concepts that are oftencomplex, abstract, and open to a range of different and evenconflicting definitions.

As is often stressed in the literature, identifying the kindsof objects and relations that can exist in the domain requirescareful analysis (Chandrasekaran, Josephson, & Benjamins,1998). Concepts and semantic relationships are not given apriori and they are not neutral, but instead they are the resultof complex interactions within a discourse community. In thecontext of the TBS ontology development, understandingthe community of users the ontology is going to serve and thefunctionality that is going to be supported influenced the mod-eling choices made that capture common and shared meaningof the domain of interest. The outcome of the conceptualiza-tion activity was a prototype model of the TBS ontology.The model has been visualized through concept maps thatprovide a clear and intuitive technique for displaying knowl-edge models. The concept models were developed usingCmap Tools software.9 The list of terms collected duringthe knowledge organization phase provided the basis for theconceptualization. These terms were analyzed, converted intoconcepts, and clustered in four upper-level categories: time,

domain concepts, people, and space. These four categorieshad emerged from the interviews with teachers and servedas a framework to aggregate the terms and organize the con-cepts to develop the conceptual structure of the ontology. Asit turned out, these upper-level categories tied in nicely withsome of the thematic strands from the National Council forthe Social Studies (NCSS) Curriculum standards for socialstudies (1994) and have been associated with the appropriateclusters of concepts on the concept maps (Figures 1–4):

• Time relates to NCSS II. Time, continuity, and change andNCSS VI. Power, authority, and governance.

• Domain concepts relate to NCSS VII. Production, distribu-tion, and consumption and NCSS VIII. Science, technology,and consumption.

• People relates to NCSS IV. Individual, groups, and institu-tions.

• Space relates to NCSS III. People, place, and environments.

Time. Temporal dimensions are notoriously challenging tomodel, and temporal reasoning is still largely an open issue.Considerable research in artificial intelligence and compu-tational linguistics has been focused on the use of linguisticmeans to represent and situate events in time. A stream ofresearch has also taken place in the semantic Web communitythat focuses on various projects, including the OWL-Time10

ontology, intended to express the temporal content of Websites and the temporal properties of Web services. At thistime, more research is needed to be able to express tempo-ral constructs in a Web-based working program that allowsadvanced temporal reasoning. Although promising for thedevelopment of temporal analysis and temporal annotation,these ideas are far beyond the requirements of the ontologyas they emerged in this study.

It is almost axiomatic to recognize that temporal enti-ties are foundational in history: “The study of history placeshuman beings and their activities in time” (N.C. Social StudyStandard Course of Study, 200611). Temporal and spatialannotations enable contextualization of content, which is oneof the tasks that teachers indicated as important for theirsearching and teaching. The relevance of context in relationto historical documents, and visual resources in particular,is highlighted by Lanzi and Besser (1998): “Deprivation ofmeaning that images ‘undergo’ when they lack contextualinformation [is one of the most] compelling and complexaspects of cultural heritage” and can make the images become‘silent’” (p. 4). In the case of the TBS collection, whereimages are a considerable part of the content, temporalannotation would provide an essential means for enhancedretrieval. Expression of temporal aspects of content is opento a wide range of modeling choices (e.g., measures of dura-tion, calendar dates, frequencies, concatenation of temporalintervals, etc.) and the tasks the ontology needs to supporthave guided modeling decisions when this was appropriate.The analysis of the domain knowledge covered by the TBScollection and the understanding of the instructional needsand tasks revealed by the teachers suggested that the tem-poral dimension can be effectively represented through core


FIG. 1. Concept map representing the category of “time” (NCSS stands for National Council for the Social Studies).

temporal entities that include “historical period” and “date.”“Historical periods,” such as Reconstruction, represents theknowledge unit for modeling the periodization of U.S. his-tory as prescribed by the curriculum and universally adoptedin history instruction.

Another core entity included in the TBS ontology is“event,”12 which pertains to both the time and the space con-ceptual dimension. Each event can express a has_sub-eventrelationship with any number of other events. As discussedearlier, teachers would benefit from the possibility of find-ing resources contextualized in time and related to historicalperiods appropriate for instruction. In addition, finding mate-rials that enable teachers to compare and contrast primarysources through time and space was seen as highly desir-able for classroom use. To this end, a set of relations hasbeen identified that may facilitate these tasks. These relations

express initiation and termination (begin/end) to specifytime intervals, concurrence (during), simultaneity or punc-tual coincidence (when), and sequence (before/after). Theconcept model of this category is illustrated in Figure 1.

Domain Concepts. Domain-specific concepts were derivedprimarily from the subject matter of the TBS learning objects.The collection illustrates various aspects of living condi-tions of home-based workers in the late 1930s in rural areasof North Carolina and Virginia and provides a remarkablesource for understanding the economics of these areas. Fromthe content analysis of the objects, families of conceptswere identified, including domestic life, labor status, fam-ily budget, home conditions, etc. These aspects were alignedto the extent possible with themes that addressed similarperspectives (e.g., farm life, home life, medical care, etc.)


from the National Council for the Social Studies (1994).They were displayed on concept maps as a visual deviceto help the teachers identify familiar instructional themes.The North Carolina Standard Course of Study also con-tributed discipline knowledge, from which additional con-cepts were derived to consolidate and broaden, when needed,the semantic coverage of the domain. In addition, conceptsand additional themes were drawn from teachers’ interviewsthat further consolidated the knowledge collected and pro-vided useful suggestions in the development of the ontology(i.e., Education: “How long students would go to school backthen?”).

As is often the case, domain concepts were sometimesambiguous and open to a spectrum of modeling possibili-ties. For example, poverty was an important concept in thecontext of the domain being modeled and it was also a chal-lenge in terms of definition. In fact, the notion of poverty isprone to changes. For example, in the 1930’s the definitionof needs used to compute poverty thresholds was broadened.Franklin Delano Roosevelt himself defined basic needs as“the opportunity to better one’s life, as well as the moreunusual resources for food, housing and shelter” (Oster, 1976,p. 6). Moreover, the federal government did not develop thedefinitions for the poverty thresholds until 1963–1964. Inthe context of this study, a definition of “poverty level” wascreated based on historical data on income taken from theHistorical Statistics of the United States (Carter, 2006). Aconcept map representing the category of domain-specificconcepts is provided in Figure 2a and 2b.

People. As noted earlier, “people” was identified as a centralcategory for the teaching of history. The relevance of this cat-egory was referenced frequently during the interviews for itspedagogical value in relating students with life histories fromthe past. The importance of personal relations, in particularfamilial and social relationships, to support teaching activitiessuch as comparisons of personal narratives and reconstruc-tion of social identities has been discussed (Pattuelli &Norberg, 2006). The category of “people” is included asone of the 10 thematic strands identified by the NCSS toguide social studies instruction. Specifically, the 4th NCSSThematic Strand, entitled Individual, Groups, and Institu-tion, stresses the importance of human beings “to understandhistorical roots and to locate themselves in time”(NationalCouncil for the Social Studies, 1994). People are a centralcomponent of the TBS collection that includes the descrip-tion and photographs of home-based workers and instancesof this category will populate the ontology as its developmentprogresses.

Space. The use of ontologies for geospatial applicationsis a rather new research topic within the ontology researchcommunity (Kauppinen, Vaatainen, & Hyvonen, 2008). Theconceptualization of spatial entities raises complex formalontological questions about the nature of spatial dimension-ality. The notion of discrete and continuum in relation to phys-ical space and the notion of boundaries and holes are among

the more controversial issues being discussed (Casati &Varzi,1994). In the context of the TBS ontology, task requirementsidentified from the user study provided the main rationalefor how to model spatial data in ways that enable usefulquery capabilities. Teachers were concerned with the lack ofgeographic precision when accessing resources via namedplaces.13 They also highlighted the need to contextualizeand compare/contrast historical information through spacein addition to time (e.g., “North vs. South during industri-alization”). Limitations of spatial indexing and geographicalaccess to library resources have been discussed in the past(Buckland, 2004). For example, Fraser and Gluck (1999)point out that in current descriptive practices, geospatialmetadata lacks “scalability of detail,” which is important forenabling users to “quickly access greater or lesser degrees ofdetail as desired” (p. 28).

In the TBS ontology, geospatial information has beenmodeled primarily, but not exclusively, in the form of admin-istrative entities (e.g., nations, regions, states, etc.) ratherthan physical entities (mountains, oceans, rivers, etc.).14

These types of entities refer to places defined by adminis-trative boundaries and conditions that would be defined asfiat objects in the context of formal ontology. Administrativeentities include political subdivisions (e.g., state, province,county), municipalities (e.g., city, town, village), residencesand street addresses, and other types of entities (e.g., Indianreservation). As suggested by several participants in the userstudy, the specification of areas at both county and statelevels are essential in their teaching practices. The teach-ers described that they typically address topics and events byrelating them by regional areas (e.g., the Great Depression inthe Midwest and in the South). While “county” was identifiedas the lowest granular level of detail useful for their queries,“city” was also added to represent the domain knowledge ofthe collection in a comprehensive way (e.g., resources relatedto Richmond were included in the collection).

One of the challenges in representing spatial concepts isthe use of name descriptors. Geopolitical entities and geo-graphical places change over time and can be known bydifferent names. The TGN15 was the reference tool usedfor this portion of the ontology and served as a placenameauthority file. Ontologies are defined at the knowledge level(Newell, 1982) and do not typically include lexical func-tionality. Yet embedding a lexical tool such as the TGN inthe ontology would provide a useful enhancement to thevocabulary. Currently, ontology development tools, includ-ing Protégé, do not provide terminology support. However,the need for ontology management tools to support term rec-onciliation has been recognized and this functionality maysoon be added to major ontology editors.

These geospatial concepts are organized in a tree structurethat, unlike the taxonomic structure of subsumption, doesnot support strict control of inheritance (Kwasnik, 1999).The relationships between these concepts are instead partitive(e.g., “Piedmont” part_of “North Carolina”). Parthood rela-tions are fundamental structuring primitives (Noy & Hafner,1997), but they are also problematic because of the variety


FIG. 2a. Concept map representing domain concepts.

FIG. 2b. Concept map representing the category of “domain concepts”.


FIG. 3. Concept map representing the category of “people”.

of different types that exist and the relative implications fordependency (Winston, Chaffin, & Herrmann, 1987; Keet,2006). For relating concepts spatially in the TBS ontology,the variants contained_in and located_in have been adoptedto express the meaning of containment and location. Thesepartitive relations are of the type “place-area”;16 they enableaggregation by composition and hold transitivity. As Keetsuggests, they can be subsumed by both part_of and spatialpart_of to ensure consistency. Another topological relation-ship included in the TBS ontology is sequential or adjacency:adjacent_to (e.g., North Carolina adjacent_to Virginia).

Reasoning over parthood is difficult to perform. However,partition can be addressed in terms of transitivity that is heldby the set of spatial relations included in the TBS ontol-ogy. This functionality may help find materials by logicallyexpanding or refining queries without the need to rely on syn-tactic matching of geographic descriptors. Also, leveragingtopological relationships of this type may facilitate tasks ofcontent contextualization and geopolitical comparisons thatrepresent important tasks for teachers to perform. It is alsoworth mentioning the role the ontology may play in seman-tic disambiguation, as in the case of a placename that doesnot have unique identification, for example, Venice in Italyand Venice in Florida, or in the context of the TBS ontology,

Columbus (county in North Carolina) and Columbus (townin Ohio). Geared specifically to the scope of the TBS ontol-ogy, the model can be viewed as a basic knowledge structurethat can be expanded to incorporate additional ontologicalconstructs as the ontology progresses in the future. A con-ceptual model of the spatial knowledge of the TBS ontologyis provided in Figure 4.

Results

The outcome of phase 2 was an ontology model prototypeinformed by the outcomes of the interviews and developedin the form of five concept maps (see Figures 1–4). The sizeof the TBS collection, which comprised approximately 150primary sources, was well-suited to be the main source ofdomain knowledge because the acquisition of knowledgecould be carried out relatively easily. As one of the toolsthat strongly influence history instruction, teaching standardswere incorporated in the ontology design. In particular, theNorth Carolina Standard Course of Study proved to be animportant source of the teachers’search terminology and wastherefore invaluable for developing a domain ontology forspecific grade level history instruction.


FIG. 4. Concept map representing the category of “space”.

Phase 3: Evaluation of the Ontology PrototypeModel

At this stage of development process, the TBS ontol-ogy had reached the form of a conceptual model, largelyun-refined, that would serve as a prototype for assessment.As part of the conceptualization phase, a sub-activity wasadded that consisted of the evaluation of the TBS ontol-ogy model with a representative sample of end users. Therationale for introducing this additional step was to verify

the appropriateness and potential usefulness of the modelin preparation for its consolidation before proceeding withformalization.

Methodology

The evaluation of ontology models is not common in ontol-ogy engineering and the lack of an established methodologyrepresents an open research issue. Ontology evaluation is


typically concerned with the evaluation of the formal qualityof the ontology and aims to measure the technical efficiencyof the ontology and, ultimately, its system implementation.Thus, evaluation is usually performed after the ontology hasbeen formalized. On the other hand, conceptual modelingevaluation has a relevant place in software engineering, wherethe assessment of the quality of conceptual models in the earlystages of software development has long been recognized asessential to determine the general quality of an informationsystem (Lindland, Sindre, & Solvberg, 1994). However, theliterature points out the lack of a clear set of methodologiesto support the practice of evaluating concept models (Moody,2005); Brewster, Alani, Dasmahapatra, & Wilks, 2004). Todate, conceptual modeling is still crafted and assessed in arelatively ad hoc fashion, relying largely on subjective viewsand common sense.

The evaluation performed in this study is more alignedwith the idea of assessing the utility and usability of theontology as a practice for “judging the ontology contentfrom the user’s point of view” as described by Gomez-Perez, Fernandez-Lopez, and Corcho (2004, p. 179). Thisapproach—qualitative in nature and user-centered—aims atassessing the adequacy of the ontology for its intended tasksand how well it represents the domain of interest.

Two major criteria have been identified to evaluate theTBS ontology model: design appropriateness and perceivedusefulness.

Design appropriateness is concerned with the ability of theontology model to capture and represent teachers’ searchingneeds. More specifically, the goal was to find evidence thatindicates whether concepts and relationships that might berelevant to the end users in their seeking process are ade-quately represented in the ontology model. The notion ofappropriateness can be viewed as context-dependent in thatit measures the quality of the ontology model based on actualend-user perspectives. In this respect, design appropriatenessis intended to assess the quality of the ontology model in termsof content, organization, and terminology. Factors that wereconsidered included:

Coverage: Is the ontology within the scope of the domain ofinterest? Are there relevant concepts missed? If not explicitlyasserted, could they be inferred?

Structure: Is the level of granularity adequate? Are theremissed relationships?

Language: Is the terminology appropriate? Do terms reflectthe users’ language?

Perceived usefulness is concerned with the capability ofthe ontology to facilitate discovery of, and access to, thedigital resources as perceived by the study participants. Oneway the ontology may prove useful is by helping the teachersformulate their information needs during the search process.This could occur when the ontology suggests query terms,connections between concepts/terms, or potential browsingcategories to support the seeking process. Other instances of

usefulness may be identified when the ontology promptsunanticipated search possibilities that would help teachersrefine or expand their information needs. Perceived useful-ness could also provide an indication of the attitude of theparticipants toward the tool and their intention to use it. Per-ceived usefulness is one of the two measures (the other onebeing ease of use) recognized by Davis (1989) as essential forassessing user acceptance of computers. Factors that measureperceived usefulness as identified in Davis’ study were takeninto consideration in the development of the study tasks. Inparticular, they inspired the last three questions of the Task 3debriefing questionnaire. The TBS ontology model was eval-uated by conducting a task-based user study. The study tookplace in the participants’ workplaces and in the usability labof UNC’s Davis Library. For each session the participant wasasked to complete a brief demographic questionnaire and tocomplete three tasks. The first two tasks required the par-ticipants to search for digital primary sources using Googleand the TBS collection. The searches were screen recorded.The third task required the examination and annotation of aseries of five paper-based concept maps during which par-ticipants were encouraged to share their thoughts out loud.Each task concluded with a debriefing interview during whichparticipants were asked to comment on their experience. Pro-tocol and session debriefings were audio-taped. The test waspiloted with a local social studies teacher to verify the timerequired to complete the test and that the tasks were well-structured, the questions were clear, and the concept mapsmade sense. Minor changes were made after the pilot test inorder to maximize the utility of the study and facilitate theflow from task to task.

Participants

A pool of 14 social studies teachers from middle schoolsand high schools in Chapel Hill, North Carolina and thesurrounding area participated in the study.As with most qual-itative studies, sampling was purposive (Hamilton & Bowers,2006). The sample was chosen based on appropriateness tothe research purpose (6th-12th-grade social studies teachers,who represent a primary community of users of the TBSontology), adequacy (the size of the sample was likely toprovide data sufficiently rich and detailed for the goal of thestudy), and available resources. To be eligible to participatein the study, teachers needed to have experience with teach-ing social studies and to have used digital primary sourcematerials for classroom instruction.Teacher participants wererecruited in several ways, including e-mails and direct solici-tation.A snowball effect was pivotal in increasing the numberof participants.

Interviews and tests took place during a 3-week periodwith 14 individual sessions that lasted 60–90 minutes each.The study participants were aged 25 to 50+ years. The major-ity of participants were female. Sixty-four percent of theparticipants held a master’s degree in education. Their teach-ing experience spanned from 1 to 30 years with a mean of6 years.


In addition to demographic information, the study par-ticipants answered several questions regarding their expe-rience with computers, the Internet, and the characteristicsof their schools. Their self-rated computer/Internet experi-ence ranged from beginner (7%) to intermediate (57%) andadvanced (36%). They all used primary source materials forinstruction and identified the Web as the main source forsearching primary sources (textbooks and libraries came sec-ond before personal collections). More than half had usedNorth Carolina digital collections and DocSouth in particular.

Suburban and rural schools were equally represented (43%each), while only 14% worked in urban schools. The studentbodies were predominantly White (57%). About half of thestudent population had a middle socioeconomic status. Allhad access to Internet in their classrooms with at least onecomputer available in the classroom. According to partic-ipants, about 75% of their students had Internet access athome.

Procedures

Task 1. Task 1 comprised searching digital primary sourcematerials with the goal of simulating a lesson preparation.Thepurpose of the task was to gather user queries that could beused to assess the appropriateness of the ontology model andobserve teachers’ normal approach to searching for primarysource materials. The searches were performed on Google,the most common search service used by the teachers asdiscovered in phase 1 interviews and in the literature. Eachparticipant was given two questions on paper that representedrealistic scenarios on the subject of the Great Depression.The task questions were inspired by the teaching activitiesdescribed in the interviews and adapted from the North Car-olina Standard Course of Study—Social Studies—UnitedStates History, adopted in North Carolina schools:17

Q1. Assess the impact of the Great Depression on the day-to-day life of a low-income family in Virginia in the 1930s.

Q2. Analyze the effects of the New Deal policies on theDepression Era life of women in North Carolina.

Participants continued to perform their searches until theyconsidered their findings satisfactory for the purpose of thetask. The searches were captured on screen. Task 1 endedwith a debriefing, during which the participants were encour-aged to share their thoughts about the searching experienceto determine whether they (a) found the search easy, (b) hadany problems thinking of suitable query terms, (c) felt theyhad all the information needed to perform the search, and(d) took a particular approach to the search and, if so, why.

Task 2. For this task, the TBS collection was introducedand the participants were asked to search the collection withthe goal of finding learning materials useful for instruction.First, the researcher gave participants a brief overview of theTBS collection. Participants were then given a few minutesto explore the Web site and to ask any questions they had.

Then, the following two questions were presented on paper.The questions reflect the scenarios derived from the phase 1interviews and based on the North Carolina Standard Courseof Study—Social Studies—United States History, adopted inNorth Carolina schools:18

Q3. Identify the role of tobacco in the economic developmentof North Carolina in the 1930s.

Q4. Describe the differing impact of the Depression onvarious minority groups.

Participants were asked to perform searches with the goalof finding learning materials useful for instruction based onthe given scenarios. Half of the participants saw question 3first; the other half saw question 4 first to reduce poten-tial learning effects. Participants performed the search twicefor each task question. The first time the search was per-formed using the search interface of the TBS collection.19

Participants were asked to continue searching until they weresatisfied with the results but without exceeding a 10-minutetime limit. The second time a faceted interface on paperwas presented to the participants (see the Appendix). Thispaper-based search interface displayed the main categoriesfrom the TBS ontology in the form of expanded facets.The facets included almost all the concepts from the ontol-ogy model developed in the study. Due to space limitations,the category “education” was excluded because it seemedunlikely to be useful in the context of the task questions. Abrief explanation informed participants that the paper-basedinterface intended to simulate a faceted search or drop-downmenus that they might have encountered and used before.Participants were asked to consult the interface on paperand then reformulate the queries they had performed ear-lier. Search terms and phrases from the entire session wererecorded through screen recording software.

Upon completion of the task, a debriefing interview wasconducted to understand whether and how the availabilityof ontology categories and concepts through the paper-basedfacets had influenced the search process. The interviews wereaudio taped.

Task 3. For Task 3, the participants were asked to manuallynavigate and annotate paper-based concept maps that repre-sented the ontology model on the basis of two scenarios. Thepurpose of the task was to (a) obtain data to assess the qualityof the ontology model through the protocol analysis and con-cept maps’annotation and (b) gain feedback on the perceivedusefulness of the ontology through the session debriefing.First, the participants were introduced to the ontology modelin the format of five concept maps corresponding to the upper-level categories of time, domain concepts, people, and space(see Figures 1–4). The maps were numbered and displayedsimultaneously and in the same order for each participant.The researcher explained the task with the following sce-nario: “Imagine that you need to prepare a class on the GreatDepression and you would like to find some primary sourcesthat illustrate the following aspect to your students: ‘What


TABLE 1. Categories of matching.

ExampleMatch category Definition user term/ontology/question

Same as ontology only Exact match between a user’s search term and an ontology term Example: Wage/wage

Same as question only Exact match between a user’s search term and a question term Example: Impact/impact

Same as both ontology and question Exact match between a user’s search term and both an Example: Tobacco/tobacco/tobaccoontology term and a question term

Same as ontology Exact match between a user’s search term and an Example: Income/income/low-incomeVariant from question ontology term, but a variant from a question term

Same as question Exact match between a user’s search term and a question term, Example: Women/women/womanVariant from ontology but a variant from an ontology term

Variant from ontology only User’s search term is a variant from an ontology term Example: African Americans/African Americans

Variant from question only User’s search term is a variant from an ontology term Example: 1930/1930s

Variant from both ontology and question User’s search term is a variant from both an ontology Example: Families/Family/familyterm and a question term

Candidate for ontology inclusion User’s search term that express a concept not present in the Example: Ruralontology but worth consideration

Out of scope User’s search term not pertaining to the subject matter, Example: Diaryfor example specifying genre.

was life like for children of your age in different areas ofNorth Carolina during the Great Depression?’” This questionworked as a model for each participant and the researcherdescribed how a search could be guided or constructed usingthe concept maps by drawing paths and circling concepts.Next, the participants were given a clean set of diagramsand were presented with questions 1 and 4 from the pre-vious two tasks, one at a time. These questions were chosenbecause they offered a range of navigation options that wouldencourage the participants to examine various sections of theontology. To complete the task, participants were requiredto perform self-directed walkthroughs of the concept maps.Specifically, participants were asked to manually navigate theconcept maps and show their exploration and seeking processby drawing their search paths with colored pencils and cir-cling target concepts they would select for their queries. Theywere also invited to annotate the diagrams and write downany questions, concerns, or suggestions they might have.

During the task, participants were encouraged to sharetheir thoughts as they navigated the maps, following thethink-aloud protocol. This method required participants toexpress their thoughts as they performed the task (Ericsson &Simon, 1993). The researcher gave only general instructions(e.g., “Try to think-aloud and verbalize everything that passesthrough your head”) to avoid influencing the participant. Theresearcher interacted occasionally with probes to remind par-ticipants of thinking aloud when they seemed to forget. Uponcompletion of the task, participants were debriefed on theexperience. The entire session was audio taped.

Findings

Task 1 yielded the queries teachers constructed to find pri-mary source materials for the topics specified in questions 1

and 2. Participants were instructed to continue searching untilthey considered their findings satisfactory; each participantspent between 5 and 10 minutes on each question. Searchkeywords and phrases were collected for each of the twotask questions and mapped onto the ontology model. Termswere also compared with the content words in the originalquestions. Each search term was categorized by the degreeof match with terms in the ontology model and in the originalquestion using the categories defined in Table 1. In this con-text, the notion of variant was interpreted in a broad sense,both as lexical (singular/plural – minority/minorities) andconceptual (synonym – impact/effect).

The mapping provides clues as to whether the users’queryterms were formulated independently or derived from thetext of the questions and whether the ontology included theusers’query terms. The data analysis was intended to identifypotential concepts/terms to be considered in the revision ofthe ontology. Table 2 presents a complete list of search termsand phrases used for both questions, along with the numberof times they were used.

Contrary to what was expected, the analysis of the searchterms did not provide much useful data to assess the ontologymodel itself. For example, only one concept was identified asa possible candidate for inclusion in the ontology. However,the results offered an insight to participants’ search strate-gies, revealing that the search terms were heavily dependenton the words and phrases in the question. The first questioncontained seven content words or phrases. Of the 13 userterms that were considered within the scope of the topic, 12were from the question as either an exact match or a vari-ant. The second question contained seven content words orphrases. Of the 12 user terms that were considered within thescope of the topic, 11 user terms were derived from the ques-tion either as an exact match or as a variant. In other words,


TABLE 2. Search terms for question 1 and question 2.

Question 1 Question 2

Categories of matching N Terms/phrases (number of occurrences) N Terms/phrases (number of occurrences)

Same as ontology only 0 1 Legislation (1)

Same as question only 3 1930s (13) 3 Effects (2)Day-to-day life (1) Life (1)Impact (1) Policies (5)

Same as both ontology 3 Great Depression (36) 2 New Deal (2)and question Virginia (41) North Carolina (27)

Family (8)

Same as ontology 1 Income (5) 1 Great Depression (2)Variant from question

Same as question 1 Low-income (6) 2 Depression Era (5)Variant from ontology Women (20)

Variant from ontology only 0 0

Variant from question only 2 Life (2) 2 Effect (1)Family life (5) Impact (2)

Variant from ontology 2 Depression (1) 1 Females (1)Variant from question Families (7)

Candidate for ontology inclusion 1 Rural (1) 0

Out of scope 9 Diary (1) 2 Personal setters (1)Firsthand account (1) Primary sources (2)Photography (1)Photograph (2)Primary resources (2)Primary sources (6)Richmond Newspaper (1)Sources (1)Waltons (1)

Total 22 14

only one user term for each question was not from the orig-inal text (rural and legislation). The limited range of searchstrategies revealed by the results of this task contrasts withthe variety of ways teachers had described how they wouldsearch for primary sources during the interviews discussedearlier.

Qualitative data were also collected during the debrief-ing, which followed the completion of each task question,where participants were asked to provide feedback aboutthe searches they had performed. All but three teacher par-ticipants found the searches to be somehow problematic.Searches to both questions did not produce results they con-sidered appropriate to satisfy the task’s scenarios. Severalparticipants found the searches related to the first questiondifficult, while they found it easier to search for the secondquestion. The main source of frustration was the difficulty infinding resources at the right level of specificity. For example,one participant commented: “You find a lot about the GreatDepression, but when you minimize it to a smaller search,I had a hard time finding an example of a Virginia family.”Another participant observed: “I could find stuff on the GreatDepression in Virginia, but not for low-income families. Icouldn’t get it narrow enough.” Other participants found veryspecific sources, but for a state other than the one indicatedin the task question. Three participants found searching for

the second question particularly frustrating when trying tolocate sources related to women. One participant commented:“It wasn’t the easiest. The thing with the second one with thelife of the women, particularly in North Carolina, I had a hardtime finding a source. There is a lot of information out thereon the Great Depression, but when you try to make the searchsmaller …at that point when I couldn’t find the informationI was looking for I would maybe go to the library.”

Task 2 yielded teachers’queries used to find relevant mate-rials in the TBS collection, first by using the existing interfacealone and then consulting the paper-based facets and terms.The objective of Task 2 was to gather evidence to assess theperceived usefulness of the ontology and collect additionaldata to be used to evaluate the design appropriateness of theontology model. The query terms before and after the use ofthe facets from both search sessions were collected and ana-lyzed. As before, users’ search terms were compared with theontology and to the original questions. A summary of resultsfrom each session is shown in Tables 3 and 4.

The researcher looked at changes in the general approachto the searches and, more specifically, at possible changes inthe formulation of the queries and at whether these changesmight have been influenced by the display of the ontologycategories through the paper-based facets (i.e., whether newquery terms were picked from the facets).


TABLE 3. Search terms for question 3.

Question 3

Before facets After facets

Terms/phrases Terms/phrasesCategories of matching N (number of occurrences) N (number of occurrences)

Same as ontology only 2 Wage (1) 14 AFDC (1)Labor (1) Agriculture (5)

Cottage industry (1)Crop (1)Family (1)Great Depression (11)New Deal (5)Piedmont (1)Program (2)Southwest (1)Stringing (1)Tobacco Bag Stringing (4)TVA (1)Wage (1)

Same as question only 3 1930s (10) 3 1930s (9)Economic development (8) role (1)Role (7) Economic development (1)

Same as both ontology 2 North Carolina (9) 2 North Carolina (13)and question Tobacco (9) Tobacco (8)Same as ontology 1 Economy (4) 2 Economy (6)Variant from question Tobacco economy (1)Same as question 0Variant from ontologyVariant from ontology only 3 Jobs (1)

Orange County (1)Welfare (1)

Variant from question only 6 Economic impact (2)Economic issues (1)Economic role (1)Economic tobacco (1)1930 (3)Development (2)

Variant from ontology 1 Tobacco industry (2) 0Variant from questionCandidate for ontology inclusion 5 Eastern North Carolina (1)

Tobacco cultivation (1)Tobacco product (1)Money (1)Poverty (1)

Out of scope 0

Total 20 24

The third question contained five content words or phrases.Users derived 13 search terms from the question betweenexact matches and variants before the use of the facets. Thepostfacet search revealed a slight decrease in the use of ques-tion terms (seven in total), but a remarkable use of termsderived from the facets (18 exact matches and 3 variants).Thistrend was confirmed by the results from the fourth question.The fourth question contained four content words or phrases.Although eight user terms were derived from the question text(three exact matches and five variants), in the postfacet searchonly one exact match and three variant terms were derivedfrom the question, but 16 exact matches and five variants werederived from the facets.

There was some learning effect, in that teachers seemed tocarry over what they had seen in the facets for their first ques-tion to their second question, regardless of the order in whichthey saw the questions. For example, for teachers who sawquestion 4 first, two exact matches from the facets were usedin the prefacet search for their second question, question 3.Similarly, teachers who saw question 3 first used one exactmatch from the facets and eight variants from the facets inthe prefacet search for question 4.

Data from the session’s debriefing were analyzed to under-stand whether the availability of ontology categories throughthe paper-based interface had an influence on participant’sapproach to searching and, if so, what kind of influence it had.


TABLE 4. Search terms for question 4.

Question 4

Before facets After facets

Terms/phrases Terms/phrasesCategories of matching N (number of occurrences) N (number of occurrences)

Same as ontology only 1 African American (3) 15 African American (1)American Indian (1)Asian (5)Farmer (1)Immigrant (1)Labor (2)Labor movement (1)Lower class (2)Middle class (1)North Carolina (2)Sharecropper (2)Social status (1)United States (1)Upper class (1)Tobacco Bag Stringing (1)

Same as question only 2 Impact (2) 1 Impact (2)Minority groups (10)

Same as both 0 0ontology and questionSame as ontology 0 1 Great Depression (9)Variant from questionSame as question 1 Depression (7) 0Variant from ontologyVariant from ontology only 8 African Americans (6) 5 African Americans (1)

Blacks (1) American Indians (2)Immigrants (1) Asians (1)Migrants (1) Wall Street (1)Migrant (1)Native Americans (1)Tobacco Bag Stringing (2)Women (4) Women (1)

Variant from question only 3 Effect (2) 2 Effects (1)Minority (6)Minorities (7) Minorities (3)

Variant from ontology 2 Depression era (2)Variant from question Great depression (12) 0Candidate for ontology inclusion 0 0Out of scope 0 0

Total 17 24

All participants found the facets useful. Several partic-ipants pointed out that the facets helped them with queryformulation. Participants observed that the facets suggestedterms they would not have thought about on their own, andseveral of them valued the fact that the facets offered them anoverview of the collection, which enabled them to make morefocused choices. One participant commented that it helpedhim to make informed decisions, which was useful for hisclass planning. Several participants also highlighted the use-fulness of the facets for narrowing down their search options,thus avoiding the need to browse the entire collection. Oneparticipant pointed out the value of the facets for large collec-tions of digital primary sources. Two other participants notedthat the facets helped them with the vocabulary. For example,one pointed out that she was not thinking of using “African

American” as a search term because she tried to think “ofwhat they would have labeled it back then, so I got no hitsfor like Blacks or Negroes.” Another commented: “I wouldtype in Native Americans and not American Indians, whichmight have affected what I pulled.” He stressed the fact that“it [is] easier to know what the keywords are that are pluggedinto the system than having to know on my own what thekeywords would be.”

When asked if the facets had prompted suggestions forteaching, several participants gave examples of ideas theygot while examining the facets. One stated: “Looking at theterms I have lots of ideas running through my head on lessonsI could relate. When it is this easy to find material, it givesyou more time to be creative, because I don’t have to spendall of my time searching for stuff.” Teachers’ reactions to


FIG. 5. Teachers’ annotated concept map for “day-to-day” for question 1.

being able to use an implemented version of the paper-basedfaceted interface were, in general, very positive. When askedto comment on the appropriateness of the terminology, thevast majority found the use of the words similar with whatthey would use: “These terms are excellent; these are theperfect terms and if you can’t find something using these,you’re in trouble.”

Task 3 data were used to (a) to assess the quality of theontology model through the protocol analysis and conceptmaps’ annotation and (b) gain feedback on the perceivedusefulness of the ontology through the session debriefing.With different levels of detail and sophistication, participantsannotated the concept maps by circling the target conceptsand drawing connections between these concepts, accordingto their flow of thinking in relation to the scenarios pro-posed by the two questions. The analysis of the drawingsand annotations revealed that participants’ pathways variedas to a starting point and direction, but they focused onsimilar clusters of concepts. For both questions, the majorityof participants started by targeting the time period and thenproceeded towards the geospatial concepts reaching their pre-ferred level of specificity. Four participants adopted a reversestrategy and started from the geospatial area. Three partici-pants chose the category of person as a starting point, but thenperformed the actual drawing by starting from either the timeor the geospatial concepts. The category of person was either

the second or the third step for the large majority of partici-pants. Within this category, the most targeted concepts werechild and adolescent, family and its various members, andracial groups, depending on the task questions. The categoriesof economy and labor were the ones more heavily marked,especially concepts and connections that would serve to rep-resent the notion of “low-income” as addressed by question 1and “differing impact of the Depression” as in question 4.A wide range of concepts for expressing the notion of“day-to-day life” from question 1 was chosen (see Figure 5).

A comparison of all the maps utilized in the study revealedthat every single concept from the category of house, trans-portation, and education had served as a target concept andwas circled in at least one session. Six participants annotatedthe maps with suggestions for additional concepts.

The protocol analysis that accompanied the navigationand annotation of the concept maps was particularly rich andrevealing. Because of the familiarity that most of the partic-ipants had with the use of concept maps as an educationaltool, the flow of thinking and the concurrent verbalizationwas in general smooth. Participants seemed to be comfort-able with this means of communication and even diverged attimes from the strict protocol to add spontaneous commentsthat were insightful for the researcher. During the sessiondebriefings, the researcher used the guided recall protocol toelicit explanations on specific issues of particular interest for


the study. Additionally, the debriefing interviews includedquestions intended to solicit feedback on the clarity of theconcept maps and their completeness and correctness fromthe perspective of teachers. Participants were asked to providesuggestions for improvements and to comment on whetherthey found the maps useful for their searching tasks.

The results of the analysis of the protocol and debrief-ing transcriptions are combined in the following section. Aseries of key themes were identified that could contribute tothe assessment of the ontology model. These themes includetemporal and geospatial dimension, coverage and modelingissues, clarity, and usefulness.

Temporal and Geospatial Dimension

As mentioned earlier, the category of time was chosen as astarting point by several study participants. The periodizationwas considered appropriate and recognized by participantsas the way they typically identify time periods in their teach-ing and learning activities. For further development of theontology model, participants suggested breaking down theperiods before the Great Depression according to the teachingstandards and incorporating categories of events, includingthe Roaring Twenties, the West, and Imperialism as theyreflect the periodization adopted in current social studiescourses of study. Annotations and comments on the conceptmap that represented the geospatial dimension indicated thatthe level of specificity was overall in line with participants’expectations and expressed needs. Several participants com-mented positively on the presence of “county,” which waschosen as a target concept by eight participants, confirmingwhat had emerged from the phase 1 interviews; “county”was definitely the lowest level of specificity that teacher par-ticipants considered useful in the context of their searches.Nobody targeted the concept of city or any of its instanceson the maps. One participant commented: “I’d never lookat the city, for instance, this area in particular, Chapel Hill,is such a transient place to live that even if I mentioned allthese places, no one in my class would know where theyare, so it is not meaningful for them. I would never look atbreaking it down below counties because they just wouldn’tcare.” In one instance, the concept of city was mentioned,but only in the specific context of a comparison betweencity life and rural life. Regional areas that include the Moun-tains, the Piedmont, and the Coastal Plain, defined in theTBS ontology as subregions of North Carolina, appearedto be useful for addressing comparisons. Also, two partic-ipants were looking for Eastern Carolina: “Everything eastthat I know grows a lot of tobacco”; “Eastern North Carolinabecause it was the poorest.” This geospatial specificationwas not represented in the TBS ontology model and par-ticipant’s comments indicated that it should be considered infuture refinements. I found it interesting that one participantbrought up the notion of neighboring state (“I want to lookat what was back then in our neighboring state to the north”;see Figure 6). Indeed, the concept map would lead to the“neighboring state” through the relation of “adjacency.” The

participant’s comment served to validate the usefulness of thisrelation.

Coverage and Modeling Issues

All participants commented in positive terms on thedomain coverage offered by the concept maps. As for fur-ther expansion of the ontology, one participant suggested toclosely consider the course of study because it is what teach-ers are going to look to. He explained that in the context of theGreat Depression, he would look for things like Social Secu-rity but he would also look for challenges from the SupremeCourt, so adding terminology such as the judicial or “thirdbranch” might be necessary.

From the analysis of maps’annotations, participants addeda few concepts that would expand the domain of the ontol-ogy in two main directions. One area where new conceptswere suggested is the one representing “the day-to-day lifeof a low-income family” from question 1. Three participantsmanually added concepts that they considered useful for thescenario proposed by the task. Most of these concepts sug-gest further specifications of, for example, the category of“education” and they represent semantic gaps to be consid-ered in future revisions of the ontology. In one instance,concepts were added that identified very specifically someof the effects of the Great Depression on people (e.g., lossof income, break up of families, or suicide). These sugges-tions seemed to express the need for a “cause-effect” typeof relationship, which the TBS ontology had not included,and highlighted an interesting modeling issue. Knowledgeof cause and effect provides the basis for historical inter-pretation, and inquiry-based learning is indeed based on theconcept of causation. For this very reason, representing thenotion of causation in a history domain involves the risk ofproducing connections that are simplistic at best, subjectiveor biased at worst. Another interesting modeling issue withboth conceptual and terminological implications is related tothe notion of minority groups addressed in question 4. Inten-tionally, the relation was expressed ambiguously (“person”has race/ethnicity) and only four groups were represented(AfricanAmerican,American Indian,Asian, andWhite). Thiscluster of concepts received a lot of attention and was heavilymarked, mostly because it was the focus of one of the taskquestions and it is a popular teaching topic. Several partici-pants commented on the usefulness of finding the differentgroups laid out. A number of additional concepts were sug-gested and various types of connections were pictured (e.g.,“I would put Latino Americans with the migrant labor move-ments”; “I would want a link to income and how it related tothese minorities”; see Figure 7).

One participant recognized the ambiguity that the concept“minority” may carry when suggesting to add the concept“Hispanic.” He was puzzled if he should consider it a race orethnicity: “I think there are lots of anthropologists that wouldargue there is no race, but ethnicity—I think you need both—leave it like that way.” Another took a different approach andrelated the concept of minority not to racial groups but to


FIG. 6. Teachers’ annotated concept map for comparison between North Carolina and “neighboring state” Virginia for question 1.

women in the workforce. One participant raised an interestingpoint about the implications of using the concept “minority.”She made the point that “the curriculum is structured withseparating groups of people. How did World War II affectAfrican Americans? How did World War II affect women?We don’t really think of it as: ‘How did World War II affectAmerica?’”Although as a teacher she was frustrated with thistype of separation, she recognized that “since that’s the modelwe’re forced to work from as teachers, I think it would beuseful to have a specific connection that pools the experienceof minority groups, because that’s something that teachersare going to be searching for.”

Clarity

The vast majority of the participants found the conceptmaps clear. Because of their familiarity with concept maps, asmentioned earlier, participants were able to gain a sense of thecontent rather quickly. One commented that “it takes a sec-ond, but once I understood what I was looking it was a goodbreakdown.” Another stated that “even if it looks clutteredyou need everything that is included to make the connec-tions.” Only one complained that she “was a little confusedwith the lines. I did not know where to start.” This comment

referred specifically to the issue of transitioning from one mapto another and it seemed to have to do more with the displayof each page on the table than with the content of the maps.In general, participants commented positively on the organi-zation of the maps and, overall, they liked the terminologyused.

Usefulness

The think-aloud protocol that accompanied the navigationand annotation of the maps seemed to suggest a good corre-spondence between the model of the ontology and the waysteachers conceive and use the domain knowledge representedin the model. In many instances, the choice of concepts andthe linking among them was in line with what the design of theontology intended to support. For instance, the ability to tar-get children or adolescents and identify relationships withinthe concept of family was perceived positively as a way “toconnect the topic to students from a personal point of view”or to “stress the deep connections in southern families atthat time.” The capability to support comparisons was high-lighted several times: “You can do so much.You can comparea person with a higher socioeconomic class to a lower. Youcan tweak your lesson to have students understand how they


FIG. 7. Teachers’ annotated concept map for “minority groups” for question 4.

differ.” Several participants recognized and appreciated theopportunities the concept maps provided to interrelate con-cepts in ways that were useful for their teaching: “I would useAfrican Americans, American Indians, and Asians and con-trast how each one was affected. I would break it down intomales and females—whether the Great Depression had moreeffects on the females or males—and then talk about the dif-ferent areas of the United States; you could talk about how theminority groups in the South were affected differently thanthe minority groups in the North.” The potential usefulness ofthe ontology as an educational tool was also noted: “It givesyou a lot of ideas about where you could go. You could cre-ate almost two weeks worth of lessons off these maps.” Alsointeresting was the way the maps were viewed by two partic-ipants in particular. While exploring and marking conceptsand connections, one participant observed that this “wouldhelp to tell a story” and constructed an articulate representa-tion of the day-to-day life of children by drawing pathwaysbetween the ontology concepts. Another participant reachedsimilar conclusions and commented that “it makes it easierto fit them into a facet of the daily life, so, you know, you cankind of tell the story: ‘This is what, you know, Johnny andSusie, a brother and sister, did from the time they got up in the

morning’ from what they saw, to what they did throughouttheir day.”

The role of ontologies for supporting construction of nar-ratives is an aspect that is still little explored in the literature(Mulholland, Zdrahal, & Collins, 2002), but the potentialusefulness of the TBS ontology to construct stories is an inter-esting approach that should be explored in the future. Indeed,history education literature has recognized the creation ofexplanatory narratives as one of the main processes involvedin thinking and learning about history (Wineburg, 1994). Thefindings from this phase 3 evaluation study provided a richset of data to be employed for the evaluation of the appro-priateness and potential usefulness of the TBS ontology. Thedata also revealed aspects of participants’ search strategy thatmay have implications on the design of the ontology. More-over, a set of suggested terms and concepts was collected thatwill be considered for future expansion and refinement of theprototype model.

Discussion of Results

Both research questions—Is the ontology model appro-priate to capture and represent teachers’ searching needs?


Is the ontology perceived to be useful by the teachers intheir seeking process?—were answered positively. Regardingthe first question, the study revealed that the TBS ontol-ogy model captured and represented teachers’ searchingneeds and expectations adequately. Study participants con-sidered it clear and comprehensive and no major flaws weredetected during the evaluation. As for the second question,the usefulness of the ontology was unanimously recognized,almost always with very positive comments from end users.Instances of usefulness were demonstrated in the way theontology helped the seeking process by suggesting searchterms and strategies for query formulation. Additional waysthe ontology seemed useful to participants was in giving thema sense of what was available and searchable by displaying theconcept space of the collection, in prompting ideas on waysprimary sources could be incorporated into lesson plans andinspiring ideas for learning activities. Overall, the study find-ings were encouraging about the potential the TBS ontologyholds to help social studies teachers find primary sources.

The nature of the domain of history, which is largely inter-pretative and less structured and specialized like the medicalor legal domains, posed a major challenge to the developmentof the ontology. The lack of existing ontologies specific to thedomain of the TBS collection to serve sources of reusableknowledge made it necessary to build the TBS ontologyfrom scratch. The development of the TBS ontology followedMethontology as the general methodology framework withthe introduction of two additional activities. The first one isrepresented by the interviews with teachers that helped toguide modeling decisions based on their expressed informa-tion needs and work practices. For example, feedback fromthe interviews encouraged the use of the teaching standardsas an integrative source for knowledge acquisition. Under-standing teachers’ context was key for gaining insight onhow the ontology should be designed to be most useful tothe teachers. For example, teacher participants indicated thatcomparing and contrasting resources and relating historicalcontent to students’ lives were important strategies to teachstudents interpretative skills. In addition, domain knowledgewas elicited that served to indicate types of concepts and theirappropriate level of specificity (e.g., education, school, one-room-school) and sets of relationships (e.g., familial relation-ships) the ontology should incorporate to support such tasks.

The second methodological addition was the evaluation ofthe ontology, introduced at the stage of model prototyping.Although the use of conceptual models to elicit user feed-back and evaluation is a rather common practice in softwaredevelopment (Beyer & Holtzblatt, 1998), it is not part ofthe current ontology engineering practice. The rationale foradding this step to the building process was the assumptionthat a sound ontology model is the basis of an effective ontol-ogy implementation. More important, modeling is a criticaltask in ontology development and it is also the most challeng-ing to perform. Gaining feedback on the validity of the modeland on its potential helpfulness early in the process may resultin an economical choice in that it can help to consolidate themodel and build a better tool.

If the results are not encouraging, then they may suggest amajor reconsideration of the purpose or scope of the ontologyor, more drastically, even an end to the project. Expandingthe methodology procedures created a way of including infor-mation gained early in the process from end users and othersources that departed from the common engineer-centric viewof ontology development where one size fits all. The back-ground knowledge from the interviews suggested modelingdecisions that contributed to make the model more adequateto the needs of its users. The outcome of the evaluation studyprovided a way to strengthen most of the modeling deci-sions and receive suggestions on how to further improve it,as discussed next. Both steps contributed to streamline theprocess in that it is much easier and less expensive to makechanges in the early stages of development than to modify itonce formalized and implemented.As discussed, the involve-ment of end users is rare in ontology development and iscertainly a new practice in the early stage of ontology con-struction. Such an approach has proven to be particularlyuseful in this preliminary phase of the ontology engineeringprocess for the richness and relevance of the backgroundknowledge collected. One reason for the effectiveness of theinterviews’outcome may be because the participants were allmembers of a specific user group that shares common workpractices and information needs, resulting in the similarityof ideas and opinions they expressed. This helped to identifya rather clear set of requirements to be translated into spec-ifications for the ontology. A more diverse set of end userswith widely differing needs might not supply such uniformguidance, but would still be informative.

Evaluation of ontology models is not a common practicein knowledge engineering, and it is even more unusual whenconducted from a user perspective. The main challenges ofthis phase of the study were related to study design issues suchas evaluation criteria. Established sets of measures for ontol-ogy evaluation already exist, but they are typically employedfor testing formal qualities of formalized ontologies (e.g.,OntoClean; Guarino & Welty, 2002) or system performance.The aim of the study was instead to assess the quality of theontology for consolidation before the transition to the formal-ization process. The ultimate goal of the study was to validatethe ontology model and to gather input on its potential useful-ness. The study was qualitative and task-based.All tasks wereframed in scenarios that were intended to represent real-worldinformation-seeking problems (Carroll, 2002) in the form of asimulated work task situation (Borlund, 2003; e.g., “Imaginethat you need to prepare a class on the Great Depression andyou would like to find some primary sources that illustratevarious aspects of the historical period”). To make the scenar-ios realistic, they were based on the feedback gathered fromthe phase 1 interviews and derived from the North CarolinaCourse of Study. The data were collected using multiple tech-niques, including demographic questionnaires, search andannotation tasks, post-task interviews, think-aloud protocol,and the researcher’s observations and field notes.

The first task, which asked participants to perform Googlesearches, did not provide particularly useful data for assessing


the ontology model as was originally expected. However,it revealed interesting aspects of participants’ search strate-gies that may have implications for the future of domainontologies, in general, and the TBS ontology, in particular.Although the large majority considered themselves com-puter and Internet proficient, with only one self-declaredbeginner, all participants showed limited capabilities to for-mulate or refine their queries independently. They all reliedprimarily on terms and phrases directly extracted from thetask questions. Even when frustrated by the search results,they did not try to modify their queries by introducing, forinstance, a new keyword. Instead, they mostly reformulatedthe queries by retyping the same words in a different orderor by making slight lexical modifications (e.g., from singularto plural). A few participants even entered the task questionsverbatim.

The difficulty teachers often have in constructing searchqueries within digital libraries has been highlighted (Recker,Dorward, & Nelson, 2004). The limited skills for formulat-ing queries are a clear indicator of the need teachers have fortools that support and facilitate their searching. The resultsmay also suggest that the teaching standards could prove tobe a major provider of search terms. The participants in vari-ous instances of both this study and the literature highlightedthe roles that standards play in lesson planning. It is reason-able to think that what the standards prescribe is also what isgoing to be searched, as one participant suggested during thesession debriefing, “because whatever terms appear there [inthe standards], they are most likely going to be your searchterms.” These considerations further strengthen the idea thatit is important to take teaching standards into considerationas a source of knowledge for the ontology and, even moregenerally, for indexing systems tailored to history teachingand learning at the grade level.

The second task asked participants to search the TBS col-lection with and without the use of paper-based facets. Theanalysis revealed a frequent use of terms from the facetsto formulate or modify their queries and to stimulate abroader range of search strategies. Participants’ feedbackfrom post-task interviews revealed the facets were unan-imously perceived useful during the search process. Theusefulness of facets and directories to support Web search-ing has been described in the literature. Benefits includeproviding context that shows relationships between topics,suggesting alternative query entries, and presenting new ideasand directions (Bates, 1989; Ellis & Vasconcelos, 1999;Lee & Olson, 2005). This type of support may, for instance,reduce the cognitive effort and the time required for inde-pendently generating search. Also, by revealing the topicalcoverage of the collection, facets help to learn about the col-lection and “form a mental map of the subject areas that arecovered and those that are excluded” that, as Shreeves andKirkham (2004) suggest, would facilitate the seeking process.

Participants’ feedback on the usefulness of the facets wasindeed aligned with the literature. The facets were consideredhelpful to: (a) formulate and/or refine queries by suggestingtopics and terms that would not have occurred to them without

the menus; (b) focus their searches at the right level of speci-ficity; (c) provide an overview of the collection that madethem aware of what was there and what was thus searchable;(d) give ideas for teaching activities; and (e) save time whenplanning a class. The third task asked participants to navigateand annotate paper-based concept maps to simulate searchesaccording to two scenarios. The design of this task was chal-lenging due to the novelty of the approach taken. Althoughgraph views of knowledge models are commonly used fortask analysis, for example, for knowledge elicitation in sys-tem development (Gordon, Schmierer, & Gill, 1993), in thecontext of this study they were employed for a different andmore elusive purpose—to gather evidence of the appropriate-ness and usefulness of the ontology model. Indeed, the use ofthe maps revealed itself to be an effective means for the goalof the task. All participants were familiar with the notion ofconcept maps because they had used them in their classroomor in their preservice training. Contrary to my expectations,minimal explanation of the task was needed and all partici-pants seemed at ease with the requirements of the task. Theuse of the think-aloud protocol to accompany the navigationof the maps was key to generate accounts of participants’actions that were particularly informative and rich withdetail.

The four upper categories of the ontology (people, time,space, and domain-specific concepts) were targeted by allthe participants in similar sequences and they appeared to beessential foci for the questions. Concepts and terms appearedto be those necessary to represent the domain and answerthe task questions. No critical semantic gaps or structuralincongruence were found. Some of the concepts/terms sug-gested by participants through annotations or verbal feedbackfilled holes in the clusters of concepts I had intentionallyleft incomplete (e.g., additional ethnic groups and the judi-cial branch) to see if teachers would contribute suggestedconcepts/terms. It was also interesting to see that the par-ticipants targeted concepts, such as “one-room school,” thatwere derived from teaching activities described in the inter-views and initially added almost reluctantly to the ontology,thus revealing that such a level of specificity in modelingthe category of education can be useful and appropriate. Bythinking of the task questions in terms of competency ques-tions, as discussed earlier, the ontology model seemed to beable to answer them in a satisfactory way. Annotations andprotocols indicated that the level of specificity was alignedoverall to the search needs of the participants. This was partic-ularly clear for the geospatial and temporal dimensions. Therole teaching standards played in teachers’ seeking processwas highlighted again in this task. It was interesting to notethat, even when participants were critical of the standards,they all agreed on the need to follow the standards’ goals andobjectives closely. An example was provided by the notionof minority. Even the participant who expressed open dis-approval about the way the notion of minority is employedin U.S. History teaching recognized the need to include it inthe concept map because “that’s something that teachers aregoing to be searching for.”


Conclusion

This study examined the potential of a domain ontology tofacilitate the search process and ultimately the use of a col-lection of primary source materials for classroom instruction.Under the assumption that a domain ontology that representscultural heritage materials is more effective when it integratesthe information requirements of end users into its design,the study tested a user-centered approach to the constructionof an ontology for the TBS collection, a digital collection ofprimary sources part from the UNC-CH University Library.

Ontologies, like any model, represent specific aspects orviews of a domain of interest. Tasks influence what and howknowledge is represented. The rationale behind having endusers participate in the early stages of the development pro-cess ensures that the system is useful and meets the needs andexpectations of the intended audience. Although this tenetmay sound axiomatic, a user-centered approach is still rarewithin the context of digital library development and is evenmore rare in the field of ontology engineering. Typically,knowledge engineers in collaboration with domain expertsconstruct ontologies. Involving representatives from the usercommunity in the design cycle of the ontology remains anunconventional choice.

The three-phase study provided evidence that the TBSontology was appropriate to support the information needs ofits intended audience and was perceived as a useful tool forhelping teachers in their seeking process. The interviews con-ducted in the initial phase of the study revealed aspects of theteachers’ work context, as well as perspectives on the knowl-edge domain that were later found to be important elementsin the design of the ontology model.

There is no doubt that conducting extensive analysis ofpotential end users would add significant time and effort tothe already demanding process of ontology construction. Inthis instance, the significant and targeted contributions of theusers guided the design of the ontology in an appropriate andfocused direction, ultimately saving time and helping to cre-ate a more useful tool. There was general agreement and ashared acceptance of the worldview offered by the ontologyand the teachers, in large part because of the common teach-ing tasks required and educational standards followed. Thereverse, of course, is also possible. That is, end users andontology developers may not be able to find a shared concep-tual framework. In those cases, the evaluation of the prototypemodel becomes even more crucial to determining whetherit would even be possible to construct a useful ontology. Itwould force the designers to consider how to accommodatethe varying views of the end users in the ontology, somethingthat is easier done at an earlier stage, rather than a later one.Such an approach may not be able to be generalized and mustbe considered on a case-by-case basis in relation to the sizeof the ontology being built, the nature of knowledge domain,and the type of end users targeted. Nevertheless, involv-ing potential end users in a preliminary stage of ontologydevelopment appears to be a worthwhile endeavor for digi-tal cultural heritage collections, including primary sources,

where few domain-specific ontologies are available for reuseand whose subject matter is largely unstructured and highlyinterpretative.

It must be acknowledged that there are limitations to thestudy. The homogeneity of the study participants, all comingfrom the same relatively small geographical area, preventsthe generalization of the results. Social studies teachers fromother parts of the country may have different informationneeds and views of the domain of interest. Moreover, thenumber of study participants was constrained by practicalissues, such as the availability of interested teachers in thevicinity.

More work needs to be conducted on developing the TBSontology for real-world application and further research isneeded to evaluate how effective the TBS ontology is in thecontext of a working application. The results from this studyencourage the continuation of the development process thatcan be done by refining and consolidating the prototype inlight of teachers’ feedback received during the evaluation andproceeding towards the representation of an ontology in aformal representation language. Such an ontology would beone of the few developed in the domain of U.S. History at thistime and could serve as a starting point for further expansions.

As ontology research in the field of cultural heritage pro-gresses, the TBS ontology may link or merge with similardevelopments and map into broader frameworks such asthat provided by CIDOC CRM. “Ontologies are agreements,made in a social context, to accomplish some objectives.It’s important to understand those objectives, and be guidedby them” (Gruber, 2003). Aligned with this tenet, this studyoffers a notion of ontologies as the result of a pragmatic pro-cess in which a user perspective is built into the design cycle.Although the methodology needs to be further tested and for-malized, this study suggests that the involvement and thecontributions of user communities to ontology developmentrepresent important areas of research to enhance communi-cation between developers and users, because “ontologies arewhat they do: artifacts to help people and their programs tocommunicate, coordinate, collaborate” (Gruber, 2003).

Endnotes1http://www.lib.unc.edu/ncc/tbs/report.html2http://www.lib.unc.edu/ncc/tbs/report.html3http://www.getty.edu/research/conducting_research/vocabularies/tgn/4This is an educational guide to the Great Depression sponsored by the

Franklin and Eleanor Roosevelt Institute at http://newdeal.feri.org/5http://www.learnnc.org/6http://www.openclinical.org/prj_galen.html7http://www.geneontology.org/8http://www-ksl.stanford.edu/knowledge-sharing/papers/engmath-tree.

html9Cmap Tools is a software product developed by the Institute for Human

and Machine Cognition (IHMC) and is freely available for educational use(http://cmap.ihmc.us/).

10http://www.w3.org/TR/owl-time/11http://www.ncpublicschools.org/curriculum/socialstudies/scos/2003-

04/007history.


12“Events are types of entities which represent occurrences in space andtime (car accident) historical event (space and time) (Perry, Sheth, &Arpinar,2006, p. 7).

13“Named Places are those entities with static spatial properties and clearspatial extents (e.g., a manufacturing plant, an apartment building, a city,etc.”; Perry et al., 2006, p. 7).

14For example, the concept of “river” was included.15http://www.getty.edu/research/conducting_research/vocabularies/tgn/

index.html16Part-place cannot be separate from the whole-area (Odell, 1998).17The questions for Task 1 were adapted from Competency Goal 6,

Objective 6.01 of the North Carolina Standard Course of Study’s EighthGrade Social Studies Curriculum: “Identify the causes and effects of theGreat Depression and analyze the impact of New Deal policies on DepressionEra life in North Carolina.” (http://www.ncpublicschools. org/curriculum/socialstudies/scos/2003-04/050eighthgrade)

18The questions for Task 2 were adapted from Competency Goal 5,Objective 5.01 of the North Carolina Standard Course of Study’s EighthGrade Social Studies Curriculum: “Identify the role played by the agri-culture, textile, tobacco, and furniture industries in North Carolina andanalyze their importance in the economic development of the state (http://www.ncpublicschools.org/curriculum/socialstudies/scos/2003-04/050eighthgrade); and Competency Goal 9, Objectives 9.04 of the North CarolinaStandard Course of Study’s Eleventh Grade Social Studies Curriculum:Describe challenges to traditional practices in religion, race, and gender”(http://www.ncpublicschools.org/curriculum/socialstudies/scos/2003-04/067eleventhgrade).

19http://dc.lib.unc.edu/cdm4/search.php

References

Baca, M., Harpring, P., Lanzi, E., McRae, L., & Baird Whiteside, A. (2006).Cataloging cultural objects: A guide to describing cultural works and theirimages. Chicago: American Library Association.

Bada, M., Stevens, R., Goble, C., Gil,Y., Ashburner, M., & Blake, J.A., et al.(2004). A short study on the success of the gene ontology. Journal of WebSemantics, 1(2), 235–240.

Bates, M.J. (1989). The design of browsing and berrypicking techniques forthe online search interface. Online Review, 13(5), 407–424.

Beck, H., & Pinto, H.S. (2002). Overview of approach, methodolo-gies, standards, and tools for ontologies. Agricultural Ontology Ser-vice, UN FAO, Rome, Italy. Retrieved September 21, 2009, fromhttp://www.fao.org/agris/aos/Documents/BackgroundAOS.html

Benjamins, V.R., Contreras, J., Blazquez, M., Dodero, J.M., Garcia, A., &Navas, E., et al. (2004). Cultural heritage and the semantic Web. LectureNotes in Computer Science, (3053), 433–444.

Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. San Francisco: Morgan Kaufmann.

Bontas, E.P., & Mochol, M. (2006). Ontology engineering cost estima-tion with ONTOCOM (Report No. TR-B-06-01). Berlin, Germany: FreeUniversity of Berlin. Retrieved September 5, 2009, from ftp://ftp.inf.fu-berlin.de/pub/reports/tr-b-06-01.pdf

Borgman, C.L., Millwood, K.A., Finley, J.R., Champeny, L., Gilliland-Swetland, A.J., & Leazer, G.H. (2005). Comparing faculty informationseeking in teaching and research: Implications for the design of digitallibraries. Journal of the American Society for Information Science andTechnology, 56(6), 636–657.

Borlund, P. (2003). The IIR evaluation model: A framework for evalua-tion of interactive information retrieval systems. Information Research,8(3). Retrieved September 5, 2009, from http://informationr.net/ir/8-3/paper152.html

Bouaud, J., Bachimont, B., Charlet, J., & Zweigenbaum, P. (1995). Method-ological principles for structuring an "ontology.” Proccedings of IJCAI’95workshop on basic ontological issues in knowledge sharing. Montreal,Canada.

Brewster, C., Alani, H., Dasmahapatra, S., & Wilks, Y. (2004). Datadriven ontology evaluation. Proceedings of International Conference onLanguage Resources and Evaluation. Lisbon, Portugal.

Buckland, M. (2004). Going places in the catalog: Enhancing scholarly andeducational resources with geospatial information. WebWise 2004: Shar-ing Digital Resources, Chicago, IL. Retrieved September 5, 2009, fromhttp://ecai.org/imls2002/WebWise.pdf

Calhoun, K. (2006). The changing nature of the catalog and its integrationwith other discovery tools. Washington, DC: Library of Congress.

Carroll, J.M. (2002). Scenarios and design cognition. Proceedings ofthe IEEE Joint International Conference on Requirements Engineering,3–5.

Carter, S.B. (2006). Historical statistics of the United States (Millennial ed.).New York: Cambridge University Press.

Casati, R., &Varzi,A.C. (1994). Holes and other superficialities. Cambridge,MA: MIT Press.

Ceusters, W., Smith, B., & Goldberg, L. (2005). A terminological andontological analysis of the NCI thesaurus. Methods of Information inMedicine, 44(4), 498–507.

Chandrasekaran, B., Josephson, J.R., & Benjamins, R.V. (1998). Ontol-ogy of tasks and methods. Proceedings of KAW’98, 11th Workshop onKnowledge Acquisition, Modeling and Management, Inn, Banff. Alberta,Canada.

Corcho, O., Fernandez-Lopez, M., & Gomez-Perez, A. (2003). Methodolo-gies, tools and languages for building ontologies: Where is their meetingpoint? Data & Knowledge Engineering, 46(1), 41.

Coventry, M., & Bass, R. (2003). Engines of inquiry:Approaches to teaching,learning & technology in American culture studies (2nd ed.). Washington,DC: American Crossroads Project.

Coyle, K., & Hillmann, D. (2007). Resource description and access (RDA).D-Lib Magazine, 13(1/2), 1082–9873.

Crofts, N., Doerr, M., Gill, T., Stead, S., & Stiff, M. (Eds.). (2010).Definition of the CIDOC conceptual reference model (Version 5.0.2).Retrieved March 4, 2010, from http://www.cidoc-crm.org/docs/cidoc_crm_version_5.0.2.pdf. Version 4.2, June.

Davis, F.D. (1989). Perceived usefulness, perceived ease of use, and useracceptance of information technology. MIS Quarterly, 13(3), 319–340.

DeRidder, J.L. (2007). The immediate prospects for the applicationof ontologies in digital libraries. Knowledge Organization: KO,34(4), 227.

Devedžic, V. (2002). Understanding ontological engineering. Communica-tions of the ACM, 45(4), 136–144.

Ding, Y., & Foo, S. (2002). Ontology research and development.Part 1-A review of ontology generation. Journal of Information Science,28(2), 123.

Ellis, D., & Vasconcelos, A. (1999). Ranganathan and the net: Using facetanalysis to search and organise the World Wide Web. Aslib Proceedings,51(1), 3–10.

Ericsson, K.A., & Simon, H.A. (1993). Protocol analysis: Verbal reports asdata (rev. ed.). Cambridge, MA: MIT Press.

Fast, K.V., & Campbell, D.G. (2001). The ontological perspectives of thesemantic Web and the metadata harvesting protocol: Applications ofmetadata for improving Web search. Canadian Journal of Library andInformation Science, 26, 5–20.

Fernandez-Lopez, M. (1999). Overview of methodologies for buildingontologies. Workshop on ontologies and problem-solving methods:Lessons learned and future trends (IJCAI99). Stockholm, Sweden.

Fernandez-Lopez, M., Gomez-Perez, A., & Juristo, N. (1997). METHON-TOLOGY: From ontological art towards ontological engineering. Pro-ceedings ofAAAI97 Spring Symposium Series, Workshop on OntologicalEngineering (pp. 33–40). Menlo Park, CA: AAAI Press.

Fischer, K.S. (2005). Critical Views of LCSH, 1990–2001: The ThirdBibliographic Essay. Cataloging and Classification Quarterly, 41(1),63–110.

Fraser, B., & Gluck, M. (1999). Usability of geospatial metadata or space-time matters. Bulletin of the American Society for Information Scienceand Technology, 25(6), 24–28.

Gill, T. (2004). Building semantic bridges between museums, libraries andarchives: The CIDOC conceptual reference model. First Monday, 9(5).Retrieved September 5, 2009, from http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/1145/1065


Gomez-Perez, A., Fernandez-Lopez, M., & Corcho, O. (2004). Ontologicalengineering: With examples from the areas of knowledge management,e-commerce and semantic Web. London: Springer Verlag.

Gordon, S.E., Schmierer, K.A., & Gill, R.T. (1993). Conceptual graphanalysis: Knowledge acquisition for instructional system design. HumanFactors, 35(3), 459–481.

Gould, J.D., Boies, S.J., & Lewis, C. (1991). Making usable, useful,productivity-enhancing computer applications. Communications of theACM, 34(1), 74–85.

Greenberg, J., & Méndez, E. (2009). Knitting the Semantic Web. London:Routledge.

Gruber, T.R. (1993). Towards principles for the design of ontologies usedfor knowledge sharing. In N. Guarino, & R. Poll (Eds.), Formal ontologyin conceptual analysis and knowledge representation, Padova, Italy.

Gruber, T.R. (2003). It is what it does: The pragmatics of ontology. InvitedPresentation to the Meeting of the CIDOC Conceptual Reference ModelCommittee, Smithsonian Museum, Washington, DC, March 26, 2003,Retrieved September 5, 2009, from http://tomgruber.org/writing/cidoc-ontology-2003.pdf

Gruninger, M., & Fox, M.S. (1994). The role of competency questions inenterprise engineering. Proceedings of the IFIP WG5.7 Workshop onBenchmarking – Theory and Practice, Trondheim, Norway, 1994.

Gruninger, M., & Fox, M.S. (1995). Methodology for the design and eval-uation of ontologies. Proceedings of the Workshop on Basic OntologicalIssues in Knowledge Sharing, IJCAI-95, Montreal.

Guarino, N. (1997). Understanding, building and using ontologies. Interna-tional Journal of Human Computer Studies, 46(2–3), 293–310.

Guarino, N., & Welty, C. (2002). Ontology applications and design: Evaluat-ing ontological decisions with OntoClean. Communications of the ACM,45(2), 61.

Guizzardi, G., Halpin, T., Guizzardi, G., & Halpin, T. (2008). Ontologicalfoundations for conceptual modelling. Applied Ontology, 3(1–2), 1–2.

Hamilton, R.J., & Bowers, B.J. (2006). Internet recruitment and e-mailinterviews in qualitative studies. Qualitative Health Research, 16(6),821–835.

Hepp, M. (2007). Possible ontologies: How reality constrains the devel-opment of relevant ontologies. [Electronic version]. Internet Computing,IEEE, 11(1), 90–96.

Hjorland, B. (2007). Semantics and Knowledge Organization. AnnualReview of Information Science and Technology, 41, 367–406.

Holsapple, C.W., & Joshi, K.D. (2002).A collaborative approach to ontologydesign. Communications of the ACM, 45(2), 42–47.

Hyvönen, E. (2009). Semantic portals for cultural heritage. In S. Staab &R. Studer (eds.), Handbook on ontologies (pp. 757–778). Berlin:Springer.

Kauppinen, T., Vaatainen, J., & Hyvonen, E. (2008). Creating and usinggeospatial ontology time series in a semantic cultural heritage portal.In Proceedings of the Fifth European Semantic Web Conference on theSemantic Web: Research and Applications. Lecture Notes in ComputerScience, 5021, 110–123.

Keet, M. (2006). Introduction to part-whole relations: Mereology, conceptualmodelling and mathematical aspects (KRDB Research Centre TechnicalReport. Bolzano, Italy: Free University Bozen-Bolzano.

Kruk, S.R., & McDaniel, W.D. (2009). Semantic digital libraries. Berlin:Springer.

Kwasnik, B.H. (1999). The role of classification in knowledge representationand discovery. Library Trends, 48(1), 22–47.

Lancaster, F.W., Connell, T.H., Bishop, N., & McCowan, S. (1991). Iden-tifying barriers to effective subject access in library catalogs. LibraryResources and Technical Services, 35(4), 377–391.

Lanzi, E., & Besser, H. (1998). Introduction to vocabularies: Enhanc-ing access to cultural heritage information. Los Angeles, CA: GettyInformation Institute.

Larson, R.R. (1999). The decline of subject searching: Long-term trends andpatterns of index use in an online catalog. Journal of American Society ofInformation Science, 42(3), 197–215.

Lee, H., & Olson, H.A. (2005). Hierarchical navigation: An exploration ofYahoo! directories. Knowledge Organization, 32(1), 10–24.

Library of Congress. (2008). On the record report of the Library ofCongress Working Group on the Future of Bibliographic Control.Washington, DC: Library of Congress. Retrieved December 18, 2009,from http://www.loc.gov/bibliographic-future/news/lcwg-ontherecord-jan08-final.pdf

Lindland, O.I., Sindre, G., & Solvberg, A. (1994). Understanding quality inconceptual modeling. Software, IEEE, 11(2), 42–49.

Lynch, C.A. (2003). Colliding with the real world: Heresies and unexploredquestions about audience, economics, and control of digital libraries. InA.P. Bishop, N.A. Van House & B.P. Buttenfield (Eds.), Digital libraryuse: Social practice in design and evaluation (pp. 191–216). Cambridge,MA: MIT Press.

Mao, J.Y., Vredenburg, K., Smith, P.W., & Carey, T. (2005). The stateof user-centered design practice. Communications of the ACM, 48(3),105–109.

McLean, N., & Lynch, C.A. (2004). Interoperability between library infor-mation services and learning environments – Bridging the gaps IMSGlobal Learning Consortium and the Coalition for Networked Informa-tion. Retrieved August 21, 2009, from www.imsglobal.org/digitalrepositories/CNIandIMS_2004.pdf

Mizoguchi, R. (2003). Tutorial on ontological engineering. New GenerationComputing, 21(4), 363–384.

Mizoguchi, R., Vanwelkenhuysen, J., & Ikeda, M. (1995). Task ontology forreuse of problem solving knowledge. Towards very Large KnowledgeBases: Knowledge Building & Knowledge Sharing (KBKS’95),Enschede, The Netherlands. 46–57.

Moody, D.L. (2005). Theoretical and practical issues in evaluating the qual-ity of conceptual models: Current state and future directions. Data &Knowledge Engineering, 55(3), 243–276.

Mulholland, P., Zdrahal, Z., & Collins, T. (2002). CIPHER: Enabling com-munities of interest to promote heritage of European regions. [Electronicversion]. Cultivate Interactive, (8). Retrieved September 5, 2009, fromhttp://www.cultivate-int.org/issue8/cipher/index.html.

Naeve, A., Lytras, M., Nejdl, W., Balacheff, N., & Harding, J. (2006).Advances of semantic Web for E-learning: Expanding learning frontiers.British Journal of Educational Technology, 37, 321.

Nagypal, G., Deswarte, R., & Oosthoek, J. (2005). Applying the seman-tic Web: The VICODI experience in creating visual contextualization forhistory. Literary and Linguistic Computing, 20(3), 327.

National Council for the Social Studies. (1994). Expectations of excel-lence: Curriculum standards for social studies. Washington, DC:NCSS.

Newell, A. (1982). The knowledge level. Artificial Intelligence, 18(1),87–127.

Noy, N.F., & Hafner, C.D. (1997). The state of the art in ontology design.AI Magazine, 18(3), 53–74.

Noy, N.F., & McGuinness, D.L. (2001). Ontology development 101: Aguide to creating your first ontology (Technical Report No. KSL-01-05).Stanford, CA: Stanford Knowledge Systems Laboratory.

Oster, S. (1976). A review of the definition and measurement of poverty:Vol. I, summary review paper;Vol. II, annotated bibliography.The measureof poverty, technical paper III. Washington, DC: Department of Health,Education and Welfare.

Simperl, E.P.B., & Tempich, C. (2006). Ontology engineering: A realitycheck. In R. Meersman et al. (Eds), On the move to meaningful Inter-net systems 2006: CooPIS, DOA and ODBASE, LNCS (Vol. 4275, pp.836–854). London: Springer-Verlag.

Pattuelli, M.C., & Norberg, L.R. (2006). Identity matters: Constructingsocial identities through ontology-based metadata. International Confer-ence on Dublin Core and Metadata Applications, Manzanillo, Mexico.215–219.

Pinto, H.S., & Martins, J.P. (2004). Ontologies: How can they be built?Knowledge and Information Systems, 6(4), 441.

Recker, M.M., Dorward, J., & Nelson, L.M. (2004). Discovery and use ofonline learning resources: Case study findings. Educational Technology& Society, 7(2), 93–104.

Ritchie, D.A. (2001). American history: The modern era since 1865.New York: Glencoe/McGraw-Hill.


Shreeves, S.L., & Kirkham, C.M. (2004). Experiences of educators usinga portal of aggregated metadata. Journal of Digital Information, 5(3),2004–2009.

Sridhar, M.S. (2004). Subject searching in the OPAC of a special library:Problems and issues. OCLC Systems & Services: International DigitalLibrary Perspectives, 20(4), 183–191.

Stutz, C. (1939). Tobacco bag stringing operations in North Carolina andVirginia. Richmond, VA: Virginia-Carolina Service Corporation.

Sumner, T., Khoo, M., Recker, M.M., & Marlino, M. (2003). Understand-ing educator perceptions of “quality” in digital libraries. Proceedings ofthe Third ACM/IEEE-CS Joint Conference on Digital Libraries, Houston,Texas. 269–279.

Sure,Y., & Studer, R. (2005). SemanticWeb technologies for digital libraries.Library Management, 26(4), 190–195.

Uschold, M. (1996). Building ontologies: Towards a unified methodol-ogy. Proceedings of Experts Systems 96, the 16th Annual Conference.Cambridge, UK.

Uschold, M., & Gruninger, M. (1996). Ontologies: Principles, methods andapplications. Knowledge Engineering Review, 11(2), 93–155.

Wilson, C.R., & Ferris, W.R. (1989). Encyclopedia of southern culture.Chapel Hill: University of North Carolina Press.

Wineburg, S. (1994). The cognitive representation of historical texts.In G. Leinhardt, I.L. Beck & C. Stainton (Eds.), Teaching and learning inhistory (pp. 83–135). Hillsdale, N.J.: Lawrence Erlbaum Associates.

Winston, M.E., Chaffin, R., & Herrmann, D. (1987). A taxonomy ofpart-whole relations. Cognitive Science, 11, 417–444.


Appendix

Paper-Based Search Faceted Interface

PEOPLE PLACES ECONOMYWoman USA IndustryMan -North ServicesChild -South -Manufacturing IndustryAdolescent –North Carolina —TextileAdult —Mountains —FurnitureSenior —-county -Cottage IndustryMarried —Piedmont —Tobacco Bag StringingNon-married –—county AgricultureDivorced —Coastal Plain FarmingWidow –—county CropFamily –Virginia Animals-member -East–father -West MOVEMENT–mother -Northeast Migration–sibling -Southeast -EmigrationRace -Northwest -Immigration-African American -Southwest-American Indian -Midwest LIVING CONDITIONS-Asian Education-White LABOR School SystemSocial Status Labor Movement –Grade School-Upper class Child Labor –Middles School-Middle class Hand Labor –High School-Working class –Cloth Washing –One-Room School-Lower class –Painting -School TermWorking Status –Sewing Transportation-Employed –Sweeping Floors Transportation Means-Unemployed –Tobacco Bag Stringing -Car

Working place -HorseHISTORICAL PERIODS –Factory -RailwayCivil War –Sawmill -TruckReconstruction –Tannery -By FootIndustrial US Jobs Health ConditionsModern US –Farmer DiseaseGreat Depression –Mill Worker -DisabilityNew Deal –Sharecropper -Heart disease-Program –Stringing -Malnutrition–AFDC Income -Mental disease–FERA -source of -Pleurisy–TVA –Insurance -Pneumonia–WPA –Means-Tested Welfare Programs -Rheumatism-Legislation Programs House–Fair Labor Standard Act –Pension -Front Porch–Social Security Act –Salary -RoomWorld War II —Wage –kitchenPostwar US –—Minimum wage –bedroomContemporary US Expense -FurniturePost 9/11 -source of -Appliances

–Grocery Bill —stoveEVENTS –Insurance -Housing Condition—Wall Street Crash –Medical Bill —Electricity—Dust Bowl –Rent —Gas

–Taxes —Indoor Plumbing—Running water


Download - Modeling a Domain Ontology for Cultural Heritage Resources ...cpattuelli.files.wordpress.com/2010/04/21453_ftp.pdfModeling a Domain Ontology for Cultural Heritage Resources: A User-Centered

Top Related