MEDICAL WORD NET AND MEDICAL FACT NET: NEW INFORMATION RESOURCES FOR CONSUMER HEALTH

5.7.2023

AbstractIf a medical information system is to mediate between experts and non-experts, then it must be able to comprehend both expert and non-expert medical vocabulary and to map between the two. Much effort has been devoted to the study of expert medical vocabulary. As computers become increasingly important to the delivery of medical information, however, then it becomes more urgent to understand also the language used by non-experts.

The English-language lexical database WordNet plays an important role in current natural language processing (NLP) applications and research, and it has inspired counterparts in some 40 languages. WordNet has wide coverage of medical terms, but its treatment of these terms is in many ways inadequate. The primary goal of this R21 proposal is to create Medical Word Net (MWN), a systematic revision and validation of WordNet’s coverage in the medical domain.

We shall draw both on our own work in medical ontology and in the construction of WordNet, and also on recent developments in information retrieval technology directed towards the construction of what are called ‘proposition banks’ or ‘fact databases.’ This means that we shall focus not just on single words – but also on the sentences in which such words occur. We will assemble a large corpus of natural-language sentences providing medically meaningful contexts for MWN terms. This will enable us to isolate errors in WordNet’s existing medical coverage and to uncover new medically relevant terms and senses in a systematic way. It will also add to the power of MWN for NLP applications.

This corpus will derive primarily from online health information sources targeted to consumers. By using expert and non-expert human validators we will construct two sub-corpora, called Medical Fact Net (MFN) and Medical Belief Net (MBN). MFN will consist of statements accredited as true on the basis of a rigorous process of validation by medical experts, MBN of statements which non-experts believe to be true.

We shall test the methodology by building an initial corpus of some 40,000 sentences and evaluating its benefits for information retrieval. If this methodology is successful, then it can be scaled to much larger corpora, embracing terms and sentences in other languages and in principle also terms and sentences used by medical experts. MFN and MBN will also support new types of research on consumer health from the perspectives of both psychology and linguistics, for example in exploring individual and group divergences in medical knowledge and vocabulary and in understanding non-expert medical reasoning and decision-making.

Draft Consortium AgreementThe contractual agreement is between SUNY Buffalo and Princeton University as sub-contractor. Barry Smith will direct the work in Buffalo, Christiane Fellbaum in Princeton.

Smith’s primary focus is the use of formal tools in the construction, integration and alignment of ontologies and terminologies in the domain of biomedical research. He has been involved in human subjects experiments designed to establish non-expert ontologies in the domain of spatial knowledge.

Fellbaum is one of the two Principal Investigators of the lexical database WordNet and is a member of the Cognitive Science Laboratory, where WordNet was created and is maintained. Dr. Fellbaum also directs a project "Collocations in the German Language of the 20th Century" at the Berlin-Brandenburg Academy of Sciences, which uses computational methods to study word-sequences. Smith and Fellbaum have been involved for some time in research at the interface between WordNet and formal ontologies.

Non-expert subjects will be recruited from the population of undergraduate students in non-medical disciplines in Princeton, using the standard methods for payment and recruitment employed by the Department of Psychology.

Expert subjects will be recruited from the population of students registered in the Primary Care Externship Program of the School of Medicine and Biomedical Sciences of the University at Buffalo.

The division of labor and responsibility reflects the expertise in Buffalo in the field of medical ontology and terminology and the long experience in Princeton in the construction of digital lexical databases.

The responsibilities will be divided as follows:

Shared between the two sites:Compilation of MBN / MFN databases

Buffalo:Expert validation of MFN databaseComputer processing of MBN / MFN / MWNTesting in query-answering systems and biomedical information retrievalMappings to standard terminology systems

Princeton:Compilation of MWN (lexical database for the medical/consumer health domain)Non-expert validation of MBN / MFN databases

2

A. SPECIFIC AIMS A1. WordNet (Miller, 1995, Fellbaum, 1998) is the principal lexical database used in NLP research and applications. While WordNet’s current version (2.0) has very broad medical coverage, it manifests a number of defects, which reflect the lack of domain expertise on the part of the lexicographers. The present proposal responds to calls from the research community to rectify these defects. (Magnini and Strapparava 2001, Bodenreider and Burgun, 2002, Burgun and Bodenreider, 2002)

We will create Medical Word Net (MWN), an open source lexical database which will revise and extend WordNet’s existing medical coverage in light of recent advances in medical terminology research. We will focus initially on the English-language single word expressions used and understood by non-experts, systematically validating WordNet’s medical coverage in a way which will lead to four types of modifications:

i. the resources of a systematically assembled and validated large corpus of sentential contexts for word usage will extend existing glosses;

ii. the definitions of existing terms and the relations linking such terms (such as is_a and part_of) will be validated for correctness by medical experts;

iii. technical terms not used by non-expert speakers of English, and obsolete terms (including some derived from medieval medicine), will be eliminated;

iv. we will map the results to existing medical terminology resources such as MeSH.

We anticipate that MWN will stabilize in a lexicon of the order of 1,500 single word expressions with some 4,000 distinguished word senses. Nearly all of the relevant word forms are present already in WordNet, but their specifically medical senses are often either unrecorded or are treated inadequately. MWN will be constructed on the basis of a scientific methodology designed (1) to document natural language sentential contexts for each relevant word sense in such a way that the expressed information can be (2) validated by medical experts and (3) accessed automatically by NLP applications used for purposes such as information retrieval, machine translation, question-answer systems, text summarization, and language generation.

A major stumbling block for existing NLP applications is that of automatic sense disambiguation. A machine can detect automatically and with high reliability that a given occurrence of the word feel is a verb. But it cannot determine which of a variety of alternative meanings it might have. WordNet 2.0 distinguishes in all 13 such meanings, of which at least three (marked ) have an obvious medical significance:

1. experience – (undergo an emotional sensation: She felt resentful; He felt regret)2. find – (come to believe on the basis of emotion, intuitions, or indefinite grounds: I feel that he doesn't like me; I find him to be obnoxious; I found the movie rather entertaining) 3. sense – (perceive by a physical sensation, e.g., coming from the skin or muscles: He felt the wind; She felt an object brushing her arm; He felt his flesh crawl; She felt the heat when she got out of the car; He feels pain when he puts pressure on his knee.) 4. feel – (seem with respect to a given sensation given: My cold is gone – I feel fine today; She felt tired after the long hike)5. feel – (have a feeling or perception about oneself in reaction to someone’s behavior or attitude: She felt small and insignificant; You make me feel naked; I made the students feel different about themselves)6. feel – (undergo passive experience of: We felt the effects of inflation; her fingers felt their way through the string quartet; she felt his contempt of her)

3

7. feel – (be felt or perceived in a certain way: The ground feels shaky; The sheets feel soft)8. feel – (grope or feel in search of something: He felt for his wallet)9. feel, finger – (examine by touch: Feel this soft cloth!; The customer fingered the sweater) 10. palpate, feel – (examine (a body part) by palpation: The nurse palpated the patient's stomach; The runner felt her pulse)11. feel – (find by testing or cautious exploration: He felt his way around the dark room)12. feel – (produce a certain impression: It feels nice to be home again)13. feel – (pass one's hands over the sexual organs of: He felt the girl in the movie theater)

WordNet’s architecture for distinguishing word senses has made an important contribution to advancing solutions to the problem of automatic sense disambiguation, and we will build upon this contribution here. To this end we shall assemble a pilot version of a large, heterogeneous, open-source corpus of sentences about medical phenomena in the English language. The corpus will be restricted to natural language grammatically complete, generic, syntactically simple sentences which have been rated as understandable by non-expert human subjects in controlled questionnaire-based experiments. The assembly of this pilot corpus, which we estimate will contain some 40,000 validated sentences, will also be used for purposes of validation of MWN. It will itself support the creation of MWN by yielding new families of words and word senses for inclusion.

Our use of human validations means that we can further extend the usefulness of the corpus for purposes of testing new applications for consumer health information retrieval and also allowing new sorts of research in linguistics and psychology. To this end we shall exploit our validation data to create two sentence subcorpora, called Medical Fact Net (MFN) and Medical Belief Net (MBN).

MFN will consist of those sentences in the pilot corpus which receive high marks for correctness on being assessed by medical experts. MFN is thus designed to constitute a representative fraction of the true beliefs about medical phenomena which are intelligible to non-expert English-speakers.

MBN will consist of those sentences in the pilot corpus which receive high marks for understandability. MBN is thus designed to constitute a representative fraction of the (true and false) beliefs about medical phenomena distributed through the population of English speakers.

Both MFN and MBN will inherit from MWN the (WordNet-based) formal architecture. (Fellbaum 1998) However, we will enhance this architecture to maximize its usefulness in information retrieval and other applications. The validation process that is involved in the construction of MFN will be used to detect errors in the existing WordNet, and also to ensure that the coverage of the natural language medical lexicon in MWN is of a scientific level sufficient to allow MWN technology to work with terminology and ontology systems designed for use by experts.

Compiling MFN and MBN in tandem will allow systematic assessment of the disparity between lay beliefs and vocabulary as concerns medical phenomena and the corresponding expert medical knowledge. The ultimate goal of our work on MFN is to document the entirety of the medical knowledge that is capable of being understood by average adult consumers of healthcare services in the United States today who have no special knowledge of medical phenomena. If the methodology for the creation and validation of the pilot corpus here described proves successful, then we believe that the

4

preconditions for the realization of this much larger goal will have been established. Already the response from NLP researchers and from online information providers to our initial work on MFN/MBN convinces us that this realization would have considerable significance for the management of consumer health information in the future.A2. EvaluationThe creation of MWN, MBN and MFN involves the use of well-tested methods for the creation of lexical databases and sentence corpora and for the handling of question-naires in the validation of sentences for understandability and correctness. We will demonstrate the scientific contributions involved in the creation of this set of resources by showing that these resources themselves bring tangible benefits to the task of con-sumer health information retrieval.

We will evaluate MWN and MFN by measuring the benefits they bring when incorporated into an existing on-line consumer health portal based on term-search technology. (We already have expressions of interest in this regard.)

We will also test whether exploiting the resources of MWN and MFN can lead to improved results in retrieval of expert information by assessing the added value generated by their use for purposes of document retrieval as measured against standard benchmarks.

The community of research groups using WordNet for applications is already very large and the results of our work will be made freely available at every stage to other researchers, who will be invited to propose further strategies for evaluation.

B. Background and SignificanceB1. The Importance of Studying Non-Expert Medical LanguageThis proposal draws on studies of computer-based tools for consumer health information retrieval. (Slaughter 2002, Smith et al., 2002, Tse, 2002, Tse and Soergel 2003, Tse 2003, Zeng et al., in press) Such studies point to a mismatch between existing tools and the non-expert language used by most consumers – the language used not only by patients but also by family members, advisors, administrators, lawyers and so forth, and to some degree also by nurses and physicians.

Where the usage of medical terms by professionals is at least in principle subject to control by standardization efforts, the highly contextually dependent usage of medical terms on the part of lay persons is much more difficult to capture in applications. The state-of-the-art of how to use a lay term is a matter of convention established more or less ephemerally by everyday talk, not only between experts and non-experts but also among the non-experts themselves.

The taxonomies reflecting popular lexicalizations in all domains are much less elabo-rate at both the upper and lower levels than in the corresponding technical lexicon-s.There are no popular terms linking infectious disease and mumps, so that in the popu-lar medical taxonomy of diseases the former immediately subsumes the latter. The pop-ular medical vocabulary naturally covers only a small segment of the encyclopedic vo-cabulary of medical professionals. It lexicalizes mainly at the level of taxonomic orders. Popular medical terms (flu) are often fuzzier than technical medical terms. Many popular terms also cover a larger range of referent types than do technical terms, others may cover only part of the extension of their technical counterparts. We hypothesize, how-ever, that with few exceptions the focal meanings (Berlin & Kay 1969) will be identical.

5

(Constructing MFN and MBN will allow us to test this and related hypotheses in a sys-tematic way.)

The lower degree of differentiation in popular language leads to intersections with families of technical terms such that the popular terms fall short of exact coverage. Many single terms used by both experts and non-experts – for example bacteria, colon, cyst, dermatitis, embryo, glucose, hepatitis, melanoma, septic, spasm – belong to much larger families of cognate terms whose remaining members (for example acystia, baeo-cystin, blastocyst, cysteamine, cysteine, polycystic) are used only by experts. B2. Mismatches in Doctor-Patient CommunicationThe skills of a physician in general practice comprise the ability to acquire relevant and reliable information through communication with patients, and then it is non-expert language that serves as the medium for knowledge exchange across the linguistic divide. The physician must also have the practical knowledge which enables him to convey diagnostic and therapeutic information in ways tailored to the individual patient.

Since the physician, too, is a member of the wider community of non-experts and continues to use the non-expert language for everyday purposes, one might assume that there are no difficulties in principle keeping him from being able to formulate medical knowledge in a vocabulary that the patient can understand. As (Slaughter 2002, Smith et al., 2002) have shown, however, there are limits to this competence. (Slaughter 2002) examines dialogue between physicians and patients in the form of question-answer pairs, focusing especially on the relations documented in the UMLS Semantic Network. Only some 30% of the relations used by professionals in their answers directly match the relations consumers used in their questions. An example of one such question-answer pair is taken from (Slaughter, p. 224):

Question Text: My seven-year-old son developed a rash today that I believe to be chickenpox. My concern is that a friend of mine had her 10-day-old baby at my home last evening before we were aware of the illness. My son had no contact with the infant, as he was in bed during the visit, but I have read that chickenpox is contagious up to two days prior to the actual rash. Is there cause for concern at this point? [...]Answer Text: (a) Chickenpox is the common name for varicella infection. [...] (b) You are correct in that a person with chickenpox can be contagious for 48 hours before the first vesicle is seen. [...] (c) The fact that your son did not come in close contact with the infant means he most likely did not transmit the virus. (d) Of concern, though, is the fact that newborns are at higher risk of complications of varicella, including pneumonia. [...] (e) There is a very effective means to prevent infection after exposure. A form of antibody to varicella called varicella-zoster immune globulin (VZIG) can be given up to 48 hours after exposure and still prevent disease. [...]

Such examples illustrate also that there are lexically rooted mismatches in communication (which may in part reflect legal and ethical considerations) between experts and non-experts. Professionals often do not re-use the concepts and relations made explicit in the questions put to them by consumers. In our example, the questioner requests a yes/no-judgment on the possibility of contagion in a 10-day-old baby. In fact, however, only section (c) of the answer responds to this question, and this in a way which involves multiple departures from the type of non-expert language which the questioner can be presumed to understand. Rather, physicians expand the range of concepts and relations addressed (for example through discussion of issues of prevention, etc.).

In all cases, the information source, whether it be a primary care physician or an online information system, must respond primarily with generic information even where

6

requests relate to specific and episodic phenomena (occurrences of pain, fever, etc.) (Compare Patel et al., 2002). In our example, all sections besides (c) are of this generic kind. They contain answers in the form of generic statements about causality, about types of persons or diseases or about typical or possible courses of a disease. MFN in this pilot phase of our work is accordingly designed to map the generic medical information which non-experts are able to understand. We anticipate the benefits of MFN in consumer information retrieval to rest primarily in the fact that it can be used as a constraint on output in such a way as to ensure that answers given are intelligible to non-experts. B3. Non-Expert Language in Online Communication Understanding patients requires both explicit medical knowledge and also tacit linguistic competence that is dispersed across large numbers of more or less isolated practitioners. This is not a problem so long as this knowledge is to be applied locally, in face-to-face communication with patients. However, as a result of recent developments in technology, including telemedicine and internet-based medical query systems, we now face a situation where such dispersed, practical (human) knowledge does not suffice.

(Ely 2000, Jacquemart 2003) have shown that clinical questions are expressed in a small number of different syntactic-semantic patterns (about 60 patterns account for 90% of the questions. Such questions are typically of the form Do hair dyes cause cancer?, Can I use aspirin to treat a hangover? With the right sort of information resource, questions such as these can easily be transformed automatically into statements providing correct answers: Hair dyes can cause bladder cancer, Aspirin doesn't help in case of a hangover – such statements being linked further to relevant and authoritative sources.

MEDLINEplus is described in its online documentation as a source of medical information for both experts and non-experts: “Health professionals and consumers alike can depend on it for information that is authoritative and up to date.” Enquirers can use MEDLINEplus like a dictionary, choosing health topics by keywords. Or they can use the system’s search feature to gain access to a database of relevant online documents selected for reliability and accessibility on the basis of pre-established criteria.

7

Table 1 Online-Inquiry to MEDLINEplus® (http://www.nlm.nih.gov/medlineplus)

Query text MEDLINEplus® response (with links to documents sorted by the following keywords)

tremor Tremor, Multiple Sclerosis, Parkinson’s Disease, Degenerative Nerve Diseases, Movement Disorders

intentional tremor Tremor, Multiple Sclerosis, Parkinson’s Disease, Spinal Muscular Atrophy, Degenerative Nerve Diseases

tremble Anxiety, Parkinson’s Disease, Panic Disorder, Caffeine, Tremortrembling Anxiety, Parkinson’s Disease, Panic Disorder, Phobias, Tremorright hand trembles Phobias, Anxiety, Infant and Toddler Development, Parkinson’s

Disease, Diabetesright hand trembles when grasping

Infant and Toddler Development, Sports Fitness, Sports Injuries, Diabetes, Rehabilitation

Table 1 shows the problems that arise when the system fails to take account of the special features of the knowledge and vocabulary of typical non-expert users. Here success in finding the needed information depends too narrowly on the precise formulation of the query text. Thus tremble and trembling call forth different responses (one lists caffeine, the other phobias), even though the terms in question differ only in a minor morphological affix, which should not bear on the semantics of the query in any way. Such problems are characteristic of information services of this kind. Experienced internet users are of course familiar with the limitations of search engines, and so they are able to manipulate their query texts in order to get more and better results. Even experienced users, however, will not be able to overcome the arbitrary sensitivities of an information system, and the latter cannot have the goal of bringing non-experts’ ways of using language into line with that of the system. B4. Corpus- and Fact-Based Approaches to Information Retrieval(Patel, Arocha and Kushniruk 2002) make clear that if a medical information system is to mediate between experts and non-experts, then it must rest on an understanding of both expert and non-expert medical vocabulary. The problem is to capture the way terms are used in their contexts in the form of a clear-cut semantic representation, and this is one of the hardest tasks facing linguistic science today. There is no effective way to disambiguate the meanings of the highly polysemous (ambiguous) terms characteristic of natural language without finding some way to take sentential contexts into account. (Pustejovsky, 1995) Standard NLP methodologies seek to solve this problem by focusing on human competence, i.e. on the combinatoric rules humans are hypothesized as using in determining specific meanings on the basis of different types of contexts. The idea underlying the present proposal draws on currently emerging NLP methodologies which draw instead on human performance and on the powers granted by computers to manipulate very large corpora. Thus it employs the strategy of storing large numbers of sentences used and understood by human beings, and exploiting standard pattern-recognition and statistical techniques for purposes of disambiguation. This methodological focus on competence is manifested in the FrameNet and Penn Propbank projects.

8

FrameNet (Baker, 1998) is a database containing not only lexical information, but information about the meaning and use of lexical items in contexts. The building blocks of FrameNet are Frames, conceptual units such as commerce, religion, and health. FrameNet has 12 medical frames:

Addiction, Birth, Biological Urge, Body Mark, Cure, Death, Health Response, Medical Conditions, Medical Instruments, Medical Professional, and Medical Specialties and Observable Body Parts.

With each frame are associated Frame Elements, for example under Cure: alleviate. v, alleviation. n, curable. a, curative. a, curative. n, cure. n, cure. v, ease. v, heal. v, healer. n, incurable. a, palliate. v, palliation. n, palliative. a, palliative. n, rehabilitate. v, rehabilitation. n, rehabilitative. a, remedy. n, resuscitate. v, therapeutic. a, therapist. n, therapy. n, treat. v, treatment. n.

Frame Elements can be mapped onto different Frames as they occur in a variety of statements, and so the underlying semantic structures of these statements are made explicit. For example, in Quinine cures malaria, quinine is mapped to medicine and malaria to illness. A major insight of FrameNet is that semantic constituents are independent of specific syntactic (part-of-speech) categories like subject, object, etc. (Fillmore, 1982) The FrameNet database contains 482 frames, each with a multiple of Frame Elements whose usage is exemplified by sentences drawn from the British National Corpus such as: He needs prolonged treatment with antibiotics; He emphasized treatment by surgical means. A lexicon shows the Frame Elements to which a given lexical item can be mapped. If a lexical item can be mapped to several Frame Elements (in distinct frames), this means that it is polysemous (ambiguous). For example, cure can be an Frame Element in a health frame and in a cooking frame. FrameNet’s lexical coverage is uneven and its medical coverage is poor. Moreover it is not accompanied by a corpus of generic statements of the sort required by MFN.

Penn’s PropBank (for “Proposition Bank”) project is in many respects similar to FrameNet, though with a greater focus on the building of a large corpus of sentences to determine context and on the association with this corpus of a specific logical (function-argument based) architecture.

Both FrameNet and Propbank are focused on usage in the general lexicon, with no concern for issues of factuality or validation or for the goal of assembling a corpus in a systematic way that is designed to ensure adequate coverage of any given domain. In this they are sharply contrasted with the project described here.

Our work is in some ways comparable also to the CYC (short for enCYClopedia) project (Guha et al. 1990, Lenat 1995), which reflects in part the idea of a fact database for naïve physics outlined in (Hayes, 1985) and pursued also in the Botany Knowledge Base (Clark and Porter 1996) and in DARPA’s Rapid Knowledge Formation project. The CYC system is composed of a knowledge base, an inference engine and tools for natural language parsing and generation. It can be used for storing and retrieving information and performing limited semantic analysis. CYC’s knowledge base is structured as a collection of hundreds of thousands of statements mostly about the external world. Examples are: The earth is round, Mountains are one kind of landform, Albany is the capital of New York, Deer live in the woods. These statements, which were entered over many years by CYC employees, are parcelled out into separate micro-

9

theories devoted to different domains. (On CYCs medical coverage see Bodenreider and Burgun, in press.)

Our project differs in a number of ways from CYC, including: (i) we focus on one single (albeit very large) domain; (ii) while CYC incorporates machinery for parsing natural language text, it does not store English sentences but rather – in keeping with its goal of being language-unspecific – statements couched in the symbolism of a modified first-order logic; (iii) only a reduced part of CYC is publicly available.

C. Preliminary Results/Progress Report C1. WordNetWordNet 2.0 is a large electronic lexical database of English that has found wide acceptance in areas as diverse as artificial intelligence, natural language processing, and psychology (Agirre et al., 2000; Al-Halimi et al., 1998, Artale et al., 1997, Basili et al., 1997, Burg and Riet, 1998, Cucciarelli and Velardi, 1997, Fellbaum, 1990, Gonzalo et al., 1998, Harabagiu et al., 1996, Magnini et al., 2001, Bewick et al., 1990). Its coverage, which is comparable to that of a collegiate dictionary, extends over some 130,000 words. The most common application is in information technology, where it is used for information retrieval, document classification, question-answer systems, language generation, and machine translation. WordNet was originally conceived as a full-scale model of human semantic organization, and its design was guided by early experiments in artificial intelligence (Collins and Quillian, 1969).

WordNet is entirely hand-built, reflecting the team’s conviction that automatically compiled dictionaries and thesauri are fraught with errors as we well as the fact that the machines are necessarily limited in mimicking the semantic intuitions and linguistic judgments of human beings. No automatically compiled lexical resource can compete in coverage and quality with WordNet, which accounts for its wide acceptance. Unfortunately, when the WordNet project was initiated, no large text corpora were available (the 1968 Brown Corpus was the only existing balanced corpus, but its coverage is woefully inadequate by today’s standards). WordNet was quickly embraced by the NLP community, a development that has guided its subsequent growth and design, and WordNet is now widely recognized as the lexical database of choice for NLP. The appeal of WordNet’s design is reflected in the fact that wordnets have been, and continue to be, built in dozens of languages. Wordnets are already available supporting many European and non-European languages. All are marked by the fact that they are open source, and by the fact that they are linked to the original English WordNet, which thereby functions as an interlingual index. In consequence, all wordnets can be mapped to one another. This means that the medical terminology that we propose to add to WordNet will ultimately be translatable into dozens of languages with very little additional effort. C2. Architecture of WordNetThe building blocks of WordNet are synonym sets (‘synsets’), which are unordered sets of distinct word forms and correspond closely to what, in medical terminology research, are called ‘concepts.’ Membership in a synset requires that the word forms express the same concept and are in this sense ’cognitively synonymous’ (Cruse, 1986). More formally, synset members must be interchangeable in some sentential contexts without

10

altering the truth-value of the sentences involved. WordNet’s architecture is thus grounded in the notion of truth-preserving interchangeability of word forms in sentential contexts, although research has not thus far focused on this feature. The construction of Medical Fact Net will allow us to rectify this gap by making explicit the contexts in which word forms are used in an environment which allows the systematic testing of the effects of word form substitution.

Examples of synsets are {car, automobile} or {shut, close}. WordNet 2.0 contains some 115,000 synsets, with many word forms (as in the case of feel above) belonging to a plurality of synsets.

WordNet is a net in virtue of the fact that the synsets are linked to one another via a small number of binary relations that differ for each of the four syntactic categories covered by WordNet: nouns, adjectives, verbs, adverbs. Noun synsets are interlinked by means of the subtype or is-a relation, as exemplified by the pair poodle-dog, and by means of the part-of relation, as exemplied by the pair tire-car. Verb synsets are connected by a variety of lexical entailment relations (Fellbaum, 1998, 2002, 2003) that express manner elaborations, temporal relations, and causation (walk-limp, forget-know, show-see). The links among the synsets structure the noun and verb lexicons into hierarchies, with noun hierarchies being considerably deeper than those for verbs.

Relations like is-a and part-of are called ‘conceptual’ or ‘semantic’ because they hold among all the members of a linked synset. In addition, WordNet records lexical relations, which hold between specific word forms above and beyond their semantic relations. This is the case with adjectives, which are organized into clusters consisting of a pair of direct antonyms (such as expensive and cheap) together with adjectives that are semantically similar to each member of such a pair (costly and low-cost, respectively). The semantically similar adjectives are said to be indirect antonyms of one member of the direct antonym pair. Thus, low-cost is an indirect antonym of expensive, and costly is an indirect antonym of cheap. Although the semantic relation of contrast holds between direct and indirect antonyms, direct antonym pairs stand out by virtue of the strong association between their members (Fellbaum 1995).

WordNet’s appeal for NLP applications stems from the fact that its synset architecture can be exploited in building NLP applications targeting the problem of automatic word sense disambiguation. Although most word forms in English are monosemous (clinician, epidemic), the most frequently occurring words are highly polysemous (dress, host). The ambiguity of a polysemous word in a context can be resolved by distinguishing the multiple senses in terms of their links to other words within the WordNet net. For example, the noun club can be disambiguated by an automatic system that considers the superordinates of the different synsets in which this word form occurs: association, playing card, and stick. C3. Uses of WordNet in Medical InformaticsWordNet’s design allows users with specific technical applications in mind to augment the database, primarily by adding new terms as leaves to the existing branches of its subsumption and part-whole hierarchies. Such enriched wordnets retain all of the origi-nal information, and the added words are semantically specified in terms of WordNet’s relations (Turcato et al., 2000). (Buitelaar and Sacaleanu 2002) describes an attempt to extend of the German wordnet with synsets pertaining to the medical domain using au-tomatic methods, in particular the detection of semantic similarity from co-occurrence

11

patterns in a domain-specific corpus. The results, while good, are hampered by prob-lems of lexical polysemy and by the characteristically German problem of the need to analyze compounds. One clear conclusion from this study is that fully automated lexical acquisition provides inadequate results, and that much of the work must be performed manually. Our proposal reflects this conclusion.

(Bodenreider and Burgun, 2002) and (Burgun and Bodenreider, 2002) characterize the definitions of anatomical concepts in WordNet and in various portions of the UMLS Metathesaurus. They found that anatomical definitions are characteristically of the form: superordinate + distinguishing feature (the latter expressed through some form of adjec-tival modification or relative clause, etc.). This way of defining words is in fact the canon-ical one (for nouns), which lexicographers observe as much as possible. MWN will con-sistently observe this standard in its augmentation and standardization of WordNet’s definitions, drawing on the results of the studies of best practice in the formulation of definitions in biomedical terminologies and ontologies in our (Smith and Rosse, 2004; Smith, Köhler, Kumar 2004).

WordNet (2.0) contains domain labels attached to thousands of synsets, a feature which allows the automatic extraction of all words that are associated with this domain. One such label is medicine. Currently, when asked to output terms associated with medicine the browser returns some 504 nouns, verbs, and adjectives (both single words and phrases), representing some 270 different senses. On the other hand, many cog-nate senses with clear medical uses are currently not labeled in this way. Table 2 pro-vides examples, with the medicine label picked out in bold:

autopsy#1 {autopsy, necropsy, postmortem, PM, postmortem examination – (an examination and dissection of a dead body to determine cause of death or the changes produced by disease)}

fester#1 {fester, maturate, suppurate – (ripen and generate pus; her wounds are festering)}

festering#1 {festering, suppuration, maturation – ((medicine) the formation of morbific matter in an abscess or a vesicle and the discharge of pus)}

festering#2 {pus, purulence, suppuration, ichor, sanies, festering – (a fluid product of inflammation)}

infection#1 {(the pathological state resulting from the invasion of the body by pathogenic microorganisms)}

infection#3 {((medicine) the invasion of the body by pathogenic microorganisms and their multiplication which can lead to tissue damage and disease)}

infection#4 {infection, contagion, transmission – (an incident in which an infectious disease is transmitted)}

maturation#2 {growth, growing, maturation, development, ontogeny, ontogenesis – ((biology) the process of an individual organism growing organically; a purely biological unfolding of events involved in an organism changing gradually from a simple to a more complex level; he proposed an indicator of osseous development in children)}

maturation#3 {festering, suppuration, maturation – ((medicine) the formation of morbific matter in an abscess or a vesicle and the discharge of pus)}

suppuration#1 {festering, suppuration, maturation – ((medicine) the formation of morbific matter in an abscess or a vesicle and the discharge of pus)}

12

suppuration#2 {pus, purulence, suppuration, ichor, sanies, festering – (a fluid product of inflammation)}

zymosis#2 {((medicine) the development and spread of an infectious disease (especially one caused by a fungus))}

Table 2. Examples of Medically Relevant Entries in WordNet 2.0

This Table also illustrates the degree to which WordNet currently includes obsolete medical terms (ichor, morbific, unction) and also terms drawn indiscriminately from both technical medical vocabularies and from natural language. Some synsets contain only folk or only technical terms, some contain a mixture of both.

To provide a preliminary estimate of the extent of WordNet’s medical coverage we derived a test lexicon of 2838 single-word medical terms from an existing digitalized lexi-cal resource for medical language processing (LinKBase of the Belgian NLP company L&C), which was constructed independently of WordNet by medical professionals. The method used was to transform LinkBAse into an alphabetically ordered term list and to eliminate automatically all acronyms, all multi-word terms, and all terms greater than 10 letters in length. Remaining technical terms were then removed manually. Of the resid-ual 2838 terms, only 11 were not present in any form in WordNet 2.0, though consider-ably more were not treated adequately in regard to their specifically medical usages, and some specifically medical syntactic modifications and compounds (bedwetting, breastfed, coldsore, ribcage) were not present.

WordNet 2.0 has inadequate treatment of the symptom role played by concepts such as redness, retching, swelling, and so forth. Thus it has Dizziness is_a sensation but not: Dizziness is_a symptom. It has A tumor is a mass of tissue and A tumor is abnormal but not Some tumors are malignant.

WordNet’s treatment of is_a, part_of and other relations, too, is marked by inadequacies in the medical domain. Thus WordNet currently contains an entailment relation exemplified by the pair snore-sleep defined as: “if someone snores, then he necessarily also sleeps (again, the reverse does not hold).” In medicine, however, it is quite possible to snore while awake, since snoring implies the respiratory induced vibration of glottal tissues as associated not only (and most usually) with sleep but also with relaxation or obesity. Constructing MFN, with its concomitant validation by experts of WordNet’s relations, will provide us with a systematic means to detect such errors. Constructing MBN in addition will give us the resources to do justice to the reason why such cases were included in WordNet as they are: People can only snore when they are asleep and similar sentences belongs precisely to the folk beliefs about medicine which MBN will document – not, however, to MFN. More generally, constructing MBN in tandem with MFN will allow us to highlight those cases where non-experts and experts use the same terms in different ways.

Another family of terms currently poorly treated are those manifesting polysemy along the medical/non-medical axis. For example, the medical senses of recession, rigors, or oppressed (as in oppressed cell growth) are not recorded. A lexical database for purposes of automatic sense disambiguation must clearly differentiate all such meanings. For computerized medical information systems do not offer the possibility of follow-up in cases of misunderstanding of the sort which we have in communication between laypersons and medical practitioners. Thus while MWN will contain only word

13

forms that are used by non-experts (and thus part of natural rather than technical language), it must for practical reasons record word senses that are peculiar to the technical vocabulary.C4. Extracting Sentences from Online Consumer Health Information SourcesWe carried out experiments designed to test a variety of methodologies for deriving terms and sentences for our corpus, including elicitation experiments with expert and non-expert human subjects and data-mining from on-line bulletin boards. We established that the most promising sources for both term- and sentence-generation are certain online information sources targeted specifically to non-specialist users of the type included within MEDLINEplus.

In one experiment sentences were derived by researchers in medical informatics from factsheets on Airborne allergens in NIAID’s Health Information Publications and on Hay fever and perennial allergic rhinitis in the UK NetDoctor’s Diseases Encyclopedia. Method The initial documents were divided into paragraph-length sections and subjects were instructed to associate with each section complete self-contained sentences expressing the generic medical knowledge it contains. Sentences were to be formed using simple syntax and as far as possible drawing on terms used in the original sources. Anaphora, indexical expressions, and formulations of instructions, warnings and the like were to be eliminated. Subjects were instructed to include only such terms and information which they themselves judged would be intelligible to non-experts. (We regard this informal reliance on subjective judgments to be admissible given the rigorous processes of validation described below.)Results 1644 sentences were produced, representing some 10 person hours of effort, samples of which are presented in Table 3.

Table 3: Sample sentences derived from online medical information sourcesThere is no good way to tell the difference between allergy symptoms of runny nose, coughing, and sneezing and cold symptoms. Allergy symptoms, however, may last longer than cold symptoms.

from NIAID HealthInfo (information also included in MEDLINEplus)

1. Allergies have symptoms.2. Colds have symptoms.3. A runny nose is a symptom of an allergy.4. Coughing is a symptom of an allergy.5. Sneezing is a symptom of an allergy.6. Cold symptoms are similar to allergy symptoms.7. A cold is not an allergy.8. Allergy symptoms usually last longer than cold symp-

toms.

What is hay fever? Hay fever, otherwise known as seasonal allergic rhinitis, is an allergic reaction to airborne substances such as pollen that get into the upper respiratory passages - the nose, sinus, throat - and also the eyes.

from NetDoctor

1. Hay fever is an allergy2. Hay fever is an allergic reaction3. Hay fever is a type of allergy4. Hay fever is a type of allergic reaction5. Hay fever is a reaction to pollen6. Hay favor is a reaction to airborne substances7. In hay fever airborne substances get into the nose8. In hay fever airborne substances get into the throat9. In hay fever airborne substances get into the eyes

D. Research Design and MethodsD1. Relevant expertise of principal investigators:

14

Smith has expertise in ontologies in biomedicine and other fields (Burkhardt and Smith 1991) , in the use of ontologies in computational linguistics (Ceusters, Desimpel, Smith, Schulz 2003) and in the role of formal ontologies and formal architectures for quality assurance and alignment of large-scale terminology systems (Smith, Williams, Schulze-Kremer 2003, Kumar, Smith 2003, Smith, Köhler, Kumar 2004, Fielding, Simon, Ceusters, Smith, 2004, Ceusters, Smith, Kumar, Dhaen, 2004, Smith, Rosse 2004) He was involved in a project designed to elicit the spatial ontology of non-experts using questionnaire and other methodologies in experiments involving large numbers of subjects. (Mark et al 1999, Smith and Mark, 1998, 2001) Fellbaum has considerable experience building large-scale lexical resources and is one of two principal authors of WordNet. (Fellbaum ed 1998, Fellbaum ETC…) She participated in a series of experiments collecting extensive word association data from English speakers in order to validate semantic relations among verbs in WordNet. (Fellbaum and Chaffin1990)D2. Sources and Selection The primary sources for terms in MWN and for sentences in our test corpus will be the relevant general lexical information contained in WordNet, medical dictionaries and large medical terminology and ontology systems such as MeSH and LinKBase, together with internet resources such as MEDLINEPlus. We shall maintain an internet portal through which these sources, and the results of our term- and sentence-extraction, will be made available online as raw data for use by other researchers.

In this initial phase of our project we are interested primarily in self-contained generic statements with a relatively simple syntax. To derive such sentences we use two methods:Method 1 derives sentences from a lexical database such as WordNet. We treat the database as a set of links between terms of the form tLu (where L ranges over 'is-a', 'part-of', 'is-caused-by', etc. relations) and t, u range over terms which occur in the medical sublexicon. Some members of the resulting class of tLu formula can be transformed automatically into English sentences with a minimal amount of post-processing. For example each t is-a u formula can be transformed into a sentence of the forms ‘a t is a u’ or 'a t is a type of u’ (with corrections for articles and plurals, as in: A cut is a type of wound; An abrasion is a wound; Patients are people). Others must be subject to manual extraction, which can be carried out by native English-speakers (linguists or others trained in manipulation of lexical databases) with no special medical expertise. Each extraction sentence will be given a precise identifying numberMethod 2 derives sentences from on-line consumer health information sources along the lines of the pilot experiment described in C4 above. Each sentence in the source documentation will be given a precise identifying number, indicating source document, position in this document, and section from which sentences have been inferred. The latter too will be given precise identifying numbers and associated with metadata documenting section and document of origin, date of processing, and also individual responsible for extraction.D3. Human Subject ValidationsThe output sentences from the above will serve, together with a random infusion of non-medical, folk-medical-but-false and medical-but-technical sentences, as inputs to validations carried out by human subjects. These will be of three primary types, referred to in what follows as Pu, Pb and Bc, for Princeton and Buffalo, and for understanding,

15

belief, and correctness, respectively. All sentences will pass through the Pu filter, in which laypersons will be recruited to rate sentences for understandability. Those sentences which survive will pass on to Pb and Bc. In Pb laypersons will rate sentences for degree of belief, in Bc medically trained participants (“experts”) will rate sentences for correctness. All validations will be carried out online.Participants (Pu and Pb): Undergraduates will be recruited with backgrounds in all areas of study, but excluding pre-medical majors and also those with considerable recent experience of medical care or of medical care institutions. 250 participants will be asked to fill out a questionnaire, including information about age, gender, their major and possible course work related to the medical domain, together with a list of sentences to be rated. Each will be paid $8 for the total task, which we do not anticipate will require more than one hour. Method (Pu): Each of 400 statements, randomized across the questionnaires, will be rated for understandability by two participants, making for a total of 50,000 (400 x 250 / 2) ratings. Rating will be in terms of responses to the question: on a scale from 1-5, would you describe this sentence as hard to understand or easy to understand? Raters will be encouraged not to reflect on successive statements but to pass immediately onto the next statement (if they are unsure they are to leave the corresponding checkbox blank, will will be counted as equivalent to a score of 1). Only those statements which receive a score of at least 4 from each of 2 subjects will be form the pool for use in Pb

and Bc.Method (Pb): Each of 200 statements from the pool will be rated by each of 250 participants for assent. Raters will be asked to respond to the question: on a scale from 1-5, would you describe this sentence with the words do not agree at all or agree completely ? Raters in Pb will be encouraged to reflect upon their answers if necessary. Statements receiving from 2 raters a total score of at least 8 will be stored as components of Medical Belief Net (MBN). Participants (Bc): Raters will be selected from the Primary Care Externship Program of the Buffalo School of Medicine, and will be subject to a pre-evaluation as follows. Some of the sentences in the pool will derive from MEDLINEplus and similar sources which have already been pre-validated by experts. These sentences, together with a corpus of medically questionable sentences from a variety of sources, will be validated as true or false by experts with specialty training in the appropriate fields. This benchmark set will then be issued in the form of a randomized list to persons who are candidate participants in Bc as a validation exercise. Those candidate participants with very high scores in this exercise will then be selected to serve as raters in a series of validations of the outputs of Pu. Since we are dealing with generic sentences with simple syntax we do not believe that it will be necessary to match sentences with raters drawn from particular subject-domains of medicine. However we will modify our rating methodology in this and other respects if this proves necessary.Method (Bc): Here rating method will involve no time constraints and raters will be encouraged to use reference works for problematic cases. Raters will be asked to respond to the question: on a scale from 1-5, would you describe this sentence with the words is not true at all or is completely true? Only those sentences which receive a score of 5 from each of two raters will be added to the MFN database. Various means will be used to counter the effects of routinization. Thus in relation to those sentences

16

which receive a score of less than 5, raters will be encouraged to propose one or more alternatives which would qualify for a perfect score and which would at the same time be intelligible to non-experts. These proposal sentences will then return to the Pu phase for assessment.

Raters in Bc will be provided with instructions as to the sorts of deviations from correctness which should lead to a score of less than 5. Above all we will draw attention to the problem created by shortfalls from genericity. Many sentences expressing generic medical knowledge (consider: smoking causes cancer) relate only to what holds for the most part or in most cases or in a statistically significant fraction of cases.

All sentences entering the Pu, Pb and Bc process will be stored together with metadata recording scores along the three axes rated, together with information pertaining to evaluators and to source of data (including exact documentation for entries derived from medical literature).D4. Formal ArchitectureThe formal architecture for MWN will model very closely that of WordNet 2.0, with refinements of the type discussed in (Smith, Rosse, 2004) concerning part_of and in (Burgun et al. 2002) concerning roles. The formal architecture for MFN will be constructed as follows.Method The sentences which survive Bc validation will be linguistically processed by both automatic and manual methods. First, the corpus will be part-of-speech tagged with the Brill tagger. All nouns, verbs, and adjectives will then be linked to the appropriate synsets in MWN. This last step cannot be performed fully automatically, as many words will be polysemous. We expect this to be the case for verbs and nouns more so than for adjectives. After automatically tagging the monosemous words, polysemous words will be linked manually. The MWN-links between words thus tagged will be exported automatically to MFN, and thus WordNet’s net structure will be inherited automatically. In addition, by permutating through MWN-synsets we can generate new sentences as input into Pu.D5. EvaluationMethod 1 We will measure the degree to which using the resources of MWN and MFN to direct users to information sources results in greater user satisfaction by setting up an experiment in which users of consumer health information portal are randomly assigned to one of four groups: the basic group involves the unsupplemented portal, which we can assume to be one in which simple term-searching is used; the second and third group will have access in addition to the resources of MWN and MFN respectively; the fourth group will have access to both.

Each time a user connects to the portal, he will be informed (by means of a pop-up or a special advertisement, or whatever mechanism the portal owner might prefer) about the possibility of participating in a study designed to improve search technology. Goals and objectives of the study will be described in simple terms, together with a description of the procedure that will be followed if he agrees to participate and confirmation that no privacy related data will be stored, that all input and output will remain anonymous, and so forth. The user is then given the option to click on a "yes" or "no" button, thereby explicitly communicating his (non-)willingness to participate. In case of a “yes" response, the user will be randomly assigned to one of the two groups as described above.

17

To measure individual user satisfaction we will follow a modified version of the procedure as described in (Saracevic, and Kantor 1988). We will ask users to provide the following information:a) related to clarity of problem definition (on a scale from 1-5, would you describe your problem as weakly defined or clearly defined?) b) related to their intent (on a scale from 1-5, would you say that your use of this information will be open to many avenues, or for a specifically defined purpose);c) related to amount of prior knowledge (on a scale from 1-5, how would you rank the amount of knowledge you possess in relation to the problem which motivated you request?) d) related to their expectation (on a scale from 1-5, how would you rank the probability that information about the problem which motivated this research question will be found in the literature?). By applying population-statistical techniques to these data in conjunction with the associated satisfaction scores we can then assess the benefits brought by MWN and MFN to different types of populations. The nul-hypothesis is that there is no difference inthe two populations, and we would apply the relevant statistical tests to check that hypothesis. The four questions are then used to identify subpopulations. It might for instance be that MFN is of little benefit for people who know very well what they are looking for and who have good insight into the problem in hand. The scale in the satisfaction question can be used to assess also whether people with little background information about a problem have an intrinsic bias for being unsatisfied (or over-satisfied) independently of what services are offered by the system.Method 2 We will measure the degree to which using the resources of MWN and MFN can lead to improved results in retrieval of expert information. Here we will follow the methodology described in (Jackson and Ceusters 2002). We will take as baseline the performance of an existing information retrieval system on OHSUMED, a corpus of documents collated from MEDLINE abstracts (Hersh et al., 1994). We will then compare this baseline with the performance obtained when the system is given additional access to MWN/MFN. The OHSUMED corpus contains a collection of questions together with relevance judgments about documents with respect to these questions. Both these relevance judgments and the results achieved by various research groups that participated in the TREC-2002 (Voorhees and Buckland, 2002) competition will be used to compare our results.D6. Future WorkIn the fields of medical education and medical literacy we envisage MBN/MFN being used to evaluate the reliability of the medical knowledge of different non-expert communities. On the basis of MFN we can imagine the development of tools to support the face-to-face education of lay people in the fields of medicine and health care, e.g. for the purpose of providing a general orientation guidance about a disease or giving general instructions concerning lifestyle, nutrition, etc. On the basis of metadata pertaining to the sources of entries in MBN it will be possible to keep track of specific kinds of false beliefs as originating in specific kinds of informants. This may prove a valuable source of information for example in targeting specific groups for specific types of remedial medical education.

18

In addition, we believe that the extended MBN will provide opportunities for a new type of research in the field of consumer health. Specifically, we envisage experiments to investigate how the domain of medical phenomena is conceptualized by non-expert human subjects. Cognitive psychologists and anthropologists such as E. Rosch and others (Rosch 1973, 1975, 1978) have postulated a level of lexical specification that they call “basic level.” Basic level words correspond to basic kinds in the ontology of language-using subjects. Such words exist in all semantic domains, but they have been studied predominantly among words denoting natural kinds, such as animals, vegetables, and fruit. For example, tomato is often cited as an example of a basic level word, whereas “vegetable” is a superordinate, and cherry tomato is a subordinate. Basic level words have many striking properties: they are universally lexicalized, characterized by high frequency of occurrence, and they are learned first by children. The concepts they denote have properties that differ maximally from each other (e.g., a tomato is very different from a cabbage or a bean), but the difference between a basic level word and a subordinate (such as between a tomato and a cherry tomato) is less pronounced. The basic level lexicon in the medical domain has thus far not been explored, but such research promises important theoretical benefits. MBN might be used to determine the basic level in the domain under investigation by examining the difference in the frequency of occurrence of synonyms: highly frequent terms are good candidates for basic level words. Following the precedent set by (Rosch 1975) we can then use the results of this work to provide a specification of the non-expert ontology of the medical domain and begin to explore differences between it and the expert ontology of medicine documents. In later iterations we envisage pursuing experiments along the lines described in (Keil et al., 1999), designed to elicit a counterpart of MBN representing the ontology of the medical domain as apprehended by children at various ages.

Note that MBN and MFN have characteristically played different roles in the above. Thus where MBN has been associated with research, for example regarding what people believe about medical phenomena, MFN has been associated with constructing practical tools designed to assist them in coming to believe what is true. We estimate that the two documents discussed in CX above together constitute some 0.2 % of the collection of similar documents available on these two sites, the whole constituting a comprehensive survey of consumer health knowledge. This suggests that a future comprehensive version of MFN might contain some 250,000 sentences. The prospect of constructing and managing an sentence-based information resource of this size would until very recently have rightly been considered overwhelming. The success of WordNet gives us confidence that this problem, too, can be solved. The development of MWN, MFN and MBN should be seen as part and parcel of recent advances in the biomedical sciences of similar scope, above all in the development of large lexical resources such as SNOMED and the UMLS, and of large fact-repositories such as KEGG or Swiss-Prot.

19

LiteratureAgirre E, Martinez D. Exploring automatic word sense disambiguation with decision lists and the Web. Proceedings of the Semantic Annotation and Intelligent Annotation Workshop organized by COLING, Luxembourg, 2000.

Al-Halimi R, Kazman R. Temporal indexing through lexical chaining. Fellbaum C. (ed.), WordNet: An Electronic Lexical Database, MIT Press, Cambridge, Maryland, May 1998.

Artale A, Magnini B, Strapparava C. WordNet for Italian and its use for lexical discrimination. Proceedings of the 5th Congresso dell’Associazione Italiana per l'Intelligenza Artificiale, Rome, September 1997; 16-19.

Baker CF, Fillmore CJ, Cronin B. The structure of the framenet database. International Journal of Lexicography, 2003; 16.3: 281-296.

Baker CF, Fillmore CJ, Lowe JB. The Berkeley FrameNet project. Proceedings of the COLING-ACL, Montreal, Canada, 1998.

Basili R, DellaRocca M, Pazienza MT. Contextual word sense tuning and disambiguation. Applied Artificial Intelligence 1997; 11 (3): 235-262.

Beckwith R, Fellbaum C, Gross D, Miller G. WordNet: A lexical database organized on psycholinguistic principles. In Zernik U (ed.), Using On-line Resources to Build a Lexicon. Erlbaum, Hillsdale, NJ, Erlbaum, 1990; Chapter 9: 211-231.

Berlin B, Kay P. Basic color terms. Berkeley/Los Angeles: University of California Press, 1969.

Bodenreider O, Burgun A. Characterizing the definitions of anatomical concepts in WordNet and specialized sources. Proceedings of the First Global WordNet Conference, January 2002; 223-230.

Bodenreider O, Burgun A. Ontologies in the biomedical domain. Part II: examples. Journal of the American Medical Informatics Association (in press).

Bodenreider O, Burgun A, Mitchell JA. Evaluation of WordNet as a source of lay knowledge for molecular biology and genetic diseases: a feasibility study. Studies in Health Technology and Informatics 2003; 95: 379-384.

Buitelaar P, Sacaleanu B. Extending synsets with medical terms: ranking and selecting synsets by domain relevance. Proceedings of the First Global WordNet Conference, Mysore, India, January 2002.

20

Burg JFM, van de Riet RP. COLOR-X: using knowledge from WordNet for conceptual modeling. Fellbaum C (ed.), WordNet: An Electronic Lexical Database, MIT Press, Cambridge, Maryland, May 1998.

Burgun A, Bodenreider O. Comparing terms, concepts and semantic classes in WordNet and the Unified Medical Language System. Proceedings of the NAACL Workshop on WordNet and Other Lexical Resources, Pittsburg, 2001; 77-82.

Burgun A, Bodenreider O, Le Duff F, Moussouni F, Loréal O. Representation of roles in biomedical ontologies: a case study in functional genomics, Proc AMIA Symp 2002; 86-91.

Burkhardt H, Smith B. (eds.), Handbook of Metaphysics and Ontology, 2 vols., Munich/Philadelphia/Vienna, 1991.

Ceusters W, Desimpel I, Smith B, Schulz S. Using Cross-Lingual Information to Cope with Underspecification in Formal Ontologies, Proc Medical Informatics Europe 2003, Amsterdam: IOS Press, 2003; 391–396.

Ceusters W, Smith B, Kumar A, Dhaen C. Ontology-based error detection in SNOMED-CT®”, in Proc Medinfo, San Francisco, September 7–11, 2004.

Ceusters W, Smith B, Fielding JM. LinkSuite: Formally robust software tools for ontology-based data and information integration. Proceedings of DILS 2004 (Data Integration in the Life Sciences), (Lecture Notes in Computer Science), Springer, Berlin, 2004, in press.

Clark P, Porter B. Building domain representations from components, Technical Report AI96-241, Department of computer science, University of Texas at Austin, 1996.

Collins AM, Quillian MR. Retrieval time from semantic memory, Journal of Verbal Learning and Verbal Behavior 1969; 8: 240-248.

Cristianini N, Shawe-Taylor J. An introduction to support vector machines (and other kernel-based learning methods), Cambridge University Press, Cambridge, UK, 2000.

Cruse DA. Lexical semantics. Cambridge University Press, Cambridge, UK, 1986.

Cucchiarelli A, Velardi P. Automatic selection of class labels from a thesaurus for an effective semantic tagging of corpora. Proceedings of the 5th Conference on Applied Natural Language Processing, Washington, 1997; 380-387.

Ely JW, Osheroff JA, Gorman PN, Ebell MHl, Chambliss ML, Pifer EA, Stavri PZ. A taxonomy of generic clinical questions: classification study. British Medical Journal, 2000; 321: 429-432.

Fellbaum C. Co-occurrence and antonymy. International Journal of Lexicography 1995; 8 (4): 281-303.

21

Fellbaum C. Distinguishing verb types in a lexical ontology. Proceedings of the Second International Workshop on Generative Approaches to the Lexicon. ISSCO, Geneva, 2003.

Fellbaum C. English verbs as a semantic net. International Journal of Lexicography 1990; 3 (4): 278-301.

Fellbaum C. On the semantics of troponymy. The Semantics of Relationships: an Interdisciplinary Perspective. Green R, Bean CA, Myaeng SH (eds.), Dordrecht, Kluwer, 2002; 23-34.

Fellbaum C (ed.). WordNet: an electronic lexical database. MIT Press, Cambridge, MA, 1998.

Fellbaum C, Chaffin R. Some Principles of the Organization of the Verb Lexicon. Proc 12th Annual Conf Cognitive Science Society, 1990; 420-428.

Fielding JM, Simon J, Ceusters W, Smith B. Ontological Theory for Ontology Engineering, in KR 2004. Proc Ninth Internat Conf Knowledge Representation and Reasoning.

Fillmore CJ. Frame semantics. In: Linguistics in the morning calm, Hanshin Publishing Co, Seoul, 1982; 111-137.

Fogg BJ, Soohoo C, Danielson D, Marable L, Stanford J, Tauber ER. How do people evaluate a Web site's credibility? Results from a large study. A Consumer WebWatch research report, prepared by Stanford Persuasive Technology Lab, Stanford University, Stanford, Connecticut, 2002.

Foltz PW, Kintsch W, Landauer TK. The measurement of textual coherence with latent semantic analysis. Discourse Processes, 1998; 25: 285-307.

Gonzalo J, Verdejo F, Chugur I, Cigarran J. Indexing with WordNet synsets can improve text retrieval. Proceedings of the COLING/ACL Workshop on Usage of WordNet in Natural Language Processing Systems, Montreal, 1998.

Good BJ. Medicine, Rationality, and Experience: An Anthropological Perspective. Cambridge University Press, Cambridge, 1984.

Good BJ, DelVecchio, Good MJ. "Fiction" and "historicity" in doctor's stories: social and narrative dimensions of learning medicine. Mattingly, Garro (eds.), 2000; 50-69.

Grenon P, Smith B, Goldberg L. Towards a dynamic biomedical ontology. Pisanelli D (ed.), Ontologies in Medicine: Proceedings of the Workshop on Medical Ontologies, Rome, October 2003, IOS Press, Amsterdam, forthcoming.

22

Guha R, Lenat D, Pittman K, Pratt D, Shepherd M. Cyc: A midterm report. Communications of the ACP 1990; 33 (8).

Harabagiu SM, Moldovan DI. A marker propagation text understanding and inference system. Stewman JH (ed.), Proceedings of the 9th Florida Artificial Intelligence Research Symposium, Key West, 1996; 55-59.

Hayes PJ. The second naive physics manifesto. Hobbs JR, Moore RC (eds.), Formal Theories of the Common-sense World, Ablex, Norwood, 1985; 1-36.

Hersh W, Buckley C, Leone T, Hickam D. OHSUMED: An interactive retrieval evaluation and new large test collection for research. SIGIR '94, Proceedings of the 17th Annual International ACM IST Project 2001-33052 WonderWeb, 1994.

Horton, R. Tradition and modernity revisited. Hollis M, Lukes S (eds.), Rationality and Relativism, Blackwell, Oxford, 1982; 201-260.

Jackson B, Ceusters W. A novel approach to semantic indexing combining ontology-based semantic weights and in-document concept co-occurrences. Baud R, Ruch P (eds), EFMI Workshop on Natural Language Processing in Biomedical Applications, Cyprus, March 2002; 75-80.

Jacquemart P, Zweigenbaum P. Towards a medical question-answering system: a feasibility study. Le Beux P, Baud R (eds.), Proceedings of Medical Informatics Europe, IOS Press, Amsterdam, 2003; 463-468.

Keil FC, Levin DT, Richman BA, Gutheil G. Mechanism and explanation in the development of biological thought: the case of disease. Atran S (ed.), Folkbiology, MIT Press, Cambridge, 1999; 285-343.

Kingsbury P, Palmer M. From TreeBank to PropBank. Proceedings 3rd Int Conf Language Resources and Evaluation (LREC-2002), Las Palmas, Spain.

Kirmayer LJ. Broken narratives: clinical encounters and the poetics of illness experience. Mattingly, Garro (eds.), 2000; 153-180.

Kleinman A. The illness narratives. Basic Books, New York, 1988.

Kumar A, Smith B. The Unified Medical Language System and the Gene Ontology, KI 2003: Advances in Artificial Intelligence (Lecture Notes in Artificial Intelligence 2821), 2003; 135–148.

Lenat D. Cyc: a large-scale investment in knowledge infrastructure. Communications of the ACM 1995; 38 (11).

23

Lewalle P. Terminology: harmonizing health-related terminology – a task of many methods. ISO (International Organization for Standardization) Bulletin, October 2000; 14-16.

Magnini B, Strapparava C. Using WordNet to improve user modelling in a web document recommender system. Proceedings of the NAACL 2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, June 2001.

Mark DM, Smith B, Tversky B. Ontology and geographic objects: An empirical study of cognitive categorization, Spatial Information Theory. Cognitive and Computational Foundations (Lecture Notes in Computer Science 1661), 1999, 283–298.

Masolo C, Borgo S, Gangemi A, Guarino N, Oltramari A. Ontology Library, WonderWeb Deliverable D18 Ontology Infrastructure for the Semantic Web IST Project 2001-33052, 2003, http://wonderweb.semanticweb.org/deliverables/documents/D18.pdf.

Mattingly C, Garro LC (eds.) Narrative and the Cultural Construction of Illness and Healing. University of California Press, Berkeley, Los Angeles, London, 2000.

McCray AT, Srinivasan S, Browne AC. Lexical Methods for Managing Variation in Biomedical Terminologies. SCAMC ‘94, 1994; 235-239.

Miller GA. WordNet: a lexical database for English. Comm ACM 38, 11, November 1995; 39-41.

Patel VL, Arocha JF, Kushniruk A. Patients' and physicians' understanding of health and biomedical concepts: relationship to the design of EMR systems. Journal of Biomedical Informatics, 2002; 35(1): 8-16.

Pustejovsky J. The generative lexicon. MIT Press, Cambridge, 1995.

Rehder B, Schreiner ME, Wolfe MB, Laham D, Landauer TK, Kintsch W. Using latent semantic analysis to assess knowledge: some technical considerations. Discourse Processes 1998; 25: 337-354.

Rosch E. Cognitive representations of semantic categories. Journal of Experimental Psychology, General 1975; 104: 192-253.

Rosch E. On the internal structure of perceptual and semantic categories. Cognitive Development and the Acquisition of Language, Moore TE (ed.), Academic Press, New York, 1973.

Rosch E. Principles of categorization. Cognition and Categorization. Rosch E, Lloyd BB (eds.), Erlbaum, Hillsdale, NJ, 1978.

24

Saracevic T, Kantor P. A study of information seeking and retrieving. II. Users, questions and effectiveness. J Am Soc Inf Sci, 1988; 39(3): 177-196.

Schulze-Kremer S, Smith B, Kumar A. Revising the UMLS Semantic Network (manuscript) (See Appendix A3).

Slaughter L. Semantic relationships in health consumer questions and physicians’ answers: a basis for representing medical knowledge and for concept exploration interfaces. Doctoral dissertation, University of Maryland at College Park, 2002.

Smith B, Köhler J, Kumar A. On the application of formal principles to life science data: a case study in the Gene Ontology. Proceedings of DILS 2004 (Data Integration in the Life Sciences), (Lecture Notes in Computer Science), Berlin, Springer, 2004, in press.

Smith B, Mark DM. Ontology and geographic kinds, Proc 8th Int Symp Spatial Data Handling (SDH ’98), Vancouver, 1998; 308–320.

Smith B, Mark DM. Geographical Categories: An Ontological Investigation, Int J Geographical Information Science, 2001; 15(7): 591–612.

Smith B, Papakin I, Munn K. Bodily systems and the modular structure of the human body. Proceedings of the 9th Conference on Artificial Intelligence in Medicine (Lecture Notes on Artificial Intelligence 2780), Springer, Berlin, 2003; 86–90.

Smith B, Rosse C. The role of foundational relations in the alignment of biomedical ontologies. Proc Medinfo, San Francisco, 7-11 September, 2004.

Smith B, Williams J, Schulze-Kremer S. The Ontology of the Gene Ontology, in Biomedical and Health Informatics: From Foundations to Applications, Proc AMIA Symp, Washington DC, Nov. 2003; 609–613.

Smith CA, Stavri PZ, Chapman WW. In their own words? A terminological analysis of e-mail to a cancer information service. Proc AMIA Symp. 2002; : 697-701.

Stanford J, Tauber ER, Fogg BJ, Marable L. Experts vs. online consumers: a comparative credibility study of health and finance Web sites. Consumer WebWatch Research Report, Sliced Bread Design, Stanford, 2002.

Streiter O. Corpus-based parsing and treebank development, ICCPOL, 19th International Conference on Computer Processing of Oriental Languages, 2001.

Tse AY. Identifying and characterizing a "consumer medical vocabulary." Doctoral dissertation, College of Information Studies, University of Maryland, College Park, Maryland, March 2003.

25

Tse T, Soergel D. Procedures for mapping vocabularies from non-professional discourse: a case study: "consumer medical vocabulary". Proceedings of the Annual Meeting of the American Society for Information, 2003.

Turcato D, Fass D, Tisher G, Popowich F. Fully automatic bilingual lexical acquisition from EuroWordNet. Proceedings of NAACL 2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, June 2001.

Vasconcelos N, Lippman A. A bayesian framework for semantic content characterization. Computer Vision and Pattern Recognition, 1998; 566-571.

Vintar S, Buitelaar P, Volk M. Semantic relations in concept-based cross-language medical information retrieval. Proceedings of the ECML/PKDD Workshop on Adaptive Text Extraction and Mining (ATEM), Cavtat-Dubrovnik, Croatia, September 22, 2003.

Voorhees EM, Buckland LP (eds.) NIST Special Publication 500-251: The Eleventh Text REtrieval Conference (TREC 2002), Department of Commerce, National Institute of Standards and Technology, 2002.

Zeng Q, Kogan S, Ash N, Greenes RA, Boxwala AA. Characteristics of consumer terminology for health information retrieval: A formal study of use of a health information service. Methods of Information in Medicine, in press.

Zhang J. Representations of health concepts: a cognitive perspective. Journal of Biomedical Informatics 2002; 35 (1): 17- 24.

26