designing a search strategy to identify and retrieve articles onevidence-based health care using...

Designing a search strategy to identify and retrieve articleson evidence-based health care using MEDLINE

JANE HARRISON School of Health and Related Research (ScHARR), University of Sheffield,Regent Court, 30 Regent Street, Sheffield S1 4DA, UK

The practice of evidence-based health care requires that information on methodology be identifiedfrom databases such as MEDLINE. Up until this year there have been no designated medical subjectheadings (MeSH) for evidence-based health care. ‘EVIDENCE-BASED MEDICINE’ appearsas a MeSH term from 1997. The absence of designated MeSH for this concept prior to 1997 providesa challenge to the searcher. This paper describes the creation of a MEDLINE search strategy to retrievearticles on the methods of evidence-based health care published prior to the introduction of the newterm, where an optimal combination of free-text and MeSH terms is required to identify relevantmaterial. The study examines both free-text and subject heading searching and attempts an optimalbalance of sensitivity and specificity. It begins by examining separate free-text and subject headingsearches. Sensitivity of the subject heading search was 33% and specificity 80%, while the free-textsearch produced a sensitivity of 50% and a specificity of 67%. The final strategy, combining bothapproaches, was more successful with sensitivity reaching between 82 and 90% and specificity 83%. It is therefore possible to devise a search strategy to retrieve articles on the methods of evidence-basedhealth care with relatively successful rates of sensitivity and specificity. The limitations of MEDLINE,

however, necessitate the use of additional approaches in identifying articles on the methods ofevidence-based health care.

The background of evidence-based health care

Evidence-based medicine has been defined as ‘an approach to health care that promotesthe collection, interpretation and integration of valid, important and applicable patient-reported, clinician-observed and research-derived evidence.’1 It embodies the wholeprocess from finding systematic evaluations of health care interventions, through ap-praising them for validity, reliability and applicability, to disseminating and implement-ing the findings of research. Its origins lie in the discipline of clinical epidemiology withits application of research methodologies to clinical decision-making about the care ofindividual patients. As this suggests, an inclusive term ‘evidence-based health care’ ismore appropriate terminology than the narrower term ‘evidence-based medicine’, andis the term which will be used to refer to the discipline throughout this paper. Themove towards evidence-based health care creates the need not only to identify andappraise research such as randomized controlled trials and systematic reviews but totrack down literature on its methods. This will encompass articles introducing clinicaleffectiveness and evidence-based health care, comments and critiques, searching forliterature, managing the review process, critical appraisal, economic evaluation, meta-

Health Libraries Review 1997, 14, 33–42

© 1997 Blackwell Science Ltd 33

analysis, publication bias, guidelines, presentation of results and dissemination of researchfindings. The objective of this study was to identify such articles (using MEDLINE) by developinga search strategy to retrieve such material. A strategy is required to maintain a resourceguide on evidence-based practice2 and to support a complementary current awarenessservice. Given the relative infancy of evidence-based health care, and its multifariousaspects, the searcher is faced with a particular difficulty. Until this year there have beenno designated medical subject headings (MeSH) for this area, and thus in developing thesearch strategy, the searcher is faced with harnessing MeSH terms that may have beenused inconsistently by indexers. The introduction of the new MeSH term ‘EVIDENCE-BASED MEDICINE’ will necessitate its inclusion in subsequent search strategies in thisarea. Because the searches for this study were carried out over a period of MEDLINE whichcovers 1993–96, the term has not been incorporated into this strategy. In contrast withpublished strategies to identify studies for systematic review, such as that by Dickersinand co-workers3 where it is important to recover all relevant articles (high sensitivity),the aim of this study was to produce a strategy which arrives at an optimal balance ofsensitivity and specificity.

Phase I of the study—developing retrieval strategies

A successful search strategy will usually include core keywords, other partly relevantkeywords and uncontrolled vocabulary4 (or natural language terms, i.e. free-text termsused by the author in the title or abstract, which do not conform to an indexing standard).For this particular study it was decided to examine the relative advantages and disadvan-tages of subject heading and free-text searching. For the first phase of the study, twoseparate search strategies were created for retrieving items on the methods of evidence-based health care.

Subject heading search

MEDLINE provides several tools to aid retrieval by subject heading. MeSH terms are usedto index each MEDLINE reference and have a ‘major descriptor’ designation to allowretrieval to be limited to those articles where MeSH terms represent the main focus. Athesaurus function is used to identify relevant MeSH terms to be included in the search,while a ‘suggest’ function, where available, provides the most frequently used MeSHterms for a particular word or phrase. Alternatively, some versions of MEDLINE use anautomatic concept-mapping facility. Such tools can provide a sound means for retrievingreferences by standardized subject headings. When attempting to retrieve items on the methods of evidence-based health care usingsubject headings one is faced with the aforementioned difficulty of there being no obviousdesignated MeSH terms in this subject area. To overcome this problem, two strategies

© 1997 Blackwell Science Ltd, Health Libraries Review 14, 33–42

34 J. Harrison

were adopted. First, the ‘suggest’ function was used in conjunction with the term‘evidence-based medicine’ and other commonly used terms to produce ‘surrogate’ subjectheadings. Second, relevant known articles were reviewed to identify frequently usedindex terms for such items. Combinations of such terms were then identified in order toheighten precision. Most single subject heading terms proved inadequate and retrieveda high proportion of irrelevant references. The review of known articles was importantin identifying such combinations of terms. At this stage the subject heading searchcomprised the following combinations:

• CLINICAL MEDICINE and DECISION MAKING• CLINICAL MEDICINE and DECISION SUPPORT TECHNIQUES• (META-ANALYSIS or explode CLINICAL TRIAL) and MEDLINE

• (META-ANALYSIS or explode CLINICAL TRIAL) and REVIEWLITERATURE

• (META-ANALYSIS or explode CLINICAL TRIAL) and PUBLICATIONBIAS

• META-ANALYSIS and REPRODUCIBILITY OF RESULTS• explode GUIDELINES and REPRODUCIBILITY OF RESULTS• REVIEW-LITERATURE and explode RESEARCH(all)

The single term COST– BENEFIT ANALYSIS was also included to pick up items onthe methods of economic evaluation. This was refined by limiting retrieval to the‘methods’ subheading, i.e.

• COST– BENEFIT ANALYSIS/methods

The free-text search

From the wide range of uncontrolled vocabulary used in the area of evidence–basedhealth care, four terms were identified as commonly occurring:

• evidence based (to retrieve items with evidence-based health care, practice, policymaking, etc.)

• economic appraisal• systematic review• critical appraisal

The truncation function (*) was used for the last two terms to pick up variants invocabulary around those terms:

• systematic review*• critical* apprais*


Designing a search strategy using MEDLINE 35

Some relevant author names were also used and limited to the Author field:

• Sackett-D* in AU• McKibbon-K* in AU

Free-text and subject heading searches—sensitivity and specificity

The searcher is concerned with two measures of the search strategy. First, sensitivity (orits parallel term recall) may be defined as the number of relevant documents retrievedfrom the total number of relevant citations in a database, or (in the absence of a totalreview of the database) from a ‘population’ of identified references. Second, specificity(or its parallel term precision) is the number of references judged to be relevant as aproportion of the total number of references retrieved. Research has documented that free-text searches have a lower rate of sensitivity thansubject heading searches. Lowe and Barnett5 recorded how the STAIRS automatic retrievalsystem was applied to a collection of 40 000 free-text documents, yielding a sensitivity of20% on average, which led to a recommendation that the use of an indexing scheme suchas MeSH would achieve more acceptable levels of sensitivity. The same study, however,indicated that the specificity of free-text searching was more successful than its sensitivity,and yielded a result of 75%. From these findings, one would expect a subject headingsearch to achieve higher sensitivity than a free-text search, and that within free-textsearching, specificity would be higher than sensitivity.

Phase II of the study— refining the search strategies

The aim of phase II was to refine the search strategies and produce a combined free-textand subject heading search, achieving an optimal balance of sensitivity and specificity (i.e.achieving the highest possible sensitivity without jeopardizing specificity, and vice versa).The terms used in the existing separate strategies were critically reviewed. The subjectheading terms had already been systematically identified, as described in Phase I. Thefree-text searches, however, were subject to modification. There was some difficulty with the first term used, ‘evidence based’. It was foundthat although this term yielded relevant references, irrelevant references which used thephrase ‘evidence based on…’ were also retrieved. This could not be ameliorated merelyby using a term such as ‘evidence-based health care’, because items of relevance on‘evidence-based policy making’, ‘evidence-based practice’ and terms from evidence-based specialities such as ‘evidence-based dentistry’ would not be retrieved. A strategyusing the term ‘evidence based’ in the title field was implemented and tested. However,although this eliminated the irrelevant references which contained ‘evidence basedon…’ in the abstract, this also lost 36 relevant references which referred to ‘evidence-based medicine’ in the abstract but did not mention it in the title. A check was madeto see if any of these references would be picked up by the rest of the strategy, butonly three were picked up elsewhere. This potential loss of 33 relevant references would


36 J. Harrison

adversely affect the recall of the search. It was therefore decided to keep the originalterm ‘evidence based’, at the expense of retrieving some irrelevant references. There were similar problems with the next two terms, ‘systematic review’ and ‘criticalappraisal’. Both of these phrases are common outside the sphere of evidence-based healthcare. Their use therefore yielded a high proportion of irrelevant references. A means ofadding to the precision of these terms was required. This was achieved by identifyingthose words which commonly accompanied them in the title, abstract or MeSH headingsof relevant references, terms such as literature, MEDLINE, COHORT-STUDIES,article*, journal*, report*, literature search*. These terms were thus combined with‘systematic review*’ and ‘critical* apprais*’. The term ‘economic evaluation’ was addedand combined with ‘method*’ (truncated) to broaden the recall of items in the area ofeconomic evaluation. ‘Economic evaluation’ alone would have picked up examples ofeconomic evaluations of interventions rather than those where methodology was themain feature. Search terms for the area of economic evaluation had previously beenlimited to ‘economic appraisal’ in the free-text search and ‘COST– BENEFITANALYSIS/methods’ in the subject heading search. The author component of the search was also modified. In using the author variant‘Sackett-D*’ the strategy picked up a few irrelevant articles by another D. Sackettpractising in an unrelated field. However, as a key figure in the sphere of evidence-basedhealth care, it was decided that this author’s name should be kept at the expense ofpicking up a few irrelevant references. The author variant ‘McKibbon-K*’, which hadinitially picked up relevant references in the area of literature searching for qualityevidence, was replaced with phrases in the Address field that aimed to pick up theseplus additional relevant references. These address terms related to the two departmentsat McMaster University, Canada, known for their pioneering work in evidence-basedhealth care; the Department of Clinical Epidemiology and Biostatistics, and the HealthInformation Research Unit. The word ‘Cochrane’ was also added to the strategy in the Title and Abstract fields,to retrieve articles on the ‘Cochrane Collaboration’, the organization whose aim is toprepare and maintain systematic reviews of the effects of health care. The section of thestrategy devised by Dickersin and co-workers3 which excludes animal studies, but whichincludes human and animal studies, was added to heighten specificity. Finally, each element of the subject heading search was tested against the results of thefree-text strategy in order to see whether any of these terms could be dropped. It wasfound that although there was some overlap, this was not sufficient to require the removalof terms. This final search strategy can be found in the Appendix.

Phase III—creating a ‘gold standard’

Two sets of references were created against which to evaluate the sensitivity of the refinedsearch strategy.



Set one—an internal ‘gold standard’

Set one comprised relevant references from the free-text and subject heading searchesplus relevant references from the refined search strategy plus any other known relevantcitations, not picked up by these searches, from journals indexed on MEDLINE. Theseknown references were taken from an existing bibliography2 of articles on evidence-basedhealth care.

Set two—an external ‘gold standard’

Set two comprised the same components as set one, but with additional known relevantcitations not indexed on MEDLINE. This gold standard aimed to indicate the limitationof using only MEDLINE, as opposed to a range of databases, to search for items on themethods of evidence-based health care. These other known references were also takenfrom the bibliography cited above. Relevance judgements were made, independently of the investigator, by the compilerof the bibliography. In this way the refinement of the search strategies was not merelyan artefact of the investigation, but a means of objectively identifying potential candidatesfor inclusion in future editions of the bibliography.

Results (see Table 1)

Phase I—specificity and sensitivity of free-text and subject heading searches

The subject heading search resulted in the retrieval of 256 references, of which 204 werejudged to be relevant. This yielded a specificity of 80%. Sensitivity was calculated as apercentage of all relevant references (610 references) in order to determine the effective-ness of this strategy alone to recall relevant references. The subject heading searchproduced a sensitivity of 33% (204/610). The free-text search strategy resulted in the

Table 1. Summary of references retrieved

Total number of Number of Number of relevantSearch strategy relevant references references retrieved references retrieved Specificity (%) Sensitivity (%)

Phase I Subject heading

610*256 204 80 33

Free text 457 307 67 50

Phase II Final strategy 610*

660 548 8390*

665† 82†

*Gold standard one, †Gold standard two (includes 55 non-MEDLINE references from bibliography).2

Searches were run during March 1996 and covered the period of MEDLINE 1993–96.


38 J. Harrison

retrieval of 457 references, 307 of which were relevant, a specificity of 67%. Sensitivity,at 50% (307/610) was higher than the sensitivity of the subject heading search. It was pointed out earlier that studies have shown that free-text searches have a lowersensitivity than subject heading searches, but that the specificity of free-text searchingis better than its sensitivity. Is this borne out in the results above? The free-text searchhad a higher sensitivity rate than the subject heading search—50% versus 33%, incontrast to what is suggested by the research. However, the specificity of the free-textwas better than its sensitivity (67% and 50%, respectively) which does support the resultsof existing studies. Nevertheless, one should be aware that the results of searches canbe dependent on a number of different variables. Searcher expertise, indexer consistencyand the nature of the subject of the information query all impinge upon the results.The absence of designated MeSH terms for the topic ‘evidence-based health care’ couldexplain the limited sensitivity of the subject heading search. The free-text strategy itselfholds the key to why sensitivity is better with the free-text search. The term ‘evidencebased’, which though picking up some irrelevant articles, also picks up many articleson evidence-based health care which are otherwise elusive. Also, the use of truncationof some free-text terms aided sensitivity. Similarly, the use of specific author names hasimproved both the sensitivity and specificity of the free-text search.

Phase II—the final search strategy

The final search strategy resulted in the retrieval of 660 references, of which 548 werejudged to be relevant, a specificity of 83%. This was superior to both the single free-textand the subject heading searches. Sensitivity was significantly higher at 90%, at leastwhen compared to set one, where retrieval was measured against a population ofrelevant references from cumulated initial and final search strategies, plus those articleson MEDLINE from a known population of relevant references. The result for sensitivityagainst set two (gold standard one plus those articles from a population of relevantreferences not indexed on MEDLINE) was lower that that for set one, at 82%. Thus it was possible to combine and refine the two initial strategies and yield higherrates of sensitivity and specificity without compromising either. Twenty-three relevantreferences were lost from the pre-refined strategies of Phase 1 of the study, but 105new references gained. Many of these were in the area of economic evaluation,under-represented in the first searches, and also articles of high relevance on theCochrane Collaboration.

Discussion

The final strategy produced relatively successful results given the nature of the subjectarea, with sensitivity at 90% and specificity at 83%. This strategy can now be used toperiodically retrieve articles from MEDLINE as it is updated, to produce a current awarenessbulletin of articles on the methods of evidence-based health care.



The results were not, however, achieved without difficulty. A complex searchstrategy was devised to retrieve relevant citations based on common index terms andby reviewing the index terms of known relevant references. The use of MeSH toimprove sensitivity and specificity relies heavily on high quality consistent indexing.4

In this study, the investigator had to navigate the MeSH terminology and to harnesscommonly used MeSH terms in devising the strategy. The study by Dickersin andco-workers3 notes that ‘until relatively recently there has been an emphasis on devel-oping MeSH terms for subject matter rather than methodology’. It looks at the problemof identifying randomized controlled trials for systematic review, with the relativelyrecent addition of MeSH terms such as ‘RANDOMIZED CONTROLLED TRIAL’as a publication type in 1991 and CONTROLLED CLINICAL TRIAL as a publicationtype in 1995. The lack of designated MeSH for randomized controlled trials was a majorproblem before the introduction of these terms. Other problems arise in the area of indexing. One is dependent on the consistency ofindexers,6 who face difficulties similar to those encountered when devising the strategy—that is, a lack of designated terms. Thus there is a greater likelihood of inconsistency inindexing such items if one has to identify ‘surrogate’ terms. Furthermore, it is important to note that MEDLINE does not automatically includebibliographic details of all articles in journals. Thirteen articles identified from thebibliography were not indexed on MEDLINE, notwithstanding the fact that the journalsin which they appeared were indexed on MEDLINE; these being mainly correspondenceor comments. Thus a further limitation of MEDLINE is that only selected or major articlesare included, as opposed to databases such as the Science Citation Index where morecomprehensive coverage is achieved. The search strategies were designed to retrieve articles from the key areas of themethods of evidence-based health care. Articles on specific meta-analyses and economicevaluations, for example, were only judged relevant where methodology was a keyconcern of the article. It is recommended, however, that a further, more comprehensive,strategy is designed for the area of economic evaluation; a large enough subject area towarrant a separate strategy. A full economic evaluation strategy would have been difficultto incorporate into this all encompassing search. It is also recommended that similar search strategies are carried out on other databasessuch as EMBASE or on unindexed current awareness sources such as the British LibraryInside Information database to complement this strategy. Within a British context itwould also be useful to add the term ‘clinical effectiveness’ to pick up NHS service-ledinitiatives in addition to the more academically derived articles on ‘evidence-based healthcare’. It has also been found worthwhile to conduct a citation search on Science CitationIndex using seminal works by Cochrane-A7 and Sackett-D,8 which provide access toimportant articles in this area that might otherwise be missed. This is, to a great extent,a characteristic of new fields of research, where single works exert a disproportionateinfluence on the development of a discipline. Finally, it should be pointed out that this is a single MEDLINE variant, rather than adefinitive strategy, with which to retrieve articles on the methods of evidence-based


40 J. Harrison

health care. This strategy will continue to be refined as new phrases and concepts areadded to the literature.

Acknowledgement

Thanks are due to Andrew Booth (Head of Information Resources at ScHARR) foradvising on study design, judging the relevance of retrieved items and reviewing themanuscript for this study.

References

1 McKibbon, K.A., Wilczynski, N., Hayward, R.S., Walker-Dilks, C.J. & Haynes, R.B. The Medical Literature as a Resourcefor Evidence Based Care at http://hiru.mcmaster.c./hiru/medline/mdl-ebc.htm.

2 Booth, A. The SCHARR Guide to Evidence-based Practice. SCHARR Occasional Paper no. 2. Sheffield: Sheffield Centrefor Health and Related Research, University of Sheffield, 1995.

3 Dickersin, K., Scherer, R. & Lefebvre, C. Identifying relevant studies for systematic reviews. British Medical Journal1994, 309, 1286–91.

4 Booth, A. In search of the evidence: informing effective practice. Journal of Clinical Effectiveness 1996, 1(1), 25–9. 5 Lowe, H.J. & Barnett, G.O. Understanding and using the Medical Subject Headings (MeSH) vocabulary to perform

literature searches. Journal of the American Medical Association 1994, 271, 1103–8. 6 Booth, A. How consistent is MEDLINE indexing? A few reservations. Health Libraries Review 1990, 7(1), 22–6. 7 Cochrane, A.L. Effectiveness and efficiency. Random reflections on health services. London: Nuffield Provincial Hospitals Trust,

1972. 8 Sackett, D.L., Haynes, R.B., Guyatt, G.M. & Tugwell, P. Clinical Epidemiology: A Basic Science for Clinical Medicine. 2nd

edn. Little, Brown & Company, 1991.

Appendix I

Search strategy to retrieve articles on the methods of evidence-based health care(Silverplatter MEDLINE)

1 CLINICAL MEDICINE and DECISION MAKING 2 CLINICAL MEDICINE and DECISION SUPPORT TECHNIQUES 3 (META-ANALYSIS or explode CLINICAL TRIAL) and MEDLINE

4 (META-ANALYSIS or explode CLINICAL TRIAL) and REVIEWLITERATURE

5 (META-ANALYSIS or explode CLINICAL TRIAL) and PUBLICATION BIAS 6 META-ANALYSIS and REPRODUCIBILITY OF RESULTS 7 explode GUIDELINES and REPRODUCIBILITY OF RESULTS 8 REVIEW LITERATURE and explode RESEARCH(all) 9 COST-BENEFIT ANALYSIS/methods10 evidence based11 systematic review* or critical* apprais*12 literature or MEDLINE or COHORT-STUDIES or article* or journal* or report* or literature search*13 economic appraisal14 economic evaluation and method*15 Sackett-D* in AU



16 Cochrane in TI17 Cochrane in AB18 Clinical Epidemiology and Biostatistics in AD19 Health Information Research Unit in AD20 McMaster in AD21 Set numbers 11 and 1222 Set numbers (18 or 19) and 2023 Set numbers 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 13 or 14 or 15 or 16 or 17 or 21 or 2224 TG=ANIMAL not (TG=HUMAN and TG=ANIMAL)25 Set 23 not set 24


42 J. Harrison

designing a search strategy to identify and retrieve articles onevidence-based health care using...

Documents