![Page 1: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/1.jpg)
Improving search efficiency for economic evaluations in
major databases using semantic technology
Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill Porter
[email protected] 2010
![Page 2: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/2.jpg)
Overview
Why are we interested in economic evaluations? Can economic evaluations be identified efficiently at
present? This research project
Methods Results Discussion
Next steps
![Page 3: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/3.jpg)
Why are we interested in economic evaluations?
Systematic reviews and technology assessments frequently consider cost-effectiveness as well as effectiveness outcomes
This information is published in economic evaluations Cost-effectiveness analyses Cost-utility analyses Cost-benefit analyses
Issues in identifying reports of economic evaluations Poor reporting
abstracts may contain terms which signal an economic evaluation but not an explicit term
Economics is often mentioned in passing in abstracts Increases number of irrelevant records retrieved
![Page 4: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/4.jpg)
Can economic evaluations be identified efficiently?
In healthcare databases Yes and No Specific economic evaluation databases are available (NHS
EED and HEED) BUT may need to carry out top up/supplementary searches
in large bibliographic databases
Beyond healthcare Seem to be no economic evaluation databases Need to search large bibliographic databases such as ERIC
and Criminal Justice Abstracts
![Page 5: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/5.jpg)
What about search filters?
![Page 6: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/6.jpg)
Can search filters help?
In healthcare databases Many search filters search filters to find economic evaluations in EMBASE and
MEDLINE achieve high sensitivity (100%) (1) BUT they have poor precision (less than 4%): very high proportion
of irrelevant studies are retrieved (1)
Beyond health Few filters available Issues of precision likely to be similar to health
(1)Glanville J, Kaunelis D, Mensinkai S. How well do search filters perform in identifying economic evaluations in MEDLINE and EMBASE. Int J Tech Assess Hlth Care 2009;25:522-529
![Page 7: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/7.jpg)
This research project
How can we improve efficiency of retrieval of economic evaluations in large bibliographic databases?
Traditional Boolean approaches don’t seem to be helping
Indexing isn’t very helpful at present Can semantic analysis software help? Collaboration with Expert System to explore potential
for identifying economic evaluations using their Cogito software
![Page 8: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/8.jpg)
Semantic Net
![Page 9: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/9.jpg)
Semantic analysisAnalysis hat assigns a meaning, a sense, to a syntactic structure and consequently to a linguistic unit, according to the knowledge contained in the semantic network.
![Page 10: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/10.jpg)
Methods
Gold standard set of 1950 economic evaluation records (published 2000, 2003, 2006) identified from NHS EED and then downloaded from MEDLINE.
Comparator set of 4136 matching MEDLINE records for the 3 years (2000, 2003, 2006) not economic evaluations But identified using the NHS EED filter
Loaded into Cogito Divided randomly into test sets and validation sets Used in-built semantic analysis and also created new rules to
categorise economic evaluations to categorise records as economic evaluations or non-economic evaluations
![Page 11: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/11.jpg)
Testing and validation
![Page 12: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/12.jpg)
ResultsTest set
(Gold Standard records=975)
(Comparator records = 2068)
Validation set(Gold Standard records=975)
(Comparator records = 2068)
Number of gold standard (GS) records retrieved 975 975
Number of comparator records retrieved 203 385
Sensitivity(number GS retrieved/number of
GS records) 100% 100%
Precision(number of GS retrieved/number
of records retrieved) 82.77% 71.69%
![Page 13: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/13.jpg)
Results, 2
Precision (combined Test and Validation sets)
Sensitivity (combined Test and Validation sets)
Using Cogito in-built semantic rules (no filter) 77.23% 100%
Using filter with records scoring 50
78% 90%
Using filter with records scoring 100 80% 85%
Using filter with records scoring 200 81% 83%
![Page 14: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/14.jpg)
Discussion
Cogito performs as well as Boolean searching in terms of sensitivity
Cogito has a much improved precision score compared to performance of Boolean filters Over 70% (Cogito) compared to under 10% (Glanville et al)
Cogito performs well ‘out of the box’ Although early training efforts did not improve precision, further
exploration might yield improved results
![Page 15: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/15.jpg)
Next steps
Identifying funding to carry out further exploration Exploring economic evaluation identification optimisation further Exploring the effects of importing results from a range of databases into
Cogito Exploring whether semantic analysis has potential to achieve
improvements in retrieval of other hard to find research where filters do not perform well diagnostic test accuracy studies and quality of life research Exploring the potential of semantic analysis for analysing records
by study design obtained from a range of databases in healthcare, social care, education and criminal justice contexts
in-built rules are database independent.
![Page 16: Improving search efficiency for economic evaluations in major databases using semantic technology Julie Glanville, Carol Lefebvre, Pamela Negosanti, Bill](https://reader030.vdocuments.net/reader030/viewer/2022032612/56649ec65503460f94bd1f30/html5/thumbnails/16.jpg)
For further information
Julie Glanville, York Health Economics Consortium
Bill Porter at Expert System
http://www.expertsystem.net/