Electronic literature searching for

qualitative research

Rachel L Shaw

with Andrew Booth, Alex J Sutton, Mary Dixon-Woods, David R Jones, Tina Miller,

Jonathan A Smith & Bridget Young

http://www.prw.le.ac.uk/research/qualquan/index.htm

The Project

• How can systematic reviews incorporate qualitative research?

1) Searching the literature

2) Appraising the evidence

3) Synthesis of qualitative evidence

4) Synthesis of qualitative & quantitative evidence

– (Dec 02-Oct 04)

Aims of Phase 1

• To systematically search the literature for qualitative evidence in order to update an existing Cochrane Review of quantitative evidence on support for breast-feeding women

• To evaluate the recall & precision of 3 electronic search strategies using 6 bibliographic databases for identifying qualitative research about breast-feeding support

Search strategies

1) Subject headings (thesaurus): using 20 thesaurus terms specific to each database

2) Free text terms: using over 40 qualitative methodology terms & authors

3) Broad-based: using 3 generic terms, ‘qualitative’, ‘interviews’, ‘findings’

Bibliographic databases

• Medicine:– MEDLINE– EMBASE

• Nursing:– CINAHL– British Nursing Index

• Social Sciences:– ASSIA– Social Sciences Citation Index

Definitions

• Recall: – Potentially relevant studies (“tested positive”)– How well a search strategy identifies what is

relevant to the research question

• Precision: – Actually relevant studies (“diagnosed positive”)– How well a search strategy screens out what is

not relevant

Discard duplicates

Discard non-human

Total initial yieldN=7420

“Raw” yieldN=17045

Total number of records retrieved in each database across all strategies

4142

12422168

47 19 315

0

1000

2000

3000

4000

5000

medline embase cinahl bni assia ssci

Total number of records retrieved by each strategy across all databases

3537 3451

3912

32003400360038004000

subjectheadings

free text broad-based

Abstract screening

• Records were screened against 2 criteria:– Topic relevance: breast-feeding support– Methodology relevance: qualitative methods

(studies using mixed methods – qualitative & quantitative – included at this stage)

Discard duplicates

Discard non-human

Total initial yieldN=7420

“Raw” yieldN=17045

Abstract screening

Screened positive sampleN=587

Getting strict on method

• Many mixed methods studies identified during abstract screening (predominantly quantitative with small qualitative component)

• Decision taken to limit inclusion to ‘qualitative studies’ only (& if mixed methods used then qualified only if predominantly qualitative)

• This demonstrates the iterative process required in such a new field of literature searching

Defining qualitative research

• Qualitative research is an umbrella term for diverse research methods in many different disciplines

• An operational definition?

• We defined qualitative research by what it is NOT: e.g. research that is dependent on numeric analysis was screened not relevant

Discard duplicates

Discard non-human

Total initial yieldN=7420

“Raw” yieldN=17045

Abstract screening

Screened positive sampleN=587

Strict on method

Candidate sampleN=262

Calculating recall & precision

• Modelled from the screening test formula

• Recall ~ sensitivity– Potentially relevant (“tested positive”) records

from each strategy as a percentage of initial total yield

• Precision ~ positive predictive value– Actually relevant (“diagnosed positive”)

records as a percentage of potentially relevant records (“tested positive”)

Potentially relevant (“tested positive”) records identified by each strategy

Recall: potentially relevant records as % of total initial yield

1) Subject headings

3537 47.6% (3537/7420)

2) Free-text 3451 46.5% (3451/7420)

3) Broad-based

3912 52.7% (3912/7420)

Total initial yield

7420 -

Actually relevant records retrieved (“diagnosed positive”) identified by each strategy

Precision: actually relevant records as % of potentially relevant records

1) Subject-headings

191 5.4% (191/3537)

2) Free-text 172 4.9% (172/3451)

3) Broad-based 187 4.7% (187/3912)

Total actually relevant records

262 -

Raw yield N=17045

Total initial yield N=7420

Broad-based strategy has highest recall

Candidate sample N=262

Subject-headings has highest precision (all strategies have generally poor precision)

Many non-human & non-qualitative records identified

Trading relevance against comprehensiveness?

• Price for high recall is poor precision

• Trade-off is traditionally expected in information retrieval circles

• Strategies designed to maximise recall

• 96% of total initial yield were irrelevant

• Screening was time consuming & costly

Conclusions• Subject headings have recognised high precision –

BUT in this case - only slightly higher than broad-based strategy

• No one strategy was sufficiently comprehensive to identify all relevant records

• A combination of search terms (subject headings & free-text) is necessary to maximise recall

• BUT high recall leads to laborious screening for very small return

• Action needed to increase precision

Recommendations for high recall, high precision strategy

• Better indexing of qualitative research on bibliographic databases

• Authors use methodology descriptors to describe their work & use structured abstracts where possible

• Systematic analysis of individual search terms is required

• Further research in different substantive areas is required

Future questions & outputs• Methods question:

– Which search terms are most successful at identifying qualitative research?

– Papers in progress for JMLA & J Adv Nurs

• Substantive question:– What does the qualitative evidence about

support for breast-feeding women add to the quantitative evidence in the existing Cochrane Review?

– A new review of qualitative & quantitative evidence will be our substantive output.

Further dissemination• 47th Scientific Annual meeting of the Society for Social

Medicine, Edinburgh, September 2003• Submitted Finding qualitative research to the BMC

Research Methodology open access journal• Provisionally accepted The problem of quality in

qualitative research by Quality & Safety in Health Care• In progress:

– Using citation searching to identify qualitative research for the International Journal of Social Research Methodology

– Defining qualitative research for Qualitative Research