Review Professor Jan Illing

Centre for Medical Education ResearchSchool of Medicine, Pharmacy and Health

Durham University

∂

Overview of the lectureThe role of the literature review in research Types of evidence: Logic of aggregation vs. logic of configurationDifferent types of literature review• Narrative• Systematic• Meta Analysis• Realist synthesis• Synthesising qualitative research

Overview of the stages of a systematic review: research question, inclusion and exclusion criteria, searchers of title,abstract and papers, data extraction critical appraisal, assessing quality.

Reviews: Cochrane collaboration, Campbell collaboration, BEME

∂

The role of the literature review • Need to understand the topic and what is already known • The review places research in the context of a body of literature• Informs what still needs to be researched, before it is known• Learn about the methods used previously – what methods are

appropriate, or not, and where new approaches are needed e.g. need for a qualitative approach• Avoids repeating what is known, or repeating studies that have

already failed.• Provides information on the strength of available evidence, quality of the studies • and identifies gaps

∂

How does a literature review help you?

• Helps to understand how one study is related to others

• Highlights conflict in findings

• Can suggests new interpretations and new insights

• Can identify what research is needed next

∂

Different types of literature review answer different types of question

∂

Types of evidenceAggregate (counting evidence) vs. Interpretive (seeks understanding)Each study adds ‘weight’ Broadens or deepens understanding

Studies add more weight Add insight to a pattern or emerging viewwith sample size, Add theorystudy design and strength of findings

Reporting statistics Reporting themes

∂

Types of review: NarrativeNarrative – also called non-systematic, traditional and conventional Provides a snap shot of prevalence knowledge

Criticisms include:• The selection and inclusion of papers are not clear, Incomplete and sometime missing• Risk of bias from selection of included studies• Not easily to reproduce• Conclusions are questionable from the quality of methodology used and

quality of reporting

∂

Systematic literature review Unlike the narrative review, the systematic approach aims to search all (not

just part) of the relevant literature in a systematic way, aiming to use –

“a reproducible method for identifying, evaluating, and synthesising the existing body of completed and recorded work produced by researchers, scholars, and practitioners” (Fink 2005)

Systematic reviews aim to inform decisions and need to be made on the best available evidence (Booth et al, 2012)

Seen as ‘a scientific activity’ (Mulrow, 1995)

Often continuously updated (Cochrane reviews)

∂

Systematic literature reviewWeaknessesThe criteria used to identify the studies can also be a source of bias or a limitation (Song et al., 2010)

• Limited by year published e.g. last 10 years may miss important research published before this time

• Limited by language e.g. publication in English may focus on positive findings, publications in authors own language may report more negative findings

• Limiting searches to well-known databases can result in missing information. E.g. An audit of sources for a review of complex interventions (including

qualitative evidence) found that only 30% were identified from databases and hand searches. About half of studies were identified by ‘snowballing’ and another 24% by personal knowledge or personal contact (Greenhalgh 2005).

∂

What a review adds• Can help answer important questions that inform the value of an intervention,

policy, practice, and identify what new research to do• Can identify if finding tend to be consistent across a theme

Areas of conflict

“Disagreements among findings are valuable… conflict can teach us a lot”

• Conflicts often identify important differences in sample and context• May indicate a need for more research • Could be used to illustrate a theory• Highlight poor research methods.

∂

Hierarchy of research evidence

∂

Stages of a systematic review Devise a research question Develop a protocol – objectives, search strategy, inclusion and

exclusion criteria Search literature – define search terms and databases and

relevant grey literature, and searches from references of key papers

Search for relevant studies abstract, then papers Data extraction and critical appraise studies Synthesis of findings Interpretation Write up

∂

A Scoping ReviewProvides a snap shot on a topic

• The focus is on the range of studies that have been done

• They may focus on an issue e.g. what theoretical approaches have been used to study x

• There is no attempt to examine quality

(Arksey and O’Malley, 2005)

∂

Meta Analysis Helps to answer questions about interventions when sample sizes need to be increased

• Systematic search with inclusion/exclusion criteria• Focus is to find studies on the same topic using the same study design e.g. RCT• Combines results of quantitative studies • Conducts statistical tests by combining data• The analysis is numerical • focus on the size of the effect

∂

Realist synthesisRealist synthesis is used mainly for the assessment of complex interventions

• Starts with a systematic search of the literature• Uses an interpretive approach, looking for patterns• Aims to explain how outcomes occur• Theory-driven approach• Using evidence from qualitative and quantitative research methods• The goal is to identify mid-range theories

∂

Context Mechanism Outcome The realist research question is often summarized as: "What works for whom under what circumstances, how and why?"

Realist inquiry is based on a realist philosophy of science (reality can be known indirectly) and considers the interaction between context, mechanism and outcome

Each study is read and the reviewers seek to identify: the contextual (C) influences that are hypothesized to have triggered the relevant mechanism(s) (M) and generated the outcome(s) (O). Known as CMOs

“Education is a complex intervention which produces different outcomes in different circumstances” (Wong et al 2012 Medical Education p89-96)

∂

Development of theoryRealist synthesis looks for mechanisms and middle-range theoriesThe aim is to test and build theory

• Mechanisms can be defined as: “…underlying entities, processes, or structures which operate in particular

contexts to generate outcomes of interest” (Astbury and Leeuw, 2010)

• Mechanisms tend to generate outcomes, and are usually hidden and are sensitive to changes in context.

• Middle-range theories are developed from real data, so can be tested.

• Programme theory explains how the intervention is expected to reach its objectives.

∂

Realist synthesis and secondary searchesRealist reviews start with a systematic review of the literature

Assess the included papers and identify the CMOs

Often involve a second search to find more studies to help explainand test out possible theories

Appraising the evidence1. Relevance - can the study contribute to theory building or theory

testing2. Rigour - whether the method used to generate data is credible and

trustworthy

∂

Example: Realist synthesis of the literature on workplace bullying in the NHSA systematic literature review focuses on research designand outcomes, but a realist review looks for patterns in context.

A realist review aims to explain why an intervention may work in one situation but not in another

Findings from a realist synthesis focused on producing a theory to explain patterns in the data.

The findings from the study on workplace bullying highlighted that interventions were more likely to be successful if thesenior managers were fully committed to them. Illing, J, et al (2013). Evidence Synthesis on the occurrence, causes, consequences, prevention

and management of bullying and harassing behaviours to inform decision-making in the NHS. HMSO.

∂

Qualitative reviews The focus is on understanding rather than effectiveness The focus on effectiveness has privileged quantitative evidence over qualitative, and reduced consideration of other options to understand more about the impact of interventions.

Complexity and context also need to be considered by practitioners and policy-makers

∂

Searching for qualitative research “It is increasingly being recognized that evidence from qualitative studies

that explore the experience of those involved in providing and receiving interventions, and studies evaluating factors that shape the implementation of interventions, have an important role in ensuring that systematic reviews are of maximum value to policy, practice and consumer decision-making” (Mays 2005, Arai 2005, Popay 2005).

Databases have improved filters to include qualitative research for MEDLINE (Wong 2004), CINAHL (Wilczynski 2007), PsycINFO (McKibbon 2006) and EMBASE (Walters 2006).

However, identifying qualitative research is problematic due to the varied use of the term ‘qualitative’ (Grant 2004).

Others argue for a purposive sampling approach, where searching is driven by the need to reach theoretical saturation and the identification of the ‘disconfirming case’, may be more appropriate (Dixon-Woods 2006).

∂

Critical appraisal of studies

Weaknesses and flaws• Need to consider which are fatal and which are non-critical• There is no such thing as the perfect study and avoid nihilism (Sackett, 1997)• Need to consider how much a weaknesses impacts on the findings

Right research design and method for the research questionCan the results be relied upon (valid, true)Are the results trustworthy, not just due to chance (reliable)Rigour – the results are not biased, confounding factors e.g. age and genderLimitations from selection bias, measurement of outcome, and analysis

∂

Assessment of quality • Quality and relevance check lists vary by discipline• For novice researchers a check list can be helpful,• Provides a list of everything that should be considered.• Aim to use check lists that are widely accepted or validated• Checklists that use a scoring system are not recommended (Higgins and Green, 2008)

∂

The problem with scales The use of scales for assessing quality is discouraged in Cochrane reviews. Offers appealing simplicity, not supported by evidence (Emerson 1990, Schulz 1995b).

A score involves assigning ‘weights’to items in the scale, difficult to justify the weights assigned.

Scales have been shown to be unreliable assessments of validity (Jüni 1999) They are less transparent to users of the review.

Preferable to use simple approaches for assessing validity that can be fully reported (i.e. how each trial was rated on each criterion)

∂

Assessment of quality:in qualitative studies

Numerous tools are available to assess the quality of qualitative research (Cote, 2005, Medical Teacher).

However, three structured appraisal approaches were compared and found that structured assessment tools may not produce greater consistency of judgements (Dixon-Woods, 2007Journal of Health Services Research and Policy)

Check lists• Evidence for Policy and Practice Information (EPPI) • Joanna Briggs Institute (JBI) Web site: www.joannabriggs.edu.au/cqrmg

• Spencer et al. 2003 conducted a review of many of the current appraisal frameworks and checklists, which may be helpful

• Critical Appraisal Skills Programme 2006 : http://www.phru.nhs.uk/Pages/PHD/resources.htm

• Expert judgement is also an important factor

∂

Quality prompts in qualitative research

• Are the aims and objectives clearly stated?

• Is the research design clear and appropriate for the research question?

• Are the processes of obtaining the data clear?

• Is sufficient data provided to link interpretation and conclusions back to the data?

• Is the method of analysis appropriate and adequately explained?

(Dixon-Woods et al, 2006, BMC Medical Research Methodology)

∂

Quality indicators for all studies.1. Is the research question(s) or hypothesis clearly stated?2. Is the subject group appropriate for the study (number, characteristics)?3. Are the methods used (qualitative/quantitative) reliable and valid for the

research question and context?4. Completeness of ‘data’ Have subjects dropped out? attrition rate < 50%? 5. Control for confounding Have multiple factors/variables been removed or

accounted for where possible?6. Are the statistical or other methods of analysis used appropriate?7. Is it clear that the data justify the conclusions drawn?8. Could the study be repeated by other researchers?9. Does the study look forwards in time (prospective) rather than backwards?10.Ethical issues, Were all relevant ethical issues addressed?11. Triangulation, Were results supported by data from more than one source?

(Buckley et al, 2009, BEME review 11).

∂

Collaborations • Cochrane collaboration

• Joanna Briggs Institute

• Campbell collaboration,

• Best Evidence Medical Education

∂

The Cochrane CollaborationThe Cochrane Collaboration was set up over 40 years ago by the epidemiologist Archie Cochrane.

The Cochrane Collaboration was set up to promoteClinical trails, using systematic review and meta-analysisto synthesise data.

Evidence was rated strong or weak according to standardcriteria which included research design and statistical effect size.

Helpful in making decisions about treatments – single variables, focused on outcomes.

∂

Other approaches to systematic review

In Health Joanna Briggs Institute (http://www.joannabriggs.edu.au) The central focus not only effectiveness, but also appropriateness,

meaningfulness and feasibility of health practices and delivery methods. These questions are often answered by considering other forms of research evidence

Also W K Kellogg Foundation (http://www.wkkf.org) Aim to improve health care practice, but do not place randomised

controlled as high up in the hierarchy. In Medical Education Best Evidence Medical Education (http://bemecollaboration.org) Realist synthesis – RAMESUS project –what works, for whom, under

what circumstances.

∂

Campbell collaborationThe Campbell collaboration hosts reviews on:• education, • crime and justice, • social welfare and • international development.

The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions.

http:wwwcampbellcollaboration.org/

∂

Best available evidence Traditionally, the evidence based practice movement has focussed on

the results of quantitative evidence (considering the RCT as the gold standard) to answer questions of effectiveness.

Joanna Briggs Institute (JBI) has as its central focus not only effectiveness, but also appropriateness, meaningfulness and feasibility of health practices and delivery methods.

Best answered by considering other forms of evidence. The JBI regards the results of well-designed research studies grounded in any methodological position as providing more credible evidence than anecdotes or personal opinion. However, when no research evidence exists, expert opinion can be seen to represent the ”best available” evidence.

∂

Conclusions

Literature reviews provide a range of purposes, including knowing what has already been done and what needs to be done

The approach used depends on the question and purpose of the review

There are a range of different types of review: narrative, systematic, meta analysis, realist synthesis

Most of these start with a systematic review

Assessment of quality can challenging, useful tools to support focus

CollaborationsCochrane collaboration, Campbell collaboration, BEME effectiveness,

As well as effectiveness we need to think about using reviews to answer questions about appropriateness, meaningfulness and feasibility

Thank youQuestions?

j.c.illing@dur.ac.uk

Centre for Medical Education ResearchSchool of Medicine, Pharmacy and Health

Durham University