Systematic reviews: how we use them in health care research and

how they can be used in other disciplines

Dr Mona Nasser DDS, MSc, PhD

“If, as is sometimes supposed, science consisted in nothing but thelaborious accumulation of facts, it would soon come to a standstill,crushed, as it were, under its own weight…. Two processes are thusat work side by side, the reception of new material and thedigestion and assimilation of the old…The work which deserves,but I am afraid does not always receive, the most credit is that inwhich discovery and explanation go hand in hand, in which notonly are new facts presented, but their relation to old ones ispointed out.”

(Rayleigh 1885)

The James Lind Library 4.2 Preparing and maintaining systematic reviews of all the relevant evidence (https://www.jameslindlibrary.org/essays/4-2-preparing-and-maintaining-systematic-reviews-of-all-the-relevant-evidence/)

"It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials"

Archie Cochrane, 1979

“Systematic reviews seek to collate evidence that fits pre-specifiedeligibility criteria in order to answer a specific research question. Theyaim to minimize bias by using explicit, systematic methods documentedin advance with a protocol.”

Chandler J, Cumpston M, Thomas J, Higgins JPT, Deeks JJ, Clarke MJ. Chapter I: Introduction. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020. Available from www.training.cochrane.org/handbook.

Steps involved in conducting systematic reviews• Define and Frame a review question• Developing and Running a search strategy • Data Screening and Data Extraction • Quality Assessment of individual studies• Synthesis of the data (qualitative and/or quantitative synthesis)• Judging the quality of the body of the evidence• Developing recommendation for future practice and research

(Certain steps might involve consulting stakeholders)

1. exp dentistry/2. exp dental facilities/3. infection control, dental/4. exp dentists/5. dental staff/6. exp dental auxiliaries/7. (dental or dentist$ or hygienist$).mp.8. ((oral or maxillofacial) adj5 (care$ or procedure$ or surgery or surgical or medicine)).mp.9. orthodonti$.mp.10. periodont$.mp.11. (tooth or teeth or gum$ or endodont$ or plaque$ or pulpotom$ or pulpectom$ or "cavity prep$" or molar$ or bicuspid$ or premolar$ or pre-molar$ or incisor$ or canine$ or eyetooth or eyeteeth or cuspid$).mp.12. ((scal$ adj2 polish$) or "root canal" or (root adj6 resect$) or (root$ adj3 planing) or apicectom$ or apicoectom$).mp.13. ((root$ or periodont$ or dental or subgingiv$ or gingiv$ or supragingiv$) adj5 (scale or scaling or scaler$ or curettage)).mp.14. Dental high speed equipment/15. ("high speed air rotor$" or "low speed handpiece$" or "low speed hand piece$" or micromotor$ or "turbine handpiece$" or "electrosurgery unit" or "air polisher$" or "prophy angle$" or "air-water syringe$" or "high speed hand piece$" or "high speed handpiece$" or "three-way air syringe$" or "threewayair syringe$" or "ultrasonic scaler$" or "hard-tissue laser$" or "dental drill$" or "piezo unit$" or "piezo hand piece$" or "piezo handpiece$" or "rotary instrument$" or "air abrasion" or "water spray$").mp.16. or/1-1517. Air microbiology/18. Air pollution, indoor/19. Aerosols/20. Inhalation exposure/21. (aerosol$ or bioaerosol$).mp.22. (droplet$ or splatter$ or spatter$ or microbe$ or bacillus or germ$ or microorganism$ or virus$ or viral or coronavirus$ or COVID$ or "middle east? respiratory syndrome$" or MERS or MERS-CoV or "camel flu" or SARS or "sudden acute respiratory syndrome$" or "Wuhan virus$" or 2019-nCoV or SARS-CoV-2 or SARS-CoV or SARS-CoV-1 or SARS-1).mp.23. (air adj5 (pollut$ or quality or impur$)).mp.24. or/17-2325. Decontamination/26. ("high volume evacuat$" or HVE or "high volume aspirat$").mp.27. Rubber dams/28. ((rubber adj dam$) or (oral adj dam$) or (dental adj dam$) or (latex adj dam$) or Kofferdam).mp.29. ("Optra Dam" or "OptraDam Plus" or OptiDam or FlexiDam or "Hygenic Fiesta").mp.30. Suction/31. ("saliva ejector" or "low volume aspirat$" or (suction adj2 saliva)).mp.32. Air filters/33. (air adj5 (filter$ or filtration or purif$ or clean$)).mp.34. ((HEPA or "High Efficiency Particulate Air" or "High Efficiency Particulate Arrestance") adj5 filter$).mp.35. Air ionization/36. (ionis$ or ioniz$).mp.37. Ozone/38. (ozonis$ or ozoniz$).mp.39. Ultraviolet rays/40. (ultraviolet or UV or ultra-violet or actinic).mp.41. ((aerosol$ or bioaerosol$ or droplet$ or spatter or splatter) adj2 reduc$).mp.42. Fumigation/43. (fog$ or fumigat$ or decontaminat$ or "smoke out" or smokeout or depollut$ or depurat$).mp.44. or/25-4345. 16 and 24 and 44This subject search will be linked to the Cochrane Highly Sensitive Search Strategy (CHSSS) for identifying randomised trials in MEDLINE: sensitivity-maximising version (2008 revision) (Lefebvre 2019).1. randomized controlled trial.pt.2. controlled clinical trial.pt.3. randomized.ab.4. placebo.ab.5. drug therapy.fs.6. randomly.ab.7. trial.ab/ not humans.sh.11. 9 not 10.8. groups.ab.9. or/1-810. exp animals•

1. exp dentistry/2. exp dental facilities/3. infection control, dental/4. exp dentists/5. dental staff/6. exp dental auxiliaries/7. (dental or dentist$ or hygienist$).mp.8. ((oral or maxillofacial) adj5 (care$ or procedure$ or surgery or surgical or medicine)).mp.9. orthodonti$.mp.10. periodont$.mp.11. (tooth or teeth or gum$ or endodont$ or plaque$ or pulpotom$ or pulpectom$ or "cavity prep$" or molar$ or bicuspid$ or premolar$ or pre-molar$ or incisor$ or canine$ or eyetooth or eyeteeth or cuspid$).mp.12. ((scal$ adj2 polish$) or "root canal" or (root adj6 resect$) or (root$ adj3 planing) or apicectom$ or apicoectom$).mp.13. ((root$ or periodont$ or dental or subgingiv$ or gingiv$ or supragingiv$) adj5 (scale or scaling or scaler$ or curettage)).mp.14. Dental high speed equipment/15. ("high speed air rotor$" or "low speed handpiece$" or "low speed hand piece$" or micromotor$ or "turbine handpiece$" or "electrosurgery unit" or "air polisher$" or "prophy angle$" or "air-water syringe$" or "high speed hand piece$" or "high speed handpiece$" or "three-way air syringe$" or "threeway air syringe$" or "ultrasonic scaler$" or "hard-tissue laser$" or "dental drill$" or "piezo unit$" or "piezo hand piece$" or "piezo handpiece$" or "rotary instrument$" or "air abrasion" or "water spray$").mp.16. or/1-15…

1. randomized controlled trial.pt.2. controlled clinical trial.pt.3. randomized.ab.4. placebo.ab.5. drug therapy.fs.6. randomly.ab.7. trial.ab/ not humans.sh.11. 9 not 10.8. groups.ab.9. or/1-810. exp animals

We found 16 studies that involved a total of 425 people. Studies involved between one and 80 participants, who were aged between 5 and 69 years. Six studies were conducted in the USA, five in India, two in the UK and one each in Egypt, the Netherlands and the United Arab Emirates.

The studies evaluated one or more of the following devices:

- high-volume evacuator (7 studies);- hands-free suction device (2 studies);- saliva ejector (1 study);- rubber dam (3 studies);- rubber dam with a high-volume evacuator (1 study); or- air cleaning system (1 study).

None of the studies evaluated the risk infectious disease transmission. Nor did they evaluate cost, acceptability or ease of implementation.

Publication Bias

• We care both about sources of biases (systematic error) in individual studies but also in the body of the evidence. One well known bias is publication bias.

• We use empirical evidence in the form of meta-epidemiological studies to inform our approach to assess the risk of bias in studies or reviews.

• Non-reporting biases lead to bias due to missing results in a systematic review.

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020. Available from www.training.cochrane.org/handbook.

Page MJ, Higgins JPT, Sterne JAC. Chapter 13: Assessing risk of bias due to missing results in a synthesis. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020. Available from www.training.cochrane.org/handbook.

• Hosseini S, Turhan B, A Systematic Literature Review and Meta-analysis on Cross Project Defect Prediction. IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. Y, DECEMBER 2016. https://bura.brunel.ac.uk/bitstream/2438/15341/1/Fulltext.pdf

• Kitcheman B. Madeyski L, Brereton P, Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment. Empirical Software Engineering. Issue 1/2020

• Grimstad S. Jorgensen M, Molokken-Ostvold KM. The Clients’ Impact on Effort Estimation Accuracy in Software Development Projects. 11th IEEE International Software Metrics Symposium (METRICS 2005). https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1509288

• Kitchenham B, Pretorius R, Budgen D, et al. Systematic literature reviews in software engineering – A tertiary study. Information and Software Technology. Volume 52, Issue 8, August 2010, Pages 792-805

• Systematic mapping/Evidence mapping/Scoping• Configurative versus aggregating results• Emerging concepts versus estimates of magnitudes and precision • Meta-analysis (and other quantative synthesis methods)• Thematic Analysis• Narrative synthesis• Network meta-analysis• Umbrella reviews or overview of reviews• Realist review• Mixed methods review• Meta-narratives• Meta-etnography• Rapid review

Littell, J.H. (2018), Conceptual and practical classification of research reviews and other evidence synthesis products. Campbell Systematic Reviews, 14: 1-21. doi:10.4073/cmdp.2018.1

Network Meta analysis“When multiple interventions have been used and compared for the same disease and outcomes, network meta-analysis (also commonly referred to as a multiple treatment comparison meta-analysis or mixed treatment meta-analysis) offers a set of methods to visualize and interpret the wider picture of the evidence and to understand the relative merits of these multiple interventions”

Mills EJ, Thorlund K, Ioannidis JP. Demystifying trial networks and network meta-analysis. BMJ. 2013 May 14;346:f2914. doi: 10.1136/bmj.f2914. PMID: 23674332.

S. Ghanavati, D. Amyot and L. Peyton, "A systematic review of goal-oriented requirements management frameworks for business process compliance," 2011 Fourth International Workshop on Requirements Engineering and Law, Trento, 2011, pp. 25-34.

• They did not use a quantative synthesis to answer the question, they had a predefined criteria (categorised the question further into – requirements engineering framework for managing compliance, goal modelling approach, business process compliance, legal requirements extraction, law complaint business process templates, legal requirements prioritisation, tool support, other legal compliance issues.

• They categorised data based on the framework and then summarised and compared the studies and data – they identified specific areas that required further work in relation to prioritization to improve compliance, templates for generating law compliant processes, general links between legal requirements, goal models, and business processes, and semi-automation of legal compliance and analysis

Systematic mapping“Systematic mapping does not aim to answer a specific question as does a systematic review, but instead collates, describes and catalogues available evidence (e.g. primary, secondary, quantitative or qualitative) relating to a topic of interest. The included studies can be used to develop a greater understanding of concepts, identify evidence for policy-relevant questions, knowledge gaps (topics that are underrepresented in the literature that would benefit from primary research), and knowledge clusters (sub-sets of evidence that may be suitable for secondary research, for example using systematic review).”

James, K.L., Randall, N.P. & Haddaway, N.R. A methodology for systematic mapping in environmental sciences. Environ Evid 5, 7 (2016). https://doi.org/10.1186/s13750-016-0059-6

Cheng, S.H., MacLeod, K., Ahlroth, S. et al. A systematic map of evidence on the contribution of forests to poverty alleviation. Environ Evid 8, 3 (2019). https://doi.org/10.1186/s13750-019-0148-4

Configurative versus Aggregative reviews• Focus on aggregating or adding up data versus focus on configuring

data from the included studies • They address different questions but it is more a continuum (not a

dichotomy)• Example: Do students in classes where there is a classroom teaching

assistant get higher or lower scores on test scores? VERSUS How can we conceptualise the way that the presence of classroom assistants changes relationships between students and teachers and between teachers in class?

Gough D, Oliver S, Thomas J, Introducing systematic reviews. Sage Publicadtions

The first part of the PhD was a systematic review focused on the question ‘How do referrals operate within the UK primary dental care setting?’. It was a configurative systematic that used critical interpretive synthesis.

Overview of reviews

“Overviews are known by a variety of different names, all potentially reflecting different aspects and aims of the syntheses. Terms used include: overview; umbrella review; meta-review; (systematic) review of (systematic) reviews; synthesis of systematic reviews; and summary of systematic reviews. The common feature of the methods associated with all of these terms is the fundamental process of synthesising evidence which is derived, often exclusively, from systematic reviews. The systematic review forms the primary ‘unit of analysis’ and is the basis upon which an overview is built”

Hunt, H., Pollock, A., Campbell, P. et al. An introduction to overviews of reviews: planning a relevant research question and objective for an overview. Syst Rev 7, 39 (2018). https://doi.org/10.1186/s13643-018-0695-8

What do we know about evidence-informed priority setting processes to set a population-level health-research agenda: An overview of reviews Audrey tan, Sumanth Kumbargere, Mona Nasser, Tarang Sharma, Tanja Kuchenmuller

• Objective- This overview aimed to synthesize existing systematic reviews to produce a draft framework of evidence-informed health priority setting processes that can be considered by WHO European Region countries to support national research agendas and highlight guidance to choose appropriate methodologies for conducting such exercises.

• Study design and setting- We searched Ovid MEDLINE® and the WHO Institutional Repository for Information Sharing from 2010-2020 for critical or systematic reviews that evaluated research priority setting exercises. We adapted the AMSTAR checklist to assess the quality of included reviews and used adapted frameworks for data extraction and analysis.

• Results- The search resulted in 1918 titles, of which 26 were synthesized. All of the included systematic reviews, were of low quality. We categorized the included studies into three broad categories based on the primary focus of the review: engaging with stakeholders; methods; context; or health area. We appraised and synthesized these priority-setting reviews to propose a draft evidence-informed framework for countries to consider while developing their national research agendas.

• Conclusion- The draft framework could support countries in answering questions that are of value to society and whose outcomes inform policies that improve the health and well-being of populations.

What’s the future

• Living systematic reviews or living maps • Rapid reviews• Re-thinking the evidence ecosystem or evidence universe

-Evidence synthesis 2.0 model

Evidence for future research?

Cummulative meta-analysis showing proven benefit of aprotinin on need for blood transfusion during cardiac surgery (Fergusson et al 2005)

Are applicants who seek support for new research required to refer to systematic reviews of existing evidence?

Funding Agencies Country

National Institute for Health Research (NIHR) UK/England

Patient-Centered Outcomes Research Institute (PCORI) USA

Medical Research Council (MRC) UK

Deutsche Forschungsgemeinschaft (DFG) Germany

The Netherlands Organisation for Health Research and Development (ZonMw) Netherlands

Canadian Institutes of Health Research (CIHR) Canada

French Ministry of Health (FMoH) France

l’Agence Nationale de la Recherche (ANR) France

Danske Regioner (DR ) Denmark

Regional Health Authorities (RHA ) Norway

National Health and Medical Research Council (NHMRC) Australia

National Institutes of Health (NIH) USA

Nasser M, Clarke M, Chalmers I, Brurberg KG, Nykvist H, Lund H, Glasziou P. What are funders doing to minimise waste in research? Lancet. 2017 Mar 11;389(10073):1006-1007. doi: 10.1016/S0140-6736(17)30657-8. Nasser M, Clarke M, Chalmers I, Brurberg KG, Nykvist H, Lund H, Glasziou P. How can research funders add value to research as part of a special session on responsible research conduct for funding agencies. Work Research integrity conference, Amsterdam, Netherlands. 28-31 May 2017