critical appraisal of the medical literature - jane hirst · •critical appraisal helps us decide...
TRANSCRIPT
![Page 1: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/1.jpg)
Critical appraisal of the medical literature
Jane Hirst
MBBS, MPH, PhD FRANZCOG
University of Oxford
Training course in research methodology and research protocol development
Geneva 2018
![Page 2: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/2.jpg)
What makes a study reliable?
![Page 3: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/3.jpg)
Nine out of ten women we asked believe that “Youth Code” makes their skin firmer and younger looking
Which design maximises the chance of getting the result the company wants?
a) Ask women buying “Youth Code” in the shops whether they agree their skin is firmer and younger looking?
a) Ask a random sample of women to try “Youth Code” and then comment on whether they agree that their skin is firmer and younger looking?
![Page 4: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/4.jpg)
What is critical appraisal?
• Carefully and systematically evaluate research to assess:• Validity (is these findings trustworthy?)
• Value (what do the results show?)
• Relevance (How do these results relate to my clinical practice?)
Burls 2009
![Page 5: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/5.jpg)
Critical appraisal: a key component of evidence based medicine
Clinical Problem
Define the
question
Search for the
evidence
Critical Appraisal
Decide what
action to take
Evaluate your new practice
![Page 6: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/6.jpg)
Asking the right question
P Population
I Intervention
C Comparator
O Outcome
![Page 7: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/7.jpg)
cebm.net
![Page 8: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/8.jpg)
Choosing right study design
• Some study designs are not appropriate to answer certain questions
• All study designs are prone to different biases
![Page 9: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/9.jpg)
Pyramid of evidence
![Page 10: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/10.jpg)
So are RCTs the gold standard for evidence?
…..dependsSlides from: K Mahtani, CEBM Oxford
![Page 11: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/11.jpg)
Limitations of RCTs
• Excellent vs Poor RCTs – quality varies• Impact on interpretation of result (external validity)?
• Expensive and time consuming• £250k - £millions over 2-5 years+
• May not always be the right study design to answer that question
![Page 12: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/12.jpg)
A RCT to examine if smoking causes lung cancer• 30 healthy Oxford Students
• Randomise to 2 groups• Gp1 smokes 20 cigarettes per day every day
• Gp2 no smoking
UNLUCKYSTRIKE
![Page 13: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/13.jpg)
Types of research
• What is the best study design for answering this type of question?
• Aetiology
• Diagnosis
• Prognosis
• Harm
• Effectiveness
• Qualitative
![Page 14: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/14.jpg)
How to critically appraise an article
• Validity: methods to check that the biases for which that particular study design is prone have been minimised
• Results
• Clinical relevance
![Page 15: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/15.jpg)
Validity
Internal External
![Page 16: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/16.jpg)
Bias
“the systematic deviation of the results of a study from the truthbecause of the way it has been conducted, analysed or reported”
Burls, “What is Critical Appraisal” 2009
![Page 17: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/17.jpg)
Bias
Design
Recruitment
Selection
InterviewerChronologyRecall
Confounding
Citation
Exposure or Outcome
Misclassification
![Page 18: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/18.jpg)
Sources of bias in clinical trials
Juni, BMJ 2001
![Page 19: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/19.jpg)
Assessing Trials of effectiveness
Questions to ask:
1. Are the results of the trial valid?
2. What are the results?
3. Will the results help locally?
From: Critical Appraisal Skills Program, Oxfordhttps://casp-uk.net/
![Page 20: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/20.jpg)
Checklists for clinical trials
![Page 21: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/21.jpg)
11 useful questions for critical appraisal of a randomised trial
Oxford Critical Appraisal Skills Programme (CASP)
![Page 22: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/22.jpg)
Representative: Are the trial subjects representative of patients in this setting?
All people in study setting
Eligible participants
In trial
![Page 23: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/23.jpg)
![Page 24: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/24.jpg)
Why randomise?
• Minimises measured and unmeasured confounding
![Page 25: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/25.jpg)
Minimising allocation bias
• Centralised computer randomisation the best
• Other methods such as sealed envelopes doubtful
• Non randomised: date of birth, alternate patients alternate days, etc.
![Page 26: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/26.jpg)
If answer to first two questions is no….
![Page 27: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/27.jpg)
![Page 28: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/28.jpg)
![Page 29: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/29.jpg)
Maintenance: Were the groups treated equally?
![Page 30: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/30.jpg)
From: Critical Appraisal Skills Program, Oxfordhttps://casp-uk.net/
![Page 31: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/31.jpg)
From: Critical Appraisal Skills Program, Oxfordhttps://casp-uk.net/
![Page 32: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/32.jpg)
Intention to treat
• Once a participant is randomised, they should be analysed to the group they were assigned to
• Pros• Reflects “real life” e.g. non compliance
• Unbiased estimate of true effect
• Maintains sample size thus maintaining statistical power
• Cons• Noncompliance provides little data on efficacy
• Treatment effect may be conservative
• Dropouts/non-compliant/compliant subjects are different
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3159210/
![Page 33: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/33.jpg)
What does this study tell us?
• P values (hypothesis testing):• Tests to exclude the null hypothesis
• Confidence intervals (estimation of effect)• Range of values within which the true effect is likely to
lie
• Wider the confidence interval, less precision in result
• Relative Risk
• Absolute Risk
• Odds Ratios
• Number needed to treat
![Page 34: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/34.jpg)
![Page 35: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/35.jpg)
From: Critical Appraisal Skills Program, Oxfordhttps://casp-uk.net/
![Page 36: Critical appraisal of the medical literature - Jane Hirst · •Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant](https://reader034.vdocuments.net/reader034/viewer/2022050407/5f846516ad1ed31f0271b379/html5/thumbnails/36.jpg)
Conclusion
• Critical appraisal helps us decide whether evidence is valid, what the results tell us and whether the study is relevant to our setting
• Checklists are available to help
• Don’t believe everything you read in journals!