Critical Appraisal of Published Medical Research

Dr. Tarek AminProfessor of Public Health

Cairo University

Background

• Every day … – ~ 46 randomized clinical trials are

published – ~ 1000 new Medline articles – ~ 6,000 new articles in biomedical

journals• Every year … – ~ 3 million articles published in ~

30,000 journals

Background

• Most research published in medical journals is either

– Too poorly done – Insufficiently relevant to be

clinically useful• Besieged with too much information

to keep up to date. • High quality information is often not

easy to find.

Critical appraisal is not

o Negative dismissal of any piece of research

o Assessment on results aloneo Based entirely on statistical analysiso Undertaken by experts only

Why critically appraise?

• To find out the validity of the study– Are the methods robust?

• To find out the reliability of the study– What are the results and are they credible?

• To find out the applicability of the study– Is it important enough to change my

practice?

What do I need to know?

o Awareness of study designso Levels of evidenceo Statistics!!o CA checklistso CA resources

Roadmap

1. Describe the evidence2. Internal validity3. External validity4. Comparison with other evidence5. Resources

1 -Describe the evidence

What relationship being evaluated and what hypothesis was tested?

What were the exposure and the outcome variable? What was the study design? Case report, series Survey Clinical trial Case-control Prospective or retrospective cohort study Cross sectional study or Others

1 -Describe the evidence Definition of participants in terms of: – Source populations – Time frame – Eligibility criteria – Participation rates of the different groups

compared Summary of the main results: What is the result in terms of association between

exposure and outcome?

Should be possible to express the main result in a simple table and obtain from the paper the means to calculate the appropriate measure of association.

2- Internal Validity

o The truthfulness of inferences about the study population.

o Causal relationship between exposure and outcome or just an association?

2 -Internal Validity

Two aspects of internal validity1. Non-causal explanations2. Causal explanations

Non-causal explanations

• Bias• Confounding• Chance variation

Non-causal explanations

• Bias – Selection bias, surveillance,

diagnosis, referral, non-response, length of stay, survival bias

– Misclassification bias: recall, interviewer, improper analysis, etc.,

Confounding

Smoking is a risk factor for cancer of the larynx

• we’d like to quantify the strength of the association between smoking and laryngeal cancer, but …

• many smokers are also drinkers (which is also a risk factor for cancer of the larynx)

• drinking is said to confound the association between smoking and risk of laryngeal cancer

Confounding definition

Confounding (the formal definition)• The effect of an extraneous variable

that wholly or partially accounts for the apparent effect of the study exposure, or masks an underlying true association

A variable is confounder

A variable is a confounder if:1. It is causally associated with the

outcome; and2. It is non-causally associated with

the exposure; and3. It and the exposure variable are on

two separate causal pathways

Confounding.

A confounding variable is associated with the exposure and it affects the outcome, but it is not an intermediate

link in the chain of causation between exposure and outcome.

Myocardial infarctionOral contraceptive

Smoking

IUD insertion

STDs

Salpingitis

Chance variation

A relationship between exposure and outcome identified by chance?

Type I error: null hypothesis is rejected when, in reality, it is true.

Non-causal explanations

The order of these non-causal explanations is important:

o Observation (information) bias, analytical manipulation of the data will not overcome the problem

o Confounding, then appropriate analysis will (in most cases) overcome the problem

Five aspects of causal explanations

1. Is there a correct temporal relationship?

2. Is the relationship strong?3. Is there a dose-response

relationship?4. Consistency of the association?5. Specificity of association

1 .Is there a correct temporal relationship?

o The exposure must act before the outcome occurs

o No problem with prospective study designs

o Difficult in retrospective studies

2 .Is the relationship strong?

Larger relative risks (and Odds) are more likely to reflect causal relationships.

3 .Is there a dose-response relationship?

The greater the exposure, the greater the risk of disease.

4 .Consistency of the association

Expected to apply across a wide range of subjects.

An association identified in one study that is consistent with the same association identified in a different groups of subjects.

5 .Specificity of association

Specificity: exposure produces a specific outcome (e.g. asbestos and mesothelioma)

3 -External Validity

External validity: can the results be applied to populations other than that which was studied?

• If the internal validity of a study is poor, the answer is no

Aspects of external validity:1. Applied to the eligible population?2. Applied to the source population?3. Applied to other relevant populations?

1.Can the results be applied to the eligible population?

– The relationship between the study population (the population from which samples are taken) and the eligible population (those that met the study inclusion criteria but did not take part) should be well documented.

– Non-participation have to be considered carefully as they are likely to be non-random.

2 .Can the results be applied to the source population?

Whether the association between outcome and exposure given by the study participants is likely to apply to other groups

3 .Can the results be applied to other relevant populations?

The difficulties of applying results from one group of subjects to another will be minimal for issues of basic physiology and maximal for effects in which cultural and psycho-social aspects are dominant

4 -Comparison with other evidence

Useful to consider a hierarchy of evidence .

1. Randomized [clinical] trials2. Cohort and case-control studies3. Other comparative studies4. Descriptive studies, case series,

case studies, clinical experience

Three aspects of comparison should be considered:

1. Results consistent with other evidence?

2. Results plausible biologically?3. Coherency with the existing

knowledge.

Are the results consistent with other evidence?

• Most important characteristic used in the judgment that an association is causal

• Lack of consistency argues against causality

Plausibility

Is the observed association biologically understandable?

Coherency

• An association is regarded as coherent if it fits the general features of the distribution of both the exposure and the outcome under assessment.

Summary

1. Describe the evidence• – type of study, outcome measure, population investigated,• results2. Internal validity• – non-causal explanations• • bias• • confounding• • chance• – causal explanations• • temporal relationship• • strength of relationship• • dose-response• • consistency• • specificity

Summary

3. External validity• – can the results be applied to the eligible population?• – can the results be applied to the source population?• – can the results be applied to other relevant populations?4. Comparison of the results with other evidence• – are the results consistent with other evidence?• – are the results plausible biologically?• – is there coherency with the distribution of the exposure

and the• outcome?• Can we apply these results to other populations? Are

the findings reported here consistent with other studies that looked at the same thing?

Resources

Web sites– Users’ Guides to Evidence-Based Practice

http://www.cche.net/usersguides/main.asp– A Student’s Guide to the Medical Literature

http://grinch.uchsc.edu/sg/– Pearls for Residents: Annotated Critical

Appraisal References http://www.cfpc.ca/English/cfpc/cme/pearls/pearls%20residents/default.asp?s=1

• Critical Appraisal of Bio-medical Literature http://www.shef.ac.uk/scharr/ir/units/critapp/resources.htm

• Critical Appraisal Resources for Assessing Health and Medical Research http://www.etsu.edu/health/index_files/harvill_handout.pdf

• Bandolier http://www.medicine.ox.ac.uk/bandolier/

Thank you