1

How to Read a Paper:

2 – Assessing the Methodological Quality of Papers

Dr Luke KaneMay 2014

2

References

• This lecture was made from T Greenhalgh’s papers in the BMJ Series “How to Read a Paper”

• Greenhalgh, T. (1997) “How to read a paper” British Medical Journal. Web, accessed April-May 2014 at <http://www.bmj.com/about-bmj/resources-readers/publications/how-read-paper>

3

Outline

• Five questions to ask yourself when looking at a paper

4

Five Essential Questions

• To decided whether methods used are valid:1. Was the study original?2. Whom is the study about?3. Was the design of the study sensible?4. Was systematic bias avoided or minimised?5. Was the study large enough and/or long enough

to make the results credible?

5

Q1: Was the study original?

• Only a very small amount of research is done that is entirely “new”

• You need to ask if the study adds to the literature• For example:– Is it bigger or longer than previous ones?– Is the methodology more rigorous?– Does it address methodological criticisms of previous

ones?– Will the results add to a meta-analysis?– Is the population studied different? Ages? Sex?

6

Q2: Whom is the Study about?

• How were the subjects recruited?– Is it likely to bias the results?• E.g. how happy are you with services of hospital?• If you place an ad in a newspaper asking people to

email in their answers then you are only sampling people that read the newspaper and are motivated to email you• Better to ask 1 in 10 people in a waiting room for

example

7

Q2: Whom is the Study about?

• Who was included?– Most trials in the West only include healthy young adults – Often trials are run on people with severe forms of a disease –

e.g. end stage heart failure– This could mean that the results are not valid for those with

milder forms• Who was excluded?

– Many studies exclude people who are illiterate– People with co-existing illness– People taking other medications

• Is it fair to extrapolate this to patients taking multiple medications with numerous diseases?

8

Q2: Whom is the Study about?

• Were the subjects studied in “real life” circumstances?– Trials can often be run very differently to how

people practise medicine in the real world– i.e. special equipment – More time, more information

9

Q3: Was the design of the Study Sensible?

• Critical appraisal:• Start with two fundamental questions– What specific intervention was being considered

and what was it being compared with?– What outcome was measured and how?

– This isn’t always what is written down! – You need to work this out for yourself

10

Q3: Is the design sensible?

• What specific intervention was being considered and what was it being compared with?– Don’t take the written statements at face value– Authors often misrepresent what they did– They often overestimate importance

– For example…

11

Q3: Is the design sensible?

12

Q3: Is the design sensible?

13

Q3: Is the design sensible?

14

Q3: Is the design Sensible?

• What outcome was measured, how?– Is the outcome a sensible choice?

If not then why did they choose it?– For example, the best way to see if a new drug works is

to see if it makes you live longer and whether the quality of life it gives you is good

– If the trial is not based on the above outcome, what and why?

– Is it based on a level of an enzyme in the blood which the pharmaceutical company says is a good indicator of survival?

15

Q4: Was Systematic Bias Avoided or Minimsed?

• Systematic bias is anything that erroneously influences the conclusions about groups and distorts comparisons

• In any study the groups being compared must be as similar as possible to each other

• Different types of study designs have different steps to reduce bias:

16

Q4: Avoiding Bias in RCTs

• Importance of random allocation!

• Need to check for bias at following stages:

17

Q4: Avoiding Bias in Non-randomised clinial trials

• You need to use common sense to decide if differences between intervention and control groups are likely to invalidate differences ascribed to the intervention

• If non-randomised then "almost always" the effects are biased

• e.g. testing a new drug between two groups: one is 75% female, one is 30% female

18

Q4: Reducing Bias in Cohort Studies

• Reminder: Cohort study

• Very difficult to get two groups who have same age, sex, socioeconomic status etc with only difference the exposure

Q4: Reducing Bias in Cohort Studies

• In practice, much of the “controlling” in cohort studies occurs at analysis stage

• Complex statistical adjustment is made for baseline differences in key variables

• Unless this is done adequately, statistical tests of probability and confidence intervals will be dangerously misleading.

• Good example is alcohol and teetotallers

20

Q4: Reducing Bias in Case-Control Studies

• Process most open to bias is not the assessment of outcome, but the diagnosis of the “cases” and the decision as to when the individual became a case

21

Q4: Reducing bias, Was Assessment Blind?

• Assessment bias from people who assess the outcome if they know what group (treatment or control) someone is in

• E.g. I knew that a patient had been randomised to an active drug to lower blood pressure rather than to a placebo, I might be more likely to recheck a reading which was surprisingly high.

22

Q5: : Were preliminary statistical questions dealt with?

• Three important numbers can often be found in the methods section of a paper:

1.the size of the sample; 2.the duration of follow up; 3.and the completeness of follow up.

23

Q5: Statistical Questions

• Sample size: How big?• The sample needs to be big enough to have a

high chance of detecting a statistically significant effect if one exists

• And therefore you can be reasonably sure that no benefit exists if one is not found in the study

Q5: Statistical Questions• Sample size - how to calculate it• Two things need to be decided:• What level between the two groups would constitute a

significant effect?• What is the mean and standard deviation of the primary outcome

variable• Then with a statistical nomogram you can work out how large a sample

you need to have a moderate, high, or very high chance of detecting a

true difference between the groups—the power of the study

• It is common for studies to stipulate a power of between 80% and

90%.

Q5: statistical Questions, ERRORs

• Underpowered studies are common• Usually because hard to recruit subjects• Such studies typically lead to a type II or ß error• This is the erroneous conclusion that an

intervention has no effect. • A type I or α error is the conclusion that a

difference is significant when in fact it is due to sampling error

Q5: Stats, duration of follow up

• Even if sample size was adequate, a study must continue long enough for the effect of the intervention to be reflected in the outcome variable

• A study looking at effect of a new painkiller on degree of postop pain may only need a follow up period of 48 hours

• Study of effect of nutritional supplementation in the preschool years on final adult height, follow up should be measured in decades

Q5:Stats, Completeness of Follow up

• Subjects who drop out ofstudies are less likely to have taken their tablets as directed, more likely to have missed their checkups, and more likely to have experienced side effects

• Reasons why patients withdraw:• Incorrect entry of patient into trial (did not fulfil the entry criteria)• Suspected adverse reaction to the trial drug• Loss of motivation• Loss of patient motivation• Withdrawal by clinician for clinical reasons (such as concurrent

illness or pregnancy)• Loss to follow up (patient moves away, etc)• Death

Q5: completeness

• You still need to analyse the people who dropped out

• This can be done on an intention to treat basis

Conclusion

• First essential question to ask about the methods section of a published paper is: was the study original?

• The second is: whom is the study about?• Thirdly, was the design of the study sensible?• Fourthly, was systematic bias avoided or

minimised?• Finally, was the study large enough, and continued

for long enough, to make the results credible?

30

References

• Bowers, D. (2008) Medical Statistics from Scratch: An Introduction for Health Professionals. USA: Wiley-Interscience.

• Grant, A. (2014) “Epidemiology for tropical doctors”. Lecture (S6) from the Diploma of Tropical Medicine & Hygiene, London School of Hygiene & Tropical Medicine.

• Greenhalgh, T. (1997) “How to read a paper” British Medical Journal. Web, accessed April-May 2014 at <http://www.bmj.com/about-bmj/resources-readers/publications/how-read-paper>