before and after studies: a reminder
DESCRIPTION
Before and After Studies: A Reminder. Introduction to the Design and Analysis of Trials can be found on: http://www-users.york.ac.uk/~djt6/. Background. Many researchers (?) use before and after studies – they are, of course, nearly completely useless. Why? This is because of: - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/1.jpg)
Introduction to the Design and Analysis of Trials can be found on:http://www-users.york.ac.uk/~djt6/
Before and After Studies: A Reminder
![Page 2: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/2.jpg)
Background Many researchers (?) use before
and after studies – they are, of course, nearly completely useless.
Why? This is because of: Regression to the mean Temporal changes
![Page 3: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/3.jpg)
Which Researchers (?) use before and after? Clinicians, teachers assessing
individuals. Action researchers. Audit.
![Page 4: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/4.jpg)
Temporal Change Things change, people get better,
policy changes all of which may make a difference.
A before and after study CANNOT possibly cope with these temporal events.
![Page 5: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/5.jpg)
Regression to the Mean Is a group phenomenon applies
when we measure a group of people and re-measure them.
Those with values below or above the mean will tend to regress back towards the mean on re-measurement.
![Page 6: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/6.jpg)
Before and after treatment for neck pain
8.5
9
9.5
10
10.5
11
1st Qtr 2nd Qtr
Improvement highly significant p < 0.0001
![Page 7: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/7.jpg)
Plot of difference scores A symptom of regression to the
mean is if you plot change scores (baseline – follow up) against baseline scores. A correlation indicates RTM.
Thus, those with the lowest baseline improve the most and those with the highest improve the least.
![Page 8: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/8.jpg)
Scatterplot showing RTM
NPQ: baseline (adjusted for m.v.)
3020100
CH
G3
NP
Q
20
10
0
-10
-20
Correlation of Change Score with baseline values = 0.33 p < 0.0001
![Page 9: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/9.jpg)
Some benefit of vaccination is due to regression to mean
![Page 10: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/10.jpg)
Meningitis After vaccination new cases of
meningitis fell from about 240 to 35 an 85% decrease. HOWEVER, of the 205 cases that were ‘prevented’ the majority 120 were due to ‘regression to the mean’ effects ONLY 41% were probably due to the efficacy of the vaccine.
![Page 11: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/11.jpg)
Education intervention Wheldall selected 40 pupils whose
reading was at least 2 years behind their peers.
Half were exposed to an intervention.
Wheldall Educational Review 2000;52:29.
![Page 12: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/12.jpg)
Before and after reading programme
80
85
90
95
100
105
Pre Post
InterventionControl
Difference highly statistically significant p < 0.001
![Page 13: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/13.jpg)
Before and after reading programme
80
85
90
95
100
105
110
Pre Post
InterventionControl
Differences between groups NOT statistically significant
![Page 14: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/14.jpg)
RTM misunderstanding “the mean gain scores translated to
impressive effect sizes of 0.6.” “It could be argued that it is asking too
much of any program to demonstrate enhanced efficacy on top of such high existing efficacy”
“…control group gains were largely attributable to pre-existing …literacy programme..”
Perhaps, BUT much of the gain will be due to RTM.
![Page 15: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/15.jpg)
Evaluation of School intervention
A secondary school routinely offered children who scored badly on a reading test an ICT intervention.
This was shown to improve children’s literacy.
![Page 16: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/16.jpg)
ICT and Reading
10.5
11
11.5
12
12.5
13
Before After
![Page 17: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/17.jpg)
Did it work? Impossible to tell. Regression to
the mean and temporal effects does not allow us to find this out.
Fortunately, we are doing a RCT of ICT and reading.
![Page 18: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/18.jpg)
RTM and Policy Decisions
Government policy targets 10 worst areas for street crime. 1 year later 17% fall in crime – some or all due to RTM.
40% increase in gun crime results in a month’s amnesty for fire arms – will probably work through RTM.
![Page 19: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/19.jpg)
Annual Increase in offences with firearms
0
5
10
15
20
25
30
35
40
2000 2001/ 02 2002/ 03Amnesty
![Page 20: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/20.jpg)
Exam marking In MSc double blind marking. Two
markers disagree at the extremes of the distribution.
We might fool ourselves that one marker is ‘hard’ and the other a ‘softie’ but really it is RTM.
![Page 21: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/21.jpg)
RTM and exam scripts
R2 = 0.7788
-30
0
30
0 50 100
![Page 22: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/22.jpg)
Policy Changes Regression to the mean is an
excellent method of ‘proving’ something works;
Failing schools or hospitals can have an ‘expensive’ management change and there is a good chance that regression to the mean will do the job.
![Page 23: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/23.jpg)
Proving ‘Effective’ Treatments RTM is an excellent phenomenon to
‘prove’ to doubting clinicians the value of a new treatment.
Choose an outcome measure with a high variance (e.g., single BP measure, FEV). Identify patients with extreme values (preferably only measured once), treat and re-measure. The group mean ought to decline (not all patients will improve but most will).
![Page 24: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/24.jpg)
Dealing with RTM Sequential measurements taking an
average (e.g., 3 BP measurements averaged out) will reduce the problem.
The only way to reliably deal with the problem is through randomised trials.
Which is why before and after data are generally regarded as almost USELESS.
![Page 25: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/25.jpg)
Ceiling and Floor Effects As well as RTM before and after studies
are blighted by ceiling and floor problems.
Often measurement instruments have a floor (e.g., 0) or a ceiling (e.g., 100%), which means if someone’s value is close to either of these extremes they cannot change much except towards the mean.
![Page 26: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/26.jpg)
League Tables Classic problem of RTM with ceiling and
floor effects. For example, schools that get close to 100% 5 GCSEs cannot do any better, whereas schools with very low levels can only go upwards. This phenomenon is skillfully exploited by politicians to show an effect. Similarly with hospital league tables.
Same problem applies to quality of life measures. EuroQol for example, has ceiling problems.
![Page 27: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/27.jpg)
Summary Before and after studies are the
weakest evaluative method of proving something does or does not work.
To control for temporal changes and regression to the mean controlled trials are required.
![Page 28: Before and After Studies: A Reminder](https://reader036.vdocuments.net/reader036/viewer/2022062517/56813c1a550346895da58dcf/html5/thumbnails/28.jpg)
Conclusion You can prove virtually any
‘crackpot’ theory using RTM. NEED a control group.