avoiding bias in rcts david torgerson director, york trials unit [email protected]
TRANSCRIPT
A reminder
• Randomised trials are needed for 4 reasons:» Avoiding selection bias;» Controlling for temporal changes;» Controlling for regression to the mean;» Basis for statistical inference.
Background
• Randomisation, when undertaken properly, prevents selection bias.
• Selection bias occurs when participants are allocated in such a way that allocation correlates with outcome.
• Selection bias is one of the main threats to validity that randomisation seeks to avoid.
• However, forms of selection and other sources of bias can still undermine the validity of a RCT.
Non-random methods:Alternation
• Alternation is where trial participants are alternated between treatments.
• EXCELLENT at forming similar groups if alternation is strictly adhered to.
• Problems because allocation can be predicted and lead to people withholding certain participants leading to selection bias.
Non-Random MethodsQuasi-Alternation
• Dreadful method of forming groups.
• This is where participants are allocated to groups by month of birth or first letter of surname or some other approach.
• Can lead to bias in own right as well as potentially being subverted.
Example: Quasi alternation
• “Before mailing, recipients were randomized by rearranging them in alphabetical order according to the first name of each person. The first 250 received one scratch ticket for a lottery conducted by the Norwegian Society for the Blind, the second 250 received two such scratch tickets, and the third 250 were promised two scratch tickets if they replied within one week.”(Finsen and Storeheier, Biomed Central 2006)
Randomisation
• Some text books typically suggest the use of random number tables or coin tossing to allocate participants.
What is wrong with that?
• Coin tossing there is no ‘audit’ trail. We have to take your word for what you did.
• Random number tables are ‘open’ and again we have to take the researcher’s word that they did what the said.
• A separate computer allocation system from a third party is best.
Simple or restricted?
• Simple allocation is best when the number of units to be allocated >50. Because it is always unpredictable it is difficult to subvert or sabotage.
• Restricted randomisation has statistical advantages for <50 (e.g., cluster trial of 20 schools/classes would benefit from using a restricted technique such as blocking or minimisation).
Will people subvert the allocation?
• Schulz [1] has described, anecdotally, a number of incidents of researchers subverting allocation by looking at sealed envelopes through x-ray lights.
• Researchers have confessed to breaking open filing cabinets to obtain the randomisation code.
• In a survey [2] of 25 researchers 4 admitted to keeping ‘a log’ of previous allocations to try and predict future allocations.
[1] Schulz JAMA 1995;274:1456.
[2] Brown et al. Stats in Medicine, 2005,24:3715.
Case Study
• Subversion is rarely reported for individual studies.
• One study where it has been reported was for a large, multicentred surgical trial.
• Participants were being randomised to 5+ centres using opaque, sequentially numbered, sealed envelopes.
Case-study (cont)
• After several hundred participants had been allocated the study statistician noticed that there was an imbalance in age.
• This age imbalance was occurring in 3 out of the 5 centres.
• Independently 3 clinical researchers were subverting the allocation.
Mean ages of groups
Clinician Experimental Control
All p < 0.01 59 63
1 p < 0.01 57 72
2 p < 0.001 33 69
3 p = 0.03 47 72
Kennedy & Grant. 1997;Controlled Clin Trials 18,3S,77-78S
Recent Blocked Trial
“This was a block randomised study (four patients
to each block) with separate randomisation at each of the three centres. Blocks of four cards were produced, each containing two cards marked with "nurse" and two marked with "house officer." Each card was placed into an opaque envelope and the envelope sealed. The block was shuffled and, after shuffling, was placed in a box.”
Kinley et al., BMJ 2002 325:1323.
What is wrong here?
Southampton Sheffield Doncaster
Doctor Nurse Doctor Nurse Doctor Nurse
500 511 308 319 118 118
Kinley et al., BMJ 325:1323.
Problem?
• If block randomisation of 4 were used then each centre should not be different by more than 2 patients in terms of group sizes.
• Two centres had a numerical disparity of 11. Either blocks of 4 were not used or the sequence was not followed.
Randomisation summary
• Very important to avoid possible problems of subversion:» Who do you trust? » Need independent allocation, third party, need
to be convincing.
Selection bias after randomisation
• Selection bias is avoided if ALL participants who are randomised are completely followed up.
• Often there is some attrition – after randomisation some refuse to continue to take part.
• Or some may refuse the intervention but can still be tracked – IMPORTANT to distinguish between these.
What is wrong here?
Random allocation
160 children 8 from
Each school
76 children allocated to control 76 allocated to intervention group
1 school 8 children withdrew
N = 17 children replaced following
discussion with teacher
Selection bias
• POSSIBLE selection bias has been introduced in this trial.
• About 10% of the sample have been non-randomly allocated to the interventions.
• What should have happened?
• The authors should have retained the 17 refusers in an intention to teach (treat) analysis.
Ascertainment bias
• This occurs when the person reporting the outcome can be biased.
• A particular problem when outcomes are not ‘objective’ and there is uncertainty as to whether an event has occurred.
• Example, of homeopathy study of histamine, showed an effect when researchers were not blind to the allocation but no effect when they were.
• Multiple sclerosis treatment appeared to be effective when clinicians unblinded but ineffective when blinded.
Avoiding ascertainment bias
• Post-tests should be administered by someone who is unaware of the group allocation. Pre-tests before allocation is made.
• Record searches (e.g., arrest records) should be done by a researcher blind to the participant’s group allocation.
Resentful Demoralisation
• This can occur when participants are randomised to treatment they do not want.
• This may lead to them reporting outcomes badly in ‘revenge’.
• This can lead to bias.
Resentful Demoralisation
• One solution is to use a participant preference design where only participants who are ‘indifferent’ to the treatment they receive are allocated.
• This should remove its effects.
Example of preference interaction
• SPRINTER was an RCT of treatments for neckpain.
• Two treatments: a Brief Intervention (1-2 sessions with a physio using CBT) vs usual care (5+ sessions).
• BEFORE randomisation we asked patients their treatment preference.
SPRINTER Preferences
• In SPRINTER preferences were mixed» 53% did not have a preference;» 16% wanted brief intervention;» 31% wanted usual care.
• ALL patients were randomised IRRESPECTIVE of their preference.
Patient Flow and 12 month Results - SPRINTER
-1 .0 07
In d iffe re n t
-2 .8 11
P re fe rB I
0 .567
P re fe rU C
B rie fIn te rve n tion
-3 .0 94
In d iffe re n t
-2 .1 42
P re fe rB I
-2 .75
P re fe r U C
U su a lC a re
Overall 12 month improvement-0.840
Overall 12 month improvement-2.825
SPRINTER Results
Examples of educational trials
• As part of a collaboration between educational researchers and trial methodologists we have completed two RCTs in education.
• We tried to conduct them according to health care standards (i.e., CONSORT).» Computers to teach spelling to children;» Incentives to retain adult learners in evening
classes.
Computer trial
• Since 1997 the Government has boasted that > £1.7 billion of taxpayers money has been put into equipping schools with computer technology. Very few RCTs have been undertaken to evaluate and substantiate this investment.
• We supported the largest RCT of computer technology ever undertaken in the UK.
Methodological challenges
• Contamination children in the control group may access the technology.» Deliver intervention on a lap top.
• Secure randomisation.» Randomisation done by York Trials Unit – series of blocked
allocation to match availability of laptops (pre-test data did not inform allocation).
• Avoiding ascertainment bias.» Post-tests given to all children in same room by invigilators blind
to allocation, test marked by marker blind to group allocation. Pre-tests done blindly.
• Resentful demoralisation.» Waiting list control – control children to get intervention at end of
term.
Methods
• Based in one school. Lack of funding meant that it couldn’t be extended elsewhere. All year 7 pupils were randomised to receive literacy instruction via computer or not.
• Sample size (n = 157) allowed us to have > 80% power to show 0.5 of effect size between groups. Sample size fixed we couldn’t increase it, if we had wanted.
• Waiting list design all pupils received intervention half at the beginning of term half later in the term (normal practice was to arbitrarily assign pupils).
7 5 va lid p re -tes ts7 1 va lid p os t-tes ts
6 8 va lid p re - & p os t-tes ts
IC T g rou pN = 7 9
N = 3 le ft s ch oo lb e fo re p re -tes t
7 0 va lid p re -tes ts6 7 va lid p os t-tes ts
6 3 va lid p re - an d p os t-tes ts
N o IC T g rou pN = 7 8
N = 1 le ft s ch oo lb e fo re p re -tes t
Y ear 7 P u p ilsN = 1 5 7
R an d om ised
Participant Flow in Trial
Results
Variable ICT GroupMeans(SD)
No ICT GroupMeans(SD)
AdjustedDifference
(Intervention – Control)(95% CI)
P value of adjusted
Difference
Spelling 38.57(18.00) 37.24
(18.33)
0.954 (-1.83, 3.74)
0.500
Reading 18.97(6.11) 20.38
(6.64)
-2.33 (-0.96, -3.71)
0.001
Brooks et al, Ed Studies 2006
Computers and Literacy
• In this study we found no benefit of using computers for teaching literacy skills.
• A larger recently completed trial in the USA (n = 512) found similar results – no benefit.
• > $90 million had been spent implementing Fast forword before it was found to be ineffective.
Rouse et al, 2004; NBER Working Paper Series, 10315.
Incentives
• Poor attendance to adult literacy and numeracy classes is a major problem.
• Recent government policy has offered either financial incentives to attend or financial penalties (reduction in benefit) if attendance is poor.
• Neither approach has been tested in an RCT.• With colleagues in Sheffield we undertook a
RCT of incentives.
Methodological challenges
• Incentive may lead to resentful demoralisation if we randomised individuals.» Cluster or class allocation of 28 classes.
• Small number of units of allocation may result in chance bias or numerical imbalance.» Used a minimisation algorithm to achieve
balance on observed variables.
Design
• Cluster or class based RCT.
• 28 classes were randomised to either receive £5 per class attended or no incentive.
• Outcome was number of sessions attended.
Variable Control(n = 70)
Intervention(n = 82)
Mean (SD) number of sessions attended
6.69(2.71)
5.28(2.79)
Mean (SD) post-test literacy scores 21.14
(8.84)19.01(8.68)
Significant reduction of about 1.5 sessions
(95% confidence interval 0.28, 2.79; p = 0.019) (adjusted for clustering, cluster size and pre-test literacy scores
Incentives
• Small £5 incentives are counter productive and reduce attendance.
• Larger incentives may work but we need to evaluate these in a large RCT with educational achievement as main outcome.
Differences between health and education
• Education trials are much easier and quicker.
• Pre-test variables strongly predict post-test, which usually isn’t the case in health care trials, consequently pre-test measurement is particularly useful to gain more statistical power.
Trial designs
• Unequal allocation» Good to use when resources limit us to the amount of
one intervention (e.g., mentoring sessions).• Cluster designs
» Avoiding contamination, care needs to be taken on using ITT and recruitment bias.
• Factorial designs» Two trials for the price of one – good value!
• Balanced design» Controls for Hawthorne effect (e.g., Intervention group
gets maths, control group gets English and 2 trials for the price of one).
Factorial Design
Computer teaching No computers and no assistants.
Computer teaching
Class room assistants Class room assistants
Challenges
• Follow-up – this is crucial to maximise this.• Schools are good children are there and apart
from (random?) absences most will be present for post-test
• Adults may need encouragement (e.g., possible incentives to complete follow-up) or statutory data collection (e.g., arrests) allows ITT to be completed.
• Convincing other researchers that RCTs are feasible and worth doing.
A few fallacies
• We can’t accurately measure an important variable (e.g., social class) that really predicts outcome – our trial will be biased.» No it won’t, good measurement of confounders may
be helpful but randomisation will cancel their effects out.
• Non compliance means randomisation is not possible.» Poor compliance will dilute the treatment effects,
however, using ITT analysis will give a public policy estimate of effect and other statistical techniques (e.g., CACE) may adjust for non-compliance.
Can trials overcome strong beliefs?
• Often not.
• RCT of 25 schools to be given an enhanced sex education programme – aim to reduce abortions among teenage women. New package cost £900 per teacher compared with £20 for conventional package.
Results
• Termination rate among intervention schools was higher (15.7 per 1,000 women – 95% CI -10.7 to 42.1, p = 0.26).
• Difference not statistically significant, but point estimate is a 14% increase risk of terminations.
What did the authors conclude?
• “High quality sex education should be continued, but to reduce unwanted pregancies complementary, longer term interventions that address socioeconomic inequalities and the influence of parents should be developed and rigorously evaluated”.
• How does this recommendation derive from the data?
Summary
• Trials have challenges – some simple mistakes can result in a trial losing its internal validity.
• Many of the problems of trials can be overcome with careful thought and preparation.