THE STEPPED-WEDGE CLUSTER RANDOMISED TRIAL:

RISKS OF BIAS AND LEVELS OF EVIDENCE

SECOND INTERNATIONAL CONFERENCE ON

STEPPED WEDGE TRIAL DESIGN

KARLA HEMMING

19TH MARCH, 2018; YORK

BRIEF HISTORY - THE FIRST SW-CRT

▶ Gambia Hepatitis Intervention Study (GHIS):

• Nation-wide trial of the HBV vaccine

• Vaccine programme rolled-out in a phased way (random) over 4 years (1986 to 1990)

• Outcome: Hepato Cellular Carcinoma (HCC) ascertained from a national cancer registry. 30 years follow-up

• “This was because instantaneous universal vaccination in the country was impossible for logistic and financial reasons.”

2

Shimakawa Y, Lemoine M, Mendy M, Njai HF, D'Alessandro U, Hall A, Thursz M, Njie R. Population-based interventions to reduce the public health

burden related with hepatitis B virus infection in the gambia, west Africa. Trop Med Health. 2014 Jun;42(2 Suppl):59-64.

The Gambia Hepatitis Intervention Study. The Gambia Hepatitis Study Group. Cancer Res. 1987 Nov 1;47(21):5782-7

WHAT HAS HAPPENED SINCE THE GAMBIA TRIAL

▶ Increasing uptake:

• Protocols

• Full trials

• Methodological research

▶ Funding supported by many

national funding streams such as:

• NIHR (UK)

• NIH (US)

3

77 17079787

Grayling MJ, Wason JM, Mander AP. Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features. Trials. 2017

Jan 21;18(1):33.

METHODOLOGICAL PROGRESS – QUICK OVERVIEW

▶ Sample size methodology:

• Hussey and Hughes

• Design Effects / Software

• Extended correlation structures

• Statistical efficiency

▶ Design and analysis:

• Incomplete designs / hybrid designs

• Model misspecification

• Methodological and quality of

reporting reviews

4

Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007 Feb;28(2):182-91.

Woertman W, de Hoop E, …Teerenstra S. Stepped wedge designs could reduce the required sample size in cluster randomized trials. J Clin Epidemiol. 2013 Jul;66(7):752-8.

Hooper R, Teerenstra S, de Hoop E, Eldridge S. Sample size calculation for stepped wedge and longitudinal cluster randomised trials. Stat Med. 2016 Nov 20;35(26):4718-4728.

Kasza J, Hemming K, Hooper R, Matthews J, Forbes AB Impact of non-uniform correlation structure on sample size and power in multiple-period cluster randomised trials.

Stat Methods Med Res. 2017

Copas AJ, Lewis JJ, Thompson JA, Davey C, Baio G, Hargreaves JR. Designing a stepped wedge trial: three main designs, carry-over effects and randomisation

approaches. Trials. 2015 Aug 17;16:352.

https://clusterrcts.shinyapps.io/rshinyapp/

SETTING THE SCENE

▶ The SW-CRT is the epitome of the pragmatic trial;

▶ Presents an unprecedented opportunity to increase the robustness with

which healthcare-policy interventions are evaluated.

▶ Questions:

• Under what circumstances is the estimate of treatment-effect from a SW-CRT is

as robust as that of a parallel-CRT?

• When should the SW-CRT be used in preference to a parallel CRT?

5

ISSUES TO CONSIDER

▶ How to design these studies so as those people (or data) that are

included are not selectively different between those who receive different

interventions/treatments (i.e. healthcare policies) – these are referred to

as selection biases.

▶ The other issue is how to analyse the data from this complicated design

so as to obtain a fair estimate of how the intervention is working – these

are referred to as analysis biases.

6

AN EXAMPLE OF A MORE RECENT TRIAL: THE EPOCH STUDY

▶ Evaluation of an evidence based integrated care pathway (ICP)

▶ Setting:

• 90 hospitals (15 geographical clusters); Emergency laparotomy

▶ Outcome:

• 90 day mortality

▶ Sample size:

• 27,500 patients

• 90% power to detect a change from 25% to 22%

▶ Routinely collected outcome

▶ Funded by UK NIHR

Pearse R, Pedan C, Bion J, Faiz O, Holt P, Girling A, et al. HS&DR - 12/5005/10. Enhanced Peri-Operative Care for High-risk patients (EPOCH) trial: a stepped wedge cluster

randomised trial of a quality improvement intervention for patients undergoing emergency laparotomy [protocol]. 2013. www.nets.nihr.ac.uk/projects/hsdr/12500510.

DEMOGRAPHICS OF SW-CRTS PUBLISHED TO DATE

Average number of clusters:

Median 17 [IQR: 8 to 38]

Average number of steps:

Median 4 [IQR: 2 to 6]

Average study duration (months):

Median 16 [IQR: 8 to 24]

8Martin J, Taljaard M, Girling A, Hemming K. Systematic review finds major deficiencies in sample size methodology and reporting for stepped-wedge cluster

randomised trials. BMJ Open. 2016 Feb 4;6(2)

WHEN IS THE SW-CRT BEING USED….

▶ When no other randomised study

design is feasible:

• Gambia study

▶ When the alternative might be the

parallel cluster randomised trial:

• EPOCH study

9Taljaard M, Hemming K, Shah L, Giraudeau B, Grimshaw JM, Weijer C. Inadequacy of ethical conduct and reporting of stepped wedge cluster randomized trials:

Results from a systematic review. Clin Trials. 2017 Aug;14(4):333-341.

WHY IS A RATIONALE IMPORTANT

▶ In the SW-CRT, this justification

includes:

• Cluster randomisation;

• Roll-out of the intervention to all

clusters;

• Staggered roll-out of the

intervention.

▶ Why this is important:

• Cluster randomisation increases the

sample size;

• Roll-out to all clusters might also

increase number of participants and

clusters exposed;

• Delayed exposure where

intervention works.

10

Reporting of Stepped-Wedge Cluster Randomised Trials: Extension of the CONSORT 2010 statement with explanation and elaboration

K Hemming, M Taljaard, JE McKenzie, R Hooper, A Copas, JA Thompson, M Dixon-Woods, A Aldcroft, A Doussau, M Grayling, C Kristunas, CE Goldstein, MK Campbell, A

Girling, S Eldridge, MJ Campbell, RJ Lilford, C Weijer, A Forbes, JM Grimshaw submitted to BMJ

RISKS OF BIAS

▶ The SW-CRT is more complicated in its design, analysis, and

implementation than the parallel CRT.

▶ Risks of bias in the SW-CRT may be higher than in a parallel CRT.

11

WHERE ON THE HIERARCHY OF EVIDENCE IS THE SW-CRT?

▶ Is there a distinction between the

level of evidence provided from:

• A parallel CRT;

• A SW-CRT?

▶ Why this is important:

• Inform when the parallel CRT is

preferential to the SW-CRT.

12

?

?

13

Consider the march of progress from the parallel CRT to the SW-CRT

Look at some possible risks of bias in turn

Weigh up the likelihood of these risks by design

Recommendations for when the SW-CRT should be used

MARCH OF PROGRESS

14

THE PARALLEL CLUSTER RANDOMISED TRIAL

▶ Typically have a small number of

clusters.

• Average 21 [IQR: 12-52]

▶ Finding of baseline imbalance is

not uncommon.

15

Rutterford C, Taljaard M, Dixon S, Copas A, Eldridge S. Reporting and methodological quality of sample size calculations in cluster randomized trials could be

improved: a review. J Clin Epidemiol. 2015 Jun;68(6):716-23.

BASELINE IMBALANCE

▶ Imbalance of key characteristics

between participants exposed to

intervention and control.

▶ Chance imbalance likely with a

small numbers of clusters.

▶ Imbalance due to selection /

recruitment biases likely when

recruitment takes place after

randomisation.

16IPuffer S, Torgerson D, Watson J. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ. 2003;327:785–789.

Perineal Assessment and Repair Longitudinal Study (PEARLS): a matched-pair cluster randomized trial Khaled M K Ismail, Christine Kettle, Sue E Macdonald, Sue Tohill, Peter W Thomas, Debra Bick BMC Med. 2013; 11:

209.

Baseline imbalance

MITIGATION STRATEGIES

▶ Imbalance due to selection /

recruitment biases likely when

recruitment takes place after

randomisation.

▶ Recruitment:

• Before randomisation

• Someone independent to study

• Complete enumeration

17

▶ Chance imbalance likely with a

small numbers of clusters.

▶ Restricted randomisation:

• For known confounders

• Not for unknown confounders

Ivers NM, Halperin IJ, Barnsley J, Grimshaw JM, Shah BR, Tu K, Upshur R, Zwarenstein M. Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials. 2012

Caille A, Kerry S, Tavernier E, Leyrat C, Eldridge S, Giraudeau B. Timeline cluster: a graphical tool to identify risk of bias in cluster randomised trials. BMJ. 2016

Systematic imbalance Chance imbalance

PARALLEL CRTS WITH BASELINE PERIOD

▶ Can adjust for baseline values (of

outcome).

▶ Less than ideal?

• Adjusted and unadjusted treatment

effects might be discordant.

• Statistical adjustment assumes a

common relationship between baseline

values and outcome in all clusters.

18

Adab P, Pallan MJ, Lancashire ER, Hemming K, Frew E, Barrett T, Bhopal R, Cade JE, Canaway A, Clarke JL, Daley A, Deeks JJ, Duda JL, Ekelund U, Gill P, Griffin

T, McGee E, Hurley K, Martin J, Parry J, Passmore S, Cheng KK. Effectiveness of a childhood obesity prevention programme delivered through schools, targeting 6

and 7 year olds: cluster randomised controlled trial (WAVES study). BMJ. 2018 Feb 7;360:k211

MARCH OF PROGRESS TO THE SW-CRT

▶ Includes within-cluster

comparison;

▶ Every cluster contributes to

control and intervention;

▶ Design should be less susceptible

(compared to a parallel CRT) to

risks of chance baseline

imbalance.

19Haines TP, Hemming K. Stepped-wedge cluster-randomised trials: level of evidence, feasibility and reporting. J Physiother. 2018 Jan;64(1):63-66.

TEMPORAL CONFOUNDING

▶ Observations under the control

condition occur systematically

earlier under the intervention

condition.

▶ Outcomes might change over

time.

▶ Time is therefore a potential

confounder in a SW-CRT.

20

TEMPORAL CONFOUNDING IN PRACTICE

▶ SW-CRTs typically run over an

extended duration (~ 2 years);

▶ External influences can affect

outcomes of interest;

▶ Adjustment (modelling) therefore

essential;

21

Temporal trend or effect of intervention?

N Engl J Med. 2016 Mar 17;374(11):1053-64.

Dreischulte T, Donnan P, Grant A, Hapca A, McCowan C, Guthrie B. Safer Prescribing--A Trial of Education, Informatics, and Financial Incentives. N Engl

J Med. 2016 Mar 17;374(11):1053-64.

POTENTIAL RISKS OF BIAS IN THE SW-CRT

22

WHAT ARE THE POTENTIAL RISKS FOR BIAS IN A SW-CRT?

▶ Look at several common risks:

• Temporal confounding

• Selection bias

• Within-cluster contamination

• Delayed treatment effects

23

RISK OF BIAS 1:TEMPORAL CONFOUNDING

Detailed Description Why this is important Mitigating strategies

Observations collected under the

control condition are, on average,

from an earlier calendar time

than observations collected under

the intervention condition.

Changes external to the trial may create

underlying secular trends. Where the same

participants are repeatedly assessed, their

health status might improve (or worsen)

over the study.

Because time is associated with both the

treatment condition and the outcome, it

means that time is a potential confounder.

Analysis and sample size

should allow for the

confounding effect of

time.

24

RISKS OF BIAS 2: SELECTION BIAS

Detailed Description Why this is important Mitigating strategies

SW-CRTs can take a complete

enumeration of the cluster, a

random sample of individuals, or

recruit participants into the trial.

Furthermore, participants might

be continuously recruited into

the trial as they present; or all

participants might be recruited at

the beginning of the trial.

Information on how observations were

sampled is important to elicit risks of

bias. Studies which take a complete

enumeration have lower risks of bias as

do studies which recruit all participants at

a fixed point in time before

randomisation has occurred; studies

which continuously recruit participants

have higher potential for identification

and recruitment biases.

Methods to reduce the

risk of bias include taking

a complete enumeration

of the entire cluster-

period; recruiting all

participants before

randomisation; or

recruiting by someone

independent to the

study.

25

RISK OF BIAS 3: WITHIN -CLUSTER CONTAMINATION

Detailed Description Why this is important Mitigating strategies

In SW-CRTs, some or all of the

clusters will be exposed to both

the control and intervention

conditions.

Participants might only be exposed

to one of the treatment

conditions, or both.

Where the duration of exposure is long, it

may be possible to sample the same

participants under the control condition and

later under the intervention condition.

In trials with long exposure

delayed assessment of

outcome should be

avoided to prevent

participants recruited

under the control

condition might later

become exposed to the

intervention condition.

26

RISK OF BIAS: DELAYED TREATMENT EFFECTS

Detailed Description Why this is important Mitigating strategies

Sometimes the effect of the

intervention is expected to

materialise immediately, and

sometimes there is a delay

before its effect will be realised.

When there is a delay before the effect of

the intervention is realised the estimate of

effectiveness can be attenuated.

Where there is expected

to be a delay before the

effect of the intervention

is materialised a

transition period can be

built into the design of

the study.

27

SUSCEPTIBILITY OF BIAS BY DESIGN

28

SUSCEPTIBILITY TO RISKS OF BIAS BY DESIGN

▶ At risk of biases due to:

• (Selection biases)

• Temporal trends

• (Within-cluster contamination)

• (Delayed treatment effects)

▶ Protected in part from:

• Chance imbalance

▶ At risk of biases due to:

• (Selection biases)

• Chance imbalance

▶ Protected from:

• Temporal trends

• (Within-cluster contamination)

• (Delayed treatment effects)

29

SW-CRT Parallel CRT

WHERE ON THE HIERARCHY OF EVIDENCE IS THE SW-CRT?

▶ CRTs with a large number of

clusters increase likelihood of:

• Internal validity (protected from

chance imbalance)

• External validity

30

?

?

Taljaard M, Teerenstra S, Ivers NM, Fergusson DA. Substantial risks associated with few clusters in cluster randomized and stepped wedge designs. Clin Trials. 2016 Aug;13(4):459-63

WHERE ON THE HIERARCHY OF EVIDENCE IS THE SW-CRT?

▶ CRTs and SW-CRTs which take a

complete enumeration or recruit

before randomisation will reduce

risk of selection biases:

• Routinely collected data; no patient

recruitment.

31

?

?

Caille A, Kerry S, Tavernier E, Leyrat C, Eldridge S, Giraudeau B. Timeline cluster: a graphical tool to identify risk of bias in cluster randomised trials. BMJ. 2016

WHERE ON THE HIERARCHY OF EVIDENCE IS THE SW-CRT?

▶ SW-CRTs with careful design can

be protected against within-cluster

contamination:

• Transition periods

• No delayed assessment of outcome

with long treatment exposure.

32

?

?

Copas AJ, Lewis JJ, Thompson JA, Davey C, Baio G, Hargreaves JR. Designing a wedge trial: three main designs, carry-over effects and randomisation approaches.

Trials. 2015 Aug 17;16:352.

WHERE ON THE HIERARCHY OF EVIDENCE IS THE SW-CRT?

▶ SW-CRTs and CRTs with

baseline adjustment can be

protected against chance

imbalance:

• Adjust for temporal trends

33

?

?

RECOMMENDATIONSWHEN A SW-CRT IS PREFERABLE TO A PARALLEL CRT

34

LOGISTIC CONSTRAINTS (THE GAMBIA TRIAL)

▶ When a SW-CRT will almost certainly be preferable:

• When the intervention will be rolled-out irrespective of any evaluation (CRT not

feasible);

• When logistical issues mean that the intervention must be phased into practice.

35

SMALL NUMBER OF CLUSTERS

▶ When there are only a small number of clusters the parallel CRT will be

at risk of bias due to chance imbalance:

• Add baseline assessment of outcome;

• Use restricted randomisation;

▶ Consider a march of progress to a SW-CRT.

36

LARGE NUMBER OF CLUSTERS

▶ When there are a large number of clusters:

• Important to consider the susceptibility of design to bias;

• Some risks of biases can be minimised.

▶ Consider a parallel CRT to avoid over reliance on a model based

analysis.

37

CONCLUSIONS

▶ The SW-CRT is the epitome of the pragmatic trial;

▶ Presents an unprecedented opportunity to increase the robustness with

which healthcare-policy interventions are evaluated.

▶ Questions:

• Under what circumstances is the estimate of treatment-effect from a SW-CRT is

as robust as that of a parallel-CRT?

- When there are only a small number of clusters

• When should the SW-CRT be used in preference to a parallel CRT?

- When a parallel trial is not feasible

38

40

GLOSSARY OF TERMS

Term Explanation Cluster The unit of randomisation.

Cluster-period A grouping of observations by time of measurement and cluster.

StepA planned point at which a cluster or group of clusters crosses from control to intervention.

Period A grouping of observations by time of measurement. Duration of period Time (e.g. months) between each step.

Sequence of treatments (often abbreviated to sequence or allocated sequence)*

A sequence of codes defining the order of implementation of the treatment conditions for each cluster. More than one cluster can be allocated to each sequence.

Intervention condition* The treatment under evaluation. Control condition The comparator treatment.

Transition periodThe time needed to fully embed the intervention. A transition period may have the same or different duration than a measurement period.

ParticipantA participant is someone on whom investigators seek to measure the outcome of interest.

Research participantA research participant denotes a human research subject from the standpoint of ethical considerations.