pcori methodology standards: academic curriculum · presentation of simulation results under a...

PCORI Methodology Standards:

Academic Curriculum

© 2016 Patient-Centered Outcomes Research Institute. All Rights Reserved.

Prepared and presented by Gary Rosner, ScD

Module 4: Planning an Adaptive

Clinical Trial

Category 9: Adaptive and Bayesian Trial Designs

Adaptive trials are complex

Adaptive designs require more planning and prespecification of details than

standard trials

• Extra planning represents a cost associated with designing an adaptive trial

• Benefits often outweigh the costs

Plans will include:

• Patient registration and randomization system

• Data management system

• Drug procurement and dispensing (if appropriate)

• Interim and final analyses

• Evaluation of statistical properties of the design

Involve all key trial leaders and stakeholders in planning process

Planning an Adaptive or Bayesian Clinical Trial

3

Prior specification of details allows for complete evaluation of the trial’s design by key

stakeholders

Complete description of the study’s design, assumptions, and decision making help

ensure the validity and the credibility of the study and its results to the scientific

community and key stakeholders

Presentation of simulation results under a broad range of scenarios provides key

information about potential errors and misstatements (e.g., false claims of treatment

differences or overestimates of treatment effects)

Rationale for This Standard (AT-1): Specification

and Planning of an Adaptive Clinical Trial

4

Meet with trial leaders and key stakeholders to discuss study’s goals, hypotheses, etc.

Prepare documents and discuss with key stakeholders throughout planning process for

feedback

Draft study’s design

• Why is it adaptive?

• How is it adaptive?

• How do planned adaptations address study goals?

Simulate initial design under several scenarios for feedback

Iterate with stakeholders

Completely specify the final design

Write detailed report of the statistical aspects of final design

This may be separate from the trial protocol, although elements are in protocol

Planning Process Includes Communicating and

Vetting Design With Key Stakeholders

5

Key considerations in planning any clinical trial include:

Prospective planning with key stakeholders

• Participating investigators, patient representatives, funding agency, etc.

Logistics:

• Registration, randomization, data management, treatment plan, follow-up, etc.

Statistical considerations:

• Sample size, plans for interim and final analyses, etc.

Considerations When Planning an Adaptive or Bayesian Clinical Trial

6

The aforementioned considerations are even more critical for adaptive and Bayesian

trials

Trial designs are more complex

• Flexibility is planned and designed before the study even starts

Greater transparency is needed to allow others to understand the trial

• What assumptions underlie the statistical considerations?

• What methods will the study team use for interim and final analyses?

• What is the infrastructure supporting the trial?

• How does the design respond under different circumstances?

Considerations When Planning an Adaptive or Bayesian Clinical Trial

7

Adaptive trials require more planning and consideration than many nonadaptive trials

What are the decision rules for altering randomization probabilities and possibly

dropping arms?

How will the design respond to different circumstances?

• Simulations under different scenarios reflecting possible treatment-specific

outcomes

What prior data and expert opinions went into the study’s assumptions?

What are the prior distributions underlying the Bayesian computations?

On which statistical or mathematical models are the design and analyses based?

What infrastructure is in place to collect trial data and revise randomization

probabilities?

Extra planning and consideration mean extra work!

Planning Requirements for Adaptive Clinical Trials

8

Discuss with key stakeholders:

The goals of the trial:

• What do we want to learn?

• Is new treatment better?

• What is the best dose?

Disease under study

Treatment(s) evaluated

Endpoints for the study

Constraints to consider in the design:

• Financial

• Maximum sample size

What will constitute a successful trial and will constitute an unsuccessful trial

What the next steps will be as a result of this trial’s outcomes

Initial Planning: Engage Stakeholders

9

Trial results and populations that will be used in determining each adaptation

Whose data are part of the calculations, and whose data are not?

How will algorithms that generate randomization probabilities account for patients

in active follow-up if we must wait for endpoints?

Statistical models that are part of the analysis and affect interim and final decisions

Any intermediate endpoints informing adaptation via predictions?

Hierarchical models to borrow strength across subgroups?

Statistical properties of the proposed adaptive design

Describe distributions of key elements of the trial (e.g., summary statistics)

• Sample sizes (total and by treatment arm)

• Numbers of patients treated on inferior treatment arms

• Number of times the design picks the best treatments

• Etc.

Key Elements for Evaluating Adaptive Designs

10

May present results as tables or graphs

Presentation of Simulation Results (Operating Characteristics)

11

Scenario Trt 1,

Low Dose

Trt 2,

Low Dose

Trt 1,

High Dose

Trt 2,

High Dose

Standard

of Care

Null (median*,

tox risk) (6 mo, 5%) (6 mo, 5%) (6 mo, 5%) (6 mo, 5%) (6 mo, 5%)

NAVG 3.9 4.6 22.4 21.4 24.3

Prob Select 0.009 0.019 0.285 0.298 0.354

Treatment 2 at higher dose is best


12

Scenario Trt 1,

Low Dose

Trt 2,

Low Dose

Trt 1,

High Dose

Trt 2,

High Dose

Standard

of Care

Trt 2,

High Dose 2

(median*,


NAVG 4.0 4.4 16.6 35.0 17.7

Prob Select 0.011 0.016 0.033 0.862 0.051

Treatment 1 (high dose) and treatment 2 (low and high dose) too toxic


13

Scenario Trt 1,

Low Dose

Trt 2,

Low Dose

Trt 1,

High Dose

Trt 2,

High Dose

Standard

of Care

Trt 1,

Low Dose OK

(median*,


Trt 2 Too

Toxic NAVG

26.3 6.8 7.7 0.4 30.9

Prob

Select

0.336 0.031 0.028 0.004 0.572

Study’s design, with enough detail to allow someone else to implement it on the basis

of description

Specification of the design and documentation in the protocol before enrollment

begins

Prior specification is a prerequisite for valid and meaningful evaluation of an

adaptive design

What Information Needs to Be Included in the Study Documents?

14

Statistical aspects of design, possibly separate from protocol in all but the simplest

cases

All necessary detail about planned interim and final analyses

• Can be quite extensive!


15

All potential adaptations

What may change?

When may things change?

What criteria do the decision rules for change consider?

All possible adaptations should be described before the study begins!


16

You will often want a separate report that provides details about the adaptive design,

such as:

Description of the adaptive trial structure

All potential adaptations and the trial results and populations that inform each

adaptation

Statistical models used for decisions and adaptations in the adaptive design

• Including calculation details, software used, etc.

Statistical models and thresholds for the primary analyses and key analyses

• Including any calculation details or software used

Adaptive Design Report

17

Additional details to include in report:

Operating characteristics for the design (e.g., based on simulation)

Example simulated trials to illustrate the behavior of adaptive algorithms

Mode of calculating operating characteristics:

• If by simulation, assumptions used in simulations

• Assumptions about accrual rate, dropout rates, and time-to-event information

• Methods and algorithms underlying the creation of virtual participants

• Any other assumptions used for the simulation of trials

Adaptive Design Report

18

Before the trial opens to accrual, evaluate the design’s statistical properties fully

More complex designs require more intensive evaluation

• Bayesian designs, in particular, often require computer simulations to

understand and evaluate the trial’s statistical properties

Adaptive trials’ statistical considerations include more than simply Type 1 error

control, power, and sample size

Evaluation of trial may consider summary statistics for:

• Sample size for the entire trial

• Number (or percentage) of patients treated on superior and inferior treatments

• Number (or percentage) of patients experiencing clinical benefit by treatment

arm

Estimated trial properties are based on simulations

• Estimates have statistical properties of a large set of independent observations

• Number of simulations provide precise estimates of trial’s frequentist properties

Statistical Properties of the Trial

19

You want evaluation of these considerations under different scenarios to evaluate

trial design

How do the design parameters affect the trial’s statistical considerations?

Does the study adapt quickly enough, too slowly, or just right to challenges?

What is the effect of changes to the accrual rate, dropout rate, data delays,

violations of assumptions, missing data, time-dependent changes in patient

characteristics, etc.?

Statistical Properties of the Trial

20

When will interim analyses that may lead to adaptations occur?

Timing of these analyses may be based on:

• Numbers of patients enrolled

• Numbers of patients passing some evaluation milestone

• Numbers of events

• Calendar time from the start of the study

What Information to Provide About Adaptive Aspects of the Design?

21

What are the adaptive decision criteria?

• Posterior probabilities?

• Predictive probabilities?

Give specific rules

• For example, drop a treatment if the posterior probability that it provides

minimal benefit is <10%

What Information to Provide About Adaptive Aspects of the Design?

22

What is the study population analyzed for the adaptive randomization algorithm?

Provide a clear description of the study population informing potential adaptations

• Intention-to-treat population?

• Evaluable patients?

• Only patients with complete data at the time of the interim analysis?

What statistical models inform adaptive decision making?

Provide details about the models and implicit or explicit assumptions

• Parametric or semiparametric longitudinal data models?

Information to Provide About Adaptive Aspects of the Design

23

Adaptive trials are inherently flexible

By definition, design elements may change during the course of the trial in response

to accumulating data

Decision rules for possible adaptations need to be stated up front

• Promotes transparency

• Removes investigators from the controls

• Allows stakeholders to evaluate characteristics of the study

• Promotes credibility, reproducibility, and validation of results by others

Built-in flexibility has to be planned before the trial opens to patient enrollment

Changes cannot be or appear to be ad hoc

Flexibility and Transparency

24

Adaptive trials require more planning than most standard designs

Describe all possible adaptations within a design before the trial begins

Adaptive designs do not simply allow ad hoc adaptations during the course of the

trial

Prespecified adaptations are part of the protocol, not subsequent protocol

amendments

Communicate and vet the trial with key stakeholders while planning the trial and when

final

Promotes credibility, reproducibility, and validation of results by others

Summary

25