sweitzer,simulating multi phase studies

Signs of the Timings: Predicting Time of Completion in

Multiphase Survival Trials

Dennis Sweitzer Ali Falahati

Delaware Chapter of the ASA September, 2006

Study Flowchart

The Protocols Outcome: •  Time to Randomized Relapse Open Label Phase

–  Up to 36 weeks –  Patients must be stable for 12 weeks before randomization –  High withdrawal rate (30-70%) –  Assumed 50% randomize

Randomized Phase –  Up to 104 weeks –  High withdrawal rate –  Assumed 30% Relapse rate –  Trial could not end until last Patient randomized >28 weeks

Sensitivity to Relapse & Discontinuation rates (1)

Cumulative Patient statuses as trial progresses 100 Relapse ~Sep

Wrong assumptions, wait longer

Low Discontinuation relative to Relapse

Sensitivity to Relapse & Discontinuation rates (2)

Higher event rates deplete patient pool

Plan to stop enrollment as soon as certain of reaching 100

~ July

Higher Discontinuation Rate, Lower relapse Rate

Large delays May never reach goal

Stopping Criteria • At least 227 Relapses • All patients still in Randomized Phase complete at least 28 weeks of treatment

Ideally: •  227th Relapses occurs shortly after:

• All patients randomized >28 wks (Per Protocol) • Randomization closed when:

• All enrolled patients randomize or discontinue

28 week Requirement later dropped (Protocol Amendment) Presentation: use 200 Relapses

Stopping Enrollment

The Problems Long Lead times •  Up to 36 weeks before randomization •  Plus 28 weeks Minimum randomization Ideally: Stop enrollment 64 weeks before target

#Relapses Must account for •  Enrollment D/C (30%-70%) •  Randomized D/C (D/C Rate ≈ Relapse Rate) •  Relapse Rates vary (Higher Relapse Rate Early) •  Competing Relapses (D/C vs Relapse) •  Sensitivity to rates (Close Rates High Variability)

Stopping Enrollment: Issues Too Early: Fewer Patients Fewer randomized Longer wait for

target #Relapse. May never reach

target #Relapses

Too Late:  Higher certainty of

reaching Goal Patients possibly in

Open Label at End Excessive #Relapses at

end of study Ethics of Randomizing

Excess # pts

Many Management Questions When do we stop enrollment while being sure of

eventually getting target # Relapses? When can we stop randomization & ensure reaching the

target? Whats the earliest and latest we can expect to reach the target?

When will all pts be randomized >28wks? When can the trial be halted (required # Relapses & all pts randomized >28 wks)

Estimated Randomization Rate? Estimated Relapse Rate?

How many active patients at the end? How well does the outcome match our assumptions

etc etc

Simulation Solution •  Make a stochastic model of the trial •  Monthly:

–  Base model parameters on blinded data observed to date –  Incorporate assumptions where data insufficient –  Incorporate uncertainty of parameters –  Execute 1000’s of simulations of the trial –  Compute statistics from the collection of simulated trials –  Repeat with new data

Advantages

Transparency •  Modeling assumptions can be:

Specified -- Graphed -- Debated

Data Driven •  New Data updates the model •  Existing Active Patients are simulated to end •  Assumptions become less important as data

accumulates

Vision of Output

•  Simulation reports varied according to changing team needs (how many open label patients on June 1? When will we reach 150 Relapses? How many randomized patients at time of 200th Relapse? If we stop randomizing on May 15, how many open lable patiestin will there be?……………………………………………….

Stochastic Modeling Approach

1.  Make a cartoon model of a patients progress through the trial

2.  What final outcomes are possible?

3.  What could happen to the patient?

4.  Identify States through which a patient passes

5.  Identify Random Processes which take patients between states

Stochastic Model

Enrolling Patients

Open Label Patients

Discontinued Patients (Open

Label Phase)

Relapses

Randomized Patients

Discontinued Patients

(Randomized Phase)

Continuous Time Markov Chain

Markov States: the Bubbles Transitions: the Arrows

Transition Probabilities change with time in state

States

2 Transitory Markov States: ❶ Open Label Phase Randomized Phase 3 Terminal Markov States: ❸ Discontinued from Open Label Phase Discontinued from Randomized Phase Randomized Relapses

Enrolling Patients

Open Label Patients

Discontinued Patients (Open Label

Phase)

Relapses

Randomized Patients

Discontinued Patients (Randomized Phase)

❶)

)

❸))

)

Transition Processes

5 Random Transition Processes: 1.  Trial Enrollment (Start Open Label) 2.  Discontinuation from Open Label Phase 3.  Randomization (from Open Label Phase) 4.  Discontinuation from Randomized Phase 5.  Randomized Relapses

Enrolling Patients

Open Label Patients

Discontinued Patients (Open Label Phase)

Relapses

Randomized Patients

Discontinued Patients (Randomized Phase) ❶)

)

❸))

)

Trial Enrollment

For each simulated patient, generate a random length of time since the last patient

•  Pick an enrollment rate λ (Based on history & judgment) •  Assume: #pts/mo ~ Poisson process with mean λ •  Time between patients ~ Exponential(1/λ) Can expand enrollment model to evaluate management

options: •  Incorporate mixture of site performances •  Adding/changing sites during the trial

Enrolling Patients

Open Label Patients

❶)

Markov Process )

)Continuing

Discontinuation

Relapse (or Randomization)

Probability of transitioning from

state i to state j between times s and t

•  Aalen-Johansen estimator of the transition probability matrices For and

# obs. Direct transitions from states h to j, visits 1 to t

# pts in state h, just prior to visit t

Aalen-Johansen estimator of Transition Probabilities

Aalen-Johansen & Kaplan-Meier

•  Generalization of Kaplan-Meier Estimation to Non-homogeneous Markov Chains

•  K-M Estimators easier: – To program (already in SAS) – To understand (Intuitive) – To Explain (Familiar)

•  Enrollment: Poisson Process •  Open Label Phase: Competing Risk Model

•  Ramdomized Phase: Competing Risk Model

Models

0= Still in OL Phase 1= Randomized 2= Discontinue fr OL phase

0= Still in Rand Phase 1= Manic event 2= Depressed event 3= Discontinue fr Rand phase

Competing Risk Model Mutually exclusive events

(e.g., Relapse vs Discontinuation, …) 2 Approaches (Pintilie, 2006) •  Jointly distributed Random Variables •  Latent failure times

–  Assume both events eventually occur –  But we only observe the first –  Use only marginal distributions –  Assuming independence (between events) –  But cannot test for independence, if only observing 1st

–  Independence: Face validity & Simplest Assumption

Kaplan-Meier Simulation

•  Assume event are independent •  Model Each process separately using Kaplan-

Meier Estimators •  Censor on other event, current time in trial •  Simulate each event separately •  Earliest of the 2 simulated processes is taken as

simulated outcome •  Caveat: Assumes Independent processes

Intuitive, easy to understand, easy to explain

Open Label Transitions 2 Competing

Processes: Discontinuation ❸ Randomization

1.  Generate Random Discontinuation time 2.  Generate Random Randomization time 3.  Use the earliest event

Open Label Patients

Discontinued Patients (Open Label Phase)

Randomized Patients

)

❸)

Randomized Phase Processes

2 Competing Processes: Discontinuation * * Relapse

Choose event as previously described. •  Current Open Label Patients are simulated to

randomization or discontinuation •  If simulated randomization, then simulate

Randomized Discontinuation or Relapse

Relapses

Randomized Patients

Discontinued Patients (Randomized Phase) )

)

Generic Transition Process Q: When to make the transition? A: First: estimate random transition

function

State "A"

State "B"

1.  Generate K-M Survival Functions from data (censoring on all other events)

2.  Make assumptions about Survival beyond last event

ŒŽ

?

Simulated Patients

A: Second: Simulate Trials For Each Simulated Trial: •  For each simulated patient within a trial

–  Pick a random p∈(0,1)

–  Interpolate t from the graph, so that (p, t) is on graph

State "A"

State "B"

Q: When to make the transition?

p

t

(p, t)

Simulating Active Patients

For each simulated trial •  For each observed patient within state “A” for time s

–  Interpolate q∈(0,1) from the graph, so that (q, s) is on graph –  Pick a random p∈(0,1)

–  Interpolate t from the graph, so that (q*p, t) is on graph

State "A"

State "B" q

s

q*p

t

(q, s) (q*p, t)

q r

t

Incorporating Parameter Uncertainty State "A"

State "B"

For each simulated trial

• Pick a random quantile r∈(0,1)

• Simulate all patients using the r%-tile confidence level of the Kaplan-Meier Curve

Simulates: combinations of high & low estimates of Event and D/C Survival curves

Limitations Requires representative data from all phases •  K-M estimates only through last event •  Assumptions must be made about hazard rate after last

available event(s) –  If assumptions correct, point estimates should be stable while

confidence intervals narrow •  Up to date data

–  Special reporting of Relapses (faxes with follow up, monitoring)

–  IVRS, EDC, monitoring reports •  Heterogeneity:

–  Earliest sites may not be representative of all sites –  Procedures may change (hopefully improve) over time –  Regional differences (standards of care, patient attitudes, etc)

Why Not a Parametric Model? Trial Structure: •  Events tend to occur

on visits granularity ✭ Continuous

•  Visits vary in spacing ✭ Discrete

•  Active Tx mixture model Changing Hazard over time

•  Must make & defend simplifying assumptions

Diagnostic: Does It Fit?

Survival curves of: Observed data vs. Simulated Data

(Censored Observed, Active OL Pts, Active Rand. Pts, Entirely simulated Pts)

Diagnostics

Plot K-M curves for each event, time in each phase •  Review assumptions (long term behavior) •  Identify data anomalies •  Identify simulation problems

Example: Regional Heterogeneity Regional modeling (Trials A & B):

Parameters varied by region more than by trial – Estimate parameters within regions – Simulate patients with Trial and Region – Summarize results by Trial

In addition to simulations which ignored region

Survival curves followed 2 patterns by

region & trial

Reporting the Simulations For each simulated Trial: •  Sort Patient Events by occurrence date (enrollment,

randomization, relapse, etc) For each scenario •  Summarize over Event records which fit scenario Examples: •  Summarize over all patients enrolled before a potential

enrollment cutoff date. •  … over all patients randomized before a cutoff date •  Summarize with and without a subset of sites

Changing Questions •  Early in Trial

–  Are the protocol assumptions accurate? –  When to stop enrollment? –  Expected # patients (enrolled, randomized, etc)

Identify problems & Evaluate fixes •  Mid-Trial

–  When to stop randomizing patients? –  Are the revised assumptions accurate? –  Were changes effective?

•  Late Trial –  When will the last Relapse occur? –  How many patients will be active in various phases?

Plan for Closeout & Database lock

Early Trial For each Enrollment Cutoff Date •  Summarize each trial for all patients enrolled

before that date •  Compute statistics over simulated trials Some trial outcomes: •  Dates of: last Relapse; All patients randomized>28 week;

PP Completion (>target & >28 weeks) •  Event & patient counts at each of above milestones •  % of Simulations with ≥200, 190, 180,… Relapses at milestones •  # active patients (open label or randomized) at given dates &

milestones Pick a cutoff date accordingly (e.g., minimize resource with

least risk of running late)

Trial Completion (1) ≥227 Relapses & All Active Patients Randomized 28wks

Earliest Completion:

• 75% Certainty:

2 August Cutoff for Nov 2006 Completion

• 90% Certainty:

1 Sep Cutoff for Dec 2006 Completion

Trial Completion (2) ≥227 Relapses Events & All Active Patients Randomized 28wks

Earliest Completion:

• 75% Certainty:

~1775 Enrolled for Nov 2006 Completion

• 90% Certainty:

~1850 Enrolled for Dec 2006 Completion

29 Oct prediction from: 528 Enrolled, 272 Rand., & 58 Relapses On 26 June: ~1950 Enrolled, 620 Randomized, 200 Relapses

Highly uncertain: 3 Relapses, 2 Randomized D/C, 73 Randomized

1 Month Later: 16 Relapses, 7 Rand.D/Cs, 182 Randomized

Slowed Enrollment: 29 Relapses, 22 Rand. D/C, 272 Rand

Example: Will a trial end? Study E: •  Endpoint: 300 type 1, 300 type 2 events •  Slower & Fewer than expected •  Simulation predicted:

–  10% chance of 300 of each –  87% chance of 600 total

•  Interim Analysis – Supported by simulation –  300 total expected April 25

Mid-Trial Output For each Randomization Cutoff Date •  Summarize each trial for all patients

randomized before that date •  Compute statistics over simulated trials •  Generates same statistics per trial •  Summarize for each randomization cutoff date

Essentially, replace “enrollment” with “randomization” & execute as before

•  Conclusion: nothing to be gained by randomizing beyond mid June

Late-Trial Output Refine estimates of last Relapse, etc. For Milestones & Calendar Dates •  Estimate #patients in each stage (e.g., how many

patients will be active at the end?) Caveats: •  Corrected (or just collected) data may change

estimates •  Old, unreported Relapses may be discovered Useful: •  Predict time between milestone to end •  Add prediction to best guess milestone

Bottom line: How accurate? Not bad:

• Actual Date of 200th Relapse covered by predicted 80% C.I.

• Width of C.I.s narrowed over time

Value Added Early Refinement of Protocol Assumptions •  Protocol: 50% randomized, 30% Relapse rates •  Trial A: 33%, 37% Trial B: 55%, 41% Early Identification of problems Quick Response to problems •  Changed procedures to improve retention in Trial A •  Added sites to Trial B after delays in starting up sites

Better allocation of resources

Trials C, D Mid trial: Regulators requested analysis of late

Relapses •  Enrollment had already ended for Trial C

– Enough patients to reach 88 late Relapses? •  Enrollment was still ongoing for Trial D

– Extend enrollment how long? Add sites?

•  How would this affect time lines?

Data problem: Unreported Relapses

Dirty Data Problems •  Some known Relapses are not usable due to missing data

(e.g., unknown randomization date) •  Corrected (or lately collected) data may change estimates •  Old, unreported Relapses may be discovered •  Data may be collected or corrected irregularly

–  Separate data sources (e.g., Relapse log + IVRS) –  Drift & shift over time –  End of trial data clean up

Solutions: •  Estimate time between milestones

–  Anchor to a known, early milestone Future solutions: •  Estimate missing data effects

–  Use time between Relapse occurrence and Relapse reporting –  Estimate number of missing Relapses from times in past

Some Feedback •  “… the simulations are very valuable

and the only way we have to plan our timelines. As it has turned out, your simulations seems to be pretty accurate as we have increased the mood event rate significantly … as predicted...”

•  ... We would have been guessing and spinning our wheels without them.”

•  Could you simulate trials xyz & uvw?

Extra Slides

Diagnostics: Cohort Analysis Cohorts: •  By month enrolled •  By month Randomized

Calculate randomization & Relapse rates

Easy to understand  Multiple Estimates which must be reconciled  Doesn’t Provide Time to Relapses  Useful reality check on Simulation Point Estimates insufficient: need C.I.s

Cohorts by Randomization Month

Actual counts of patients by status by month

Cohorts by Month Randomized (Cumulative Count)

Height is # patients randomized in or before month

Cohorts by Month Randomized (Cumulative Percentage)

•  >40% randomized eventually have Relapse  <N/0.40 randomized to get N

Open Label Cohorts (Current Status)

Open Label Cohorts (Cumulative Statuses)

• ~ 64% eventually Randomize & >40% Relapse Rate •  # open label pt < N/(0.64*0.40)

Solutions? Crude Relapse Rates of all Patients in Phase Mixture of patients: •  Relapse & D/C rates change with exposure •  Mixture of Pt. Exposures changes with time Cohorts: Track Relapses over Time Easy to understand  Multiple Estimates which must be reconciled  Doesn’t Provide Time to Relapses  Useful reality check on Simulation Point Estimates insufficient: need C.I.s

Clinical Trial Management Planning Trials (future) •  Is a trial feasible? •  Sensitivity to assumptions? •  Costs: # Pts, # Pt-mos, #visits, #Sites, #Site-mos Trial Execution (current) •  Anticipate delays •  No information on outcome •  Could be added to simulation Program Planning (future) •  Replace “Trial Phases” with “Toll Gates” •  Enhance modeling of Trial Enrollment process

Example: Adding Site

•  Drop OL Phase, expand enrollment process •  Simulate time to start up new site, pts/mo at a new

site, etc •  Report by #Additional Sites instead of cutoff dates

Sites

New Sites

Relapses

Randomized Patients

Discontinued Patients