Projecting ‘Time-to-event’ Outcomes in

Technology Assessment: an Alternative Paradigm Adrian Bagust

CHE Economic Evaluation seminar 13th February 2014

1

Context

LRiG is one of nine independent multi-disciplinary

academic research groups providing evidence

assessment for NICE technology appraisals

Reliant on drug manufacturer for clinical evidence

– usually 1 or 2 key RCTs for STA topic

Most trials close early for commercial reasons

Access to trial data restricted to summaries and

publications – full patient data withheld.

2

Problem, Objective & Focus

TTE / Survival outcome estimation is often the

major source of uncertainty in NICE appraisals

Problem: How to estimate life time survival gains

from incomplete/immature trial data?

Objective: To estimate the expected mean survival

beyond the available trial data (Kaplan-Meier)

Focus: Appraisal of interventions for (mainly)

advanced/metastatic cancers

Survival Analysis: Kaplan-Meier

Basic data comes from Kaplan-Meier analysis of

observed events

K-M is a non-parametric technique, which

accumulates risk per unit of time between events

K-M gives unbiased estimates of survival vs time

provided any censoring is uninformative

Mean survival can be estimated as the area under

the curve (AUC) of the K-M survival plot

3

4

Censoring matters

Censoring

at last

observation

biases end

of the curve

and leads to

misfitting

parametric

functions

5

‘Standard’ method for projective modelling

Fit a limited set of ‘simple’ functions to the whole

trial data set (i.e. normal, exponential, Weibull,

Gompertz, logistic, gamma, log-normal, log-logistic,

extreme value)

Select ‘best fit’ function based on AIC / BIC scores

Apply selected function to model whole period

Often use a single model to represent both trial

arms, with treatment as a covariate

“For every complex problem there is an answer that is clear, simple and wrong.”

H.L Mencken

6

Problems with the ‘Standard’ method (1)

Essentially descriptive – not clear whether a good

description of known data will give reliable

estimates of unknown future events (projective)

Mechanistic process – is the selected function

suitable/appropriate? Is there causal logic?

No account taken of trial design (inclusion/exclusion

criteria, drug kinetics/dynamics, drug

response/resistance, monitoring protocols)

7

Problems with the ‘Standard’ method (2)

All standard functions are well-behaved smooth

continuous formulations to describe risk varying

over time according to a single mechanism

Clinical trials are designed to induce changes in

risk trajectories over time: treatment is introduced,

achieves full efficacy, loses efficacy, another

treatment may be offered, palliative care

8

Problems with the ‘Standard’ method (3)

“AIC can tell nothing about the quality of the

model in an absolute sense. If all candidate

models fit poorly, AIC will not give any warning of

that.” Wikipedia

Projection with standard functions can be highly

sensitive to the choice of model despite minimal

differences in ‘fit’ scores.

Standard functions give very different results

9

0.00

0.25

0.50

0.75

1.00

0 5 10 15 20 25

Years from randomization

PF

S

K-M data

K-M confidence limits

Limit of mature data

Weibull model

Exponential model

Log-logistic model

Log-normal model

Gompertz model

Gamma model

10

LRiG’s distinctive approach

Primacy of experimental data over projections

Understand the trial and the context

Focus on the primary objective (beyond the trial)

Search for meaning - all effects have a cause

Hypothesis formulation and testing

Avoid preconceptions (no ‘painting by numbers’)

Realism - no effect, no cause (prove otherwise!)

Parsimony – KISS / Occam’s razor

Question everything…..

All models are wrong - Nature makes fools of us all!

11

The nature of clinical trials

Trials are about altering risk

Trials look for differences

Trial patients are selected for ‘success’

Few treatments work immediately

Few treatments work indefinitely

Few treatments work for everyone

Inclusion/exclusion criteria impact on survival

Trial populations change during the trial

12

Examining the data

H(t) vs t plot

shows long-term

parallel trends

more clearly than

Ln(H(t)) vs Ln(t) plot

Typical oncology trial

New treatment initiated on Day 1 and continues

either for a specific maximum duration, or until

patient condition worsens (progression)

Progression-free survival (PFS) = time to disease

progression or death from any cause

Overall survival (OS) = time to death from any cause

Post-progression survival (PPS)* = OS – PFS

PPS may involve several subsequent different

phases of treatment

* Not usually reported

13

Typical oncology trial - PFS

14

15

Typical oncology trial - OS

16

Typical oncology trial – OS Hazard

17

Typical oncology trial - PPS

18

Post-progression

survival

Frequently, after

progression there is no

difference between

treatments – a common

PPS fixed risk applies.

This corresponds to

parallel risks at the end

of the overall survival

plot.

19

Underlying question: how to fit multi-phase

convolution functions to empirical data

Exponential PFS Exponential PPS is

straightforward

PFS(t) = exp(– r1.t); PPS(t) = exp(– r2.t)

OS(t) = p * PFS(t) + (1 - p) * PFS(t) PPS(t)

= p * exp(– r1.t)

+ (1- p) * {r1.exp(– r1.t) – r2.exp(– r2.t)} / (r2 - r1),

where p = proportion of progression events which are fatal

No difference for PPS

Difference for PFS

Convolution of two

exponential functions

generates short-term

inflection in OS curve

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 100 200 300 400 500 600 700 800 900 1000

Days

PP

S

Y K-M data

X K-M data

Fitted joint exponential model Case study

20

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 100 200 300 400 500 600 700 800 900 1000

Days

Overa

ll S

urv

ival

Y K-M survival

Y exponential convolution model

X K-M survival

X exponential convolution model

21

Basic issue: multi-phase convolution functions

in usable form (fitting to empirical data) Other combinations are more difficult

e.g. Weibull Exponential

Representing Weibull as a mixture of exponentials is

promising:

“…any Weibull distribution with shape parameter less than 1

arises as a mixture of exponentials. Also the exponential

distribution itself arises as a mixture of Weibull distributions

with fixed shape parameter p, so long as p > 1.”

Jewell NP Mixtures of exponential distributions Annals of Statistics 1982, 10(2):

479-484

22

Heterogeneity & mixed models

T

T

T

Case Study: segmented model & PH assumption

23

Case Study: segmented model & PH assumption

24

25

Case Study: segmented model & PH assumption

HR = 1.00

HR = 1.71

HR = 1.34

26

Case Study: segmented model & PH assumption

Is projective modelling always necessary?

27

Conclusion We consider that the ‘standard method’ of projection

does not provide an adequate basis for secondary

analysis of RCT data and the projection of time-to-

event outcomes data to end of life, nor does it give

sufficient regard to the primacy of experimental data.

The alternative approach outlined moves away from a

limited mechanistic procedure, and avoids many of

its unwarranted assumptions.

We believe that modelling should pursue a scientific

approach based on observation, hypothesis

formulation and testing to identify relevant and

informative models.

We are seeking to develop further the analytical

methods to support this approach.

28

Discussion

29

30

An example: data hypothesis confirmation

Long-term project begun in 1997 on cost-

effectiveness in type 2 diabetes

After working with different models & methods, we

concluded that better understanding was required

of the natural history of the disease

How does mildly elevated blood glucose develop

until patients are dependent on insulin?

How do drugs affect this progression of disease?

31

Finding data…

We identified an historic clinical trial (Belfast Diet

Study), which looked only at the effect of

controlled diet – no drugs at all!

We made contact with the PI who agreed to give

access to detailed data on results for re-analysis

We analysed the changes in beta-cell function

(from HOMA model) to look for temporal trends,

stratifying by time to diet failure

Is there a combined trend to explain data?

32

Time-shifting cohorts shows 2-phase pattern

33

Lab research indicates mechanism

34 Topp B, et al A model of b-cell mass, insulin and glucose kinetics J Theor Biol 2000; 206:605-19