james hung, 2004 fda/industry management of missing data in clinical trials from a regulatory...

James Hung, 2004 FDA/Industry

Management of Missing Data in Clinical Trials from a Regulatory Perspective

H.M. James HungDiv. of Biometrics I, OB/OPaSS/CDER/FDA

Presented in FDA/Industry Workshop, Bethesda, Maryland, September 23, 2004


Collaborators

Charles Anello, Yeh-Fong Chen, Kun Jin, Fanhui Kong, Kooros Mahjoob, Robert O’Neill, Ohid Siddiqui Office of Biostatistics, OPaSS, CDERFood and Drug Administration


DisclaimerThe views expressed in this presentationare not necessarily of the U.S. Food andDrug Administration.

Acknowledgment

O’Neill (2003, 2004)Temple (1994-2004)


Outline

• Informative dropout• Statistical analysis methods• Methodology consideration• Summary


Clinical trial focuses on intent-to-treat population (including completers and dropouts)

Response variables often measured over time (e.g., at multiple clinic or hospital visits)


Often the main clinical hypothesis concerns the effect K of a test drug r.t. a control at some time K (e.g., end of study).

Statistical null hypothesis

H0: K = 0

i.e., allow nonzero at other time

points? (make sense?)


Unclear why testing only at the last time point is most relevant (for simplicity? avoid

statistical adjustment for testing multiple times?)

Drug effects over time are important information. e.g., inconceivable to market a drug that is effective only at Week 6, say.


For drug effect over time (or some period of time, e.g., at steady state), the relevant null hypothesis is

H0: 1 = ∙∙∙ = K = 0

or H0: slope difference = 0 (if

response follows straight-line model ) or others for relevant time period.


Informative DropoutIn many disease areas, dropout rate is high and the results of any analyses for ITT population is not interpretable because of a large amount of missing data, particularly when dropouts are ‘informative’.


Dropout problems are multi-dimensional e.g., dropping out due to multiple reasons: side effects of the drug, healthstate is worsening, unperceived benefit

Little knowledge of real causes of missing data, whether missing mechanism related to study outcome or treatment


Informative dropout has many different definitions, e.g., - dependent on observed data, dependent on missing data, treatment-related dropout, … - tied in with missing mechanism MCAR, MAR, MNAR, NIM, …

O’Neill (2003, 2004)


For regulatory consideration, any treatment related dropout may be a suspect of informative dropout and missing mechanism probably needs to be considered informative (i.e., may severely bias estimates and tests) unless proven otherwise.


In a clinical trial, each cohort of dropout by reason or by dropout time can be very small. Difficult or impossible to assess whether missing values are informative.


40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

Ou

tco

me

40

60

80

10

01

20

14

01

60

Week0 Week1 Week2 Week3 Week4

Ou

tco

me

40

60

80

10

01

20

14

01

60


Ou

tco

me

40

60

80

10

01

20

14

01

60


Ou

tco

me

40

60

80

10

01

20

14

01

60


Ou

tco

me

40

60

80

10

01

20

14

01

60


placebo patients' responses


40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

40

60

80

10

01

20

14

01

60

Ou

tco

me

40

60

80

10

01

20

14

01

60


drug patients' responses


Based on visual inspection, drug seems to perform better than Placebo.


40

60

80

100

120

140

160

Out

com

e M

easu

re

40

60

80

100

120

140

160


lack of effect

placebodrug


40

60

80

10

01

20

14

01

60

Ou

tco

me

Me

asu

re

40

60

80

10

01

20

14

01

60


withdraw consent

placebodrug


40

60

80

10

01

20

14

01

60

Ou

tco

me

Me

asu

re

40

60

80

10

01

20

14

01

60


insufficient response

placebodrug


40

60

80

10

01

20

14

01

60

Ou

tco

me

Me

asu

re

40

60

80

10

01

20

14

01

60


adverse events

placebodrug


Difficult to tell whether missing mechanism is ‘ignorable’ or not…e.g., in a linear response profile, MARMay be NIM.


40

60

80

10

01

20

14

01

60

Ou

tco

me

Me

asu

re

40

60

80

10

01

20

14

01

60


completers

placebodrug


1 1

1 12 2

2

33 3 3

4

4 4

45

56

66 6 6

Out

com

e M

easu

re

4060

8010

012

014

016

0


lack of effectwithdraw consentinsufficient responseadverse eventsprotocol violationcompleters

placebo group's mean by dropout reason


11

1

2

2 2

2

3 3 3

3

44

4

45

5

5 5

6

66

6 6

Ou

tco

me

Me

asu

re

40

60

80

10

01

20

14

01

60


lack of effectwithdraw consentinsufficient responseadverse eventsnon-compliancecompleters

drug group's mean by dropout reason


These plots show difficulty in classifying dropouts (informative or not) in individual trials where each cohort of dropout is small, (though total dropout rate could be high).

These types of analysis should be donewith external historical trials, at leastfor classification purpose.


Statistical Analysis Methods

Literature guidance1) No satisfactory statistical analysis

method for handling non-ignorable missing data

2) Likelihood-based methods require assumptions about missing data mechanism (unverifiable from current trial data)


Facts1)Validity of any analysis method is very much in question.

2) Better alternative method is unclear. Use of current trial data to seek imputation method is futile.

3) Dropouts and missing data are unavoidable.


Glimpse of the analysis problem

= µ1 - µ2 at last time pointni = # of completers in group ifi = ni/Ni

If there is no missing value, we have D = Y1 – Y2 (unbiased for ) V(D) = estimated variance of D Z = D/[V(D)]1/2

2,1),/,(~:sizeofmeansample 2 iNNYN iiii


Missing values { D , V(D), Z } not obtainable. Can try to get E( D | data) and V( D | data).and construct Z* = E( D | data ) / [V( D | data )]1/2

or Z+ = E( D | data ) / [V(D)]1/2


),(:dataobservedi,groupFor ioi RY

Yoi = sample mean of completers Ri = vector of indicators for completion or dropout Ymi = unobservable sample mean of dropouts

),|()1(

),|()1(

),,,|(

2222

1111

2211

2121

RYYEf

RYYEf

YfYf

RRYYDE

om

om

oo

oo


Immediately, when f1 ≠ f2, this statistichas problem of interpretation, unlessRi and Ymi are independent (MI).

Under MI, E(Ymi | Yoi, Ri ) = E(Ymi) .

And if E(Ymi) = µi, then completer

analysis might offer a reasonable estimate of .


When f1 = f2 = f,

)},,|(

),|(){1()(

),,,|(

222

11121

2121

RYYE

RYYEfYYf

RRYYDE

om

omoo

oo

a linear combination of obs sample mean difference of completers and difference in conditional mean of dropouts (the latter requires models).


What about Var (D | data)?

Another formidable task !

Nonlikelihood-based methods are difficult to provide useful solutions unless some kind of ad-hoc conservative imputation is feasible.


LOCF (last observation carried forward)

LOCF tests H0: K = 0.

LOCF can be biased either in favor oftest drug (e.g., when its effect decaysover time*) or against test drug, evenin case of MCAR.

*Siddique and Hung (2003)


For assessing drug effect over time, LOCF can seriously underestimatevariability of measurement and isunrealistic (i.e., impute a constant valuefor every visit after the patient droppedout).


LAO (last available observation)Operationally identical to LOCF, thistests some global drug effect over time, H0: w1hµ1h = w2hµ2h

Wih= E(dropout rate of drug group i at time h)

μih = expected response of patients dropping out

after time h in drug group i

Is this null hypothesis relevant?

Shao and Zhong (2003)


1

2

1 1

1 2 23 3

v0 v1 v2 v3

Y

LOCF versus LAO (in red)


The global mean µi = wihµih can beunbiasedly estimated by the sample mean. But the usual MSE from ANOVAmay not estimate right target (Shao andZhong).

LAO results can be difficult to interpretif dropout reasons or dropout rates aredifferent in treatment groups.


If drug effect over time is at issue,why not use all the pertinent data(longitudinal data analysis should be more efficient than LAO). - need medical colleagues’ buy in

Ex. Analysis of cuff BP over time may be more powerful (value of test statistic is much larger) than LAO

Hung, Lawrence, Stockbridge, Lipicky (2000)


MMRM* (mixed-effect model repeated measure with saturated model) Response = µ + treatment + time +

treatment*time + baseline + subject (treatment) + error subject (treatment) and error are random effects treatment and time are class variables*Mallinckrodt et al (2001)


MMRM* analysis used to test H0: K = 0. - statistically valid under MAR - seem more stable in terms of type I

error rate than LOCF under MCAR or MAR*# (LOCF can be very bad, depending on at other visits)

*Mallinckrodt et al (2001) #Siddique and Hung (2003)


LOCF, LAO, MMRM can be very

problematic in case of informative

missing. Don’t know how to do ‘conservative’ imputation with these methods.


Worst rank/score analysis Test drug effect at time K in the

presence of events (e.g., death) that cause informatively missing values of the primary study outcome at time K.

Example: In congestive heart failure trials, exercise time is missing after death from heart failure.

Lachin (1999)


Assign a worst score to any informativelymissing values (due to occurrence of anabsorbing event related to progression ofdisease) and perform a nonparametricrank analysis.

Valid and efficient for testing H0:no treatment difference in distributions ofboth event time and main study outcomeLachin (1999)


For a drug having little effect on non-mortal outcome (e.g., exercise time), thisanalysis when used to test non-mortaleffect can be anti-conservative if the drugimproves survival.

Unclear how to perform a reasonable testfor the non-mortal effect alone (e.g., labeling issue)


Time to treatment failure analysis

In time to event analysis, if test drug hassevere side effects that cause moredropouts, then time to treatment failure(event or dropping out due to side effects) analysis may provide a conservative analysis.


Like the worst score/rank analysis, it is unclear how to perform a reasonable testfor time to the interested event alone- censoring on dropout due to failure ?


WLP opposite/pooled imputation

For binary outcome, opposite imputationimputes sample event rate of completers in one arm for unobservedevent rate of incompleters in the opposite arm.

Wittes, Lakatos, Prostfield (1989)

Proschan et al (2001)


Pooled imputation imputes sample eventrate of completers from both arms forunobserved event rate of noncompletersin each arm.

Treat imputed rate as ordinary rate.Compute Z statistic in the ordinarymanner using a combination of theobserved and the imputed rates. Wittes et al (1989), Proschan et al (2001)


WLP is less conservative than the worstcase analysis (assign ‘event’ to dropouts in the test drug group and ‘nonevent’ todropouts in the control group).



Partial list of other well-known methods

Likelihood-based method Pattern-mixture model selection modelNon-likelihood based method GEE Ad hoc imputation method


Methodology ConsiderationO’Neill (2003, 2004)- better assume NIM in planning stage missing data process not directly verifiable - choice of approach as the primary strategy for handling missing data ?- choice of approaches for sensitivity analysis, robustness analysis ?


Unnebrink and Windeler (2001)• adequacy of ad hoc strategy (e.g.,

LOCF, ranking, imputation of mean of other

group, etc) for handling missing value depends on whether the courses of disease are similar in the study groups

• For large dropout rates or different courses of disease, no adequate recommendations can be given


In planning strategies for handling missing values, we need to consider:

1)Null hypothesis should be carefully defined in anticipation of missing data.

It should not be altered by the presence of missing data after trial is done, regardless of their pattern.


2) For design, every attempt needs to be made to minimize dropouts. Alternative designs (e.g., enrichment design*, randomized withdrawal*) may be used to narrow the study population (recognize problem of generalizability), if ITT population cannot be properly studied.

*Temple (2004)


3) For analysis, the method needs to facilitate ‘conservative’ imputation to: - adjust the effect estimate toward null- inflate variability (double discounting for possible exaggeration from imputation of missing data), e.g., some type of worst score or rank.


4) Seek missing mechanism model to help imputation.This needs to use knowledge of disease process (how? Need to get practical experiences)The model needs to be flexible for sensitivity/robustness analysis.

Note: such model is not verifiable


5) Conduct better pilot trials or analyze historical data to explore response profiles of dropouts by reasons to see if missing mechanism may be related to outcome, and propose a reasonably conservative imputation method


Key to ‘reasonable’ imputation

= µ1 - µ2 at last time pointni = # of completers in group iIf there is no missing value, we have D = Y1 – Y2 (unbiased for ) V(D) = estimated variance of D Z = D/[V(D)]1/2

2,1),/,(~:sizeofmeansample 2 iNNYN iiii


Missing values { D , V(D), Z } not obtainable. Can try to get E( D | data) and V( D | data).And thus we construct Z* = E( D | data ) / [V( D | data )]1/2

or Z+ = E( D | data ) / [V(D)]1/2

All need models.



Goal is to use of a model such that |Z*| ≤ |Z| or |Z+| ≤ |Z| .

Since functional forms of E(D | data) and V(D | data) are unavailable, use of linear model to remove 1st-order effect of data is the first step. Then, what is the impact of imposing such model on estimation of V(D | data) or V(D)?


SUMMARY

Intent-to-treat is the goal. If the dropout rate is high, interpretable intent-to-treat analysis may not be achievable. Alternative designs (e.g., enrichment design) that narrow study population may need to be considered (caveat: generalizability of interpretation).


Intuitively, use of all data seems to be more promising than use of end point data to offer better guidance as to how to reasonably impute missing values. Yet,this advantage comes with a price that unverifiable statistical models must be dependent on.Thus, every method needs to facilitate ‘conservative’ imputation approach.


For regulatory applications, every attempt needs to be made to:- minimize dropout - explore response pattern of dropout in order to be able to propose a reasonably conservative imputation method- propose conservative strategies for primary analysis and sensitivity analyses


Selected ReferencesLachin (1999, Controlled Clinical Trials)Unnebrink, Windeler (2001, Statistics in Medicine)Shao, Zhong (2003, Statistics in Medicine)Proschan, McMahon, et al (2001, Journal of Statistical Planning and Inference)Wittes, Lakatos, Probstfield (1989, Statistic in Medicine)Mallinckrodt et al (2003, ASA JSM)Siddique, Hung (2003, ASA JSM)Hung, Lawrence, Stockbridge, Lipicky (2000, unpublished manuscript)


O’Neill (2003, ASA JSM; 2004, DIA EuroMeeting)Temple (1994-2004, Lecture notes on Clinical Trial Designs)Temple (2004, Society of Clinical Trials talk)

james hung, 2004 fda/industry management of missing data in clinical trials from a regulatory...

Documents

fdaindustry slide

james hung

treatment slide

fdaindustry y

drug administration

fdaindustry missing

current trial data slide

fdaindustry dropout