Outcomes and AEs Outcomes and AEs

Steven R. Cummings, MD Steven R. Cummings, MD

Outline: OutcomesOutline: Outcomes

• Primary and secondary outcomesPrimary and secondary outcomes

• Surrogate markersSurrogate markers

• Adverse experiencesAdverse experiences

MorboneMorbone

• A company wants help designing a A company wants help designing a trial of Morbone new treatment for trial of Morbone new treatment for osteoporosisosteoporosis

• Animal models: improves bone mass Animal models: improves bone mass and bone strengthand bone strength

• Planning a clinical trialPlanning a clinical trial

• Would like FDA approval to market Would like FDA approval to market MorboneMorbone

Morbone TrialMorbone Trial

Potential outcomesPotential outcomes

• Bone density (BMD) Bone density (BMD)

• Vertebral fractures (by x-ray)Vertebral fractures (by x-ray)

• Hip fracturesHip fractures

Which should be primary?Which should be primary?

How to start?How to start?

• Designate one “primary” and the Designate one “primary” and the others as “secondary” outcomesothers as “secondary” outcomes

Why one “primary” outcome?Why one “primary” outcome?

• To calculate sample sizeTo calculate sample size

• For testing statistical significance For testing statistical significance without penaltywithout penalty

• Greater credibilityGreater credibility

• The FDA requires that an outcome be The FDA requires that an outcome be “primary” in order to approve a drug “primary” in order to approve a drug for that indicationfor that indication–Beneficial effects on secondary Beneficial effects on secondary

outcomes can’t be used for ‘indication’outcomes can’t be used for ‘indication’

Which primary for Morbone?Which primary for Morbone?

• Bone density (BMD) Bone density (BMD)

• Vertebral fractures (by x-ray)Vertebral fractures (by x-ray)

• Hip fracturesHip fractures

ConsiderationsConsiderations

• Clinical importanceClinical importance

• FeasibilityFeasibility–Sample size and costSample size and cost

• Scientific / biological interestScientific / biological interest

• (For new drugs: What does FDA need (For new drugs: What does FDA need in order to approve an indication for in order to approve an indication for prescribing the drug?)prescribing the drug?)

Clinical importanceClinical importance

• Hip fracture: causes almost all of the Hip fracture: causes almost all of the deaths, much of the disability, and deaths, much of the disability, and 75% of the costs of fractures75% of the costs of fractures

• Vertebral fracture by x-ray: about 1/3 Vertebral fracture by x-ray: about 1/3 cause recognized pain and disability. cause recognized pain and disability. Mild changes might not be real Mild changes might not be real fractures.fractures.

• BMD: loss leads to greater risk of BMD: loss leads to greater risk of fracturesfractures

Which Primary Outcome?Which Primary Outcome?Sample sizeSample size

AlternativesAlternatives

• Improvement in BMDImprovement in BMD

• Vertebral fracturesVertebral fractures

• Hip fracturesHip fractures

Sample size/duration Sample size/duration

• 200 / 1 year200 / 1 year

• 3,000 / 3 yrs3,000 / 3 yrs

• 8,000 / 4 yrs8,000 / 4 yrs

Why not make BMD the Why not make BMD the primary outcome?primary outcome?

• Best choice to minimize costBest choice to minimize cost

• Issue: is it a valid surrogate marker Issue: is it a valid surrogate marker of clinical outcomes?of clinical outcomes?

Clean up your language!Clean up your language!

• Not all markers are ‘surrogates’Not all markers are ‘surrogates’

• BiomarkersBiomarkers– Measurement of a process or state.Measurement of a process or state.

• Surrogate markerSurrogate marker– Substitute for clinical outcomeSubstitute for clinical outcome

• Validated surrogateValidated surrogate– You can trust it. Effect of treatment on marker You can trust it. Effect of treatment on marker

has been established to consistently represent has been established to consistently represent the effect on clinical outcome.the effect on clinical outcome.

Surrogate markers for trialsSurrogate markers for trials

• ‘‘A laboratory or physical sign that is A laboratory or physical sign that is used in trials as a substitute for a used in trials as a substitute for a clinically meaningful endpoint.’ clinically meaningful endpoint.’

Criteria for validating a surrogate Criteria for validating a surrogate marker for treatmentsmarker for treatments

• Biologically plausibleBiologically plausible

• Marker strongly predicts the clinical Marker strongly predicts the clinical outcomeoutcome

• Treatment changes the markerTreatment changes the marker

• Treatment changes the rate of Treatment changes the rate of disease in the predicted direction disease in the predicted direction

* Prentice, 1989

Is BMD a valid surrogateIs BMD a valid surrogate

• Biologically plausibleBiologically plausible–Correlation between BMD and bone Correlation between BMD and bone

strength ~0.7 -0.9strength ~0.7 -0.9

Criteria for validating a surrogate Criteria for validating a surrogate marker for treatmentsmarker for treatments

• Biologically plausibleBiologically plausible

• Marker strongly predicts the clinical Marker strongly predicts the clinical outcomeoutcome

* Prentice, 1989

Risk of new vertebral fractures Risk of new vertebral fractures by quartile of BMDby quartile of BMD

Hip BMDHip BMD

00

22

44

66

88

1010

Quartile of Quartile of BMDBMD

11 22 33 44

Spine BMDSpine BMD

00

22

44

66

88

1010

Quartile of Quartile of BMDBMD

11 22 33 44

9.99.9

1.81.8

4.24.25.25.2

9.59.5

2.72.7

3.93.94.64.6

% o

f w

om

en w

ith

fra

ctu

re%

of

wo

men

wit

h f

ract

ure

Bone densityBone density

• Biologically plausible: YESBiologically plausible: YES

• Marker strongly predicts the clinical Marker strongly predicts the clinical outcome: YESoutcome: YES

• Treatment changes the marker*Treatment changes the marker*–YES: treatment improves BMD~5%YES: treatment improves BMD~5%

• Treatment changes the rate of Treatment changes the rate of disease in the predicted direction*disease in the predicted direction*–YES: other trials - 35-50% decrease risk YES: other trials - 35-50% decrease risk

* Prentice, 1989

Are we there yet?Are we there yet?

• BMD does all the right things a BMD does all the right things a surrogate marker should do.surrogate marker should do.

• Anything else?Anything else?

Treatment improves BMDTreatment improves BMD

RxRx BMDBMD

BMD predicts fractureBMD predicts fracture

FxFxBMDBMD

Bone densityBone density

• Do changes in the surrogate (bone Do changes in the surrogate (bone density) account for changes in density) account for changes in reduction in the outcome (fractures)reduction in the outcome (fractures)

* Prentice, 1989

FlourideFlouride

• Increased BMD ~10%Increased BMD ~10%

• IncreasedIncreased the risk of fractures the risk of fractures

* Prentice, 1989

Decreased Vertebral Fracture Risk: Decreased Vertebral Fracture Risk: Predicted from BMD vs. ObservedPredicted from BMD vs. Observed

80806060404020200 0

EstradiolEstradiol

EtidronateEtidronate

EtidronateEtidronate

AlendronateAlendronate

CalcitoninCalcitonin

8 8

55

1313

2020

3 3

PredictedPredicted

Cummings, ASBMR 1997Cummings, ASBMR 1997

Decreased Vertebral Fracture Risk: Decreased Vertebral Fracture Risk: Predicted from BMD vs. ObservedPredicted from BMD vs. Observed

80806060404020200 0

EstradiolEstradiol

EtidronateEtidronate

EtidronateEtidronate

AlendronateAlendronate

CalcitoninCalcitonin

6161

5656

5858

5050

6262

8 8

55

1313

2020

3 3

ObservedObservedPredictedPredicted

Cummings, ASBMR 1997Cummings, ASBMR 1997

The perfect surrogate is the causal The perfect surrogate is the causal pathway by which tx affects the diseasepathway by which tx affects the disease

RxRx FxFxBMDBMD

But treatments may have other effects But treatments may have other effects that could also influence the diseasethat could also influence the disease

RxRx FxFxBMDBMD

??????

TurnTurnoverover

Change in one marker will account forChange in one marker will account foronly part of the effect.only part of the effect.Some changes might also be harmful.Some changes might also be harmful.

Does BMD “Explain” the Reduction in Does BMD “Explain” the Reduction in Fracture Risk? Fracture Risk?

Main methodsMain methods

• Freeman Freeman –For individual data from a trialFor individual data from a trial

• Meta-analysisMeta-analysis–For aggregate data from multiple trialsFor aggregate data from multiple trials

Proving that a change in measurement Proving that a change in measurement predicts effect of treatment on fracture riskpredicts effect of treatment on fracture risk

Two approachesTwo approaches

• Individual level Individual level – How well does How well does changechange in the marker account in the marker account

for the effect in people? for the effect in people?

• Trial levelTrial level– How well does the How well does the changechange in measurement in measurement

predict the clinical results from trials? predict the clinical results from trials?

Validating that a marker is a Validating that a marker is a good surrogategood surrogate

#1#1

• Individual level Individual level –How well does change in the How well does change in the

measurement account for the decrease measurement account for the decrease in fracture risk in people? in fracture risk in people?

–The test: What percent of effect of The test: What percent of effect of decrease in fracture risk ‘explained’ by decrease in fracture risk ‘explained’ by change in the measurementchange in the measurement

Does BMD “Explain” the Reduction in Does BMD “Explain” the Reduction in Fracture Risk? Fracture Risk?

Main methodsMain methods

• Freeman Freeman –For individual data from a trialFor individual data from a trial–Estimates p = proportion of treatment Estimates p = proportion of treatment

effect “explained” by effect “explained” by change change in the in the markermarker

–ß = coefficient for treatmentß = coefficient for treatment–ß* = “adjusted for change in the markerß* = “adjusted for change in the marker– (1 - ß* / ß)(1 - ß* / ß)

Does BMD “Explain” the Reduction in Does BMD “Explain” the Reduction in Fracture Risk? Fracture Risk?

• • Freeman method applied to studies Freeman method applied to studies –FIT trial (alendronate): p = 0.16FIT trial (alendronate): p = 0.16–MORE trial (raloxifene): p = 0.05MORE trial (raloxifene): p = 0.05

• Very little of the treatment effects are Very little of the treatment effects are due to individual improvements in due to individual improvements in spine BMD, as measured by DXA.spine BMD, as measured by DXA.

Does BMD “Explain” the Reduction in Does BMD “Explain” the Reduction in Fracture Risk? Fracture Risk?

• • Freeman method applied to studies Freeman method applied to studies –FIT trial (alendronate): p = 0.16FIT trial (alendronate): p = 0.16–MORE trial (raloxifene): p = 0.05MORE trial (raloxifene): p = 0.05

• Very little of the treatment effects are Very little of the treatment effects are due to individual improvements in due to individual improvements in spine BMD, as measured by DXA.spine BMD, as measured by DXA.

Proving that a change in measurement Proving that a change in measurement predicts effect of treatment on fracture riskpredicts effect of treatment on fracture risk

2nd approach2nd approach

• Individual level Individual level

• Trial levelTrial level–How well does the change in How well does the change in

measurement predict the fracture measurement predict the fracture results from trials? results from trials?

– ‘‘Meta-analysis’ of many trialsMeta-analysis’ of many trials

““Meta-analysis” of trials of Meta-analysis” of trials of antiresorptivesantiresorptives

Each 1% improvement in spine BMD predicts Each 1% improvement in spine BMD predicts .03 (.02 to .05) reduction in risk of vertebral fracture.03 (.02 to .05) reduction in risk of vertebral fracture

““Meta-analysis” of trials of Meta-analysis” of trials of antiresorptivesantiresorptives

Each 1% improvement in spine BMD predicts Each 1% improvement in spine BMD predicts .03 (.02 to .05) reduction in risk of vertebral fracture.03 (.02 to .05) reduction in risk of vertebral fracture

ExpectedExpected

.25.25

ImplicationsImplications

• You cant trust changes in BMD in your You cant trust changes in BMD in your patients as an index of whether treatment patients as an index of whether treatment is working.is working.

• You can’t trust that a drug that improves You can’t trust that a drug that improves BMD will reduce the risk of fracture.BMD will reduce the risk of fracture.

• FDA still requires fractures as the FDA still requires fractures as the endpoint of trials for registering drugs.endpoint of trials for registering drugs.

Surrogate markers that failedSurrogate markers that failed

• Anti-arrhythmic drugs - frequency of Anti-arrhythmic drugs - frequency of ventricular arrythmias - and death ventricular arrythmias - and death

• Inotropic agents for CHF - improved Inotropic agents for CHF - improved cardiac function - mortalitycardiac function - mortality

• Estrogen - LDL/HDL cholesterol - CHDEstrogen - LDL/HDL cholesterol - CHD

• Antiresorptive drugs - BMD - fractureAntiresorptive drugs - BMD - fracture

• And more..And more..

Surrogates and safetySurrogates and safety

• Trials with surrogate endpoints are small Trials with surrogate endpoints are small and shortand short

• Generally too small to detect some Generally too small to detect some important adverse effectsimportant adverse effects

• Usually too short to determine whether Usually too short to determine whether long-term treatment continues to be long-term treatment continues to be effective (or safe)effective (or safe)

Limitations of surrogate markers Limitations of surrogate markers is the main reason for relying on is the main reason for relying on trials with clinical outcomestrials with clinical outcomes

Which Primary Outcome?Which Primary Outcome?

AlternativesAlternatives

• Improvement in BMDImprovement in BMD

• Vertebral fracturesVertebral fractures

• Hip fracturesHip fractures

Sample size/duration Sample size/duration

• 200 / 1 year200 / 1 year

• 3,000 / 3 yrs3,000 / 3 yrs

• 8,000 / 4 yrs8,000 / 4 yrs

Adverse events Adverse events

AEsAEs

• Hugely expensive & time-consumingHugely expensive & time-consuming–May account for 1/4 of the expense of May account for 1/4 of the expense of

drug trialsdrug trials

• Poorly donePoorly done

• Poorly studiedPoorly studied

• An opportunityAn opportunity

Issues re: Adverse EventsIssues re: Adverse Events

• Elicited vs. volunteeredElicited vs. volunteered

• Nuisance AEsNuisance AEs

• Attribution of causeAttribution of cause

AEs for Morbone TrialAEs for Morbone Trial

• The company’s standard approach: The company’s standard approach:

‘‘Record any symptoms or conditions the Record any symptoms or conditions the subject has experienced:subject has experienced:

________________________________________________________________

__________________________________________________________________

An alternative?An alternative?

““Since your last visit, has a doctor told Since your last visit, has a doctor told you you had (check all that apply)”you you had (check all that apply)”

__A blood clot in the leg or lung (venous __A blood clot in the leg or lung (venous thrombosis)thrombosis)

__ An ulcer __ An ulcer

……for all possible diseasesfor all possible diseases

What’s wrong with this approach?What’s wrong with this approach?

Approaches to AEsApproaches to AEsVolunteered vs. elicitedVolunteered vs. elicited

Pro Pro elicited/check listelicited/check list

• More sensitive?More sensitive?

• Easier to codeEasier to code

Con:Con:

• Miss unexpected or Miss unexpected or milder AEs milder AEs

Pro Pro volunteeredvolunteered

• Catch unexpected Catch unexpected AEs AEs

• Fewer AEs?Fewer AEs?

Con:Con:

• Hard to code; costlyHard to code; costly

• Less sensitive?Less sensitive?

Which approach is most likely to find Which approach is most likely to find real AEs? real AEs?

• Evidence is mixedEvidence is mixed

• Sensible approach: standard Sensible approach: standard questions to elicit uncommon AEs questions to elicit uncommon AEs suspected to be related to drug.suspected to be related to drug.

• Additional open ended questions to Additional open ended questions to capture unexpected AEs.capture unexpected AEs.

An approach An approach

• Evidence is mixedEvidence is mixed

• Sensible approach: standard Sensible approach: standard questions to elicit uncommon AEs questions to elicit uncommon AEs suspected to be related to drug.suspected to be related to drug.

• Additional open ended questions to Additional open ended questions to capture unexpected AEs.capture unexpected AEs.

FDA AE classificationsFDA AE classifications

• Serious AEsSerious AEs–DeathsDeaths–HospitalizedHospitalized–Cancer (except skin cancer)Cancer (except skin cancer)–Birth defectsBirth defects

• Reported within 24 hoursReported within 24 hours

• Collect extensive documentationCollect extensive documentation

FDA AE classificationsFDA AE classifications

• Non-serious AEsNon-serious AEs–Anything symptom or clinical eventAnything symptom or clinical event–Regardless of importance or Regardless of importance or

potential relevancepotential relevance

The Bunion ProblemThe Bunion Problem

• FIT Trial of alendronate in 6,400 FIT Trial of alendronate in 6,400 women for 4 yearswomen for 4 years

• Recorded over 20,000 episodes of Recorded over 20,000 episodes of URIs (and thousands of reports of URIs (and thousands of reports of bunions!)bunions!)

• Enormous data management effort Enormous data management effort and costand cost

• How could this be avoided?How could this be avoided?

How to minimize ‘nuisance’AEsHow to minimize ‘nuisance’AEs

• Elicit uncommon, plausible and Elicit uncommon, plausible and important AEsimportant AEs

• Limit collection of non-serious AEs Limit collection of non-serious AEs to samples of subjectsto samples of subjects–1st 300, then stop collecting1st 300, then stop collecting

New approach to classifying AEsNew approach to classifying AEs

• Serious AEsSerious AEs

• Important AEsImportant AEs–Limit activityLimit activity– Initiate a prescription or procedureInitiate a prescription or procedure–E.R. visitE.R. visit

• Minor AEsMinor AEs

AttributionAttribution

• SAEs must be classified asSAEs must be classified as–DefinitelyDefinitely–ProbablyProbably–Possibly, orPossibly, or–Not... Not...

...related to the study drug...related to the study drug

Did the treatment cause the AE?Did the treatment cause the AE?

• 67 year old Morbone volunteer starts 67 year old Morbone volunteer starts taking the study drug. taking the study drug.

• She reports a pruritic blistering rash She reports a pruritic blistering rash on her arms that started 7 days after on her arms that started 7 days after starting and disappeared 2 days after starting and disappeared 2 days after stopping the drug. stopping the drug.

• Your attribution?Your attribution?

AttributionAttribution

• Attributions to the drug are usually Attributions to the drug are usually wrongwrong– (Unless the treatment is known to cause (Unless the treatment is known to cause

the effect)the effect)

ValidationValidation

• Self report is sometimes inaccurateSelf report is sometimes inaccurate

• To get an accurate estimate of the To get an accurate estimate of the effect on plausible and serious effect on plausible and serious outcomes (and benefits), they should outcomes (and benefits), they should be validated.be validated.

• Expensive processExpensive process–Collection of recordsCollection of records–Central adjudication by expertsCentral adjudication by experts

Analysis of AE results of trialsAnalysis of AE results of trials

• Current practice: Current practice: – Intention-to-treat for effectiveness and Intention-to-treat for effectiveness and

safetysafety

• More conservative approachMore conservative approach– ITT for effectivenessITT for effectiveness–Per protocol for AEsPer protocol for AEs

SummarySummaryAssessing AEs in the Morbone TrialAssessing AEs in the Morbone Trial

• Elicit (check list)Elicit (check list)–Hospitalizations, birth defects..Hospitalizations, birth defects..–Disability, new prescriptions and Disability, new prescriptions and

proceduresprocedures

• Open-ended collection of minor AEs Open-ended collection of minor AEs on first 300 for 1 year on first 300 for 1 year

• No attributionNo attribution

• Per protocol analysis of AEsPer protocol analysis of AEs