bayesian hierarchical models for detecting safety signals in clinical trials h. amy xia and haijun...

Bayesian Hierarchical Models for Detecting Safety Signals

in Clinical TrialsH. Amy Xia and Haijun Ma

Amgen, Inc.

MBSW 2009, Muncie, INMarch 20, 2009

Disclaimer: The views expressed in this presentation represent personal views and do not necessarily represent the views or practices of Amgen.

Outline• Introduction• A motivating example• Bayesian Hierarchical Models

– Meta analysis of Adverse Events data from multiple studies incorporating MedDRA structure

– Incorporate patient level data– Effective graphics

• Closing Remarks

Three-Tier System for Analyzing Adverse Events in Clinical Trials

• Tier 1: Pre-specified Detailed Analysis and Hypothesis Testing– Tier 1 AEs are events for which a hypothesis has

been defined

• Tier 2: Signal Detection among Common Events– Tier 2 AEs are those that are not pre-specified and

“common”

• Tier 3: Descriptive Analysis of Infrequent AEs– Tier 3 AEs are those that are not pre-specified and

infrequent

Gould 2002 & Mehrotra 2004SPERT White Paper 2008

Multiplicity Issue in Detecting Signals Is Challenging

• Detection of safety signals from routinely collected, not pre-specified AE data in clinical trials is a critical task in drug development

• Multiplicity issue in such a setting is a challenging statistical problem – Without multiplicity considerations, there is a potential

for an excess of false positive signals – Traditional ways of adjusting for multiplicity such as

Bonferroni may lead to an excessive rate of false negatives

– The challenge is to develop a procedure for flagging safety signals which provides a proper balance between ‘no adjustment’ versus ‘too much adjustment’

Considerations Regarding Whether Flagging an Event

• Actual significance levels• Total number of types of AEs• Rates for those AEs not considered for flagging• Biologic relationships among various AEs

1st two are standard considerations in the frequentist approach. The 2nd two are not, but relevant in the Bayesian approach

-- Berry and Berry, 2004

Bayesian Work in Signal Detection

• Spontaneous adverse drug reaction reports– Gamma Poisson Shrinker (GPS) on FDA AERS

database (DuMouchel,1999)– Bayesian Confidence Propagation Neural Network

(BCPNN) on WHO database (Bate, et al. 1998)

• Clinical trial safety (AE) data– Bayesian hierarchical mixture modeling (Berry and

Berry, 2004)

Meta Analysis

• Glass (1976)Meta-analysis refers to a statistical analysis that combines the results of some collection of related studies to arrive a single conclusion to the question at hand

• Meta-analysis based on – aggregate patient data (APD meta-analysis)– Individual patient data (IPD) meta-analysis

• Bayesian modeling is a natural choice to incorporate the complex hierarchical structure of the data

George Chi, H.M. James Hung, Robert O’Neill (FDA CDER)

“Safety assessment is one area where frequentist strategies have been less applicable. Perhaps Bayesian approaches in this area have more promise.”

(Pharmaceutical Report, 2002)

An Example• Data from four double-blind placebo-controlled studies

on drug X. Study populations are similar.• Sample sizes:

• After converting all AEs into same MedDRA version, reported AEs are coded to 464 PTs under 23 SOCs and 233 HLTs

Study Drug X

N

Drug X

Subj-yr

Placebo

N

Placebo

Subj-yr

Study A 57 28.25 55 19.02

Study B 486 104.75 166 34.93

Study C 390 85.44 193 40.97

Study D 312 68.78 306 65.91

N_0: sample size in placebo arm; N_1: sample size in treatment arm

n_0: # subject with AE in placebo arm; n_1: # subject with AE in treatment arm

rt_0: subject incidence in placebo arm; rt_1: subject incidence in treatment arm

SYSTEM ORGAN CLASS HIGH LEVEL TERM PREFERRED TERM STUDY n_0 n_1 N_0 N_1 rt_0 rt_1

INFECTIONS AND INFESTATIONSHERPES VIRAL INFECTIONS HERPES SIMPLEX 1 0 3 55 57 0.00 5.26

2 0 5 166 486 0.00 1.033 1 8 193 390 0.52 2.054 2 2 306 312 0.65 0.64

INFECTIONS AND INFESTATIONSUPPER RESPIRATORY TRACT INFECTIONS SINUSITIS 1 2 8 55 57 3.64 14.04

2 6 19 166 486 3.61 3.913 0 8 193 390 0.00 2.054 4 11 306 312 1.31 3.53

INFECTIONS AND INFESTATIONSUPPER RESPIRATORY TRACT INFECTIONS RHINITIS 1 0 0 55 57 0.00 0.00

2 0 0 166 486 0.00 0.003 1 4 193 390 0.52 1.034 0 0 306 312 0.00 0.00

SKIN AND SUBCUTANEOUS TISSUE DISORDERS URTICARIAS URTICARIA 1 0 1 55 57 0.00 1.75

2 1 1 166 486 0.60 0.213 0 4 193 390 0.00 1.034 1 4 306 312 0.33 1.28

SKIN AND SUBCUTANEOUS TISSUE DISORDERS

PURPURA AND RELATED CONDITIONS ECCHYMOSIS 1 0 0 55 57 0.00 0.00

2 0 4 166 486 0.00 0.823 0 5 193 390 0.00 1.284 0 3 306 312 0.00 0.96

INJURY, POISONING AND PROCEDURAL COMPLICATIONS

NON-SITE SPECIFIC INJURIES NEC EXCORIATION 1 0 0 55 57 0.00 0.00

2 0 7 166 486 0.00 1.443 0 0 193 390 0.00 0.004 0 1 306 312 0.00 0.32

INJURY, POISONING AND PROCEDURAL COMPLICATIONS

NON-SITE SPECIFIC PROCEDURAL COMPLICATIONS PROCEDURAL PAIN 1 0 0 55 57 0.00 0.00

2 0 0 166 486 0.00 0.003 0 1 193 390 0.00 0.264 2 1 306 312 0.65 0.32

Proposed Bayesian Approach• Hierarchical mixture models for aggregated

binary responses was constructed based on the work by Berry & Berry(2004)– Explore impact of using different MedDRA hierarchy– Inclusion of study effects– Further extended to a hierarchical Poisson mixture

model, to account for different exposure/follow-up times between patients

• Individual patient level models are discussed• Implemented the above models with available

software– WinBUGS for model implementation– S-Plus graphics for inference

MedDRA

• MedDRA (the Medical Dictionary for Regulatory Activities Terminology) is a controlled vocabulary widely used as a medical coding scheme.

• MedDRA Definition (MSSO): – MedDRA is a clinically-validated international medical

terminology used by regulatory authorities and the regulated biopharmaceutical industry. The terminology is used through the entire regulatory process, from pre-marketing to post-marketing, and for data entry, retrieval, evaluation, and presentation.

https://eudract.emea.europa.eu/docs/medDRA/Introduction_To_MedDRA.ppt MSSO: Introduction to MedDRA

http://www.fda.gov/cder/present/dia-699/dia628/index.htmMedDRA and Pharmacovigilance - The Way Forward, 7/8/99

http://www.fda.gov/cder/present/dia-699/dia628/index.htm

SOC = Respiratory, thoracic and mediastinal disorders

HLGT = Respiratory tract infections

HLT =Viral upper respiratorytract infections

HLT = Influenza viral infections

HLGT = Viral infectious disorders

SOC = Infections and infestations

PT = Influenza

Example of MedDRA Hierarchy

https://eudract.emea.europa.eu/docs/medDRA/Introduction_To_MedDRA.ppt MSSO: Introduction to MedDRA

Hierarchical Structure of MedDRA• Bayesian hierarchical models allow for explicitly

modeling AEs with the existing coding structure – AEs in the same SOC more likely to be similar within than

across SOCs. – Allow for this possibility, but does not impose it, depending on

the actual data– SOC tends to be too broad. HLT is more closely related to

medical concepts.

• In fact, clinical and safety people would (informally) consider the similarity of the AEs, say, within SOCs when they review AE tables – For example, if differences in several CV events were observed,

then each would be more likely to be causal than if differences came from medically unrelated areas (eg, skin, neurological, thrombosis, cancer)

• Bayesian hierarchical modeling allows a scientific, explicit, and more formal way to take it into consideration

Notations

• Study i=1,…I, SOC b=1,…B and PT j=1,…kb

• Data: For AEibj

– Treatment group: Yibj incident events observed in Nit patients with Tibj subjects’ exposure

– Control group: Xibj incident events observed in Nic patients with Cibj subjects’ exposure

Bayesian Meta Analysis Hierarchical Logistic Regression

ly.respective groups, control

and treatmentin the AEfor ratesevent are and where

,BinomialBinomial

ibjibjibj

ibj,icibjibj,itibj

ct

) c(N ~ ), X t(N ~ Y

)1()exp(~

);1()exp(~ :prior 3 Stage

);,(~:prior 2 Stage

);,()1()0(~ :prior 1 Stage

:OR-logeffect ent for treatmPrior Mixture

),N( ~, ),N(~

)logit(t ,)logit(c

2bj

2bbbj

2bj,bjibj

bjibjibjibj ibj

pp

pp

ppb

bbbb

I

I

Beta

N

•Common treatment effect for same PT across studies

Bayesian Meta Analysis Hierarchical Logistic Regression (Cont.)

• Treatment effect with additive study effects:

• A random treatment effect/multiplicative model:

),()1()0(~, ),N(~

),N( ~, ),N(~

)logit(t ,)logit(c

2bj

2bjbjibj

2bbbj

2bj,bjibj

ibjibjibjibj ibj

bbbb N

)01.0,0(~

),()1()0(~

),N( ~, ),N(~

)logit(t ,)logit(c

2bj

2bbbj

2bj,bjibj

ibjibjibjibj ibj

N

N

Study

i

bbbb

i

Bayesian Meta Analysis Hierarchical Logistic Regression (Cont.)

• Other priors– Stage 1 λbj ~N(μλb , σ2

λb );

– Stage 2 μλb~N(μλ0, σ2λ0 ); σ2

λb ~IG(αλ, βλ);

μθb ~N(μθ0, σ2θ0 ); σ2

θb ~IG(αθ, βθ)

- Stage 3 μλ0~N(μλ00 , σ2λ00 ); σ2

λ0 ~IG(αλ00, βλ00)

μθ0~N(μθ00 , σ2θ00 ); σ2

θ0 ~IG(αθ00, βθ00)

Hyperparameters μλ00, σ2λ00, αλ00, βλ00, μθ00, σ2

θ00,

αθ00, βθ00, αλ, βλ , αθ, βθ are fixed constants

Inference

• AEbj is flagged if – Pr( θbj > d*| Data) > p, where θbj is log-OR in

Binomial models and log-RR in Poisson models.

– d* and p are all prespecified constants.

• Graphs are useful tools in deciphering data and presenting results

Model Selection

• Deviance Information Criteria (DIC) was used to compare models with same data

• Limited sensitivity analyses were done to check the robustness of the models

• Different levels of MedDRA structures were used– SOC/PT, HLT/PT and SOC/HLT/PT

• Treatment effect with additive study effects model using SOC/PT structure was chosen

Bayesian Meta Analysis Hierarchical Log-linear Regression

• Poisson models – Adjust for different exposures in treatment and control– Assume constant hazard over time– Unless AEs are fairly common or follow up of studies are

quite unbalanced between treatment arms, usually are not very different from Binomial models

Yibj~Pois(tibj Tibj ); Xbj~Pois(cibj Cibj)

where tibj and cibj are event rates, and Tibj and Cibj for AEibj are total exposure times in the treatment and control groups, respectively

log(cibj )=λibj; log(tibj )=λibj + θbj ,

Note that θbj =log(RRbj)

Fisher's Exact Test 2-sided P-values

Raw Risk Difference(%), Treatment - Placebo

- lo

g1

0 (

Fis

he

r e

xact

p-v

alu

e)

-0.01 0.0 0.01 0.02

0.0

0.5

1.0

1.5

2.0

2.5

ECCHYMOSIS

EXCORIATION

FATIGUE

HERPES SIMPLEX

SINUSITIS

GASTROINTESTINAL DISORDER...INFECTIONS AND INFESTATIO...SKIN AND SUBCUTANEOUS TIS...NEOPLASMS BENIGN, MALIGNA...GENERAL DISORDERS AND ADM...PSYCHIATRIC DISORDERSINJURY, POISONING AND PRO...RESPIRATORY, THORACIC AND...CARDIAC DISORDERSMUSCULOSKELETAL AND CONNE...REPRODUCTIVE SYSTEM AND B...RENAL AND URINARY DISORDE...EYE DISORDERSINVESTIGATIONSNERVOUS SYSTEM DISORDERSMETABOLISM AND NUTRITION ...IMMUNE SYSTEM DISORDERSEAR AND LABYRINTH DISORDE...SURGICAL AND MEDICAL PROC...VASCULAR DISORDERSENDOCRINE DISORDERSHEPATOBILIARY DISORDERSBLOOD AND LYMPHATIC SYSTE...

0 1 2 3

log Odds Ratio

GASTROENTERITIS VIRAL

INFLUENZA

BRONCHITIS

DIZZINESS

TENSION HEADACHE

BASAL CELL CARCINOMA

URTICARIA

EXCORIATION

HEADACHE

INJECTION SITE BRUISING

DYSPEPSIA

HERPES SIMPLEX

FATIGUE

SINUSITIS

ECCHYMOSIS

Peto's Method

0 1 2 3

log Odds Ratio

TENSION HEADACHE

UPPER RESPIRATORY TRACT INFECTION

PHARYNGITIS

URTICARIA

BRONCHITIS

DIZZINESS

NASOPHARYNGITIS

INFLUENZA

DYSPEPSIA

INJECTION SITE BRUISING

HEADACHE

ECCHYMOSIS

HERPES SIMPLEX

SINUSITIS

FATIGUE

M-H TA CC Method

log-OR posterior means

P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

ECCHYMOSIS

BLOOD AND LYMPHATIC SYSTE...CARDIAC DISORDERSEAR AND LABYRINTH DISORDE...ENDOCRINE DISORDERSEYE DISORDERSGASTROINTESTINAL DISORDER...GENERAL DISORDERS AND ADM...HEPATOBILIARY DISORDERSIMMUNE SYSTEM DISORDERSINFECTIONS AND INFESTATIO...INJURY, POISONING AND PRO...INVESTIGATIONSMETABOLISM AND NUTRITION ...MUSCULOSKELETAL AND CONNE...NEOPLASMS BENIGN, MALIGNA...NERVOUS SYSTEM DISORDERSPSYCHIATRIC DISORDERSRENAL AND URINARY DISORDE...REPRODUCTIVE SYSTEM AND B...RESPIRATORY, THORACIC AND...SKIN AND SUBCUTANEOUS TIS...SURGICAL AND MEDICAL PROC...VASCULAR DISORDERS


P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

ECCHYMOSIS



P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

ARRHYTHMIA ARTERIOSCLEROSIS CORONARY ARTERY BUNDLE BRANCH BLOCK LEFT EAR DISCOMFORT TINNITUS VERTIGO VERTIGO POSITIONAL CATARACT CONJUNCTIVAL HAEMORRHAGE

CONJUNCTIVITIS DIPLOPIA

DRY EYE EYE INFLAMMATION EYE PRURITUS EYELID OEDEMA ABDOMINAL DISTENSION ABDOMINAL PAIN ABDOMINAL PAIN UPPER ABDOMINAL TENDERNESS DIARRHOEA DRY MOUTH

DYSPEPSIA

FLATULENCE FOOD POISONING GASTROINTESTINAL HYPERMOTILITY GASTROINTESTINAL INFLAMMATION GASTROOESOPHAGEAL REFLUX DISEASE GINGIVAL BLEEDING NAUSEA STOMACH DISCOMFORT TOOTH DISORDER TOOTHACHE

FATIGUE

INJECTION SITE BRUISING INJECTION SITE HAEMATOMA IRRITABILITY MALAISE

LIVER DISORDER HYPERSENSITIVITY

SEASONAL ALLERGY ABDOMINAL INFECTION ACUTE SINUSITIS APPENDICITIS BRONCHITIS BRONCHITIS CHRONIC

CANDIDIASIS CAT SCRATCH DISEASE CELLULITIS STAPHYLOCOCCAL CERVICITIS CONDYLOMA ACUMINATUM

CYSTITIS DIVERTICULITIS EAR INFECTION ERYSIPELAS EYELID INFECTION

FOLLICULITIS

FUNGAL INFECTION FURUNCLE GASTROENTERITIS VIRAL

GENITAL INFECTION FEMALE GINGIVAL INFECTION

HERPES SIMPLEX

HERPES VIRUS INFECTION INFECTED INSECT BITE INFECTED SEBACEOUS CYST INFECTIOUS MONONUCLEOSIS INFLUENZA LABYRINTHITIS LARYNGITIS

LOWER RESPIRATORY TRACT INFECTION MASTITIS NASOPHARYNGITIS ONYCHOMYCOSIS

ORAL PUSTULE OTITIS EXTERNA OTITIS MEDIA

PHARYNGITIS

PNEUMONIA POSTOPERATIVE WOUND INFECTION PULPITIS DENTAL PUNCTURE SITE INFECTION RASH PUSTULAR RHINITIS SIALOADENITIS

SINUSITIS

SKIN INFECTION SUBCUTANEOUS ABSCESS TINEA CRURIS TINEA INFECTION TINEA VERSICOLOUR TONSILLITIS TOOTH ABSCESS TOOTH INFECTION VAGINAL CANDIDIASIS

VAGINAL INFECTION VIRAL INFECTION VIRAL UPPER RESPIRATORY TRACT INFECTION

ANIMAL BITE CONTUSION

EXCORIATION FALL FOOT FRACTURE HAND FRACTURE HEAT STROKE POST-TRAUMATIC PAIN ROAD TRAFFIC ACCIDENT SCAPULA FRACTURE TENDON INJURY THERMAL BURN TOOTH FRACTURE TOOTH INJURY WOUND

BLOOD CHOLESTEROL INCREASED

BLOOD GLUCOSE INCREASED

BLOOD PRESSURE INCREASED

BODY TEMPERATURE INCREASED WEIGHT DECREASED WEIGHT INCREASED HYPERCHOLESTEROLAEMIA BACK PAIN INTERVERTEBRAL DISC DEGENERATION JOINT EFFUSION MYALGIA SHOULDER PAIN TENDONITIS


FIBROUS HISTIOCYTOMA LIPOMA PROSTATE CANCER SEBORRHOEIC KERATOSIS SKIN PAPILLOMA

SQUAMOUS CELL CARCINOMA OF SKIN BURNING SENSATION

CAROTID ARTERY STENOSIS CARPAL TUNNEL SYNDROME CEREBROVASCULAR ACCIDENT CEREBROVASCULAR INSUFFICIENCY DISTURBANCE IN ATTENTION

DIZZINESS

DIZZINESS POSTURAL DYSGEUSIA FACIAL PALSY HEADACHE HYPERSOMNIA HYPERTONIA

HYPOAESTHESIA LETHARGY MEMORY IMPAIRMENT MIGRAINE NERVE COMPRESSION NEURALGIA PARAESTHESIA RESTLESS LEGS SYNDROME SCIATICA SENSORY DISTURBANCE

SINUS HEADACHE

SOMNOLENCE SYNCOPE SYNCOPE VASOVAGAL

TENSION HEADACHE

AGITATION ALCOHOLISM

ANXIETY

ATTENTION DEFICIT/HYPERACTIVITY DISORDER BIPOLAR I DISORDER DEPRESSION DYSPHORIA

INSOMNIA LIBIDO DECREASED NERVOUSNESS PREMATURE EJACULATION STRESS CALCULUS BLADDER HAEMATURIA NEPHROLITHIASIS POLLAKIURIA

STRESS INCONTINENCE TRIGONITIS URINARY INCONTINENCE URINARY RETENTION BENIGN PROSTATIC HYPERPLASIA CERVICAL DYSPLASIA ERECTILE DYSFUNCTION

OVARIAN CYST RUPTURED VAGINAL LACERATION VULVOVAGINAL DRYNESS ALLERGIC COUGH DYSPHONIA DYSPNOEA DYSPNOEA EXERTIONAL EPISTAXIS

HYPERVENTILATION NASAL CONGESTION

RHINITIS ALLERGIC RHINORRHOEA

RHONCHI SINUS CONGESTION SLEEP APNOEA SYNDROME SNEEZING SNORING

THROAT IRRITATION ACNE ACTINIC KERATOSIS

DERMAL CYST

DERMATITIS CONTACT

ECCHYMOSIS

ECZEMA GUTTATE PSORIASIS INGROWING NAIL INTERTRIGO NIGHT SWEATS ONYCHOMADESIS SKIN HAEMORRHAGE SKIN LESION

URTICARIA

ENDODONTIC PROCEDURE FLUSHING

HOT FLUSH HYPERTENSION LYMPHOEDEMA



P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0




DYSPEPSIA


FATIGUE






FOLLICULITIS



HERPES SIMPLEX




PHARYNGITIS


SINUSITIS













DIZZINESS



SINUS HEADACHE


TENSION HEADACHE


ANXIETY









DERMAL CYST

DERMATITIS CONTACT

ECCHYMOSIS


URTICARIA





P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0




DYSPEPSIA


FATIGUE






FOLLICULITIS



HERPES SIMPLEX




PHARYNGITIS


SINUSITIS













DIZZINESS



SINUS HEADACHE


TENSION HEADACHE


ANXIETY









DERMAL CYST

DERMATITIS CONTACT

ECCHYMOSIS


URTICARIA





P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0




DYSPEPSIA


FATIGUE






FOLLICULITIS



HERPES SIMPLEX




PHARYNGITIS


SINUSITIS













DIZZINESS



SINUS HEADACHE


TENSION HEADACHE


ANXIETY









DERMAL CYST

DERMATITIS CONTACT

ECCHYMOSIS


URTICARIA




Posterior Summary of Bayesian Hierarchical Meta-Analysis Model


P(O

R >

1|D

ata

)

-3 -2 -1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0




DYSPEPSIA


FATIGUE






FOLLICULITIS



HERPES SIMPLEX




PHARYNGITIS


SINUSITIS













DIZZINESS



SINUS HEADACHE


TENSION HEADACHE


ANXIETY









DERMAL CYST

DERMATITIS CONTACT

ECCHYMOSIS


URTICARIA




Inferences of Binomial Hierarchical Model with Mixture Prior

SOC PT

Two sided Fisher's Exact

Post Prob OR=1

Post Prob OR>1

Post Prob OR>1.1

Post Prob OR>1.2

Post Prob OR>2

GENERAL DISORDERS AND ADMINISTRATION SITE FATIGUE 0.019 0.82 0.18 0.18 0.18 0.10

INFECTIONS AND INFESTATIONS

HERPES SIMPLEX 0.039 0.75 0.25 0.25 0.25 0.21

INFECTIONS AND INFESTATIONS SINUSITIS 0.012 0.72 0.27 0.27 0.27 0.19SKIN AND SUBCUTANEOUS TISSUE DISORDERS ECCHYMOSIS 0.005 0.11 0.89 0.89 0.89 0.89

Bayesian Patient Level Models

• IPD models to include within patient correlation and patient level factors while incorporating MedDRA coding hierarchy

• Data from one study is used:618 subject, 207 unique AEs, N = 127926

),0(~ I), ,0MVN(~

ksubject for covariates ofset theis Z where

Z )logit(t ,Z)logit(c

),(~X),(~Y

2k

2b

k

kkbjbjkbjkkbj kbj

kbjkbj

Normal

cBernoullitBernoulli kbjkbj

Simulation Study

• Simulation scheme:– Randomly assign subjects to treatment or placebo to create a

“null” scenario– Adverse events within subject remain unchanged to maintain the

SOC/PT hierarchy– 1000 simulated datasets

• Family-wise error rates (also FDR in this case) of Fisher’s exact text unadjusted for multiplicity and Poisson regression with mixture prior are compared

• Percentage of simulated datasets yielding Y=0, 1, 2 or ≥3 incorrectly flagged adverse events out of 464 PTs are also compared

Distribution of Y (%)

FWER/ FDR

1 2 >=3

Non-adjusted 2-sided Fisher’s exact test

2-sided test, p-value <=0.05 99.1 2.7 5.4 91

Non-adjusted 1-sided Fisher’s exact test

p-value<=0.05 90.1 22.7 26.9 40.5

Bayes Hierarchical Poisson Model *

c=1, p=0.7 10.2 7.5 2.0 0.7

c=1, p=0.8 6.1 4.6 1.3 0.2

c=1, p=0.9 2.8 2.7 0.1 0.0

c=1.2, p=0.7 10.2 7.6 1.9 0.7

c=1.2, p=0.8 6.0 4.6 1.2 0.2

c=1.2, p=0.9 2.7 2.6 0.1 0.0

c=2, p=0.7 7.0 5.3 1.5 0.2

c=2, p=0.8 3.7 3.3 0.4 0.0

c=2, p=0.9 1.3 1.2 0.1 0.0

RR≠1, p=0.7 13.8 10.3 2.5 1.0

RR≠1, p=0.8 7.9 6.3 1.4 0.2

RR≠1, p=0.9 3.6 3.5 0.1 0.0

* pDatacRRbj )|Pr(

Simulation Study

• 464 independent tests with alpha=0.05 would yield in average about 23 signals and have FDR=1 if no multiplicity is adjusted for.

• Correlation of the AE data reduced the error rate in our simulation study

• But the FDR is still as high as 99.1%. For 91% cases there are at least 3 falsely identified signals.

• The FWERs/FDRs for all Bayes model results are much lower and acceptable.

Closing Remarks• Current traditional approach of flagging

routinely collected AEs based on unadjusted p-values or CIs can result in excessive false positive signals– As a result, it can cause undue concern for

approval/labeling/post marketing commitment

• Commonly used meta-analysis methods for aggregated binary outcome (OR)– Peto’s method is not recommended for

severely unbalanced studies or common events unless treatment effects are small

– MH method: needs continuity correction

Closing Remarks (Cont.)• Bayesian meta-analysis hierarchical mixture modeling

provides a useful tool to analyze data from multiple studies and address multiplicity – Allows for explicitly modeling AEs with the existing MedDRA coding

structure– Use a mixture prior by allowing a point mass on equality of the treatment

and control rates– Study differences can be accounted for– No need to add continuity correction. Double zero studies are included.– For less common AEs and studies without a great amount of follow-up

variation between treatment groups, inferences from Poisson regression and logistic regression models are very similar

• Computation for signal detection using IPD is challenging• Graphics are effective in displaying flagged signals

Future Work

• More sensitivity analysis on the performance of the models

• Further simulation study on type II error and operating characteristics of Bayesian models

• Zero-inflated Poisson model might be a good approach for relatively healthy population

• Incorporating severity information of AEs• Multi-axial structure of MedDRA coding system• The field of clinical trial signal detection is still in its

infancy– More research and practice are needed– Statisticians need to work with clinicians/safety scientists closely

to further advance this field

References• Bate A, Lindquist M, Edwards, IR, Olsson S, Orre R, Lansner A, and De

Freitas RM (1998). A Bayesian neural network method for adverse drug reaction signal detection. Eur J Clin Pharmacol 54:315-321

• Berry S and Berry D (2004) Accounting for multiplicities in assessing drug safety: a three-level hierarchical mixture model. Biometrics, 60: 418-426

• Chi G, Hung HMJ, and O’Neill R (2002). Some comments on “Adaptive Trials and Bayesian Statistics in Drug Development” by Don Berry. In Pharmaceutical Report, Vol 9, 1-11

• Crowe B, Xia A, Watson D, Shi H, Lin S, Kuebler J, Berlin J, et al. (2008). Recommendations for Safety Planning, Data Collection, Evaluation and Reporting During Drug, Biologic and Vaccine Development: A Report of the PhRMA Safety Planning, Evaluation and Reporting Team (SPERT). Manuscript in preparation.

• DuMouchel W (1999). Bayesian data mining in large frequency tables, with an application to the FDA Spontaneous Reporting System (with discussion). The American Statistician 53:177-202

• Gould AL. Drug safety evaluation in and after clinical trials. Deming Conference, Atlantic City, 3 December 2002

• Mehrotra, DV and Heyse, JF (2004). Multiplicity considerations in clinical safety analysis. Statistical Methods in Medical Research 13, 227-238

• Spiegelhalter DJ, Best NG, Carlin BP and van der Linde A (2002) Bayesian measures of model complexity and fit (with discussion). J. Roy. Statist. Soc. B. 64, 583-640.

Thank You!

Back-up Slides

Incidence Rate of Treatment Arm

0.00 0.10 0.20 0.30

010

2030

4050

Min. 0

1st Qu. 0

Median 0

Mean 0.003 3rd Qu. 0.003

Max. 0.333

Incidence Rate of Placebo Arm

0.00 0.05 0.10 0.15

020

4060

80 Min. 0

1st Qu. 0

Median 0

Mean 0.003 3rd Qu. 0

Max. 0.164

Bayesian Hierarchical Model for AE (Berry & Berry 2004)

• PT level assumptions:

• SOC level assumptions:

),()1(]0[~

),(~2

2

iiiij

iij

NI

N

• Global assumptions:

),(~

),(~

),(~2

0

20

Beta

TN

TN

i

i

i

)1()(~

)1()(~

),(~

),(~

),(~

),(~

),(~

),(~

2

2

2

2

200000

200000

IExp

IExp

IG

IG

IGT

IGT

TN

TN

TT

TT

SOC: Injury, Poisoning and Procedural Complications

Preferred TermCtrl r(%)

Trt r(%)

diff (Trt-Ctrl ) Preferred Term

Ctrl r(%)

Trt r(%)

diff (Trt-Ctrl )

1 Alcohol Poisoning 0.00 0.08 0.080 26 Limb Injury 0.14 0.16 0.0222 Animal Bite 0.14 0.24 0.102 27 Mouth Injury 0.00 0.08 0.0803 Arthropod Bite 0.56 0.24 -0.315 28 Muscle Injury 0.00 0.08 0.0804 Back Injury 0.69 0.24 -0.453 29 Muscle Strain 0.69 0.72 0.0285 Cartilage Injury 0.14 0.00 -0.139 30 Neck Injury 0.14 0.08 -0.0596 Chemical Eye Injury 0.14 0.00 -0.139 31 Pain Trauma Activated 0.14 0.40 0.2637 Closed Head Injury 0.00 0.08 0.080 32 Pneumothorax Traumatic 0.00 0.08 0.0808 Concussion 0.28 0.00 -0.278 33 Post Procedural Complication 0.14 0.00 -0.1399 Contrast Media Reaction 0.14 0.00 -0.139 34 Post Procedural Pain 0.00 0.08 0.080

10 Contusion 0.83 0.80 -0.030 35 Procedural Pain 0.28 0.08 -0.19711 Epicondylitis 0.14 0.08 -0.059 36 Rib Fracture 0.14 0.08 -0.05912 Excoriation 0.00 0.64 0.643 37 Road Traffic Accident 0.00 0.16 0.16113 Eye Injury 0.14 0.08 -0.059 38 Scapula Fracture 0.00 0.08 0.08014 Fall 0.00 0.32 0.321 39 Skeletal Injury 0.42 0.00 -0.41715 Foot Fracture 0.00 0.32 0.321 40 Skin Laceration 1.11 0.72 -0.38816 Foreign Body in Eye 0.14 0.00 -0.139 41 Splinter 0.14 0.08 -0.05917 Hand Fracture 0.00 0.24 0.241 42 Sunburn 0.28 0.24 -0.03718 Heat Exhaustion 0.14 0.00 -0.139 43 Tendon Injury 0.00 0.08 0.08019 Heat Stroke 0.00 0.08 0.080 44 Thermal Burn 0.00 0.16 0.16120 Hip Fracture 0.00 0.08 0.080 45 Tooth Fracture 0.00 0.08 0.08021 Injury Corneal 0.14 0.00 -0.139 46 Tooth Injury 0.00 0.16 0.16122 Jaw Fracture 0.14 0.08 -0.059 47 Traumatic Haematoma 0.28 0.08 -0.19723 Joint Injury 0.00 0.08 0.080 48 Wound 0.14 0.24 0.10224 Joint Sprain 0.28 0.16 -0.117 49 Wrist Fracture 0.14 0.00 -0.13925 Ligament Injury 0.14 0.08 -0.059

bayesian hierarchical models for detecting safety signals in clinical trials h. amy xia and haijun...

Documents

data slide

statistical analysis

adjustment slide

example data

aes rates

various aes

hand metaanalysis

clinical trials tier