this work is licensed under a creative commons...
TRANSCRIPT
![Page 1: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/1.jpg)
Copyright 2008, The Johns Hopkins University and Mary Foulkes. All rights reserved. Use of these materials permitted only in accordance with license rights granted. Materials provided “AS IS”; no representations or warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for obtaining permissions for use from third parties as needed.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this site.
![Page 2: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/2.jpg)
Analysis Issues II
Mary Foulkes, PhDJohns Hopkins University
![Page 3: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/3.jpg)
Section A
Missing Data
![Page 4: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/4.jpg)
4
Causes of Missing Data
DropoutNoncompliance with measurement procedure−
Missed visit
−
Refused procedureError−
Test not done
−
Test done improperly−
Results lost
![Page 5: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/5.jpg)
5
Ideal Study
No dropoutsNo noncomplianceNo errorsNo loss to follow-up
![Page 6: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/6.jpg)
6
Big Worry
Missing data may be associated with outcome Form of this association is unknownFailing to account for the (true) association, may bias results
![Page 7: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/7.jpg)
7
Bias
Null hypothesis: no differenceIf null hypothesis is true, and it systematically eliminates patients with poorer prognosis on one treatment arm, results will be biased towards the alternative hypothesisIf results are biased in this way, the type one error (false positive rate) is inflatedPresence of bias means significance tests are unreliable
![Page 8: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/8.jpg)
8
Missing Data and the Intention-to-Treat (ITT) Principle
ITT: analyze all randomized patients in groups What to do when data are unavailable?Implication of ITT principle for design: collect all required data on all patients, regardless of compliance with treatment−
Thus, you avoid missing data
![Page 9: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/9.jpg)
9
Dilemma
Excluding patients with missing values can bias results and increase type one error (false positives)Collecting and analyzing outcome data on non-compliant patients may dilute results and increase type two error (false negatives)−
But we can compensate for this dilution with sample size increases
![Page 10: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/10.jpg)
10
Different Questions
Is this treatment safe and effective when used as directed in a well-defined population?Is this drug safe and effective when used for its intended purpose by practicing physicians?
![Page 11: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/11.jpg)
11
Relevance of Methodology
Quadratic function of am’t missingMethodology matters little when . . .−
No data are missing: analysis is straightforward
−
Lots of data are missing: no analysis is credibleMethodology is more important when modest/moderate amounts of data are missing−
Not enough to demolish credibility
−
Enough to potentially shift conclusions
![Page 12: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/12.jpg)
12
Handling Missing Data: Design and Conduct
Select primary endpoints that are more “robust” to missing data−
Change from first to final visit
−
SlopeExpend effort on collecting outcome data on all patients entered−
Visit reminders
−
General patient encouragement−
Collection of data regardless of compliance with treatment
![Page 13: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/13.jpg)
13
Handling Missing Data: Analysis
Exclude subjects with missing valuesLast Observation Carried Forward (LOCF)Group meansMultiple imputationBest/worst case scenarios
![Page 14: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/14.jpg)
14
Exclusions of Subjects
Excluding subjects with missing values is the simplest approach−
Requires the assumption that excluded subjects are a random subset of all randomized subjects
−
Analysis is straightforwardAssumption noted above is difficult to justify in most instances
![Page 15: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/15.jpg)
15
Exclusion: Source of Bias
Randomization ensures that there are no systematic differences between treatment groupsExcluding subjects from analysis on the basis of post-randomization events, e.g., (non-compliance), may introduce systematic differences between treatment groups, thereby biasing the comparison
![Page 16: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/16.jpg)
16
Imputation
Determine a value that is “best guess” of true value of missing data pointSeveral approaches proposed and/or in useSimplest approaches are statistically most problematicAny approach involving “made-up” data is problematic to some degree
![Page 17: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/17.jpg)
17
Imputation
Last Observation Carried Forward (LOCF)−
Use last measurement available in patients with missing data after a certain point
Group means−
Assign average value of outcome variable among those in that treatment group with complete data
Multiple imputation*−
Predict missing outcome on the basis of outcomes for other patients with similar characteristics
*Source: Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley. New York.
![Page 18: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/18.jpg)
18
Assumptions Required
LOCF: last available measure is a good estimate of final (missing) measureMean substitution: assigning group average to missing values will result in a good estimate of treatment effectMultiple imputation: one can create a model using data on covariates to accurately estimate missing value
![Page 19: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/19.jpg)
19
Problems with Assumptions
LOCF: data may be missing because of some aspect of treatment −
Patient not responding
−
Treatment not tolerable so that early data may not reflect true treatment effect
Mean substitution is like excluding patients with missing data, except that power is (artificially) maintainedMultiple imputation approach accounts for variability in estimating missing values, but assumption that we can predict outcome accurately based on measured covariates remains problematic
![Page 20: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/20.jpg)
20
Bottom Line
Assessments based on imputation models are reliable only insofar as one has confidence in the assumptions
![Page 21: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/21.jpg)
21
Worst/Best Case Scenarios (Sensitivity Analysis)
Neither exclusions nor imputationProvides bounds for the “true” results if all planned data points had been observedProvides a sense of how far off any of the other analyses could beDoes not provide a single “answer” but aids in the interpretation of other “answers”
![Page 22: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/22.jpg)
22
An Ounce of Prevention . . .
Discourage dropoutEncourage complianceInstitute QC checks to minimize errors
![Page 23: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/23.jpg)
23
Summary
The less there is missing data, the there is less concern about . . .−
Bias
−
Reliable conclusions−
Appropriate methods of analysis
Methods to replace or otherwise account for missing data are allflawed in important waysSensitivity analyses are essential to evaluating reliability of conclusions
![Page 24: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/24.jpg)
24
In the Next Section We’ll Look at . . .
Multiplicity−
Multiple treatment arms
−
Multiple controls−
Multiple subgroups
![Page 25: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/25.jpg)
Section B
Multiplicity
![Page 26: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/26.jpg)
26
Multiplicity in Trials
Refers to the multiple judgments and inferences−
Hypothesis tests
−
Confidence intervals−
Graphical analyses
Leads to a concern about false positives or an inflation of type one error
![Page 27: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/27.jpg)
27
Example
Chance (probability) of drawing the ace of clubs by randomly selecting a card from a complete, shuffled deck is 1/52Chance of drawing the ace of clubs at least once by randomly selecting a card from a complete, shuffled deck 100 times is . . . ?
![Page 28: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/28.jpg)
28
Sources of Multiplicity
Multiple treatment arms−
Doses, regimens
−
TreatmentsMultiple controls−
Internal
−
External (historical)Multiple evaluation groups−
Evaluable subgroup
−
Per protocol subgroup−
Demographic and disease-based subgroups
![Page 29: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/29.jpg)
29
Sources of Multiplicity
Multiple endpoints−
Efficacy variables
−
Evaluation time-pointsMultiple analyses−
Statistical tests
−
Dichotomization and cut-points−
Approaches to covariates or missing data
Multiple interim analyses of accumulating dataMultiple studies, sets of studies
![Page 30: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/30.jpg)
30
Young’s Rules
With enough testing, false positives will occurInternal evidence will not contradict a false positive results (i.e., don’t expect to figure out which are the false positives)Good investigators will come up with possible explanations“It only happens to the other guy”
![Page 31: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/31.jpg)
31
Approaches to the Problem
Do only one testAdjust the p-valuesPerform the desired tests at the nominal level and warn reader that no account has been taken for multiple testingIgnore the problem
![Page 32: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/32.jpg)
32
Bonferroni Adjustment
Most common approachCan severely reduce power when many comparisons are madeConservative when comparisons are not independentMethods exist to account for inter-correlation of results in adjusting significance thresholds
![Page 33: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/33.jpg)
33
Multivariate Testing
Mutually exclusive groupsMultivariate (“global”) tests assess whether all groups are similarIf conclusion of global test is no, then do pair-wise testsIf conclusion of global test is yes, stopProblem: if all are similar but one, the power for a global test may be lowNote: if multiple arms represent different doses, you should test for dose-response rather than heterogeneity
![Page 34: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/34.jpg)
34
Limitation of Comparisons
Select a single primary hypothesis; treat others as exploratoryPre-specify comparisons of interestCreate composite variablesRefrain from data-dredging
![Page 35: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/35.jpg)
35
Multiplicity when “Lumping” or “Splitting”
Splitting a trial with focus on the positive subgroup generally leads to a misleading result−
ISIS-2 trial [Lancet. (1988). 349-360]
−
The subgroup result by astrological signs at birth of Gemini or Libra came out to be slightly unfavorable for aspirin
−
However, the subgroup result by the other astrological sign gave a result in favor of aspirin with p-value < 0.00001
Lumping trials (say, given two trials) only when at least one of them does not give statistically significant result inflates the type I error rate
Source: Lu and Huque. (2001). Proper planning for multiplicity, replicability and clinical validity at the design stage is necessary.
![Page 36: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/36.jpg)
36
Some Final Comments
Address multiplicity issues at the design stageDifferent questions require different approachesClinical subset decision rules, involving multiple endpoints, inflate type one error rate and require adjustmentsWith multiple step procedures, interpretation of conditional results and computation of confidence intervals can be problematicTesting for large families of hypotheses using methods such as FDR (False Discovery Rate) should be considered exploratory strategies
![Page 37: This work is licensed under a Creative Commons …ocw.jhsph.edu/courses/BiostatMedicalProductRegulation/...Group means − Assign average value of outcome variable among those in that](https://reader033.vdocuments.net/reader033/viewer/2022050217/5f6307814860774eb95e71d7/html5/thumbnails/37.jpg)
37
In the Next Lecture We’ll Look at . . .
Non-inferiority−
Active control trials
−
Specifying “delta”−
Assay sensitivity
−
Potential problems