Kenneth F. Schulz, PhD, MBA

Triangle Global Health Consortium Breakfast Discussion

Outcome Outcome

Placebo

40% Non-compliance

60% Compliance

Policy of no treatment

25% Non-compliance

75% Compliance

Policy of treatment

Treatment

Randomize

Participants with Infection

Outcome Outcome

Treatment A

97% Treatment A

3% Treatment B

Policy of Treatment A

33% Treatment A

67% Treatment B

Policy of Treatment B

Treatment B

Randomize

Participants with Infection

8,198 Placebo

22% Non-compliant < 4 doses Outside time period 24 HIV +

78% Compliant 50 HIV+

Policy of administering placebo

25% Non-compliant < 4 doses Outside time period 15 HIV+

75% Compliant 36 HIV+

Policy of administering vaccine

8,197 Vaccine

Randomize 16,395

Outcome Outcome

Participants without HIV Infection

Per-protocol traditionally used in preventive vaccine trials

Completed immunization series

With the assigned vaccine

Within the allotted window of time

Moreover, follow-up for counting cases of disease typically begins a few weeks after the last dose– Vaccine-induced response is believed

“adequate”

Per-protocol similar to ITT in vaccine trials

Levels of full compliance are frequently as high as 90-95% in vaccine trials

Moreover, compliance is easy to assess – Vaccinations given by health care providers

And instances of noncompliance usually appear to be unrelated to treatment– Conjecture . . . cannot be proven to be

Dropping out of a vaccine trial due to toxicity is uncommon– Vaccines extremely benign compared to therapies

Per-protocol similar to ITT in vaccine trials

If few cases of disease occur before the participants are “fully immunized” and the conditions are met from the prior slide

Then, “it may be possible to estimate the biological efficacy of a full vaccination series without a large magnitude of bias.”

“. . . There should be little risk that an ineffective preventive vaccine will be licensed if conclusions are drawn based on a per-protocol analysis.”

Horne AD, Lachenbruch PA, Goldenthal KL. FDA

Excluding Post-Randomization, Before Series Completion, Outcomes

Intuitively attractive– However, the same argument could be used to exclude all

outcomes, e.g., in a placebo group

At best, if a priori, cannot improve upon randomization– Only can worsen internal validity . . . Additional bias

If a posteriori, certainly biased – Investigators observe results, then conjure up theoretically

justifiable rules that favor their hypotheses– Severely biased, although will appear logical in the paper

• Post hoc lucidity

Exclusions After Randomization (Per-protocol)

Damage internal validity– Can introduce bias– Avoided in design and conduct– Carefully scrutinized in reports

All randomized patients should be analyzed– and analyzed as part of the group to which they

were initially assigned– ITT (intent-to-treat) analysis

Secondary, non-ITT can be performed– Labeled as non-randomized comparisons

800 Low Risk2 HIV+

800 Low Risk2 HIV+

1,000 Placebo

200 High Risk40 HIV+

800 Low Risk8 HIV+

Per-protocol Placebo160 High Risk 792 Low Risk

200 High Risk10 HIV+

Per-protocol Vaccine190 High Risk798 Low Risk

1,000 Vaccine

Randomize 2,000

Outcome Outcome

Participants without HIV Infection

EndEnd

Slides after this will only be Slides after this will only be used to clarify a point if brought used to clarify a point if brought

up in discussionup in discussion

RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had

Survived a Myocardial Infarction

Clofibrate Placebo

5 Year mortality 20.2% 20.9% (p = .55)

Authors state that:• One can justify almost any conclusion, dependent upon the analysis

chosen• Manipulating deviates leads to severe bias• Can you ever do so?

Eliminating deviates from clofibrate

(80% adherence)15.0% 20.9% (p < .05)

Eliminating deviates from both groups 15.0% 15.1%

Thank you for your thoughtful and comprehensive treatment of randomization issues in controlled trials. This email comes with hope that you have a few minutes to respond to a query.

I am currently an investigator at the end of a 4 year . . . trial, writing up a paper I would like to submit to JAMA. I prepared the randomization list and concealment according to procedures you recommended in the 1995 article. . .

During the course of the study, however, my colleagues and I (naïve about applying an intention-to-treat analytic plan) deviated from the original generation list. When, three quarters of the way through sample recruitment, treatment participants were missing all sessions, we assumed this group could be treated as "controls," as they received no treatment. Thus, I took any new "control" envelopes out of the randomization sequence, leaving only the treatment assignments.

Dear Dr Schulz (email)

Soon thereafter, realizing our mistake, but with only 8-10 slots left in enrollment, I put all the controls back in (all of the last assignments were control, in an attempt to re-balance the randomization). Again, my assignment staff were never aware of any deviations in the protocol.

The overall effect is that there are approximately 8 fewer controls than there are for the two treatment groups, but we have worked closely with a statistician to take this deviation into account in the analysis.

My question is to what extent (in how much detail) would you describe this problem to the JAMA reader, and how egregious a mistake is this, in your opinion?

Many thanks in advance for your time and consideration.

Dear Dr Schulz (email, continued)

Flow Chart: Withdrawals but ITT

The Lancet 2004; 364: 772

Exclusions of LFU Damage Internal Validity

Without outcomes from those lost to follow-up, investigators have little choice but to exclude them from the analysis

Any losses damage internal validity– However, differential rates of loss among

comparison groups cause major damage

Investigators must minimize their losses to follow-up

Summary: Per-protocol analyses and “Non-Analyzable” Outcomes

Per-protocol anlayses, if specified a priori, cannot improve upon randomization– But, could lead to bias

If instituted a posteriori, biased and unethical– Even though it may sound logical in paper

Critically, readers cannot determine if per-protocol analysis was really instituted before (protocol) or after– Therefore, using just a rule can taint an article