PHARMACEUTICAL STATISTICS

Pharmaceut. Statist. 2005; 4: 87–89

Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/pst.169

An unreasonable prejudice against

modelling?

Stephen Senn*,y

Department of Statistics, University of Glasgow, Glasgow, UK

Statistical Principles for Clinical Trials, guideline,ICH E9 of the International Conference onHarmonisation [1], is a sensible document full ofsensible advice. Here is what it has to say aboutcovariates:

In some instances an adjustment for the influence ofcovariates or for subgroup effects is an integral partof the planned analysis and hence should be set out inthe protocol. Pre-trial deliberations should identifythose covariates and factors expected to have animportant influence on the primary variable(s), andshould consider how to account for these in theanalysis in order to improve precision and tocompensate for any lack of balance between treat-ment groups. (Section 5.7)

This coverage of covariates reflects, I believe, thefact that in the pharmaceutical industry we aremore prepared to adjust for covariates thanelsewhere, and I personally think this is good.

Outside the industry you will sometimes en-counter a prejudice against sophistication inanalysis. For example, odds ratios have beencriticized because they are difficult to understand[2] and numbers needed to treat are promotedinstead (see Grieve’s article in this journal for acriticism [3]), and this prejudice also extends to theuse of covariates. Even ICH E9 does not escapeentirely; the passage I quoted continues:

When the potential value of an adjustment is indoubt, it is often advisable to nominate the un-

adjusted analysis as the one for primary attention,the adjusted analysis being supportive.

I am less keen on this. There is a suggestion,perhaps just a hint, that the unadjusted analysis issomehow more reliable than the adjusted one. Thishint becomes something approaching a prejudicein the paper by the Committee for ProprietaryMedicinal Products (now the Committee forMedicinal Products for Human Use, CHMP),‘Points to consider on adjustment for baselinecovariates’ [4], which has this to say:

Alternative analyses should always be presented toconfirm that the conclusions of the study are notsensitive to the choice of covariates included. . .Forordinary linear models, adjusted estimates of thetreatment effect should be compared to unadjustedestimates. (Section IV.5)

I suspect that many readers of PharmaceuticalStatistics will not find this unreasonable. Yet howmany would agree to the following? ‘Where ameta-analysis has been carried out, it should bedemonstrated that the conclusions are not affectedif one of the trials is dropped, even if there are onlytwo trials in the meta-analysis.’ Of course, I havemade that up. However, requiring that an analysisshould give the same answer, whether or not aprognostic covariate is dropped, is closely analo-gous to this, as may be simply illustrated when thecovariate is a baseline.

Suppose that we have an outcome Y and abaseline X and assume that s2X ¼ s2Y ¼ s2 and thatthe correlation between the two measures is r.Construct the change score D ¼ Y � X : This is, in

Copyright # 2005 John Wiley & Sons, Ltd.

Received \60\re /teci

yE-mail address: stephen@stats.gla.ac.uk

*Correspondence to: Stephen Senn, Department of Statistics,University of Glasgow, 15 University Gardens, G12 8QQ,Glasgow.

fact, frequently done and presumably, if this werethe basis of a simple analysis unadorned by anycovariate adjustments, the EMEA would requireno sensitivity analysis. Now consider, however,another score, the total score, T ¼ Y þ X : Thisscore is orthogonal to D by construction and, inthe absence of any treatment by patient interac-tion, is independent of it. Thus we have, that

var Dð Þ ¼ 2 1� rð Þs2

var Tð Þ ¼ 2 1þ rð Þs2

cov D;Tð Þ ¼ 0

Two independent unbiased estimates of the treat-ment effect are now available: one, #tD; based oncontrasts between treatment groups of D, and theother, #tT ; based on contrasts of T.

The situation is analogous to that when we havetwo identical trials, admittedly of rather differentprecision. How would we weight them in a meta-analysis? The answer is, inversely proportional totheir variances. The variances, of course, willdepend on the numbers, n1; n2; in each group,but this is irrelevant for the relative values of #tDand #tT which are identically proportional to q ¼ð1=n1 þ 1=n2Þ; so we just weight according to theinverse of the variances of D and T. Thus we havefor the two weights, wD;wT ;

wD ¼ð1þ rÞ

2; wT ¼

ð1� rÞ2

Hence our overall optimal estimator is

#t ¼1þ r2

#tD þ1� r2

#tT

However, since #tD is just a linear combination ofthe change scores D, and since #tT is the identicallinear combination of the totals T, then #t is thesame linear combination of terms

1þ r2

Dþ1� r2

T

¼1þ r2ðY � XÞ þ

1� r2ðY þ XÞ

¼ Y � rX

However, this is simply the covariance-adjustedoutcome, so that optimal weighting of the two

sources of information is equivalent to analysis ofcovariance.

So, to prefer an analysis of change scores to ananalysis of covariance is analogous to preferringthe results from the larger of two otherwiseidentical trials to the meta-analysis of them both.

What is the loss in this strategy? This can beconsidered by comparing the efficiency of the twoapproaches: change score and analysis of covar-iance. The former has a variance proportional to2ð1� rÞs2 and the latter, as is well known [5, ch.7], to ð1� r2Þs2: The ratio of the latter variance tothe former is the efficiency, and this is

ð1� r2Þs2

2ð1� rÞ2s2¼

1þ r2

Thus, for example, for a correlation of 0.7, theefficiency is only 85%.

To be fair to the ‘Points to consider’ document Iquoted, it does make it clear that some degree ofdiscrepancy between the two analyses is inevitableand should not be a cause for concern. I wonder,however, if we always keep in mind the degree ofdiscrepancy that is possible. Consider the differ-ence between the analysis of covariance andchange-score estimates

#tdiff ¼1þ r2

#tD þ1� r2

#tT

� �� #tD

¼1

2½ðr� 1Þ#tD þ ð1� rÞ#tT �

Now the variances of #tD and #tT are proportionalto 1� r and 1þ r; respectively, and the estimatesare independent. Hence, the variance of #tdiff isproportional to

1

4½ðr� 1Þ2ð1� pÞ þ ð1� rÞ2ð1þ rÞ� ¼

ð1� rÞ2

2

The ratio of this to the corresponding term for thechange-score estimate is thus

ð1� rÞ2=2ð1� rÞ

¼1� r2

so that, since the lowest plausible value of r is 0,the variance of the difference between the twoestimates could be 50% of the variance of thechange score estimate. Of course, in practice it is

Copyright # 2005 John Wiley & Sons, Ltd. Pharmaceut. Statist. 2005; 4: 87–89

88 S. Senn

likely to be much lower. Nevertheless, there is ageneral point widely applicable to statisticalinference, one that was recognized by Fisher 80years ago: if estimates have different variances, it isinevitable that they will disagree from time to time[6]. In fact, using a result of Fisher’s, which statesthat the correlation between an efficient estimateand an inefficient one is the square root of theefficiency of the latter, we have that the correlationbetween the change score and analysis of covar-iance estimate will beffiffiffiffiffiffiffiffiffiffiffi

1þ r2

r

A similar point to the one I am making, applies,of course, when different tests are applied to thesame data [7], for example rank tests and para-metric tests [5, ch. 13]. Requiring these both to besignificant reduces the power of the trial. Unfortu-nately, many CHMP guidelines now require a‘responder’ analysis in addition to a parametricanalysis of the main outcome variable [8]. This isnot only misleading but also inefficient [9,10].

So I will nail my colours to the mast. Where theydisagree, I generally prefer the results of analysisof covariance to simpler models, of proportionalhazards analyses to the log-rank test and oflogistic regression to chi-square tests and, for thatmatter, the within-patient estimates of treatmenteffects in AB/BA cross-over trials to those basedon the first period alone. Analyses that makeintelligent use of prognostic variables can con-siderably increase the precision of our inferences.This is not an objection to simple trials [11].Sometimes the most effective way to collect moreinformation is to run such a trial, and if morepatients can be studied for the same cost andeffort, it may be more efficient than the analysis ofa smaller trial using prognostic information.

However, it is an argument against ignoringprognostic information where this has beencollected.

This, however, is a viewpoint column and otherviews are welcome. I look forward to readers ofPharmaceutical Statistics joining the debate.

REFERENCES

1. International Conference on Harmonisation. Statis-tical principles for clinical trials (ICH E9). Statisticsin Medicine 1999; 18:1905–1942.

2. Altman DL, Deeks JJ, Sackett DL. Down with oddsratios. Evidence-Based Medicine 1996; 1:164–166.

3. Grieve AP. The number needed to treat: a usefulclinical measure or a case of the Emperor’s newclothes? Pharmaceutical Statistics 2003; 2:87–102.

4. Committee for Proprietary Medicinal Products.Points to consider on adjustment for baselinecovariates. European Medicines Evaluation Agency:London, 2001; pp. 1–9.

5. Senn SJ. Statistical Issues in Drug Development.Wiley: Chichester, 1997.

6. Fisher RA. Theory of statistical estimation. Pro-ceedings of the Cambridge Philosophical Society1925; 22:700–725.

7. Ruberg S, Cairns V. Providing evidence ofefficacy for a new drug. Statistics in Medicine1998; 17:1813–1823.

8. Kieser M, Rohmel J, Friede T. Power and samplesize determination when assessing the clinicalrelevance of trial results by ‘responder analyses’.Statistics in Medicine 2004; 23:3287–3305.

9. Senn S. Individual response to treatment: is it avalid assumption? British Medical Journal 2004;329:966–968.

10. Senn S. Disappointing dichotomies. PharmaceuticalStatistics 2003; 2:239–240.

11. Peto R, Collins R, Gray R. Large-scale randomizedevidence – large, simple trials and overviewsof trials. Journal of Clinical Epidemiology 1995;48:23–40.

An unreasonable prejudice against modelling? 89

Copyright # 2005 John Wiley & Sons, Ltd. Pharmaceut. Statist. 2005; 4: 87–89