PHARMACEUTICAL STATISTICS

Pharmaceut. Statist. 2003; 2: 3–4 (DOI:10.1002/pst.035)

Lost opportunities for statistics

Stephen Senn*,y

Department of Statistical Science and Department of Epidemiology and Public Health,

University College London, 1–19 Torrington Place, London WC1E 6BT, U.K.

At a recent advisory hearing in the UnitedKingdom on releases to the environment I wasasked to give evidence regarding an experimentthat had been carried out to test some geneticallymodified grain. Grain had been fed to a largenumber of chickens in a small number of pens,birds in the same pen receiving the same grain,either standard or modified, as the case might be.Readers of Pharmaceutical Statistics will recognizethis immediately as an example of cluster rando-mization and will need no instruction in the factthat the chickens within a given pen cannot betreated as independent observations. The hierar-chy of observations must be accounted for, eitherexplicitly (as in some form of mixed model) orimplicitly (say, using summary statistics per pen).

The object of the experiment was to illustratethat there was no difference in the health ofchickens fed modified or standard grain. Unfortu-nately, not only did the authors of the reportignore the hierarchical nature of the data but lackof statistical significance was taken as proof ofequivalence. As I pointed out in my commentary,it is many years since this fallacious argument wasaccepted in drug regulation. Two serious mistakeswere thus made (tending, it must be admitted, inrather different directions) in analysing thisexperiment.

At this point the reader may be forgiven forwandering what this has to do with pharmaceu-tical statistics. After all, we know that bad analysisis widespread outside the industry, and misunder-standing of statistical methods is a common

phenomenon. The reason why this case is ofrelevance here, however, is that the geneticallymodified grain was produced by a leading life-sciences company with a vigorous pharmaceuticalsdivision. Thus the management of the overallcompany had in their employ, albeit in differentdivisions, dozens of statisticians knowledgeableabout both problems of equivalence and mixedmodels, any one of whom, had she or he beenconsulted early enough, could have prevented thisamateurish waste of shareholders’ money. Thisexample is particularly shocking if one considersthat experimental design was a subject that grewup with agriculture through the work of Fisher,Yates and others employed at Rothamsted andother agricultural research stations. How is itpossible for a great company like this, with bothstatistical and agricultural expertise at its com-mand, not to secure adequate statistical advice foranalysis and, worse still, to ignore statisticalprinciples in design?

Why was no statistician amongst the co-authorsof this report? Why, for that matter, werestatisticians not employed by the agriculturaldivision of the company; or, if they wereemployed, why were they not consulted here? Ifear a depressing answer: there was no regulatorypressure to do so. As a result of this hearing, thismay change. The government advisory committeecalled for statistical advice and that is why I waspresent at the hearing. Unfortunately, the truth isthat many managers do not see statistics assomething that adds value, perceiving it insteadas something you do if you have to, which almostalways means when the regulator forces you to.Thus statistics becomes a ritual rather than a vitalactivity in obtaining and evaluating information.

Copyright # 2003 John Wiley & Sons, Ltd.

*Correspondence to: Stephen Senn, Department of StatisticalScience, University College London, 1–19 Torrington Place,London WC1E 6BT, U.K.

yE-mail: stephens@public-health.ucl.ac.uk.

I also fear that the problem lies not just withmanagers but partly with statisticians. For exam-ple, even if we ignore agriculture as being none ofour business, can we claim that as pharmaceuticalstatisticians we are doing enough to ensure thatour subject is being used for every aspect of thepharmaceutical business: not just drug develop-ment but also research, manufacturing and mar-keting? Are we trying to convert the statisticalheathen, and if so have we found the right way topreach the message? Is it possible that theobsessions we have in development regardingdesign and analysis are causing those engaged inother fields to pick up the wrong message?

I was recently involved in giving a series ofcourses on statistical methods in drug research fora leading pharmaceutical company, to whom allpraise for recognizing the importance of thesubject, not only to development but also toresearch. However, it soon became clear to me thatthe life scientists, I was teaching were concentrat-ing on some of the least important aspects ofproblems as being vital. The course was followedby a workshop in which the question mostfrequently put was how to adjust for multiplicity.The context might be one of screening a greatnumber of molecules per year using experimentsthat frequently had many arms but not invariablythe same number. I pointed out that conventionalapproaches to adjusting for multiplicity were quiteinappropriate for several reasons. First, the overalltask was to choose the best drugs to go forward.Penalizing a compound that was one of eight in anexperiment by applying a more stringent adjust-ment than for one that was one of four in anotherwas illogical: both drugs would be candidates forfurther research. The experimentwise error ratewas irrelevant. Second, the conventional 5% levelwas almost certainly inappropriate. Screen 100 000compounds at the 5% level and you send on atleast 5000 to the next stage, the lower boundapplying if none work. Surely management shouldbe engineering the process as a whole, with expertinput from statisticians based on carefully keptrecords, and issuing guidelines accordingly? Third,since so many experiments are being run, shouldnot some way be found of exploiting the mass of

data using, say, an empirical Bayes approach,rather than analysing each experiment in completeisolation? W. Edwards Deming, the guru ofquality, was adamant that it was the duty ofmanagers to study and understand the system forwhich they were responsible and that statistics(appropriately employed) had a great part to playin this. It seemed to me here that nobody in thecompany understood the system as a whole.

The point I am trying to make is thatstatisticians in the pharmaceutical industry havea great opportunity to make a difference to thesuccess or otherwise of their companies. Of coursethere are considerable opportunities in drugdevelopment. Our understanding of planning andanalysis continues to increase and we continue todo better with clinical trials. But the opportunitiesto add more to the value of what is being done aregreater elsewhere because so little use is currentlybeing made of statistical thinking outside clinicaltrials. The title of this journal is PharmaceuticalStatistics. It is to be hoped that it does not turninto Drug Development Statistics, still less intoDrug Trial Statistics. However, the matter is notentirely in the control of the editors, who aredependent on authors to submit papers across awide spectrum of applications. Of course, sincemost statisticians employed within the industrycurrently work on clinical trials a vicious circleoperates.

There are many missed opportunities for apply-ing statistics throughout the pharmaceutical in-dustry from decision analysis in drug discovery topharmacovigilance via forecasting of sales inmarketing, quality control in manufacturing, andpharmacoeconomics [1]. Failure to apply statisticsin these areas is a loss not only to our professionbut also to patients, shareholders and society. Weneed to do more to convince others of the value ofstatistics and not just rely on the regulator to makethe case.

REFERENCES

1. Grieve AP. Do Statisticians count? A personal view.Pharmaceutical Statistics 2002; 1:35–43.

Copyright # 2003 John Wiley & Sons, Ltd. Pharmaceut. Statist. 2003; 2: 3–4

4 S. Senn