Engaging in Statistical Engaging in Statistical Practice in Academia is Practice in Academia is

HonorableHonorable

Michael J. SchellMichael J. Schell

Moffitt Cancer Center & Moffitt Cancer Center &

Research InstituteResearch Institute

The Methodologist and the The Methodologist and the PractitionerPractitioner

The roles of a statistical methodologist and a The roles of a statistical methodologist and a statistical practitioner, while overlapping statistical practitioner, while overlapping somewhat, are distinct.somewhat, are distinct.

Academia still needs to learn how to Academia still needs to learn how to properly define the proper expectations properly define the proper expectations and rewards for the statistical practitionerand rewards for the statistical practitioner

FSU Builds a New Academic FSU Builds a New Academic Building Circa 1978Building Circa 1978

Two of following four departments should Two of following four departments should move to the new building: Mathematics, move to the new building: Mathematics, Meteorology, Oceanography, Statistics. Meteorology, Oceanography, Statistics. What split would be best?What split would be best?

Apparently some departments are too Apparently some departments are too academically too close for comfort.academically too close for comfort.

The Farmer and the Cowman Should be FriendsThe Farmer and the Cowman Should be FriendsOklahomaOklahoma – Rodgers and Hammerstein – Rodgers and Hammerstein

Pro-farmerPro-farmerI'd like to say a word for the farmer, I'd like to say a word for the farmer,

He come out west and made a lot of He come out west and made a lot of changes changes

Pro-cowmanPro-cowmanHe come out west and built a lot of fences, He come out west and built a lot of fences,

And built 'em right acrost our cattle ranges. And built 'em right acrost our cattle ranges.

The Farmer and the Cowman The Farmer and the Cowman Should be FriendsShould be Friends

Pro-cowmanPro-cowmanThe farmer should be sociable with the The farmer should be sociable with the

cowboy if he rides by and asks for food cowboy if he rides by and asks for food and water.and water.Don't treat him like a louse make him Don't treat him like a louse make him welcome in your house.welcome in your house.

Pro-farmerPro-farmerBut be sure that you lock up your wife and But be sure that you lock up your wife and daughters! daughters!

Societal Costs to Poor Societal Costs to Poor Medical TrainingMedical Training

While systems designed to cull the best from the While systems designed to cull the best from the rest might be desirable in situations like rest might be desirable in situations like baseball’s World Series, this baseball’s World Series, this oughtought not to be our not to be our design in the training of medical students. We design in the training of medical students. We want want eacheach medical school graduate to have medical school graduate to have mastered mastered allall the material considered important. the material considered important. At present, those doctors in need of remedial At present, those doctors in need of remedial assistance are often not identified until some assistance are often not identified until some kind of ex-post screening process is undertaken, kind of ex-post screening process is undertaken, such as in-service examinations, specialty board such as in-service examinations, specialty board certifications, or certifications, or malpractice lawsuitsmalpractice lawsuits..

Innovator’s Prescription, p. 352-3Innovator’s Prescription, p. 352-3

Medical Schools in La-La LandMedical Schools in La-La Land

What roles are today’s leading medical schools What roles are today’s leading medical schools playing in training the caregivers we’ll need? In playing in training the caregivers we’ll need? In fact, they’re training more and more of the fact, they’re training more and more of the doctors we won’t need, and leaving others to doctors we won’t need, and leaving others to train the professionals we will need. train the professionals we will need. The The medical education system in the United States medical education system in the United States has no means of coordinated planning for has no means of coordinated planning for training doctors for societal needs.training doctors for societal needs.

Innovator’s Prescription, p. 354Innovator’s Prescription, p. 354

Improper Reward Structure Improper Reward Structure in Medical Educationin Medical Education

The only resource allocation mechanism is that The only resource allocation mechanism is that medical students, like the rest of us, choose medical students, like the rest of us, choose careers that are intellectually and emotionally careers that are intellectually and emotionally engaging, with the most attractive incomes and engaging, with the most attractive incomes and lifestyles. … lifestyles. … This has led to a rapid expansion of This has led to a rapid expansion of subspecialists and a dearth of primary care subspecialists and a dearth of primary care physicians. …physicians. …

As for the shortage of primary care physicians, As for the shortage of primary care physicians, America is turning to immigrants from foreign America is turning to immigrants from foreign nursing schoolsnursing schools

Innovator’s Prescription, p. 354-5, 357Innovator’s Prescription, p. 354-5, 357

End Result: Disruption of End Result: Disruption of Medical SchoolsMedical Schools

We anticipate that, given the need for more We anticipate that, given the need for more primary care doctors and nurses that our primary care doctors and nurses that our medical and nursing schools are not medical and nursing schools are not meeting, major integrated health-care meeting, major integrated health-care providers will begin training their own providers will begin training their own caregivers.caregivers.

Innovator’s Prescription, p. 360Innovator’s Prescription, p. 360

How Many Discoveries Have Been Lost by How Many Discoveries Have Been Lost by Ignoring Modern Statistical Methods?Ignoring Modern Statistical Methods?

Rand R. Wilcox, American Psychologist, 1998Rand R. Wilcox, American Psychologist, 1998

Arbitrarily small departures from normality result in Arbitrarily small departures from normality result in low power; even when distributions are normal, low power; even when distributions are normal, heteroscedasticity can seriously lower the power heteroscedasticity can seriously lower the power of standard ANOVA and regression methods.of standard ANOVA and regression methods.

… … most quantitative articles tend to be too most quantitative articles tend to be too technical for applied researchers.technical for applied researchers.

If the goal is to avoid low power, the worst method If the goal is to avoid low power, the worst method is the ANOVA F test.is the ANOVA F test.

British Medical Journal articles by British Medical Journal articles by Doug AltmanDoug Altman

The scandal of poor medical research, 1994The scandal of poor medical research, 1994

Why are errors so common? Put simply, much Why are errors so common? Put simply, much poor research arise because researchers feel poor research arise because researchers feel compelled for career reasons to carry out compelled for career reasons to carry out research that they are ill equipped to perform, research that they are ill equipped to perform, and nobody stops them.and nobody stops them.

Statistics and ethics in medical research. The Statistics and ethics in medical research. The misuse of statistics is unethical, 1980misuse of statistics is unethical, 1980

Ethical Guidelines Discussion, Ethical Guidelines Discussion, (JH Ellenberg, TAS,1983)(JH Ellenberg, TAS,1983)

RD Remington: “The discipline of statistics, which RD Remington: “The discipline of statistics, which emphasizes the development of new emphasizes the development of new methodology and new theoretical structures, is methodology and new theoretical structures, is perhaps comparable to its parent discipline, perhaps comparable to its parent discipline, mathematics …”mathematics …”““The practice of statistics on the other hand, is a The practice of statistics on the other hand, is a profession, involving at every stage judgment profession, involving at every stage judgment and informed choice.”and informed choice.”““The guidelines themselves give only slight The guidelines themselves give only slight notice to a central feature of professional notice to a central feature of professional practice of any type – professional judgment.”practice of any type – professional judgment.”

Ethical Guidelines Discussion (2)Ethical Guidelines Discussion (2)

HV Roberts: “The Guidelines will help to remind HV Roberts: “The Guidelines will help to remind statisticians … that statistical practice requires integrity statisticians … that statistical practice requires integrity as well as professional skill. But [the Guidelines] sound as well as professional skill. But [the Guidelines] sound bland.”bland.”TemptationsTemptations are pervasive, yet subtle: are pervasive, yet subtle:1. to modify one’s best evaluation of the data by what the 1. to modify one’s best evaluation of the data by what the audience or client wants to hear.audience or client wants to hear.3. to reject needed tools on the grounds that they will 3. to reject needed tools on the grounds that they will prove too difficult to explain.prove too difficult to explain.4. 4. to be lax in seeking out the most appropriate statistical to be lax in seeking out the most appropriate statistical tools.tools.7. 7. to neglect checks and safeguards against data to neglect checks and safeguards against data problems, model failure, and processing errors.problems, model failure, and processing errors.

Marketing of PharmaceuticalsMarketing of Pharmaceuticals

1)1) Must have the produced the drug and Must have the produced the drug and shown its efficacyshown its efficacy

2)2) Need to produce the drug in mass Need to produce the drug in mass quantitiesquantities

3)3) MarketingMarketing

Marketing of Statistical IdeasMarketing of Statistical Ideas

1)1) Must have derived the statistic and Must have derived the statistic and demonstrated its efficacydemonstrated its efficacy

2)2) Need to have available softwareNeed to have available software

3)3) Need to disseminate the ideaNeed to disseminate the idea

Key PrincipleKey Principle

In an environment where In an environment where ideas are not marketed, first ideas are not marketed, first on the market winson the market wins

First-on-the-market winnersFirst-on-the-market winners

Chi-square testChi-square test

T-test, 1908T-test, 1908

ANOVAANOVA

Kolmogorov-Smirnov test, 1937Kolmogorov-Smirnov test, 1937

Duncan’s test, 1950Duncan’s test, 1950

Kaplan-Meier curves, 1958Kaplan-Meier curves, 1958

Cox regression, 1972Cox regression, 1972

Three examples where education Three examples where education and practice lags woefully behindand practice lags woefully behind

1.1. Two-sample location problemTwo-sample location problem

2.2. 2 x 2 contingency table2 x 2 contingency table

3.3. Multiple testing adjustmentMultiple testing adjustment

Multiple Testing AdjustmentMultiple Testing Adjustment

Traditional answer: BonferroniTraditional answer: Bonferroni

Improved answer: HolmImproved answer: Holm

Holm adjustmentHolm adjustment

J or Scandinavian Statistics, 1979J or Scandinavian Statistics, 1979

3,432 total citations (3/16/09)3,432 total citations (3/16/09)288 in 2007288 in 2007386 in 2008386 in 2008

Personal thanks: Gary Koch and Alex Personal thanks: Gary Koch and Alex DmietrienkoDmietrienko

Two-sample location problemTwo-sample location problem

Traditional answer: Wilcoxon rank sum test Traditional answer: Wilcoxon rank sum test for small samples, t-test for large samplesfor small samples, t-test for large samples

Better practice: Wilcoxon or normal scores Better practice: Wilcoxon or normal scores test for large samples, t-test for small test for large samples, t-test for small samples (such as n=3, but assumptions samples (such as n=3, but assumptions are critical!)are critical!)

Hodges and Lehmann , 1961Hodges and Lehmann , 196144thth Berkeley Symposium Berkeley Symposium

Hodges and Lehmann (1956) proved that the ARE Hodges and Lehmann (1956) proved that the ARE of the Wilcoxon rank sum test is at least .864of the Wilcoxon rank sum test is at least .864

Chernoff and Savage (1958) proved that the ARE Chernoff and Savage (1958) proved that the ARE of the normal scores test is at least 1of the normal scores test is at least 1

““The above results suggest that on the basis of The above results suggest that on the basis of power, at least for large samples, both the power, at least for large samples, both the Wilcoxon and normal scores tests are preferable Wilcoxon and normal scores tests are preferable to the t-test for general use.”to the t-test for general use.”

First Simulation on Robustness of t-testFirst Simulation on Robustness of t-testCA Boneau, 1960CA Boneau, 1960

320 citations320 citationsAssessed validity only, using three distributions, Assessed validity only, using three distributions,

normal, uniform, exponential and three sample normal, uniform, exponential and three sample size pairssize pairs

Conclusion: t-test is fine.Conclusion: t-test is fine.Later discovery: exponential simulation was done Later discovery: exponential simulation was done

wrongwrong

Highest citation count on any subsequent Highest citation count on any subsequent simulation study (39 articles thru 2000) = 96simulation study (39 articles thru 2000) = 96

Misconceptions Leading to Choosing the t Test Misconceptions Leading to Choosing the t Test Over the Wilcoxon-Mann Whitney Test for Over the Wilcoxon-Mann Whitney Test for

Shift in Location ParameterShift in Location Parameter

J of Modern Applied Statistical Methods, 2005J of Modern Applied Statistical Methods, 2005by SS Sawilowskyby SS Sawilowsky

““The knowledge about the large sample The knowledge about the large sample asymptotic theory “had even penetrated to the asymptotic theory “had even penetrated to the level of a book review written in 1968!” level of a book review written in 1968!”

““The Wilcoxon rank-sum test … show[s] only slight The Wilcoxon rank-sum test … show[s] only slight losses in both large and small sample efficiency losses in both large and small sample efficiency relative to the t-test in the normal case, while in relative to the t-test in the normal case, while in many non-normal cases, efficiency exceeds many non-normal cases, efficiency exceeds 100%”.100%”.

Who was the book review author?Who was the book review author?

Duane Meeter!Duane Meeter!

But who was listening?But who was listening?

On Student’s 1908 ArticleOn Student’s 1908 ArticleJASA March, 2008JASA March, 2008

Comment by Diaconis and LehmannComment by Diaconis and Lehmann

… … even under slight deviations from normality, the even under slight deviations from normality, the t-test can be far from optimal. The poor t-test can be far from optimal. The poor performance of the t-test, particularly for performance of the t-test, particularly for distributions with heavy tails, can be seen in distributions with heavy tails, can be seen in comparison with nonparametric tests, such as comparison with nonparametric tests, such as the Wilcoxon or normal scores tests.the Wilcoxon or normal scores tests.

… … for all distributions with finite variance, the for all distributions with finite variance, the asymptotic relative efficiency relative to t is asymptotic relative efficiency relative to t is ≥ .864 for Wilcoxon and ≥ 1 for normal scores.≥ .864 for Wilcoxon and ≥ 1 for normal scores.

Textbook PlacementTextbook Placement

Basic Practice of Statistics, 4Basic Practice of Statistics, 4 thth Ed. 2006 David S. Ed. 2006 David S. Moore (728 pages)Moore (728 pages)

Non-parametric tests don’t make the book; they Non-parametric tests don’t make the book; they appear in the virtual appendix.appear in the virtual appendix.

Statistics: A Biomedical Introduction, 1977Statistics: A Biomedical Introduction, 1977

Hollander and WolfeHollander and Wolfe

T-test in Chapter 5; Wilcoxon in Chapter 13T-test in Chapter 5; Wilcoxon in Chapter 13

Biostatistics, 2Biostatistics, 2ndnd Ed. van Belle, Fisher, et al., 2004 Ed. van Belle, Fisher, et al., 2004

T-test in Chapter 5; Wilcoxon in Chapter 8T-test in Chapter 5; Wilcoxon in Chapter 8

2x2 contingency table2x2 contingency table

Traditional answer: chi-square test unless Traditional answer: chi-square test unless expected cell count < 5, then Fisher exact expected cell count < 5, then Fisher exact testtest

Better answer: It depends on the design Better answer: It depends on the design (what marginals are fixed), For most (what marginals are fixed), For most practical situations: unconditional testpractical situations: unconditional test

Problems with chi-squareProblems with chi-square

It cheats (is invalid) sometimes and we now can It cheats (is invalid) sometimes and we now can tell where by exact or Monte Carlo results.tell where by exact or Monte Carlo results.

The rule of thumb reduces the violations but The rule of thumb reduces the violations but doesn’t eliminate them.doesn’t eliminate them.

D’Agostino, Chase, and Belanger recommend that D’Agostino, Chase, and Belanger recommend that one can use the chi-square test all the time! one can use the chi-square test all the time! (TAS, 1988)(TAS, 1988)

Enter Barnard’s unconditional testEnter Barnard’s unconditional test

1945,7 Barnard introduces test1945,7 Barnard introduces test1949 Barnard renounces own test1949 Barnard renounces own test19791979 Kempthorne: “The importance of the topic cannot Kempthorne: “The importance of the topic cannot

be stressed too heavily .. 2x2 contigency tables are the be stressed too heavily .. 2x2 contigency tables are the most elemental structures leading to ideas of most elemental structures leading to ideas of association.association.

… … It is remarkable that a consensus has not been It is remarkable that a consensus has not been reached”.reached”.

1985 Suissa and Shuster re-introduce test1985 Suissa and Shuster re-introduce test1985? Mehta (at JSM conference) “I’m not sure I believe in 1985? Mehta (at JSM conference) “I’m not sure I believe in

it … but if people want it (chuckles), I might include it in it … but if people want it (chuckles), I might include it in StatXact.”StatXact.”

Recent ArticlesRecent Articles

Campbell (Stat Med, 2007) recommends the Campbell (Stat Med, 2007) recommends the Mantel-Haenszel chi-square when expected cell Mantel-Haenszel chi-square when expected cell size are 1 or moresize are 1 or more

Lydersen, Fagerland, and Laake (Stat Med, 2009) Lydersen, Fagerland, and Laake (Stat Med, 2009) recommend the unconditional test unless both recommend the unconditional test unless both margins are fixedmargins are fixed

Proschan, and Nason (Bmcs, 2009) “One lesson is Proschan, and Nason (Bmcs, 2009) “One lesson is that when the design dictates the margins … that when the design dictates the margins … one should condition on them.”one should condition on them.”

Ethical ConsiderationsEthical Considerations

Ethical Guidelines for Statistical Practice, ’99Ethical Guidelines for Statistical Practice, ’99 “ “The use of statistics in medical and biomedical research The use of statistics in medical and biomedical research

may affect whether individuals live or die”may affect whether individuals live or die”“… “… society depends on sound statistical practice … society depends on sound statistical practice … many unresolved issues that deserve frank discussion.”many unresolved issues that deserve frank discussion.”

Educators have an ethical responsibility to properly train Educators have an ethical responsibility to properly train their “tool user” students in best practicestheir “tool user” students in best practices

““Tool user” statisticians have an ethical responsibility to Tool user” statisticians have an ethical responsibility to seek best practice information seek best practice information

Insufficient Focus on Insufficient Focus on Teaching GoalsTeaching Goals

… … students would rotate from one clerkship students would rotate from one clerkship to the next in an identical sequence – to the next in an identical sequence – rather than crisscrossing back and forth as rather than crisscrossing back and forth as they presently do, and where they presently do, and where the quality of the quality of students’ educational experiences students’ educational experiences depends to a frustrating degree on “the depends to a frustrating degree on “the luck of the draw”.luck of the draw”.

Innovator’s Prescription, p. 348Innovator’s Prescription, p. 348

Why Medical Schools Won’t Why Medical Schools Won’t ChangeChange

Although there are clear needs for Although there are clear needs for sustaining improvements to medical sustaining improvements to medical education, … education, … we are pessimistic that our we are pessimistic that our leading medical schools will be able to act leading medical schools will be able to act decisively on either frontdecisively on either front. … The reason . … The reason lies in the mechanisms of governance in lies in the mechanisms of governance in these institutions – which are largely these institutions – which are largely collegial and consensus-driven.collegial and consensus-driven.

Innovator’s Prescription, p. 358Innovator’s Prescription, p. 358

Oklahoma RepriseOklahoma Reprise

Territory folks should stick together, Territory folks should stick together, Territory folks should all be pals. Territory folks should all be pals. Cowboys dance with farmer's daughters, Cowboys dance with farmer's daughters, Farmers dance with the ranchers' gals.Farmers dance with the ranchers' gals.