0 THERAPEUTICS VOLUME 62 NUMBER 1 JULY 1997

COMMENTARY

Bias against the null hypothesis in maternal-fetal pharmacology and toxicology

Gideon Koren, MD, ABMT, FRCPC Toronto, Ontario, Canada

In the context of this review, bias against the null hypothesis is defined as the systematic discrimination leading to decreased scientific presentation, peer- reviewed publication, and public dissemination of studies that show no or nonsignificant differences between or among the comparison groups. Exam- ples of common comparisons are the effect of drug A versus drug B and the effect of drug C by con- trasting “before” and “after” treatment values.

From The Division of Clinical Pharmacology/Toxicology, Depart- ment of Pediatrics and Research Institute, The Hospital for Sick Children, and the Departments of Pediatrics, Pharmacol- ogy, Pharmacy, and Medicine, University of Toronto.

Supported in part by grants from the Medical Research Council of Canada, the National Health Research Development Pro- gram, Health Canada, Physician Services Inc. (Toronto), and the Motherisk Research Fund.

Based on the Nineteenth Rawls-Palmer Advance in Medicine Lec- tureship, March 5, 1997, San Diego, Calif., as presented to the American Society for Clinical Pharmacology and Therapeutics.

Received for publication March 25, 1997; accepted March 27, 1997.

Reprint requests: Gideon Koren, MD, ABMT, FRCPC, Division of Clinical Pharmacology, The Hospital for Sick Children, 555 University Ave., Toronto, Ontario M5G 1X8, Canada.

Clin Pharmacol Ther 1997;62:1-5. Copyright 0 1997 by Mosby-Year Book, Inc. 0009-9236/97/$5.00 + 0 13/l/82262

While clinicians and researchers often intuitively believe that such bias exists, it is routinely very difficult to prove it because of lack of access to source data. The potential detrimental effects of such bias on the advancement of new, balanced scientific knowledge are as difficult to ascertain be- cause misinformation may cause adverse effects that are difficult to measure.

I have chosen the areas of maternal-fetal drug therapy and maternal-fetal toxicology to explore the sources and potential effects of such bias because of the uniqueness of the clinical situation: More than half of all pregnancies are unplanned,’ leading mil- lions of women every year to unknowingly expose their fetuses to drugs they have been taking. Once such exposures become apparent, high levels of anx- iety are exhibited by women, their families, and health professionals in fear of teratogenic effects. These fears often lead women to consider or even to undergo abortions of otherwise-wanted pregnan- cies. In this context, the existence of epidemiologic studies that show a drug to be safe in pregnancy (and thus not to increase the baseline teratogenic risk of the studied population) is of paramount im- portance. If bias exists against the presentation and publication of such “negative” studies (i.e., studies

1

2 Korepz CLINICAL PHAFMA COLOGY & THERAPEUTICS

JULY 1997

that could not reject the null hypothesis), then women and their health professionals are more likely to be swayed by the “positive” studies that might have been published and brought to their attention.

Since the inception of the Motherisk program, a teratology counseling service for women and their health professionals in the Division of Clinical Pharmacology/Toxicology at the Hospital for Sick Children in Toronto, we have been repeatedly sur- prised by the immediate impact on women of “pos- itive” reports claiming teratogenic effects of various xenobiotic agents, often ignoring a large body of “negative” evidence. After developing2 and validat- ing3 tools to quantify women’s perception of terato- genie risk, we have shown that women exposed to nonteratogenic agents (e.g., dental x-rays or acet- aminophen) assign themselves a 25% risk of major malformations, which is in the range of the known risk of thalidomide.2 The same women correctly as- signed the baseline teratogenic risk in the general population at around 5%.2 These figures have been tragically supported by unfortunate events that fol- lowed the meltdown of the nuclear plant in Chernobyl 10 years ago when pregnant women in Athens, Greece, thousands of miles away from the Ukraines, were erroneously led to believe that they had a sub- stantial teratogenic risk from the radionuclides, lead- ing about half of them to abort their pregnancies.4

Several sources of misinformation have been identified by us: Largely distributed books aimed at pregnant women often contain inaccurate and mis- leading information on teratogenic risk, almost al- ways to alarm rather then to reassure.’ Unfortu- nately, we often encounter cases in which physicians are the source of misinformation, suggesting to women that they abort their pregnancies “to be on the safe side.” The severe public health impact of such misinformation has led us to initiate research aimed at quantifying the extent of the bias against the null hypothesis and at critically evaluating data coming from other centers. These studies will be briefly described and discussed.

Evidence-based characterization of the bias Bias in presentation in scientific meetings. The first

mechanism used by medical researchers to publicize new knowledge is presentation in scientific meet- ings. The Society of Pediatric Research (SPR) an- nually holds a large meeting at which new data of pregnancy outcome after various intrauterine insults are presented, along with many other types of research

topics. With maternal cocaine use achieving epidemic proportions in North America> scores of papers deal- ing with cocaine effects on neonates have been sub- mitted to the SPR’s annual meeting since the early 1980s: The SPR’s abstracts, published by the journal Pediatric Research, allow a unique and rare opportu- nity to assess whether bias against the null hypothesis exists because accepted abstracts are published side by side with rejected ones, with the former being denoted by special symbols.5 Analysis of all abstracts dealing with cocaine in pregnancy revealed that abstracts showing no adverse effects of cocaine had a marginal chance of being accepted for presentation, whereas the majority of those documenting adverse fetal or neona- tal effects were accepted.

Further analysis revealed that the “negative” pa- pers (those showing no adverse effects of cocaine) were overall of superior scientific quality, with more common validation of cocaine use and statistically larger study and control groups of patients.5 This bias would have led participants in the SPR meet- ings to obtain a distorted estimation of the terato- genie risk of cocaine. Because such meetings are the first opportunity for practicing physicians to be up- dated on new knowledge, a bias in the type of in- formation presented to them may have far-reaching effects.

For example, of six abstracts that examined whether maternal cocaine use is associated with the sudden infant death syndrome (SIDS), the two “negative” studies were rejected. In Motherisk we have encountered a perfectly healthy neonate of a cocaine-using mother whose physician put on a home apnea monitor to deal with the risk of SIDS. The single mother in this case could not cope with the instrument and the child was transferred to fos- ter care.5 To date, with a lo-year retroscope it is evident through large controlled studies that co- caine is not a factor that increases the risk of SIDS6; hence the two “negative” SPR reports that were rejected from presentation were scientifically the correct ones.

Bias in publication in peer-reviewed journals. It may be argued that although the decreased presentation of “negative” studies in scientific meetings is serious, it is mainly peer-reviewed scientific articles that shape the status of new knowledge. In 1991 Easter- brook et al7 examined the fate of research projects approved by the Central Oxford Research Ethics Committee from 1984 to 1987.

They confirmed our findings showing that studies with statistically significant results were more likely

CLINICAL P HARMACOLOGY & THEWEUTICS VOLUME 62. NUMBER 1 Koven 3

to be presented in scientific meetings than those with nonsignificant results or “null” results (i.e., the comparison groups are similar). More importantly, “negative” studies had, on average, a 2.3-fold less chance of being published in peer-reviewed journals and, once published, it was in journals with lower citation impact factors. The bias was stronger for observational studies than for interventional proto- cols. Projects producing “negative” studies had sub- stantially less chance of being further funded than those with “positive” results. The authors reiterated that “conclusions based only on a review of pub- lished data should be interpreted cautiously, espe- cially for observational studies. Improved strategies are needed to identify the results of unpublished as well as published studies.“7

Bias in media reporting. With more than 20,000 medical journals, all of us rely unintentionally on electronic media and newspaper reports to be alerted to new medical knowledge. Naturally, when one hears on the news about a new study pertaining to one’s scientific interest, one will search, find, and read that article. However, most often such new stories are not in our specific areas of research, and hence even highly sophisticated and critical scien- tists are “stuck” with what the media have told them. If media reporting is biased against negative data, then clinicians and patients alike will be victims of unbalanced information, stressing the “positive” data.

We have recently compared the rates of newspa- per reporting of two studies, one negative* and one positive,’ published back to back in the Journal of the American Medical Association. Both studies an- alyzed an area of public health concern: radiation as a risk for cancer. Seventeen newspapers publishing 19 reports on the two studies were identified. Nine reports were dedicated entirely to the positive study and 10 reports covered both studies. Of importance, none of the reports were dedicated to the negative study on1y.l’ In reports covering both studies, the mean length of the positive reports was twofold longer than the mean length of the negative reports (354 + 181 versus 192 + 178 words;p = 0.04). The mean quality score of the positive reports was sig- nificantly higher than the score of the negative re- ports (10.1 + 3.4 versus 5.9 + 4.9;~ = 0.02; quality scores could range from 0 to 12).l”

The synergistic negative effects of negative results The above evidence implies that bias against

“negative” results exists at every step of presentation

and dissemination of new medical knowledge. A study with “negative” results is less likely to be accepted for presentation in scientific meetings. If the authors try to publish these data in peer- reviewed journals, they are less likely to be success- ful, and even if they are, publication will likely be in a journal with a lower citation impact. If he or she is lucky enough to publish these data, the chance that the lay media will help disseminate such data is slim.

Although there is no simple way to estimate the cumulative effect of bias in all these steps, there is no doubt that they all act to discourage researchers from publishing “negative” studies, thus creating the file-drawer e$ect, a term coined by Rosenthal” for negative studies that remain hidden in the file drawer and are never published.

Testing the effect of misinformation on bias creation

One of the most thorny issues in the area of maternal-fetal toxicology is the potential effects of mild maternal drinking on pregnancy outcome. An estimated 50% of women of reproductive age drink socially in amounts of up to several drinks a week. With half of the pregnancies to date unplanned, this will translate to about one quarter of all fetuses exposed in utero to some alcohol. During the past 3 decades, hundreds of studies have described the numerous detrimental effects of the fetal alcohol syndrome and fetal alcohol efects, both of which result from heavy maternal drinking during preg- nancy. In contrast, a large body of evidence to date has failed to show adverse fetal outcome after mild and rare drinking in the first trimester. In two recent meta-analyses of all available studies, the Motherisk program has failed to show increased fetal risks of mild maternal drinking.12,13

Although the message that heavy maternal drink- ing may cause fetal damage is very important, nu- merous agencies, public figures, and health profes- sionals extrapolate from it to advise women that even small amounts of alcohol may cause fetal dam- age. In 1991 the Manitoba Medical Association launched a campaign to increase public awareness regarding the reproductive risks of alcohol.i4 As part of this campaign, a l-minute video advertise- ment was produced, and it was aired by several television organizations. We believed this recording to be unduly alarming,15 specifically because it sug- gests that even a single drink during pregnancy can cause fetal harm. Therefore we tried to quantify women’s perceptions of the teratogenic risk of alco-

4 Karen CLINICAL PIitUWA COLOGY &THERAPEUTICS

JULY 1997

ho1 and to evaluate whether this videotape could change their attitudes.16

We studied 30 nonpregnant adult women from 19 to 52 years of age, 20 of whom were public health professionals and 10 of whom were university stu- dents. All were first asked to estimate the amount of maternal alcohol consumption that may cause fetal damage (“even a single drink” or “small amounts every day” or “large amounts every day”). Then they watched the Manitoba videotape and subsequently completed the questionnaire again.

The video contained the following narrative text, accompanied by a view of a fetus frozen in an ice cube that dropped in slow motion into a glass of whisky14:

“Your baby. . . Drinking alcohol during pregnancy may cause physical, mental, and behavioral abnormalities in your baby. There is no known safe amount. Regardless of age or race, any drink containing alcohol puts your baby at risk. What you drink your baby drinks. Is there any choice?”

Subsequently there is a view of a pregnant woman being offered a drink to which she answers:

“No thanks, we are not drinking.”

Before watching the movie, 21 participants be- lieved that it took one drink per day throughout pregnancy to harm a baby, whereas only seven be- lieved that even a single drink during pregnancy could harm the unborn. After the movie, 19 believed that even one drink during pregnancy could harm the fetus (p < 0.0001). No woman changed her view to a less alarming one.16

It can be argued that it is safer to err on the conservative side and to advise women that medic- inal or recreational drugs are teratogenic even if existing data do not reject the null hypothesis. How- ever, at the end of this information line stand mil- lions of women in the reproductive age group who may decide, on the basis of wrong information, to abort otherwise wanted pregnancies. Although it may be politically incorrect to suggest that low doses of an “evil” such as alcohol can be safe in early pregnancy, the medical community should refrain from political or ideologic considerations when eval- uating medical evidence.

The Bendectin saga-anatomy of bias and its detrimental effects

Bendectin, a combination of the H,-blocker dox- ylamine and vitamin B,, was the most widely pre-

scribed antiemetic for nausea and vomiting of preg- nancy in the United States and Canada during the 1960s and 1970s. Because of mounting litigations claiming that the drug caused a variety of unrelated congenital malformations, the drug was withdrawn from the market by its manufacturer in 1983.17 Two different and independent meta-analyses based on more than 100,000 cases have shown that Bendectin does not increase the rates of malformations in general or of specific defects.1s,19 By 1997 most legal claims have been rejected by the courts in the United States.17

The drug is still available in Canada (Diclectin, Duchesnay, Laval, Quebec). In 1989, the Toronto Star “discovered” that a drug that was removed from the American market was still available, carrying a headline that read “Drug still on market feared another thalidomide.” This prompted the Minister of Health and Welfare to appoint an expert com- mittee, which advised him that the drug is safe and should therefore be unambiguously labeled for morning sickness.17

The severe bias against the null hypothesis in this case resulted in the removal of the only U.S. Food and Drug Administration-approved drug for nausea and vomiting in pregnancy, leaving an estimated 2 million pregnant American women each year to ex- perience the natural course of morning sickness. Not surprisingly, then, Neutel and Johansen have shown that the rate of hospitalization of women as a result of severe forms of nausea and vomiting in pregnancy has more than doubled since the removal of Bendectin, carrying direct costs of many millions of dollars every year.20 The indirect costs and detri- mental effects on women’s quality of life have not been yet defined. We recently showed that a sub- stantial number of women who are not offered an antiemetic for severe forms of nausea and vomiting in pregnancy choose to abort their otherwise-wanted pregnancies.21 Because of the common occurrence of nausea and vomiting in pregnancy (50% to 70% of all pregnant women), the bias against the null hypothesis in this case has led to tangible and readily measurable ill effects on the health and well-being of large numbers of women, who remained orphaned from the benefits of pharmacotherapy during a very vulnerable period of their lives.

The effects of the bias on meta-analysis The file-drawer e$ect results in an undefined num-

ber of “negative” studies that remain unknown to the scientific community.” The systematic, quanti-

CLINICAL PHARMACOLOGY & THERAPEUTICS VOLUME 62, NUMBER 1 ICore7.t 5

tative review of studies through meta-analysis is based on the ability to retrieve and analyze all avail- able studies. The fear of bias through missing un- published “negative” reports has led meta-analysts to add to their design a sensitivity analysis that estimates how many unknown “negative” studies with a given effect size are needed to change a significantly positive odds ratio. For example, if a single additional negative study would nullify a sta- tistically significant odds ratio, then the investiga- tors, as well as their readers, should interpret such result extremely cautiously.

Educational implications The notion that “negative” results are not “good

news” is not hidden from graduate and postgraduate students. This may lead them to “toil” to get the data to look “positive” through “massage” of data, by forging results, or by highlighting a spurious nonhypothesis-driven “positive” result in the pres- ence of a largely “negative” message. It is of utmost importance for mentors, supervisors, and academic program directors to make their students aware of the existence of bias against the null hypothesis and its attendant risks. Students must understand that the importance of results should not be dependent on their directionality, but rather on the a priori questions that have been asked and the hypothesis that has driven them. Well-designed studies should be planned in a way that their message will be important with either “negative” or “positive” re- sults.

References 1. [Anonymous]. Better news on population. Lancet

1992;339:1600. 2. Koren G, Bologa M, Long D, Henderson K, Feldman

Y, Shear N. Perception of teratogenic risk by preg- nant women exposed to drugs and chemicals in early pregnancy. Am J Obstet Gynecol 1989;160:1190-4.

3. Koren G, Pastuszak A, Pellegrini E. Prevention of unnecessary pregnancy termination by counseling women on drug, chemical and radiation exposure dur- ing the first trimester. Teratology 1990;41:657-62.

4. Trichopoulos D, Zavitsanos X, Koutis C, Drogari P. The victims of Chernobyl in Greece: induced abor- tions after the accident. Br Med J 1987;295:1100.

5. Koren G, Graham K, Shear N, Einarson T. Bias against the null hypothesis: the reproductive hazards of cocaine. Lancet 1982;2:1440-2.

6. Kandall SR, Gaines J, Habel L, Davidson G, Jessop D. Relationship of maternal substance abuse to sub- sequent sudden infant death syndrome in offspring. J Pediatr 1993;123:120-6.

7. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991; 337:867-72.

8. Jablon S, Hrubec Z, Boia JD Jr. Cancer in population living near nuclear facilities: a survey of mortality nationwide and incidence in two states. JAMA 1991; 265:1403-S.

9. Wing S, Shy CM, Wood JL, Wolf S, Cragle DL, From EL. Mortality among workers at Oak Ridge National Laboratory. JAMA 1991;265:1397-402.

10. Koren G, Klein N. Bias against negative studies in newspaper reports of medical research. JAMA 1991; 266:1824-6.

11. Rosenthal R. The file drawer problem and tolerance for null results. Psycho1 Bull 1979;86:638-41.

12. Devlin J, Agro K, Trepanier E, Makarechian N, Ein- arson T, Koren G. The effect of moderate alcohol consumption on spontaneous abortion, still birth and premature birth: a meta-analysis. Biol Neonate [in press].

13. Polygenis D, Wharton S, Malmberg C, Sherman N, Kennedy D, Koren G, et al. Moderate alcohol con- sumption during pregnancy and fetal malformations; a meta-analysis. Neurotoxicol Teratol [in press].

14. The Manitoba Medical Association. Fetal alcohol syn- drome: the preventable birth defect [videotape]. Win- nipeg, Manitoba: Manitoba Medical Association, 1991.

15. Koren G. Drinking and pregnancy [letter]. Can Med Assoc J 1991;145:1552.

16. Koren G, Koren T, Gladstone J. Mild maternal drink- ing and pregnancy outcome: perceived versus true risks. Clin Chim Acta 1996;246:155-62.

17. Orenstein M, Einarson A, Koren G. Bendectin/ diclectin for morning sickness: a Canadian follow-up of an American tragedy. Reprod Toxic01 1995;9:1-6.

18. Einarson TR, Leeder JS, Koren G. A method of meta-analysis of epidemiological studies. Drug Intel1 Clin Pharm 1988;22:813-24.

19. McKeigne PM, Lamm SH, Linn S, Kutcher JS. Ben- dectin and birth defects: I: meta-analysis of epidemi- ologic studies. Teratology 1994;50:27-37.

20. Neutel CI, Johansen HL. Measuring drug effective- ness by default: the case of Bendectin. Can J Pub1 Health 1995;86:66-70.

21. Mazotta P, Magee L, Koren G. Therapeutic abortions due to severe morning sickness: an unacceptable com- bination. Can Fam Physician [in press].