Controversial Issues Require Clear and Objective Methods to Minimize Biases: Response to

Schumm et al. (2014)

Robert E. Larzelere, Ph.D.

Oklahoma State University

I decided to become a family scientist partly because I thought that children and families deserve

the best science that we can achieve. Science should be used to improve the quality of lives and

relationships as much as it has been used to improve technology. Strongly held values can either

help or hinder achieving that goal. Because we all spend important segments of our lives in some

type of family-like relationships, personal values are both more important and more problematic

in family science than in the natural sciences. For a variety of reasons, our culture often seems

incapable of having civil constructive discussions among people with a diversity of values on

important issues. Instead such discussions usually produce more heat than light, especially on

controversial issues. Family science should be shedding more light than heat than other segments

of society, but we are creatures of our own culture and of our own unique family-related

experiences and values. Although the old ideal of value-free science has been shown to favor

insiders’ values over under-represented values, it is not clear that we have developed fairer

alternatives. The ideal goal of unbiased objectivity in science can be challenged by the strong

values people have about issues related to family-like relationships. Arguably no issue in family

science provokes more polarized values than issues relevant to marriage and parenting for gays

and lesbians. As a family methodologist, my response therefore focuses on issues about using

our diversity of values to enhance family science as well as critiquing specific aspects of this

particular study.

This paper by Schumm, Landess, and Williams tries to address one major criticism of a

controversial study on parenting by gays and lesbians (Regnerus, 2012). The criticism is that, in

order to maximize sample size, Regnerus (2012) included any fleeting homosexual activity on

the part of a parent as sufficient to qualify for being counted as part of his gay or lesbian

parenting groups. His homosexual parenting groups were therefore biased, partly because

instability of parents’ intimate relationships was greater in his homosexual parenting groups than

in the comparison groups of heterosexuals. A strength of this new study by Schumm et al. is that

it directly investigates the effect of the stability of parental relationships on outcomes in grown

children within both heterosexual and homosexual parenting groups. Their finding that stability

improved outcomes within both types of parenting couples is an important contribution,

especially now that gay and lesbian marriage is rapidly becoming legal everywhere in the United

States. Children of the most stable parenting partners scored better than less stable parenting

partners on 21 relevant outcomes for heterosexual parents and on 17 relevant outcomes for

lesbian parents. This is remarkably similar given the disadvantages faced by homosexual couples

and their children due to discrimination. It would be less biased to compare effect sizes than to

compare the number of differences to determine whether the stability of parenting couples

matters as much or more among homosexual parents than among heterosexual parents.

Especially on controversial issues, it is crucial to be clear and fair in designing the groups to be

compared, in choice and operationalization of outcome variables, and in reporting the results to

maximize clear interpretations by readers across the entire spectrum of relevant values. Probably

because this paper is a first draft, it seems to fall short of those optimal characteristics at times.

2

First, the study would be strengthened by making the groups as similar as possible except for the

central independent variables (stability and sexual orientation). Although distinguishing a stable

lesbian group is a major improvement on Regnerus’ (2012) operationalization of his same-sex

parenting categories, careful matching would have been better (e.g., Wainright & Patterson,

2008). Using 2+ matches for each same-sex parenting couple would improve statistical power

beyond using only one-to-one matches. Although there is a small sample of stable lesbian same-

sex parents, there is a much larger sample of heterosexual parenting couples from which matched

cases could be drawn to reduce biases in the comparisons being made. It would also be better to

match the groups with the same sexual orientation on irrelevant variables that might bias the

stability results due to confounding if possible. The authors acknowledge that they did not select

an equally stable group from the heterosexual stepfamilies to match the stable lesbian group, but

this is one example of the kind of matching that would improve the group comparisons.

The authors should also specify any important details about outliers, which they mention in the

last section. Especially in small groups, an outlier could dramatically change the results. It would

be helpful to know which conclusions depend upon extreme outliers and which do not. Given the

difficulty of matching closely on multiple variables, some variables could be incorporated as

covariates. For example, the authors could consider controlling for the extent to which the

children and the parents report experiencing discrimination due to homosexuality. Such a

variable might also be a moderator of the effects of the sexual orientation or of parenting stability

on their children. Such a moderator might be useful to predict how much better the children of

homosexual parents could fare in our society if discrimination against them were minimized or

eliminated.

To make the results more clear and transparent, it would be better to report both effect sizes and

statistical significance more consistently and clearly. The reported differences due to stability

within heterosexual families were apparently all significant at the .05 level. The last sentence of

the first paragraph under Results uses the ambiguous term “only small differences,” without

clarifying whether those are marginally significant or not significant. Instead of limiting the

section on stability effects within heterosexual families to that topic, that section concludes by

comparing same-sex families vs. stable heterosexual families on only one outcome, childhood

sexual abuse. Then it features only one misleading statistic, that the highest rate of childhood

sexual abuse occurred for nonheterosexual children from stable lesbian families (43%). This is

most likely a small-sample outlier, since it is based on the 3 of 7 nonheterosexual children who

reported having been sexually abused by a parent or caretaker. The abusers were most likely to

be males, according to rape statistics available to me. Consistent with other results in this study,

this 43% rate is not significantly higher than the rate for nonheterosexual children of

heterosexual stepfamilies. Furthermore, Schumm et al. omitted the fact that the lowest rate of

childhood sexual abuse was for heterosexual children of stable lesbian parents (0 of 21).

Altogether, the children of stable lesbian parents were not significantly more likely to experience

childhood sexual abuse than were children of intact heterosexual couples, p = .08, Fisher’s exact

test.

To the authors’ credit, the next section on the effects of stability for same-sex families

acknowledges the difficulty of detecting significant differences because of the smaller sample

3

sizes. But the authors never clarify which reported differences were statistically significant.

Comparing effect size differences for the most stable vs. less stable family structures would

provide a fairer test of whether stability effects are equivalent within homosexual and

heterosexual families. Matching on important confounding variables or controlling for them

would make the comparisons less biased as well.

For some reason, this draft includes much less information about the third hypothesis, which

compares children of the most stable lesbian families with the two heterosexual groups. It almost

looks like the authors ran up against limits of pages or time to describe their results carefully and

fairly. Although Table 3 orders the outcome variables by effect size of their differences, there is

no information on statistical significance, although I later discovered that the significance was

reported in Table 2 for continuous outcomes, but that information is lacking for binary outcomes

in Table 1. This could be done via 2 X 2 chi-squared analyses, analogous to pairwise

comparisons following ANOVA, using both liberal and conservative controls for Type I errors.

Although the children of intact heterosexual parents were doing significantly better on several

outcomes, that may be due to the greater stability in their family of upbringing than experienced

by any other group. The initial sentence comparing the children of stable lesbian parents vs.

heterosexual stepfamilies seems misleading. The statement is “Second, the extent of differences

between heterosexual stepfamilies and same-sex stable families was much reduced” [compared

to differences favoring intact heterosexual families]. It turns out that the only significant

difference on child outcomes between stable lesbian families and heterosexual stepfamilies

favors lesbian families: Their children had significantly higher self-esteem than do children of

heterosexual stepfamilies. But you have to dig quite a bit to figure that out, because that

conclusion is not obvious either in the text or in Table 3. (It can be deduced from Table 2 if the

reader notices the subtitles of the 1st and 2

nd data columns, which are presented in parentheses.)

The only outcome on which the children of stable same-sex parents are even marginally worse

than children of heterosexual stepfamilies is physical health. But physical health seems to be an

idiosyncratic outcome, given that it is the only variable for which the children of stable same-sex

parenting couples score worse than children of less-stable same-sex parenting couples.

The interaction effects of sexual orientation of children with sexual orientation of parents are

intriguing, but require replications to determine their reliability.

A careful revision of this study, which incorporates this critique and other critiques, could be an

important contribution to the literature and to the welfare of children who may be affected by

policies related to this topic. Family science has benefited from those whose experiences and

values are under-represented in the dominant values in our discipline (Larzelere & Skeen, 1984).

To take an obvious example, the conclusions made during the days of value-free social science

were biased in favor of the values that dominated social science then. Those biases were never

adequately corrected until under-represented voices were able to contribute the kind of

scholarship necessary to correct those biases. Family science always needs to hear credible

critiques and research from peoples and values that are under-represented in our discipline. Two

overlapping recommendations for using the diversity of family-related values to improve family

methodology are (1) to include insider as well as outsider perspectives at all stages of a research

study and (2) to include the widest possible diversity of relevant values on the research team

4

(Larzelere & Klein, 1987). No value position should have veto power, but all value positions

should be heard and evaluated fairly.

Family science would benefit in many ways by incorporating a wider diversity of relevant values

in research deliberations and collaborations. Consider, for example, the relevance of attribution

theory for how family science gets perceived by others. Attribution theory is “the theory of how

people explain others’ behavior” (Myers, 1983, p. 73). For example, how do other social

scientists and/or the general public explain the behavior of a social scientist who makes a

conclusion that supports parenting by gays and lesbians? According to attribution theory’s

discounting principle, people hearing that conclusion will discount its scientific credibility if they

perceive pro-homosexual motivations for that conclusion (Ross & Fletcher, 1985). On the other

hand, the augmentation principle states that the credibility of a conclusion supporting

homosexuals will be strengthened if the social scientist is perceived as having motivations

discouraging such conclusions. That is, to the extent people believe that personal attitudes or

external social pressures favor one conclusion on a controversial issue, they will discount the

perceived scientific objectivity of conclusions supporting the favored conclusion, while they will

augment the scientific credibility of conclusions that disagree with the favored conclusion. The

perceptions of personal biases can arise in several ways, but it is a more viable possibility when a

social scientist’s conclusions invariably support one side of a controversial issue, especially if the

many judgment calls inherent in science seem to consistently favor making the apparently

preferred conclusions (Schumm, 2012). Note that neither the discounting nor the augmentation

principle proves that a conclusion is incorrect, even though both principles can influence the

perceived credibility of that conclusion. It is dangerous, however, for a social science to be too

quick to dismiss conclusions as biased based on under-represented values, because it is precisely

those under-represented values that can correct blind spots due to the dominant mainstream

biases in that social science. Recall that it was primarily minorities and women who exposed the

White male biases when social science pretended to be value-free.

The discounting and augmentation principles can also be based on perceived external pressures

to support one side of a controversial issue. The external pressures for anti-homosexual or pro-

homosexual conclusions have changed dramatically over the last 50 years. It used to be

considered hazardous to a professional career to do pro-homosexual research 50 years ago

(Galliher, Brekhus, & Keys, 2004). Galliher et al. (2004) claim that Laud Humphreys was the

first sociologist to come out as gay in 1974. Attribution theory implies that his pro-homosexual

conclusions would have then been seen as more credibly based on science due to external

pressures in the opposite direction, although his coming out as gay made a discounting principle

more apparent to others. Currently, however, conclusions that fail to support homosexual

advocacy are considered dangerous to one’s professional career, which decreases the perceived

credibility of pro-homosexual conclusions according to the discounting principle, and increases

the perceived credibility of alternative conclusions. A previous president of the American

Psychological Association, who advocated for homosexual rights during the 1970s and served as

a consultant only for Democratic administrations, says that the American Psychological

Association’s credibility as a scientific society has been undermined because its resolutions

invariably support the politically liberal side of any controversial issue (Cummings &

O'Donohue, 2008). From an attribution perspective, this new study by Schumm et al. has the

potential to provide the most convincing evidence to date that children of stable lesbian parents

5

have outcomes that are at least equivalent to children of heterosexual parents who have also

experienced some disruptions in their family structure. If this study can be improved along the

lines suggested in this critique, this resulting paper can provide even stronger evidence that

stable lesbian parents provide a parenting environment that is at least equivalent to heterosexual

parents, with the important exception of heterosexual parents who remain together throughout

the growing up years of their children.

Of course, this study also has other implications that seem somewhat consistent with other

relevant literature. There is a shortage of adequate studies comparing the children of stable same-

sex father couples to comparable heterosexual two-parent couples. Societal support for gay

marriage should increase the number and stability of gay couples who decide to include

biological or adopted children in their family. It would be ideal if objective research could

evaluate such parenting as it increases in quantity and quality in the coming years. Will the

stability of gay and lesbian couples catch up with the stability of heterosexual couples? Can these

stabilities be predicted in a way that would improve the welfare of children affected by

placement decisions regardless of the sexual orientation of the prospective parents?

It would also be ideal if well-trained social scientists across a wide range of relevant value

perspectives could collaborate on major studies or meta-analyses on controversial topics such as

this one. The range of value positions among social scientists who collaborate and will defend

their ultimate conclusions can expand the range of the value positions that will perceive the

studies’ conclusions to be credible. On the other hand, if only a narrow range of dominant value

positions produce conclusions in the name of family science, those conclusions may be viewed

as credible only by an equally narrow range of policy makers. The best examples of experts

trying to incorporate the full range of values on this topic are Loren Marks (2012), David

Blankenhorn (2012, June 22), and Bill Doherty (2013, July 27). Marks sought a wide diversity of

critiques of drafts of his critique of the APA Brief on Lesbian and Gay Parenting, which led him

to be “repeatedly humbled” leading to “a wider scope of vision” (p. 784). Blankenhorn and

Doherty have both advocated for strengthening all marriages regardless of sexual orientation.

Daniel Kahneman, who is the only psychologist known to have won a Nobel prize, claimed that

one key to his success was adversarial collaboration (Kahneman, 2003). Many of his

accomplishments emerged from constructive collaborations with adversaries who disagreed with

his initial research interpretations. Such collaborations are admittedly easier to achieve on less

controversial issues, but may be even more important for controversial issues that matter for

children. There are many forces that are increasing the polarization of the modern world on most

important issues. Civil adversarial collaboration has the potential of representing a much-needed

example of using our diversity to bring out the best in all of us rather than using our value

diversity to produce polarized extremes trying to shout each other down.

6

References Blankenhorn, D. (2012, June 22). How my view on gay marriage changed. New York Times.

http://www.nytimes.com/2012/06/23/opinion/how-my-view-on-gay-marriage-changed.html?_r=1&

Cummings, N. A., & O'Donohue, W. T. (2008). Eleven blunders that cripple psychotherapy in America. New York:

Routledge.

Doherty, W. (2013, July 27). Marriage comes out of the closet. Minneapolis StarTribune.

http://www.startribune.com/opinion/commentaries/217173411.html

Galliher, J. F., Brekhus, W. H., & Keys, D. P. (2004). Laud Humphreys: Prophet of homosexuality and sociology.

Madison: University of Wisconsin Press.

Kahneman, D. (2003). Experiences of collaborative research. American Psychologist, 58, 723-730. doi:

10.1037/0003-066X.58.9.723

Larzelere, R. E., & Klein, D. M. (1987). Methodology. In M. B. Sussman & S. K. Steinmetz (Eds.), Handbook of

marriage and the family (pp. 125-155). New York: Plenum Press.

Larzelere, R. E., & Skeen, J. H. (1984). The method of multiple hypotheses: A neglected research strategy in family

studies. Journal of Family Issues, 5, 474-492. doi: 10.1177/019251384005004004

Marks, L. (2012). "We see what we seek": A rejoinder to the responses of Amato, Eggebeen, and Osborne. Social

Science Research, 41, 784-785.

Myers, D. G. (1983). Social psychology. New York: McGraw-Hill.

Regnerus, M. (2012). How different are the adult children of parents who have same-sex relationships? Findings

from the New Family Structures Study. Social Science Research, 41, 752-770.

Ross, M., & Fletcher, G. J. O. (1985). Attribution and social perception. In G. Lindzey & E. Aronson (Eds.),

Handbook of social psychology (3rd ed., Vol. 2, pp. 73-122). New York: Random House.

Schumm, W. R. (2012). Methodological decisions and the evaluation of possible effects of different family

structures on children: The new family structures survey (NFSS). Social Science Research, 41(6), 1357-

1366. doi: 10.1016/j.ssresearch.2012.08.011

Wainright, J. L., & Patterson, C. J. (2008). Peer relations among adolescents with female same-sex parents.

Developmental Psychology, 44(1), 117-126. doi: 10.1037/0012-1649.44.1.117