Diffusion and Distortion of Evidence in Science

Dr. Rhodri Ivor LengDepartment of Science, Technology,

and Innovation Studies, University of Edinburgh

Rhodri.leng@ed.ac.uk

Introduction

“The anthropologist feels vindicated in having retained his

anthropological perspective in the face of the beguiling charms of

his informants: they claimed merely to be scientists discovering

facts; he doggedly argued that they were writers and readers in the

business of being convinced and convincing others”.

Latour and Woolgar (1979, p.88)

A vast and steeply growing literature

Fig.1: Cumulative growth of articles and reviews

in Web of Science S SCI, 1900-2019. Exponential

fit.

Fig 2: Figure displays growth of publication

and authors in physics since 1900: Source:

Sinatra et al. (2015).

The growth of a topic: Intranasal oxytocin

Fig 3: The left axis shows the number of articles and reviews published per year and indexed in

the Web of Science (WoS) with terms indicating a focus on intranasal oxytocin published since

2000 as of Jan 2021, while the right axis shows the annual citations received by these papers.

0

2000

4000

6000

8000

10000

0

50

100

150

200

250

2000 2005 2010 2015 2020

Publications

Citations

Citation networks (CNA)

• We can represent the scientific literature on any given area of

interest as an interconnected network (graph) of papers – with

connections from a citing document to the cited document.

• Nodes represent single documents, depicted as blue circle

• Edges represent citations from a document to another document,

depicted as blue arrows

Intranasal oxytocin: 2000-2019

Fig 4: Citation network of articles and reviews with terms indicating a focus on intranasal oxytocin in the titles,

abstracts, and associated keywords 2000-2020 (n=1,354; m=22,511) Data extracted from Web of Science.

Nodes coloured by cluster membership and edges between nodes coloured by the colour of the target node.

100 years of oxytocin research

Fig 5: Left image is a citation network clustered by the Leiden algorithm, with nodes coloured by cluster

membership and edges between nodes coloured by the colour of the target node (n=10,357; m=163,668). The image

in the top right is the same network with edges removed

Skewed distribution of citation

• A given network is dominated by a few,

highly cited papers – citation distribution

are heavily right skewed.

• Typically 80% of citations are directed at

only 20% of all publications if we include

articles, reviews, and other documents

types (Gingras 2014)

• The ‘Rich get Richer’ phenomenon; the

‘Matthew Effect’ (Merton 1968);

Cumulative Advantage (Price 1965;

1976)

Fig 6. How often articles and reviews with “oxytocin” in the

title have been cited? Citation distribution of 6,329 oxytocin

articles and reviews published <2010 (total citations:

266,510). Y-axis shows proportion of papers, X-axis shows

proportion of citations.

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

• Fanelli (2012) examined hypothesis-testing papers. In a sample of ~2000 papers derived from

20 disciplines between 2000–2007 and comparing these to ~2000 papers published in the

1990s, Fanelli found that ‘negative findings’ appeared to be disappearing from the literature in

all disciplines. Of papers published between 1990 and 1991, 70% returned findings supportive

to the hypothesis being tested in those papers, but by 2005 this had risen to 89%.

• Many possible contributing factors. Few direct replication studies conducted and Questionable

Research Practices (QRPs), such as P-hacking, HARKing, publication bias, etc. See Ioannidis

(2005), Gelman and Loken (2014) Song et al. (2010), Baker (2016).

A positive skew

Positive Skew II

• Not only are ‘positive findings’ more likely to be published, they are also more likely to

be cited.

• Citation bias refers to the act of preferentially citing evidence of a particular direction

(typically significant ‘positive’ results) and ignoring inconvenient evidence. On

average, it seems, scientists tend to cite studies returning statistically significant

‘positive’ results more than twice as often than those reporting equivocal or negative

findings (Duyx et al. 2017). The authors concluded: “Our results suggest that

citations are mostly based on the conclusion that authors draw rather

than the underlying data.” (p.97)

Citation Bias I

Greenberg (2009) How citation distortions create unfounded authority: analysis of a citation network. BMJ 2009; 339

:b2680

The claim

β amyloid is implicated in inclusion body myositis

The evidence

11 primary studies

5 supported the claim

6 contradicted it

Citation Bias II

Leng RI (2018) A network analysis of the propagation of evidence

regarding the effectiveness of fat-controlled diets in the secondary

prevention of coronary heart disease (CHD): Selective citation in

reviews. PLOS ONE 13(5): e0197716

The claim:

Evidence from RCTs supports the use of dietary

fat modification for treating CHD

The evidence

4 RCTs

1 supported the claim

3 contradicted it

Citations

By 1984, the supportive trial accumulated 259citations, the highest cited unsupportive trial

received just 60.

How were reviews citing these trials?

Distortion of MeaningIn 2010, Andreas Stang published a critique of the Newcastle–Ottawa scale (NOS), a scale introduced by Wells et al.

to assess the quality of non-randomised studies for the purposes of meta-analysis. He came to an unequivocally

critical conclusion:

“I believe Wells et al. provide a quality score that has unknown validity at best, or that includes quality items that

are even invalid. The current version appears to be inacceptable for the quality ranking of both case-control

studies and cohort studies in meta-analyses. The use of this score in evidence-based reviews and meta-analyses

may produce highly arbitrary results” (2010, p.6).

Eight years later, Stang and colleagues (2018) published a second paper calling attention to the widespread inaccurate

citation to ‘Stang (2010)’. In the intervening years, ‘Stang (2010)’ had become the ‘go to’ reference for scientists

seeking something to reference when they used NOS in their publications – accumulating, by 2016, 1,250 citations.

Stan. In a sample of 96 citing systematic reviews, ‘Stang (2010)’ was cited in a manner that suggested it supported use

of the NOS in 94 – and only 18 of these systematic reviews provided any other reference in support of its use.

“…the vast majority of indirect quotations of the commentary have been misleading. It appears that authors of

systematic reviews who quote the commentary most likely did not read it” (2018 p.1030).

Fig 3. Citations to Stang 2010 as of October 2020 as recorded in the

Web of Science Core Collection

0

200

400

600

800

1000

1200

1400

2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Nu

mb

er o

f ci

tin

g re

cord

s

Year of Publication

Distortion of Meaning II

The most recent is a meta-analysis by Xu et al. [7] in

Gene (2020):

“Two investigators independently assessed the

quality of the included literature using the

Newcastle-Ottawa Scale (NOS) scoring system

(Stang, 2010)” (p.2)

Next, is a systematic review and meta-analysis by Zeng

et al. [8] in Nutritional Neuroscience: “We used the

Newcastle-Ottawa scale (NOS) for methodological

quality assessment of cohort and case–control studies

[16]” (p.812; 16 is Stang 2010)

The attempt by Stang et al. (2018) to combat this

erroneous interpretation appears to have fallen on

deaf ears – since 2018, Stang et al.’s analysis of this

misquotation has been cited only six times (as of

October 12 2020, Web of Science).

The Phantom Reference

Sun et al. (2008)

“Rutin is often used as a therapeutical medicine …

which can dilute the blood, reduce capillary

permeability and lower blood pressure [1]”

1. Van der Geer J, Hanraads JAJ, Lupton RA (2000) The

art of writing a scientific article J Sci Commun 163:51–59

By 2019, this phantom had been cited more 480 times,

including 79 times in peer-reviewed journal papers.

In 13 journal papers, this phantom was used to support the

claim that rutin has health benefits

Read before you cite…

• Citations are not a proxy for study quality.

• You cannot take the description of previous findings or the references used to previous studies in

other scientific papers as an accurate or representative account of the existing evidence. You need

to read the original papers yourself.

• Importantly, you need be aware that authors can, and do, selectively cite the evidence that supports

their preferred interpretation. So if you follow the papers that an author cites, you will tend to find

the literature that supports the author’s position, and may miss the ‘inconvenient’ evidence.

Relevant Reading

• Greenberg, SA. (2009). How citation distortions create unfounded authority: analysis of a citation network. BMJ. 339:b2680• Leng, RI. (2018). A network analysis of the propagation of evidence regarding the effectiveness of fat-controlled diets in the secondary prevention of

coronary heart disease (CHD): Selective citation in reviews. PLoS One. 13(5):e0197716. DOI: 10.1371/journal.pone.0197716• Leng, RI. (2020). The phantom reference and the propagation of error. Stable URL:• https://www.the-matter-of-facts.com/post/the-phantom-reference-and-the-propagation-oferror• Leng G., Leng RI. (2020). The Matter of Facts: Skepticism, Persuasion, and Evidence in Science. MIT Press, Cambridge, Massachusetts• Leng, G., Leng RI., Maclean, S. (2019). The vasopressin-memory hypothesis: a citation network analysis of a debate. Ann N Y Acad Sci. 1455(1):126–140• Duyx, B., Urlings, MJ., Swaen, GM., et al. (2017). Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol.

88:92–101• Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics. 90(3):891–904• Latour, B., Woolgar, S. (1979). Laboratory Life, the Social Construction of Scientific Facts. Beverly Hills, CA: Sage• Price, DJ. ([1963] 1986). Little Science, Big Science …and Beyond. New York, NY: Columbia University Press• Price, DJ. (1965). Networks of scientific papers. Science. 149:510–5• Price, DJ. (1976). A general theory of bibliometric and other cumulative advantage processes. J Am Soc Inform Sci Tec. 27(5):292–306• Leng G., Leng RI. (2020). The Matter of Facts: Skepticism, Persuasion, and Evidence in Science. MIT Press, Cambridge, Massachusetts• Gingras, Y. (2014). Bibliometrics and Research Evaluation: Uses and Abuses. Cambridge, MA: MIT Press• Merton, RK. (1968). The Matthew Effect in science. Science. 159(3810):56–63• Sinatra, R., Deville, P., Szell, M., et al. (2015). A century of physics. Nature Physics. 11(10):791–6• Stang, A. (2010). Critical evaluation of the Newcastle–Ottawa scale for the assessment of the quality of nonrandomized studies in meta analyses. Euro

J Epidemiol. 25(9):603–5• Stang, A., Jonas, S., Poole, C. (2018). Case Study in Major Quotation Errors: A Critical Commentary of the Newcastle–Ottawa Scale. Euro J Epidemiol.

33(11):1025–31• Ioannidis, JP. (2005). Why most published research findings are false. PLoS Med. 2(8):e124• Gelman, A., Loken, E. (2014). The Statistical Crisis in Science. American Scientist 102:460• Song, P., Parekh, S., Hooper, L., et al. (2010). Dissemination and publication of research findings: an updated review of related biases. Health Technol

Assess. 14(8):iii, ix-xi, 1–193• Baker, M. (2016). 1500 scientists lift the lid on reproducibility. Nature. 533:452–4