it's all a game: the twin fallacies of epistemic purity and the scholarly invisible hand

��It’s all a game: ��

��The twin fallacies of epistemic purity

and the scholarly invisible hand

ICIS Gaming Metrics Carl T. BergstromUC Davis Department of BiologyFebruary 4th, 2016 University of Washington#GamingMetrics @CT_Bergstrom

Responsible science Gaming the System

Image: ASU Center for Biology and Society

1)  The fallacy of epistemic purity��

2)  The fallacy of the invisible hand

Whatmotivatesscientists?

Imagina7on

Reality

Epistemically pure

Epistemically sullied

The field of economics in four words:

peoplerespond toincentives

E.g. Dasgupta and David (1994) Research Policy

•  Non-rivalrous

Scientific discoveries (made public) are public goods.

E.g. Dasgupta and David (1994) Research Policy

•  Non-rivalrous

•  Non-excludable (Arrow’s paradox)

Scientific discoveries (made public) are public goods.

Dasgupta and David (1994) Research Policy

Markets typically under-allocate to the production of public goods because producers are not sufficiently rewarded.


Scientists are rewarded for discoveries in recognition and prestige which are rivalrous and excludable.


Scientists may enjoy solving puzzles and learning about the world, but they are highly motivated to seek recognition for their discoveries.

Research strategy

Publi

catio

n str

ateg

y

Epistemically pure

Socially op7mal

Fraud

Plagiarist


A hypothetical space of scholarly ��

practice

Research strategy

Publi

catio

n str

ateg

y

Epistemically pure

Socially op7mal

Fraud

Plagiarist

most of us most of the 7me

A hypothetical space of scholarly ��

practice

Research strategy

Publi

catio

n str

ateg

y

Science as usual

Social op7mality

Norms of behavior��attempt to carve

up the space of practice

We are struggling to define these norms and figure out where various practices fall.

RESEARCH STRATEGIES

Outcome switching P-‐hacking Garden of forking paths

Uncorrected mul7ple comparison

Icons from Jeff Leek, http://simplystatistics.org/2016/02/01/

We are struggling to define these norms and figure out where various practices fall.

PUBLICATION STRATEGIES

Courtesy authorship Salami slicing Medical ghostwri7ng Cita7on rings

CC images: pixabay.comBroader authorship Cita7on infla7on Self-‐cita7on

Careful: some seemingly questionable strategies��may have beneficial consequences.

Broader authorship

Salami slicing

Cita7on infla7on CC images: pixabay.com

More rapid publication

Better, more interdisciplinary work

Denser citation network

“I’m exploring the fit of my mathematical models of scientific activity to historical data because I think an empirical component will make it easier to find funding for my work.”



Gaming the system?

“

Are scientist, like Adam Smith’s��economic actors,

led by an invisible hand to �� promote an end which was �� no part of [their] intention ?

David Hull 1988; 1997,Brad Wray 2000, etc.

”

Research strategy

Publi

catio

n str

ateg

y

Science as usual

Social op7mality


The invisible hand is the hopethat somehow individual

incentives will lead to social optimality ��

E.g. Kitcher (1993) The Advancement of Science

Modeling efforts attempt to support the invisible hand hypothesis, e.g. by showing how competition can cause scientists to disperse across problems instead of clumping on the most promising one.

Song, Bergstrom, and Foster in prep.

But we’ve recently developed an explicit social welfare analysis of problem selection by scientists.

We find that individual incentives do not, in general lead to socially optimal outcomes.

Oren and Kleinberg 2013

Another model, by Oren and Kleinberg, finds that if scientists differ in aptitude for different problems, there is no way society can incentivize socially optimal behavior.

In science, the invisible hand doesn’t do its job.

Even the good guys are trying to work the system.

Let’s design our norms and institutions accordingly.

Example: self-citation ��

Responsible practice?��

Or gaming the system?

Self-citation rate by gender

●

●

●●●

●●

●

●●●

●●

●●

●●●

●

●●●●

●●●

●●●●

●●●●●●●●●

●●●

●●●●●●●●●●●●●●●

●

●●

●●

●●●

●●●

●●●●●●

●

●●●●●

●●●●●

●●●●●●

●●●●●●●

●●●●●●●●●●●●●

●●●●●●

●●●

●

●

●●

Women's and men's rates of self-citation

��

��

��

��

��

��

��

��-�� / ��

Based on > 3 million papers from JSTOR King et al. in prep.

Rates of rates of self-citation do make a difference to impact metrics, particularly the h-index.

– Cameron et al 2016 Bioscience

Self-citations per authorship

”“

King et al. in prep.

If men self-cite more, are they ��gaming the system?��

��

We face a complicated interplay of norms, strategic incentives and gender differences.��

��

Gaming is a spectrum; the ambiguities matter. ��