replichi: graduate student perspectives
Post on 30-Nov-2014
925 Views
Preview:
DESCRIPTION
TRANSCRIPT
RepliCan’tGraduate Student Perspectives
MIT HUMAN-COMPUTER INTERACTION
Michael BernsteinMIT Computer Science and Artificial Intelligence Laboratorymsbernst@mit.edu | @msbernst
Speaking for every graduate student in SIGCHI, I can say one thing:
I can’t speak for everygraduate studentin SIGCHI.
My Unassailable, Extremely ScientificData Collection Protocol:Survey of CHI student volunteers, CHI-students ACM listserv, and snowballed recruitment through Facebook and Twitter.
N=93 responses
Have you ever replicateda study or system?
No
Yes
0 10 20 30 40 50 60 70 80 90
Respondents
17% yes, 83% no
Do you ever plan to replicatea study or system?
38% “Hell yes”, 62% “Hell no”
Hell no
Hell yes
0 10 20 30 40 50 60 70 80 90
Respondents
Set Yourself Apart
The point of research is to come up with exciting ideas that solve problems. Not copy others’ work.
“
”
Set Yourself Apart
I'm more creative than that.“ ”
There’s No Reward
New studies confirming old studies have no chance of publication.
“”
There’s No Reward
Reviewers […] didn't feel replication was necessary even though the original study was specific to a single company.
“
”
There’s No Reward
I very frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”.
“
”
There’s No Reward
I very frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”.
“
”
Responding to IncentivesOpen access and replication. A true scientist’s ideals, but see:The grad student must conform.
“”
Haikusstudies should break
ground replication wastes our
timelet's find new
problems
“”
Haikus
think analyzingCMC is tough? try it
reproducibly!
“”
Haikus
repeat to be surewe stand on giants’
shouldersbut do so on faith
“”
83% have not62% will not
(But we’ll need to replicate the study to be sure.)
Why?
repeat to be surewe stand on giant's shouldersbut do so on faith
“
”
Why?
Replication is a critical component of scientific research, and it should be encouraged and rewarded. The lack of it is detrimental to the scientific soundness of our discipline.
“
”
Why?
we lack the time forreplication of studiesjust review strictly
“
”
Why?
think analyzingCMC is tough? try it reproducibly!
“
”
Why?
It seems like the best outcome for a replication, rather than success, is actually a refutation of the original study.
“
”
Why?
Because its's a waste of time: HCI studies are so small, I know they surely WON'T replicate, so why bother!
“
”
Why?
CHI is too cutting edge for things like replication, or good science, or careful analysis, or the humility to accept that other topics besides Fitts' Law deserve dozens of nearly-identical studies.
“
”
Why?
I do not intend on taking the risk of replicating some of my favorite works unless I see evidence that the CHI community supports such a thing.
“
”
There’s No RewardCase A. Confirmation of the earlier results (very boring)
Case B. Conflict with earlier results (unpublishable problem)
“
”
top related