So#ware reward, cita.on, a0ribu.on
Tracking usage and impact
Neil Chue Hong, Alberto Di Meglio, Josh Greenberg, Juan Lalinde, Kevin
Jorissen
Models of a0ribu.on • Tradi.onal nota.on of cita.ons -‐ authority flows from paper to paper through
cita.on chains – Lots of murkiness when it comes to so#ware. – Cita.on is one way of measuring impact but only one.
• Papers are completed and published before people “use” them so impact is always downstream – So#ware can be published mul.ple .mes. – You write a paper so someone else can read it. Only fix bugs in pre-‐print. – You don't maintain the paper, you publish new work, papers. – We don't check papers for their dependencies and revise them without new work.
• So#ware is more like a long term research project which has many versions (akin to results)
• If you create things which are higher quality, have to be rewarded. – Helping out on forums -‐ huge impact, but recogni.on is zero. – Reward for the so#ware itself should be more than the paper that describes it. – Impact of so#ware should be even greater than the impact of a single paper because it
provides tools for doing many things.
Ways in which we like to be rewarded
• Money – Salary – Prizes
• Recogni.on and Respect – Academic – Peers – Public
• Achievement of long term pla[orm funding • Promo.on and tenure • Being featured by others • Being curated • Chocolate cake
Ways in which we can measure usage and impact
• coun.ng downloads • coun.ng cita.ons on related papers • coun.ng direct cita.ons of so#ware
– about box should give a very clear cita.on that can be copied and pasted • coun.ng numbers of licenses granted • pu]ng in constraints asking for updates on usage as part of the licenses • logging usage through checking for updates (e.g. in Zotero) • webanaly.cs techniques • sta.s.cs from so#ware catalogues, marketplaces, science gateways (e.g. in
nanoHUB) • We want to measure how people are using the so#ware (not just when they are
using it – collect sta.s.cs manually through site administrators registering services at their sites (could
be automa.c) – cita.on of so#ware, generate data when it's used (version used, authors, size of usage) – number of commi0ers, contributors, par.cipants, vitality of community – surveys, site visits, observa.on of scien.sts in daily rou.ne
Changes to make it easier to track usage and impact of so#ware
• Formal way of tracking – DOIs for so#ware? So#ware cita.ons.
• So#ware depositories for reproducible papers (e.g. RunMyCode)
• Be0er upstream prac.ces e.g. always using networked code repositories
• Bu0on in so#ware for "prepare my results and other stuff for publica.on"
What are the biggest issues
• changing the culture surrounding the value and importance of so#ware when looking at career progression (stopping the self-‐reinforcing process)
• how do you rela.vely value someone's contribu.on, and appor.on credit (ar.cula.on of roles?)
• do we understand the core community who can judge the value and impact
• understanding how to cite so#ware so it can be tracked is difficult
Things we’d like to understand • What’s the model of credit for the impact of so#ware on the work it
enables (i.e. what lets you rack up points?) – 1 point every .me a paper cites you or 50 points if a paper that uses you is
cited 50 .mes? • Is there a scien.fic community, many scien.fic communi.es?
– From which communi.es do people want to get recogni.on, and from whom within the communi.es?
• Are there examples where removing the "hierarchical value/weigh.ng" or hyperdifferen.a.ng (extreme differen.a.on of roles) models of a0ribu.on work well in the world of regular scholarly communica.on?
• Should there be a differen.al weigh.ng of the respect that an individual gives (Tripadvisor model vs "wise ones"/Faculty of the 1000) – Who is important in the community for giving out “respected” rewards?
• Can we pick a handful of rela.vely complex pieces of so#ware and ask people involved in the development to assign rela.ve values to each others contribu.ons? Does it change over .me?