value impact researchdataservices_esip_2017

Earth Science Information Partners (ESIP) Workshop July 2017

Neil Beagrie (Charles Beagrie Ltd)

Measuring the Value and (Economic)

Impact of Research Data Services

Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark

About us

• Charles Beagrie Ltd – independent consultancy in

digital archive, library, science and research

sectors.

• Neil Beagrie consultant specialising in cost-benefit

analysis, research data curation and digital

preservation, e.g. KRDS projects.

• Prof John Houghton economist specialising in

applying economic methods to information

technology policy, science and technology policy,

and Open Access.

Presentation Overview

• Previous work(note focussing on cost-benefit)

• ESDS Impact Study

• ADS/BADC/EBI Impact Studies

• CESSDA Cost-Benefit Advocacy Toolkit

• Key Challenges and Lessons

Informal – happy to take questions as we go

Why measure value and

(economic) impact?• Advocacy: proving (to non-researchers) that current services

or future services are a worthwhile investment;

• Complementary to other approaches such as data/article

citation analysis;

• Can show more of the impact on research, teaching &

learning, etc.

Note however:

• It is challenging to do;

• Methods well-established but impact data hard to obtain and

to use.

Previous Work

Big Science and Innovation Study for BIS July 2013

• Desk review of c. 100 studies internationally;

• 3 studies highlighted to BIS as being particularly good

examples of ‘good practice’ in the measurement of

economic impacts:

• Berkeley Lab 2010

• Human Genome Project 2011

• Economic and Social Data Service (ESDS) 2012

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/249715/bis-13-861-big-science-and-innovation.pdf

http://www.esrc.ac.uk/files/research/research-and-impact-evaluation/economic-impact-evaluation-of-the-economic-and-social-data-service/

The ESDS Study

Best Practice from ESDS study

• Applies range of

methods;

• Includes counter-

factual;

• Data collection:

users & depositors,

research & teaching;

• Data weighting -

survey value

responses weighted

to reflect the overall

pattern of use from

weblogs;

• Case studies/ KRDS

benefits illustrate

benefits and impact

pathways.

ESDS Study:

Returns on Investment (ROI)

ROI - service Benefit/cost ratio of

net economic value to

ESDS operational costs

ROI - research data creation

scenarios -Increase in returns

from data re-use

facilitated by ESDS (counterfactual)

5.4 to 1

Up to 10 to 1

ESDS Study:

Researcher Efficiency Gains

Impact of using ESDS data and services on research efficiency

(after Beagrie et al 2012, p77, Figure 15)

Economic and Social Research Council © 2012 CC-BY licensed

The Value and Impact Studies

Economic & Social Data Service (ESDS)

Archaeology Data Service

British Atmospheric Data Centre

European Bioinformatics Institute

Value + Impact Analyses

of Data Services

John Houghton + Neil Beagrie 4 joint studies to date.

5th in progress. Methods applied to:

http://www.esrc.ac.uk/files/research/research-and-impact-evaluation/economic-impact-evaluation-of-the-economic-and-social-data-service/

http://repository.jisc.ac.uk/5509/1/ADSReport_final.pdf

http://repository.jisc.ac.uk/5382/1/BADCReport_Final.pdf

http://www.beagrie.com/EBI-impact-summary.pdf

Approaches and Methods

• We combined quantitative and qualitative approaches.

• All studies used:

• Desk-based analysis of existing literature and reports;

• Existing management and internal data collected by the data

services; and

• Original data collection in the form of online surveys, together with

semi-structured interviews.

Indicative Scale (currently £1.00 = $1.31)

The Value and Impact of the British Atmospheric Data Centre

Key Findings The economic analysis shows that:

• increase in research efficiency estimated as at least £10 million per

annum.

• Increase in ROI from the data re-use facilitated by BADC estimated

as between £11 million and £34 million over thirty years (net present

value) from one-year’s investment.

The qualitative analysis in the user survey showed that:

• 81% of the academic respondents reported that BADC was very or

extremely important for their academic research, and

• 53% of respondents reported that it would have a major or severe

impact on their work if they could not access BADC data and

services.

In the depositor survey:

• 52% suggested that being unable to deposit data with BADC would

have a major or severe impact on them.

This project has received funding from the European Union’s Horizon 2020

research and innovation programme under grant agreement number 674939.

CESSDA SaW Toolkit

ESDS Study: Context

• UK Data Archive one of the largest European

social science data archives: perhaps the largest

• UK Data Archive established for 40 years: has

built up collections and users over time

• Only economic impact study for any social

science data archive to date

• How to extend work for other social science

archives?



CESSDA-SaW Task 4.6

Capturing and communicating the value and

economic impact of social science data services.

Develop a benefit/cost advocacy programme and

supporting tools; assembling an evidence base to support

the negotiation with ministries and funding organisations;

support advocacy with other core stakeholders such as

data creators and data users.

Toolkit published April 2017.



CESSDA-SaW

User Requirements Survey



Survey Q2

THE DATA SERVICE'S CURRENT STAFFING IS

APPROXIMATELY:



Toolkit Components

• Factsheets

• ROI, Benefits, Costs

• Worksheets

• Benefits Summary for a Data Archive

• Archive Development Canvas

• Case studies

• ADP, FSD, LiDA, UKDS

• Selected External Tools

• CCeX, KRDS, CDMA, ESDS Impact, ADP surveys , etc

• User Guide



The Factsheets



ROI Factsheet (1)



ROI Factsheet (2)



Counter-factuals – “Costs of Inaction”

“Ideally, economic impact assessments should estimate the

counterfactual – i.e. what would occur in the absence of the

facility…However, counterfactuals are rarely addressed in the

[c.100] studies reviewed due to lack of data. We found two

exceptions that address this issue partially. One is the

evaluation of the economic impacts of ESDS (2012) which

partially explores the counterfactual through a users’

survey…Another exception is a review of economic impacts of

large-scale science facilities in the UK (SQW, 2008) …

however, this estimation is not done rigorously and relies

mostly on the estimation of the local benefits.”

Big Science and Innovation - Report to BIS - Technopolis

2013



ROI Factsheet (3)



Benefits

Factsheet (1)



Benefits

Factsheet (2)



Benefits Factsheet (3)



Costs

Factsheet (1)



Costs

Factsheet (2)



Bringing it all together:

The Archive Development Canvas



The Archive Development Canvas (detailed)



The Archive Development Canvas (mapped)

Key Challenges and Lessons

EBI Report: User population• How ‘actual direct users’ were estimated from web stats:

• Unique host log of 10.8 million (Annual figure for 2014) adjusted

to reflect key considerations:

• People behind unique IPs in web stats usually over-estimated by a

factor of 7 (Formitchev, 2010)

• EMBL-EBI accessed by organisational IPs (survey: 4185 responses

from 3622 unique IPs): Factor in 1.16 per unique IP

• Respondents accessed 9 services on average – 10.8 million figure

captured across EMBL-EBI resources

• Total user population estimate: 198,000 unique, actual

direct users

Online Surveys

(BADC)

• Two independent online surveys were conducted by CB

Ltd;

• Survey questionnaires developed iteratively - small group

of BADC customers helped test pilot versions;

• Given the nature of subject and diverse customer base

substantial effort was needed to design questionnaires;

• Significant effort spent to reduce the likely burden on

questionnaire recipients.

Online surveys (BADC)

Significant time spent on:

• wording of the invitations to participate

• advance notice of the surveys via websites, news feeds, social

media and email shots

Five £50 Amazon vouchers offered in a draw for participants

Surveys had high response rates and completion rates



Toolkit challenges & lessons

• diverse scale / type of organisations

• complexity of topic / local resources



Effort Grading Levels

hours days months



Design criteria

• Easy for overloaded individuals/ smaller

services

• Short documents

• Good Infographics

• Synthesis

• Making existing tools easier to use/tailored to

(social science) data services

Conclusions

• Value and (economic) impact analysis doable but

challenging;

• Difficulties and critical points for economic analysis include:

• Limited knowledge of real user populations;

• Survey sample and survey design;

• Complexity of mature data services and their resources;

• "Costs of inaction" data - counter-factuals for data

services.

• CESSDA SaW toolkit approach adaptable to other disciplines

and can tailor for/utilise sector specific data and needs.

Questions?