value&impact research dataservices_idcc_2017

This project has received funding from the European Union’s Horizon 2020

research and innovation programme under grant agreement number 674939.

IDCC Workshop February 2017 - Neil Beagrie (Charles Beagrie Ltd)

Demonstrating the Value and Impact of Research Data

Services

Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark



Workshop Programme

13.30-13.45 Brief introduction to CESSDA and the CESSDA-SaW project (Peter Doorn)

13.45-14.30 Presentation and discussion of the Cost-Benefit Advocacy Toolkit (Neil Beagrie)

14.30-15.00 Bringing it together: The Archive Development Canvas

15.00-15.25 Coffee/tea break

15.30-15.45 Presentation and discussion of the Capability Development Model (Mike Priddy)

15.45-16.45 Breakout groups with hands-on opportunities to use and discuss the tools presented

16.45-17.00 Final discussion and close

Strengthening and Widening

Title Text

Brief introduction to CESSDA

and the CESSDA-SaW project

IDCC17 Workshop - Demonstrating the Value and Impact of Research Data Services

12.30-17.00 Monday 20 February 2017

Quincentenary Conference Centre Royal College of Surgeons of Edinburgh

Peter Doorn

Director DANS

Vice-Chair

CESSDA G.A.

What is CESSDA?15 (16) members:

1. Austria

2. Belgium

3. Czech Republic

4. Denmark

5. Finland

6. France

7. Germany

8. Hellas

9. Lithuania

10. Netherlands

11. Norway

12. Slovenia

13. Sweden

14. Switzerland

15. UK

16. Slovak Republic (Observer)

CESSDA provides large-scale, integrated

and sustainable data services to the social

sciences.

CESSDA brings together social science

data archives across Europe, with the aim

of promoting the results of social science

research and supporting national and

international research and cooperation.

CESSDA is now a limited company under

Norwegian law and is owned and financed

by the individual member states’ ministries

of research.

Norway is hosting CESSDA, and the main

office is located in Bergen.

Originally established as an informal

umbrella organisation of data archives in

1976 (> 40 years ago!)

CESSDA will become an ESFRI ERIC later

this year

10 countries with activities aiming at membership: Bulgaria,

Croatia, Estonia, Hungary, Ireland, Italy, Portugal, Romania,

Serbia, Spain

ESFRI Projects

ESFRI Projects have been selected for scientific

excellence and maturity

included in the Roadmap in order to underline their

strategic importance

ESFRI Landmarks

CESSDA and 28 other RIs that were implemented under

earlier ESFRI Roadmaps and that are now established

as “major elements of competitiveness of the European

Research Area”

“These are

successfully

implemented ESFRI

projects that are

featuring top science

services or effectively

advancing in their

construction.”

ESFRI Roadmap 2016

ESFRI = European Strategy Forum for Research Infrastructures

ERIC = European Research Infrastructure Consortium

Service Providers (data archives)

Constitute the operational distributed network

Participate in the governance in an advisory capacity

Deliver data services Service Level Agreements

Adhere to common quality rules and standards (such as the Data Seal of

Approval)

Carry out the CESSDA work plan jointly with Main Office

CESSDA Activities

Coordinating the network of European data service providers and

promotion of the results of social sciences;

Facilitating researcher access to important data resources regardless of

the location of either researcher or data;

Working continuously to include further data sources from Europe and

beyond, into the infrastructure;

Providing training within CESSDA and beyond on best practices of data

services and data management;

Promoting and facilitating wider participation in CESSDA;

Developing and coordinating of standards, protocols and professional

best practices pertaining to the preservation and dissemination of data

and associated digital objects.

Examples of CESSDA Activities

CESSDA Work Plan

Joint Catalogue of Data Resources and Services

Compliance with Data Seal of Approval

Common Data Access and Dissemination Policies (based on FAIR

data principles)

Metadata Management

Open Source Metadata Harvester

CESSDA Persistent Identier Policy

CESSDA Training & Expert Seminars

Euro Question Bank

Research Infrastructure Projects

Strengthening and Widening the European

infrastructure for social science data archives

Overall ambition: to establish the conditions for, and to initiate the

movement towards, a seamless social science data archive service for

the whole of the European Research Area (ERA), which is capable of

supporting the research needs of the next generation of social

scientists wherever in Europe they may be, or beyond.

Aims:

to achieve full European coverage of CESSDA

to strengthen the existing network

to ensure sustainability of its data for the widened network.

Overarching impact: transform the evidence base and research insights

available to those tackling the social and economic challenges of

Europe.



Measuring Value and

Economic Impact of

Research Data

Infrastructure



Presentation Overview

• Measuring Value and Impact of Research Data

Infrastructure (briefly – note focussing on cost-benefit)

• ESDS Impact Study (briefly)

• CESSDA SaW and User requirements survey (briefly)

• Cost-Benefit Advocacy Toolkit (less briefly)

Informal – happy to take questions as we go



Previous Work

Big Science and Innovation Study for BIS July 2013

• Desk review of c. 100 studies internationally;

• 3 studies highlighted to BIS as being particularly good

examples of ‘good practice’ in the measurement of

economic impacts:

• Berkeley Lab 2010

• Human Genome Project 2011

• Economic and Social Data Service 2012

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/249715/bis-13-861-big-science-and-innovation.pdf

http://www.esrc.ac.uk/files/research/research-and-impact-evaluation/economic-impact-evaluation-of-the-economic-and-social-data-service/



John Houghton + Neil Beagrie 4 joint studies to date. 5th in

progress. Methods applied to:

Economic & Social Data Service (ESDS)

Archaeology Data Service

British Atmospheric Data Centre

European Bioinformatics Institute

Swiss Institute of Bioinformatics (in progress)

Value + Impact Analysis of Data Services

http://www.esrc.ac.uk/files/research/research-and-impact-evaluation/economic-impact-evaluation-of-the-economic-and-social-data-service/

http://repository.jisc.ac.uk/5509/1/ADSReport_final.pdf

http://repository.jisc.ac.uk/5382/1/BADCReport_Final.pdf

http://www.beagrie.com/EBI-impact-summary.pdf



Indicative Scale (1)



Indicative Scale (2)



Best Practice from ESDS study• Applies range of

methods;

• Includes counter-

factual;

• Data collection

tailored to different

stakeholders:

depositors, users,

research, teaching;

• Data weighting -

survey value

responses weighted

to reflect the overall

pattern of use from

weblogs;

• Case studies/ KRDS

benefits illustrate

benefits and impact

pathways.



ESDS Study: Return on Investment (ROI)

(ROI - service) Benefit/cost ratio of

net economic value to

ESDS operational costs

5.4 to 1

(ROI - research data creation)

scenarios -Increase in returns

on investment in data creation

arising from additional use

facilitated by ESDS -counterfactual

up to 10 to 1



ESDS Study: Researcher Efficiency Gains

Impact of using ESDS data and services on research efficiency

(after Beagrie et al 2012, p77, Figure 15)

Economic and Social Research Council © 2012 CC-BY licensed



• UK Data Archive one of the largest European social

science data archives: perhaps the largest

• UK Data Archive established for 40 years: has built up

collections and users over time

• Only economic impact study for any social science

data archive to date

• How to extend work for other social science archives?

ESDS Study: Context



Task 4.6: Cost-Benefit Advocacy Programme

Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark



CESSDA-SaW Task 4.6Capturing and communicating the value and economic impact of social science

data services. Develop a benefit/cost advocacy programme and supporting tools;

assembling an evidence base to support the negotiation with ministries and funding

organisations; support advocacy with other core stakeholders such as data creators

and data users. Final publication this summer – presentation of advanced draft

materials.Timetable

Milestone

number

Milestone name Due date

MS26 Survey to gather

and

validate

requirements

April 2016

MS27 Draft Toolset and

testing

Oct 2016

MS28 Focus groups and

case studies

March

2017



CESSDA-SaW

User Requirements Survey



Survey Q1

IN WHICH COUNTRY IS THE DATA SERVICE BASED?



Survey Q2THE DATA SERVICE'S CURRENT STAFFING IS

APPROXIMATELY:



Key Stakeholder Focus Groups



Toolkit Responses



Effort Grading Levels

hours days months



Design criteria

• Easy for overloaded individuals/ smaller services

• Short documents

• Good Infographics

• Synthesis

• Making existing tools easier to use/tailored to

(social science) data services

• Creative Commons CC-BY wherever possible for

ease of re-use



Toolkit Components• Factsheets

• ROI, Benefits, Costs

• Worksheets

• Benefits Summary for a Data Archive

• Archive Development Canvas

• Case studies

• ADP, FSD, LiDA, UKDS

• Selected External Tools

• CCeX, KRDS, CDMA, ESDS Impact, ADP surveys , etc

• User Guide



The Factsheets



ROI Factsheet (1)



ROI Factsheet (2)



Counter-factuals – “Costs of Inaction”

“Ideally, economic impact assessments should estimate the counterfactual – i.e. what would occur in the absence of the facility…However, counterfactuals are rarely addressed in the [c.100] studies reviewed due to lack of data. We found two exceptions that address this issue partially. One is the evaluation of the economic impacts of ESDS (2012) which partially explores the counterfactual through a users’ survey…Another exception is a review of economic impacts of large-scale science facilities in the UK (SQW, 2008) … however, this estimation is not done rigorously and relies mostly on the estimation of the local benefits.”

Big Science and Innovation - Report to BIS -Technopolis 2013



ROI Factsheet (3)



ROI Factsheet (4)



Benefits Factsheet (1)



Benefits Factsheet(2)



Costs Factsheet(1)



Costs Factsheet(2)



Costs Factsheet(3)



Bringing it all together:

The Archive Development

Canvas



The Archive Development Canvas (high-level)



The Archive Development Canvas (detailed)



The Archive Development Canvas (mapped)



Capability Maturity Model

Mike Priddy DANS



Breakout Groups

Flexible session – you choose what you want to

focus on and if you do it individually or in group

(s)

• Factsheets

• ROI, Benefits - in paper copy – read, discuss

content, how you would use it

• Worksheets Benefits Summary for a Data Archive

• Archive Development Canvas Paper version

with prompts

• Capability Maturity Model

—thanks for your attention!

website: www.cessda.net / twitter: @CESSDA_Data