PaNdata Photon and Neutron Data Infrastructure

3 November 2011

Juan Bicarregui

Safety Information for use at RAL

Fire bells... Please leave the building by the closest route

and go to the fire assembly point.

Klaxon... Please stay inside the building and close any

doors or windows having external access.

Emergency Number for FIRE, AMBULANCE or FIRST AID.

short code from any internal phone 2222

or from a mobile phone 01235 778888

Additionally... Please take a few moments to check that any equipment that may be plugged into the mains supply has undamaged leads, no exposed cables, a secure plug and that you have not created a trip hazard especially if you are on an escape route!

Agenda - 3rd Nov 2011 12:00 Arrivals and Lunch14:00 Start of meeting

14:00 Introduction, Introductions and review agenda (Juan Bicarregui )14:05-15:30 Overview of projects (6 projects, 10 minutes each)“Vertical” projects• PaNdata (Juan Bicarregui and others)• CRISP (Jean Francois Perrin)• HDRI (Rainer Gehrke)“Horizontal” Projects• EUDAT (David Corney)• OpenAirePlus (Natalia Manola)• ORCID (Cameron Neylon)Any other related projects or activities (Round Table)15:30 Break16:00-17:30 Discussions on technical areas. Identification of areas for cooperation. Eg • Metadata catalogues and cross searching;• Ids for Data and/or publications. • Unique identification and disambiguation of people; 17:30 End (Taxis to MHH at 17:30, Dinner at MHH at 20:00)

Agenda – 4th Nov 2011 Morning (ISIS TS2 CR16-17, Taxis from MHH at 8:30)

9:00 Introduction (Juan Bicarregui)• Review agenda, Overview, Introductions (JCNS, MaxLab), etc.• Review of any actions from the day before9:15 - 10:45 PaNdata ODI Service Activities • WP3 User AAA Service - Heinz Joseph Weyer (20 mins)• WP4 Data Catalogue Service - George Kourousias (20 mins)

– Common Data Model Access project - Alain Buteau (15 mins)• WP5 Virtual Laboratories - Frank Schluenzen (15 mins)• Discussion (20 mins)10:45 Break11:00 PaNdata ODI Joint Research Activities (3 x 20 minutes + 15)• WP6 Provenance - Brian Matthews (20 mins)• WP7 Preservation - Jean Francoise Perrin (20 mins)• WP8 Scalability - Bill Pulford (20 mins)• Discussion (15 mins)12:15 Lunch

Agenda – 4th Nov 2011 Afternoon (ISIS TS2 CR16-17)

13:15 Dissemination and Engagement• Dissemination (both projects) - Frank Schluenzen (15 mins)• IUCr Working group on Data Deposition - Heinz Joseph Weyer and Alun Ashton (15 mins)13:45 Finalising PaNdata Europe• Review of actions from Period 1 review - Simon Lambert (5 mins)• Remaining deliverables - status and plans • D2.4 Integrated Policy (Due Sept) - Rudolf Dimper (5 mins)• D3.4 Dissemination (Due Nov) - Frank Schluenzen (5 mins - covered above?)• D6.3 and 6.4 Software (Due Sept and Nov) - Mark Johnson (10 mins)• D7.3 and 7.4 Integration (Due Sept and Nov) - Brian Matthews (10 mins) • D1.4 Final Management Report (Due Nov) - Simon (5 mins) • Finances - Juan (5 mins)• Initial planning for the final review – Juan (5 mins) • Collaboration with the US (INFRA-2012-3.2 and 3.3) - Juan (10 mins)14:45 Break15:00 Starting PaNdata ODI.• Management and Administration (Juan Simon, Denise) (30 mins)• Contractual Agreements (GA and CA)• Prepayments• Procedures• Reporting• Review of Early Deliverables and short term plans - Juan (15 mins)• Action review – Juan (15 mins)16:00 Close

Agenda - 3rd Nov 2011 12:00 Arrivals and Lunch14:00 Start of meeting

14:00 Introduction and review agenda (Juan Bicarregui )14:05-15:30 Overview of projects (6 projects, 10 minutes each)“Vertical” projects• PaNdata (Juan Bicarregui and others)• CRISP (Jean Francois Perrin)• HDRI (Rainer Gehrke)“Horizontal” Projects• EUDAT (David Corney)• OpenAirePlus (Natalia Manola)• ORCID (Cameron Neylon)Any other related projects or activities (Round Table)15:30 Break16:00-17:30 Discussions on technical areas. Identification of areas for cooperation. Eg • Metadata catalogues and cross searching;• Ids for Data and/or publications. • Unique identification and disambiguation of people; 17:30 End

Introduction and review agenda

• Thank you!• Introductions (round table)• Structure of EU e-Infrastructure Programme• Reason for invitation

– Technical and organisation benefits• Aims for the afternoon

– Schedule for the afternoon

••• 8

OpenAIREplus

M€ 95

2.3.5. PRACE 3.4 SA – 3.5 NCPs 1.2.2 Data 1.2.1 e-Science env.

Bdg

M€18 M€ 5 M€ 45 M€ 27

EarthServer

BioVeL DRIHM

SCI-BUS

VERCE N4U

GLORIA

SCIDIP-ES ESPAS

transPLANT

PanDataODI ENGAGE

diXa iMarine

agINFRA

EUDAT

PRACE-2IP Discover the COSMOS

FISCAL

ELLA

Virtual Campus Hub

GLOBAL excursion

M€ 18 M€ 5 M€ 45 M€ 27

EuroRIs-Net+

ORIENTplus

FP7-Infrastructures Call 9 Projects Overview

In addition: Exa-scale HPC Call – 3 projects - M€ 25

Tools for virtual research environments

Tools for virtual research environments

Generic services, storage and computation

OA participatory infrastructure

Agricultur

e

Environment

Physics, Engineering

Biolo

gy

Medici

n

e

Atmosphere/Space Physics

Social SciencesScientific Data

(Discipline Specific)

Other Data

Researcher 1

Non Scientific World

Scientific WorldResearcher 2

Aggregated Data Sets(Temporary or Permanent)

Workflows

Aggregation Path

transPLANT

EUDAT

AgINFRA

iMarine

OPENAire Plus

diXa

SCIDIP-ES

ESPAS

ENGAGE

PanDataODI

Scientific Data Landscape of Initiatives – results from call9

VREs

VREs

Introduction and review agenda

• Thank you!• Introductions (round table)• Structure of EU e-Infrastructure Programme• Reason for invitation

– Technical and organisation benefits• Aims for the afternoon

– Schedule for the afternoon

Agenda - 3rd Nov 2011 12:00 Arrivals and Lunch14:00 Start of meeting

14:00 Introduction and review agenda (Juan Bicarregui )14:05-15:30 Overview of projects (6 projects, 10 minutes each)“Vertical” projects• PaNdata (Juan Bicarregui and others)• CRISP (Jean Francois Perrin)• HDRI (Rainer Gehrke)“Horizontal” Projects• EUDAT (David Corney)• OpenAirePlus (Natalia Manola)• ORCID (Cameron Neylon)Any other related projects or activities (Round Table)15:30 Break16:00-17:30 Discussions on technical areas. Identification of areas for cooperation. Eg • Metadata catalogues and cross searching;• Ids for Data and/or publications. • Unique identification and disambiguation of people; 17:30 End

Overview

The PaNdata Collaboration

The Vision

The PaNdata Europe Project

The PaNdata Open Data Infrastructure Project

Looking Forwards

The PaNdata Collaboration• Established 2007 with 4 partners• Expanded since to 11 (now 13) organisations

(see next slide)

• Aims: – “...to construct and operate a shared data

infrastructure for Neutron and Photon laboratories...”

2007 2008 2009 2010 2011 2012 2013 2014 EDNS (4) EDNP (10) PaNdataEurope(11) Pandata ODI(11)

PaN-data bring together 11 major European Research Infrastructures

PaN-data is coordinated by the e-Science Department at the Rutherford Appleton Laboratory, UK

ISIS is the world’s leading pulsed spallation neutron source

ILL operates the most intense slow neutron source in the world

PSI operates the Swiss Light Source, SLS, and Neutron Spallation Source, SINQ, and is developing the SwissFEL Free Electron Laser

HZB operates the BER II research reactor the BESSY II synchrotron

CEA/LLB operates neutron scattering spectrometers from the Orphée fission reactor

ESRF is a third generation synchrotron light source jointly funded by 19 European countries

Diamond is new 3rd generation synchrotron funded by the UK and the Wellcome Trust

DESY operates two synchrotrons, Doris III and Petra III, and the FLASH free electron laser

Soleil is a 2.75 GeV synchrotron radiation facility in operation since 2007

ELETTRA operates a 2-2.4 GeV synchrotron and is building the FERMI Free Electron Laser

ALBA is a new 3 GeV synchrotron facility due to become operational in 2010

PaN-data Partners

JCNS Juelich Centre for Neutron Science MaxLab, Max IV Synchrotron

PaN-data Applications

The partners operate hundreds of instruments used by over 30,000 scientists each year

These instruments support scientific fields as varied as:• Physics, Chemistry, Biology, Material sciences, Energy technology,

Environmental science, Medical technology and Cultural heritage

Applications include:

• crystallography that reveals the structures of viruses and proteins important

for the development of new drugs

• neutron scattering that identifies stresses within engineering components

such as turbine blades

• tomography that can image microscopic details of the 3D-structure of the

brain

Industrial applications include pharmaceuticals, petrochemicals and microelectronics

PaN-data Europe – building a sustainable data infrastructure for Neutron and Photon laboratories

Overview

The PaNdata Collaboration

The Vision

The PaNdata Europe Project

The PaNdata Open Data Infrastructure Project

Looking Forwards

Science driver – Data IntegrationNeutron diffraction X-ray diffraction

}NMR

High-qualitystructure refinement

}

What is e-Infrastructure?

DataCreation

Archival

Access

Storage ComputeNetwork

Services

Curation

the researcher actsthrough ingest and access

Virtual Research Environment

the researcher shouldn’t have to worry about the information infrastructure

Information Infrastructure

EDNS - European Data Infrastructure for Neutron and Synchrotron Sources

PaNdata Vision

Single Infrastructure Single User Experience

CapacityStorage

Publications Repositories

Data Repositories

Software Repositories

Raw Data

Data Analysis

Analysed Data

Publication Data

Publications

Facility 1

Raw Data

Data Analysis

Analysed Data

Publication Data

Publications

Facility 2

Raw Data

Data Analysis

Analysed Data

Publication Data

Publications

Facility 3

Different Infrastructures Different User ExperiencesRaw Data Catalogue

Data Analysis

Analysed Data Catalogue

Publication Data Catalogue

Publications Catalogue

In words:PANdata will provide our user communities with data repositories and data management tools to: • deal with large sets and large data rates from the experiments, • enable easy and standardised annotation of data, • allow transparent and secure remote access to data, • establish sustainable and compatible data catalogues, allow long-term preservation of data, and • provide compatible open source data analysis software.

This will have a major impact on our scientific user community because it will offer: • cross facility and cross discipline data analysis, • secure access to large data sets over the network instead of using portable media, • maintaining the records of science by having properly annotated data, • linking publications to data, • allowing efficient software developments, and

• efficient scientific collaborations across Europe by providing compatible data formats and analysis software.

Metadata and Digital Curation

Proposal

Approval

SchedulingExperiment

Data cleansing

Record Publication

Scientist submits application for

beamtime

Facility committee approves application

Facility registers, trains, and schedules

scientist’s visit

Scientists visits, facility run’s experiment

Subsequent publication registered

with facility

Raw data filtered and cleansed

Data analysis

Tools for processing made available

Overview

The PaNdata Collaboration

The Vision

The PaNdata Europe Project

The PaNdata Open Data Infrastructure Project

Looking Forwards

PaN-data Standardisation

PaN-data Europe is undertaking 5 standardisation activities:

1. Development of a common data policy framework

2. Agreement on protocols for shared user information exchange

3. Definition of standards for common scientific data formats

4. Strategy for the interoperation of data analysis software enabling the most appropriate software to be used independently of where the data is collected

5. Integration and cross-linking of research outputs completing the lifecycle of research, linking all information underpinning publications, and supporting the long-term preservation of the research outputs

PaN-data Europe – building a sustainable data infrastructure for Neutron and Photon laboratories

PaN-data Europe TimelinePaN-data Europe runs from June 2010 until December 2011 with workshops in Spring and Autumn 2011.

PaN-data Europe – building a sustainable data infrastructure for Neutron and Photon laboratories

Workpackage (abbreviated title) Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

Milestones M1 M2 W1 M3 M4 W2

WP1 Management D D D D WP1 Management

WP2 Common data policy framework D D D D WP2 Common data policy framework

WP3 Knowledge exchange/dissemination D D D D WP3 Knowledge exchange/dissemination

WP4 Common user information exchange D D D WP4 Common user information exchange

WP5 Scientific data D D D WP5 Scientific data

WP6 Data analysis software infrastructure D D D D WP6 Data analysis software infrastructure

WP7 Integration and cross-linking D D D WP7 Integration and cross-linking

Key

D - Deliverable

M - MilestoneW - Workshop

Workpackage (abbreviated title)

Workshops

Data Policy

Development and delivery

of the comm

on data policy

User and Data Standards

Delivery of draft standards

for data and user information

Baseline for integration

Delivery of policy on user inform

ation, first report on

publications and integration

Integration proposalDelivery of policy and

first proposal on integration and on analysis soft

ware

Final Workshop

Final reports on standards

M1

M2

M3

M4

2.1 Data Policy

2.2Software

Policy

2.3UserPolicy

2.4Integrated

Policy

4.1User

Proposal

4.2User

Workshop

4.3User

Revision

5.1Data

Proposal

5.2Data

Workshop

5.3Data

Revision

6.1SoftwareReview

6.2Software

Workshop

6.3SoftwareProposal

6.4Software Revision

7.1Integration

Report

7.2Integration Proposal

7.3Integration Revision

3.4

Final

Workshop

Project Management, Knowledge Exchange and Dissemination Activities

Dependencies between the major project tasks

Dependencies

Overview

The PaNdata Collaboration

The Vision

The PaNdata Europe Project

The PaNdata Open Data Infrastructure Project

Looking Forwards

ERA Open Access Sharing Initiatives (examples, etc)

ERA Infrastructure Platform Initiatives (EGI, etc)

PaNdata Support Action

(Ends 30 Nov 11)

Policies and Standards

PaNdataODI

(begins end2011)

JRAs

Users

Data

Software

Integration

Provenance

Preservation

Scalability

PaNdataODI

(begins end2011)

ServicesUsers

Data

PaNdataODI

Virtual Labs

Policies Powder Diff

SAXS & SANS

Tomography

ObjectivesObjective 2 – UsersTo deploy, operate and evaluate a system for pan-European user identification across the participating facilities and

implement common processes for the joint maintenance of that system.

Objective 3 – DataTo deploy, operate and evaluate a generic catalogue of scientific data across the participating facilities and promote

its integration with other catalogues beyond the project.

Objective 4 – Provenance To research and develop a conceptual framework, defined as a metadata model, which can record the analysis

process, and to provide a software infrastructure which implements that model to record analysis steps hence enabling the tracing of the derivation of analysed data outputs.

Objective 5 – PreservationTo add to the PaNdata infrastructure extra capabilities oriented towards long-term preservation and to integrate

these within selected virtual laboratories of the project to demonstrate benefits. These capabilities should, as for the developments in the provenance JRA, be integrated into the normal scientific lifecycle as far as possible. The conceptual foundations will be the OAIS standard and the NeXus file format.

Objective 6 – Scalability To develop a scalable data processing framework, combining parallel filesystems with a parallelized standard data

formats (pNexus pHDF5) to permit applications to make most efficient use of dedicated multi-core environments and to permit simultaneous ingest of data from various sources, while maintaining the possibility for real-time data processing.

Objective 7 – DemonstrationTo deploy and operate the services and technology developed in the project in virtual laboratories for three specific

techniques providing a set of integrated end-to-end data services.

PaNdata ODI Joint Research Activities

PaNdata ODI Service Activities

PaNdata ODI Service ReleasesStandards from

PaNdataSupport Action

uCat

dCat

vLabs

Prov

Pres

Scale

Rel 1 Rel 2 Rel 3 Rel 4

users

data

s/w

Integ

Jun 2014Jun 2013 Dec 2013Dec 2012

Overview

The PaNdata Collaboration

The Vision

The PaNdata Europe Project

The PaNdata Open Data Infrastructure Project

Looking Forwards

Data

The Research Lifecycle

the researcher actsthrough ingest and access

Research Environment

Creation

Archival

Access

Storage ComputeNetwork

Data

Services

the researcher shouldn’t have to worry about the information infrastructure

Information Infrastructure

ICAT

TopCAT

EGIGEANT

Local resources

User Info feedDAQ feed

Data Analysis feed Provenanced Data

OECD Principles and Guidelines for Access to Research Data from Public Funding

13 principles

A – Openness • Openness means access on equal terms for the international research community at

the lowest possible cost, ....

B – Flexibility, C – Transparency, D – Legal conformity, E – Protection of intellectual property, F – Formal responsibility, G – Professionalism

H – Interoperability• Technological and semantic interoperability is a key consideration in enabling and

promoting international and interdisciplinary access to and use of research data. ...

I – Quality, J – Security, K – Efficiency, L – Accountability

M – Sustainability• ... taking administrative responsibility for the measures to guarantee permanent access

to data that have been determined to require long-term retention.

[http://www.oecd.org/dataoecd/9/61/38500813.pdf]

The 7 C’s

Creation Collection

Capacity

Computation

Curation

Collaboration Communication

PaNdataEurope SA

PaNdata ODI

PaNdata VRE

DataCreation

Archival

Access

Storage ComputeNetworkServices

Curation

Overview

The PaNdata Collaboration

The Vision

The PaNdata Europe Project

The PaNdata Open Data Infrastructure Project

Looking Forwards

www.pan-data.eu

Thank You

Agenda - 3rd Nov 2011 12:00 Arrivals and Lunch14:00 Start of meeting

14:00 Introduction and review agenda (Juan Bicarregui )14:05-15:30 Overview of projects (6 projects, 10 minutes each)“Vertical” projects• PaNdata (Juan Bicarregui and others)• CRISP (Jean Francois Perrin)• HDRI (Rainer Gehrke)“Horizontal” Projects• EUDAT (David Corney)• OpenAirePlus (Natalia Manola)• ORCID (Cameron Neylon)Any other related projects or activities (Round Table)15:30 Break16:00-17:30 Discussions on technical areas. Identification of areas for cooperation. Eg • Metadata catalogues and cross searching• Ids for Data and/or publications• Unique identification and disambiguation of people17:30 End

Agenda – 4th Nov 2011 Morning (ISIS TS2 CR16-17)

• 9:00 Introduction (Juan Bicarregui)• Review agenda, Overview, Introductions (JCNS, MaxLab), etc.• Review of any actions from the day before• 9:15 - 10:45 PaNdata ODI Service Activities • WP3 User AAA Service - Heinz Joseph Weyer (20 mins)• WP4 Data Catalogue Service - George Kourousias (20 mins)

– Common Data Model Access project - Alain Buteau (15 mins)• WP5 Virtual Laboratories - Frank Schluenzen (15 mins)• Discussion (20 mins)• 10:45 Break• 11:00 PaNdata ODI Joint Research Activities (3 x 20 minutes + 15)• WP6 Provenance - Brian Matthews (20 mins)• WP7 Preservation - Jean Francoise Perrin (20 mins)• WP8 Scalability - Bill Pulford (20 mins)• Discussion (15 mins)• 12:15 Lunch

Agenda – 4th Nov 2011 Afternoon (ISIS TS2 CR16-17)

13:15 Dissemination and EngagementDissemination (both projects) - Frank Schluenzen (15 mins)IUCr Working group on Data Deposition - Heinz Joseph Weyer and Alun Ashton (15 mins)13:45 Finalising PaNdata EuropeReview of actions from Period 1 review - Simon Lambert (5 mins)Remaining deliverables - status and plans D2.4 Integrated Policy (Due Sept) - Rudolf Dimper (5 mins)D3.4 Dissemination (Due Nov) - Frank Schluenzen (5 mins - covered above?)D6.3 and 6.4 Software (Due Sept and Nov) - Mark Johnson (10 mins)D7.3 and 7.4 Integration (Due Sept and Nov) - Brian Matthews (10 mins) D1.4 Final Management Report (Due Nov) - Simon (5 mins) Finances - Juan (5 mins)Initial planning for the final review – Juan (5 mins) Collaboration with the US (INFRA-2012-3.2 and 3.3) - Juan (10 mins)14:45 Break15:00 Starting PaNdata ODI.Management and Administration (Juan Simon, Denise) (30 mins)Contractual Agreements (GA and CA)PrepaymentsProceduresReportingReview of Early Deliverables and short term plans - Juan (15 mins)Action review – Juan (15 mins)16:00 Close

WP2 DisseminationObjectives Engagement with other initiatives and dissemination of project results, in particular to other

research infrastructures.Task 2.1. Establish an external web site as an extension to the existing website for the PaNdata

collaboration (www.pandata.eu). Task 2.2. Establish an interest group for project news items via community channels, informing

them of project progress. Task 2.3. Presentations to relevant international audiences at conferences, symposia, other

project meetings etc. Task 2.4. Provision of the open source software and appropriate documentation to potential

partner bodies. Task 2.5. Workshops to present the integrated systems to user and facility communities.

D2.1 : Project Website (M1) – November 2011D2.2 : Dissemination plan (M3)D2.3 : First Open Workshop (M15) – January 2013D2.4 : Open Source software distribution procedure (M21)D2.5 : Second Open Workshop (M27) - January 2014

WP3 User Catalogue and AAA ServiceObjectives To deploy, operate and evaluate a protocol for pan-European user identification across the participating facilities and implement common processes for the joint maintenance of that system.Task1: Consultation on existing software components recommendations for technologies to be implemented.Task 2: Set up team includes representatives from the user office and/or IT staff of the partners.Task 3: Specify an architecture which ... builds on the IRUVX "umbrella" concept. Task 4: Implement ... the necessary local modifications (including trust management). Task 5: Implement a standard affiliation database which is accessible for update and use by the participating facilities ...

Introduce a central affiliation database according to the PaNdata de-facto standard.Provide an interface of the local WUO systems to this standard. Organise and support the migration of the local WUOs to this new affiliation database.

Task 6: Deploy the user management system at all participating facilities. A major factor will be the integration with the facility's bespoke user administration systems. The deployment will include setting up of an administration authority for the system.

Task 7: Evaluate the system within a subset of the collaborating facilities. Task 8: Operate and report on the AAA trust system for the remainder of the project. Task 9: Maintain communication with other user authentication systems (through Workpackage 2) ...

D3.1 : Specification of AAA infrastructure (M6) Apr 2012D3.2 : Pilot deployment of initial AAA service infrastructure (M12) Nov 2012D3.3 : Production deployment of AAA service infrastructure (M18) Apr 2013D3.4 : Evaluation of initial AAA service infrastructure (M24) Nov 2014

WP4 Data CatalogueObjectives To deploy, operate and evaluate a generic catalogue of scientific data across the participating facilities and promote its integration with other catalogues beyond the project• develop generic software infrastructure to support interoperation of facility data catalogues• deploy this software to establish a federated catalogue of data across the partners, • provide data services based upon this generic framework which will enable users to deposit, search, visualise, and analyse data across the partners’ data repositories, • evaluate this service from the perspective of facility users, • manage jointly the evolution of this software and the services based upon it, • promote the take up of this technology and the services based upon it beyond the projectTask 4.1. Survey the features of existing implementations of metadata catalogues ...Task 4.2. ... deploy the chosen metadata catalogue in the legacy context of the facilities. Task 4.3. Provide remote API access to the individual catalogues and integrate to provide a single search capability across the collaborating facilities. Task 4.4. Evaluate the performance of searching the metadata catalogue and retrieving data.

D4.1. Requirements analysis for common data catalogue (M9) D4.2. Populated metadata catalogue with data from the virtual laboratories (M15) D4.3. Deployment of cross-facility metadata searching (M21) D4.4. Benchmark of performance of the metadata catalogue (M27)

WP5 Virtual Laboratories (Service)Objectives To deploy a set of integrated end-to-end user and data services supporting three specific

techniques: • Structural 'joint refinement' against X-ray & neutron powder diffraction data • Simultaneous analysis of SAXS and SANS data for large scale structures • Access to tomography data exemplified through paleontological samples

D5.1: Specific requirements for the virtual laboratories (M6) Apr 2012

D5.2: Deployment of Specification of the three virtual laboratories (incorporating any specific requirements software to support them) (M18) Apr 2013

D5.3: Report on the implementation of the three virtual laboratories (M30) Apr 2014

WP6 Provenance (JRA)Objectives To develop a conceptual framework, which can record and recall the data continuum, and

especially the analysis process, and to provide a software infrastructure which implements that model to record analysis steps hence enabling the tracing of the derivation of analysed data outputs

Task 1: Requirements for Provenance Task 2: Modelling the data continuum Task 3: Ontologies for specific instruments/techniques Task 4: Tool Support for the Data Continuum Task 5: Tracing the Data Continuum Task 6: Evaluation

D6.1: Model of the data continuum in Photon and Neutron Facilities (M12) Nov 2012D6.2: Common ontology definition and definition of tools to support the use of provenance

for Photon and Neutron Facilities (M18) Apr 2012D6.3: Tools for building research objects in Photon and Neutron Facilities (M24) Nov 2013D6.5: Evaluation report on provenance management in Photon and Neutron Facilities (M30)

Apr 2014

WP7 Preservation (JRA)Objectives To incorporate models and tools oriented towards long-term data preservation into the

PaNdata infrastructure, focussing on several aspects considered of benefit: an OAIS-based infrastructure; persistent identifiers; and certification of authenticity and integrity

Task 1. Baseline and OAIS application Task 2. Persistent identifiers (for datasets)Task 3. Representation information and archiving

RI for datasets, and AIPs (Archival Information Packages)This will include software as a kind of representation information, and the need to preserve the software itself.

Task 4. Integrity of datasets Mechanisms for maintaining and checking integrity of datasets. (for individual datasets (as preservation actions are

performed) and for data holdings as a whole.

Task 5. Evaluation and reporting

D7.1 Implementation of persistent identifiers for PaNdata datasets (M15) Jan 2013D7.2 Mechanisms and tools for representation information and archiving (M21) July 2013D7.3 Mechanisms and tools for integrity of datasets(M27) Jan 2014D7.4 Report on evaluation of preservation mechanisms (M30) Apr 2014

WP8 Scalability (JRA)Objectives To develop a scalable data processing framework combining parallel filesystems with a

parallelized standard data format (pNexus pHDF5) to permit applications to make most efficient use of dedicated multi-core environments and to permit simultaneous ingest of data from various sources, while maintaining the possibility for real-time data processing.

Task 1: pNexus API (Develop a pHDF5 compliant Nexus API.)Task 2: Investigate parallel file systems. Task 3: Investigate implementations on specific file systems

MPI-I/O implementations and pHDF5/pNexus on an even smaller number of preselected file systems.

Task 4: Coupling of advanced (pre-)processing engines.– Test the capability of the system to cope with multiple parallel data streams. This will contain for example

explicit tests feeding a pHDF5-file consisting of a large number of individual images into a multi-core analysis engine.

Task 5: Demonstration.D8.1: Definition of pHDF5 capable Nexus implementation (M9) - Software D8.2: Evaluation of Parallel filesystems and MPI I/O implementations (M9) - Report D8.3: Implementation of pNexus and MPI I/O on parallel filesystems (M21) - Prototype D8.5: Examination of Distributed parallel filesystem (M21) - Report D8.6: Demonstrate capabilities on selected applications (M21) - Demonstrator D8.7: Evaluation of coupling of prototype to multi-core architectures (M30) - Report

PaN-data Europe: actions fromPeriod 1 review

• Review report received 11 October• All deliverables accepted except D1.1.3 “Second

(annual) management report”– Revise financial statements; clarification of Table 3.5

(PSI) required– For Recommendation 4 see page 18, 2nd sentence:

“organized transition” to PaN-data ODI• But is D1.1.3 the correct place for this?

PaN-data Europe: actions fromPeriod 1 review

D2.4

D3.4

D2.4?

???

PaN-data Europe: actions fromPeriod 1 review

• We still need to resubmit the cost claims for Period 1

• To be submitted along with the revised management report in the NEF session

PaN-data Europe deliverable D1.4• D1.1.4 = Final Management Report• Due Month 18 = end of November• Also cost claim submission for last six months

PaN-data Europe deliverable D1.4• Standard template for final report

– Final publishable summary report• Executive summary (1 page)• Summary description of project context and objectives (≤ 4

pages)• Description of main S&T results/foregrounds (≤ 25 pages)• Potential impact (including the socio-economic impact and

the wider societal implications of the project so far) and the main dissemination activities and exploitation of results (≤ 10 pages)

• Address of the project public website and relevant contact details

PaN-data Europe deliverable D1.4• Standard template for final report

– Use and dissemination of foreground• Section A: List of scientific papers and dissemination activities• Section B: Specifies the exploitable foreground and provides

the plans for exploitation

– Report on societal implications• Ethics, workforce statistics, gender aspects, synergies with

science education, interdisciplinarity, engaging with civil society and policy makers, use and dissemination, media and communication to the general public

WP1 ManagementObjectives To establish an effective and efficient collaboration between the partners... To ensure that the project achieves its objectives ... To report to the Commission as required...

Task 1.1: Set up mechanisms to run the project through the rest of its duration (M1–M2).Task 1.2: Monitor progress of project activities and put in place appropriate corrective actionsTask 1.3: Organise general meetings of the project (kick-off and bi-annually thereafter).Task 1.4: Report to EC on the technical and financial progress of the project (annually and at

the end of the project).

D1.1: Project management structures, reporting, risk and quality ... procedures (M3)D1.2: First annual management report (M12)D1.3: Second annual management report (M24)D1.4: Final management report (M30)

Project procedures• Start up• Governance• Communications and meetings• Deliverables• Reporting and cost statements

Start-up• Grant Preparation Forms (GPFs)

– All done!• Grant Agreement

– All done!• Consortium Agreement

– DESCA v2 agreed in principle– Draft distributed and feedback received– Almost ready

• Pre-financing– Need to check

GovernanceProject Management Board (PMB).

Responsible for : Budget, consortium, activities, performance of the contractors, arbitrating on any conflict, IPR, risks, approve all new contractors etc

From Proposal:“The PMB will be chaired by a senior representative from the coordinating partner and include the Project Manager and one voting representative from each of the partners. Dr. Robert McGreevy will be the initial chair of the PMB his possible replacement could be undertaken by a majority vote of the PMB. A meeting of the PMB will be held at the Project Kick Off for validating the activities, the structural methods, the planning and the budget, and then at least 4 times a year.”

From CA: For the purposes of this Consortium Agreement references to the General Assembly shall mean the Project Board.The General Assembly shall consist of one representative of each Party (hereinafter referred to as “Member”).Each Member shall be deemed to be duly authorised to deliberate, negotiate and decide on all matters listed in Article 6.3.6 of this Consortium Agreement.The Coordinator shall chair all meetings of the General Assembly, unless decided otherwise by the General Assembly.

Project Manager (PM) ...interface between the Consortium and the European Commission.

“The PM is in charge of all administrative and financial matters, included in WP1, ...The PM is responsible for the follow up of the deliverables and milestones with help from WP WPLs. ... chairs the monthly project meetings via teleconference,Dr. Juan Bicarregui from the e-Science Centre, STFC will be appointed project manager for the duration of the project. His possible replacement is the responsibility of the Project Management Board.”

GovernanceWork Package Leader (WPL).

“the WPL will be responsible for scheduling work tasks, allocating resources available, and coordinating the production of deliverables to time and budget. The WPL will report on progress to the PM ...The PM and WPLs will consult regularly, with monthly teleconferences. ...Tolerances will be agreed between PM and WPL on each of the workpackages, ....

Management STFC Juan BicarreguiDissemination DESY Rainer GehrkeUsers PSI Heinz Joseph WeyerData ELETTRA George Kourousias ?Virtual Laboratories DESY Thorsten KrachtProvenance STFC Brian MatthewsPreservation ILL Jean-François Perrin ?Scalability Diamond Bill Pulford

Decision-making Process• The ultimate decision making entity of the project is the PMB. However, day to day decisions will be made by

either the PM or the WPLs as required. Decisions within the PMB are reached by consensus. In the event that no consensus is reached, decisions will be made by simple majority vote of the project partners. ...

Management of Knowledge and IPR• ... scientific publications and presentations at conferences or exhibitions. • ...Software and standards arising from the project will be available on an open-source basis and will be

disseminated to other large-scale scientific facilities. These activities will be under the co-ordination of the WP3 Leader.

• The Consortium Agreement will lay down rules for the ownership and protection of knowledge as well as for access rights. ...

• the WP3 leader will be in charge of collecting and proposing matters referring to the results for dissemination. ...Open Access• In accordance with the European Commission‘s Open Access Pilot (see for example

ftp://ftp.cordis.europa.eu/pub/fp7/docs/open-access-pilot_en.pdf), the project team will deposit peer-reviewed articles arising from the project into suitable institutional or subject-based repositories, using best efforts to ensure open access to the articles within six months. An example of such a repository already well established within the consortium is STFC‘s ePubs (http://epubs.stfc.ac.uk). (Or the Wiki)

Risk Management and Mitigation Plan• ...Section 1.3.5 gives a summary of the initial high level risks and a prevention and remedy strategy for each.• The project management, coordinated by the PM, shall identify and monitor risks that may have an impact on the

project schedule and outcomes and shall take appropriate measures to limit and/or mitigate their effects. ...• Risk management will be a standing agenda item of all PMB meetings.Quality Management • ...The project will establish a quality assurance system, under the responsibility of the PM, and devolved to WPLs

for each work package. Each deliverable will be subject to internal review for completeness, accuracy and consistency.

Governance

Communications and meetings• Monthly telecon• Project Management Board – four times per year

– Voting procedure etc. is in the CA• Aim for three face to face meetings per year

Communications and meetings• We have 3 or 4 official names per partner

– signatories, admin contact, scientific contact• We have the PANDATA mailing list (54 names)

– PANDATA smaller list?– PANDATA WP leaders (and friends)

Communications and meetingsfreddie.akeroyd@STFC.AC.UK Freddie AkeroydDebbie.Greenfield@STFC.AC.UK Debbie Greenfieldtom.griffin@STFC.AC.UK Tom Griffinneil.geddes@STFC.AC.UK Neil Geddesjuan.bicarregui@STFC.AC.UK Juan Bicarreguisimon.lambert@STFC.AC.UK Simon Lambertbrian.matthews@STFC.AC.UK Brian Matthewsrobert.mcgreevy@STFC.AC.UK Robert McGreevyk.shankland@READING.AC.UK Kenneth Shanklanddenise.small@STFC.AC.UK Denise Smallalun.ashton@DIAMOND.AC.UK Alun Ashtonbill.pulford@DIAMOND.AC.UK Bill Pulfordfulvio.bille@ELETTRA.TRIESTE.IT Fulvio BilleMirjam.vanDaalen@PSI.CH Mirjam van Daalendimper@ESRF.FR Rudolf Dimperstephan.egli@PSI.CH Stephan Egliderek.feichtinger@PSI.CH Derek Feichtingerdfernandez@CELLS.ES David Fernandezpicca@SYNCHROTRON-SOLEIL.FR picca frédéric-emmanuelbrigitte.gagey@SYNCHROTRON-SOLEIL.FR Brigitte GAGEYcgirbau@CELLS.ES Conchi Girbaugoetz@ESRF.FR Andy Goetzandy.gotz@ESRF.FR Andy Gotzvolker.guelzow@DESY.DE Volker Guelzowdietmar.herrendoerfer@HELMHOLTZ-BERLIN.DE Dietmar Herrendoerferdeborah.iorio@SYNCHROTRON-SOLEIL.FR Deborah IORIOjohnson@ILL.FR Mark Johnson

The PaN-data email list: stephane.longeville@CEA.FR Stephane LONGEVILLEphilippe.martinez@SYNCHROTRON-SOLEIL.FR Philippe Martinezchristian.jung@HELMHOLTZ-BERLIN.DE Christian Jungklora@CELLS.ES Jorg KloraMark.koennecke@PSI.CH Mark Koenneckkegeorge.kourousias@GMAIL.COM George Kourousias(2)thorsten.kracht@DESY.DE Thorsten Krachtute.krell@DESY.DE Ute Krellmetge@CELLS.ES Joachim Metgemutti@ILL.EU Paolo Muttijean.pearce@STFC.AC.UK Jean Pearceperrin@ILL.EU Jean-François Perrinstephane.poirier@SYNCHROTRON-SOLEIL.FR Stephane Poirierporte@ESRF.FR Dominique Portemilan.prica@ELETTRA.TRIESTE.IT Milan Pricapascale.prigent@SYNCHROTRON-SOLEIL.FR Pascale Prigentpugliese@ELETTRA.TRIESTE.IT Roberto Pugliesermpugliese@GMAIL.COM Roberto Pugliese(2)isabelle.rauschenbach@DESY.DE Isabelle RauschenbachTobias.Richter@DIAMOND.AC.UK Tobias Richterdsalvat@CELLS.ES Daniel Salvatfrank.schluenzen@DESY.DE Frank Schluenzenschwarzkopf@BESSY.DE Olaf Schwarzkopfsole@ESRF.FR Armando Soleheinz-josef.weyer@PSI.CH Heinz-Joseph Weyermichael.wilson@STFC.AC.UK Michael Wilsonalain.buteau@SYNCHROTRON-SOLEIL.FR BUTEAU Alain

* Total number of users subscribed to the list: 54

Deliverables• No period of grace in FP7 – the due date is the due

date!• Internal review procedure

Reporting and cost statements• Quarterly activity progress reports with effort

estimates• Six-monthly effort reports with progress• Formal report annually

– Including costs– STFC can input into NEF

• Prepayment plus annual payments and final payment

Website and Dissemination• Website• Open workshops Month 15, 27

Actions• User: Harmonisation mtg Dec 8 Hamburg• 2012 - 1 Feb - friendly user phase start• Ask US facility if they want to be involved in

counting users• End Spring 2012, review firnedly user phase• Data: ISIS – Elettra meeting mid Nov.• Comparison of alternatives: input to ICAT

roadmap.• Vlabs: identify software, definitin of Wflows

Action• Work with vlabs on user cases• IUCr liaison (HJW)• Scaleability – list of whos involved and VC.• Brian and Heinz to liasie with ORCid• Communicate with OpenAire (Brian)• Send list of review reply plans (Simon) • Next Mtg – February (before M6 deliverable)• Plan review – suggest Feb • Vlabs reqs mtg/ ICAT mtg (all 3 at DESY or ESRF)

• Propose dates for PMB mtgs. ( Denise) • Propose dates for monthly telecons.• Check single PMB member per organisation

(Denise)• Check WP leaders