Impact & ResultsEGI-Engage:

Advanced Computing for Researchwww.egi.eu

The EGI-Engage project (full name: Engaging the Research Community towards an Open Science Commons)ran from March 2015 to August 2017, coordinated by EGI and co-funded by the European Union (EU) Horizon2020 program under grant number 654142.

EGI-Engage had a mission to expand the capabilities of a backbone of federated services for compute,storage, data, communication, knowledge and expertise, complementing community-specific capabilities.

This report publication showcases the results of the project and their impact on science and society.

Purpose

1

8millioneuros

8competence

centres

43partners

11business

cases

6flagshipevents

40trainingevents

EGI-Engage in numbers:

13key exploitable

results

EGI-Engage supported science at all scales

730thousand

cores

300 PBonlinestorage

346 PBnearlinestorage

"You can see this increasing demand for distributed computing at every scale, from the theoretical chemistusing 5 million core hours a year, through to major collaborations like WeNMR in structural biology or theLarge Hadron Collider, which bring together thousands of scientists and routinely transfer something like50 petabytes of data per month."

Tiziana Ferrari, EGI-Engage Project Coordinator.

2

Building on the EGI service catalogue, EGI-Engage supported a wide range of scientificdisciplines at all scales, from large research communities & Research Infrastructuresto small research groups and individual researchers.

Figures correct as of November 2017

From individual researchers...Studying chemical reactionsChemical reactions are at the core ofeverything in the Universe.

Ernesto García, based at theUniversity of the Basque Country inSpain creates computational modelsto describe chemical reactions.

In the last two years, García hassubmitted about 2.5 million High-Throughput Compute jobs for atotal of 31 million CPU hours andpublished papers in MNRAS andChemical Physics Letters.

31millioncore

hours

2.5million

computejobs

Providers: compchem VO, supported by 17data centres in FR, GR, IT, PL & ES.

Predicting the onset of epilepsyEpilepsy affects about 2.4 million ofpeople per year.

Massimo Rizzi and his colleagues atthe Mario Negri Institute for Pharma-cological Research researched themarkers that predict the start ofepilepy before symptoms emerge.

By using High-Throughput Computeto perform their calculations, theysaved years of research time. Theresults of the study are published inScientific Reports.

Detecting social media trendsSocial networks nowadays are bigdata production engines.

Athena Vakali and her colleagues atthe Aristotle University of Thessa-loniki in Greece worked on a newmodel of detecting social media trends.

Vakali used Cloud Computeresources to help them run theexperiments. They used about 48CPU cores and 46 GB of availablememory. The results are published inAdvances in Big Data.

200thousandcompute

jobs

Providers: the biomed VO, supported by 60data centres & the Italian Grid Infrastructure.

48CPUcores

46GB

memory

Providers: GRNET & Okeanos, part of the EGIFederated Cloud.

3

to research infrastructures...MoBrain: from molecule to brainThe MoBrain Competence Center (CC)has developed online portals for lifescientists worldwide. In 2016, theMobrain CC partnered with seven EGIdata centres to secure High-Throughput Compute and OnlineStorage resources for their activities.

In total, the data centres offeredaround 75 million hours of computingtime & over 50 TB storage capacity.

The MoBrain portals powered by theEGI are: HADDOCK, DisVis, AMBER, CS-Rosetta, FANTEN & PowerFit.

71millioncore

hours

50TB

storage

EMSO: large-scale, marine RIThe EMSODEV project developed aData Management Platform (DMP) toset up a flexible and scalable datamanagement service for a long-termand (near)-real-time monitoring.

EMSODEV developed the DMP ontop of the EGI Federated Cloud withthe EGI support on virtualisation,storage, networking and security.

The experimental prototype of theDMP is now deployed in the RECAS-cloud and fully integrated with EGIAAI services.

ESA: European Space AgencyTerradue is a SME tasked by ESA tolead the development of a cloudinfrastructure that supports theGeohazards and Hydrology thematicexploitation platforms.

Terradue needed cloud resources tomake this possible and to be able tohandle massive data streams.

Seven EGI cloud providers fromItaly, UK, Greece, Germany, Poland,Belgium and Spain committed thecloud resources necessary to makethe project happen.

340CPU

cores

360CPU

cores

800GB

memory

More use cases are available on the EGI website: https://www.egi.eu/use-cases/

9TB

storage

4

and large research collaborations.CTA: Cherenkov Telescope ArrayThe Cherenkov Telescope Array (CTA)will be the world’s leading gamma-raypublic observatory.

CTA is using EGI High-ThroughputCompute & Online Storage servicesto handle the computational demandsduring the project’s first phase.

Between 2013-2016, CTA consumed:

• 360 million CPU hours

• 11 Petabytes of data transferred

• 2 Petabytes currently in storage• 11 million computation jobs.

Worldwide LHC Computing GridWLCG is a global collaboration ofmore than 170 computing centres in42 countries, linking up national andinternational grid infrastructures.

The collaboration between WLCGand what is now EGI spans over tenyears old: WLCG has been involvedin every step of the development ofEGI and is the biggest consumer ofEGI resources.

The four largest EGI Virtual Organi-sations are all LHC experiments:ATLAS, ALICE, CMS and LHCb.

LIGO/Virgo collaborationThe work on detecting gravitationalwaves by the Virgo & LIGO ScientificCollaborations won them thePhysics Nobel Prize in 2017.

Virgo data is collected at the Euro-pean Gravitational Observatory site,but its final repositories are theCCIN2P3 computing centre in Lyon &the INFN-CNAF centres in Bologna.

Wave analyses are run through EGIvia the Virgo Virtual Organisation(VO), which consumed collectively 40million CPU hours in 2015 & 2016.

5

Adoption of computing & storage resources

Research communities+ 30% registered users, increase driven by:

• Physics, Research Infrastructures

• SaaS operated on top of

the EGI Federated Cloud

+ 40 collaborations, including:

• 19 Research Infrastructures

• 31 RIs and e-Infrastructures integrated

+ 38 virtual organisations,

in response to outreach activities

& the Competence Centre programme

+ 11 service level agreements (SLAs):

• Peachnote (music platform)

• EMSOdev (ESFRI)

Business programmeEGI partnered with industry and SMEs to co-developsupport solutions for their computing needs.

Use cases:

• 150 recorded

• 20 in progress

• 11 completed• 4 memorandums of understanding (MoUs)

Selected business partners:

6

The adoption of EGI services increased during EGI-Engage. Below are the trends observed during the project:

OpenCoasts portal2 TB per year

Increased availability of scientific data& efficient use of IT for research

During the EGI-Engage project, the number of international research collaborations and infrastructures supported bythe EGI Federation increased by 48%. EGI supported data & analysis needs at all scales:

The compute capacity increased substantiallyduring EGI-Engage. Today, more than 200research collaborations benefit from the EGItechnical infrastructure.

Belle experiment10s PB per year

LHC+ 200 PB archived data

*Project year 1#Project year 2

300 PBonlinestorage

346 PBnearlinestorage

730kcores

+ 23% *+ 12% #

+ 12% *+ 5% #

+ 42% *+ 23% #

Increase of compute capacity:

7

From: And: To:

Contribution to cloud megatrends

8

In the area of cloud computing, the EGI-Engage project establisheda blueprint consisting of best practices to achieve interoperabilityacross multiple cloud providers.

To date, the EGI Federated Cloud is the only existing publicly-funded distributed research cloud in Europe, offering on average7,000,000 CPU hours per year to researchers from all disciplines. Itis now made of 21 publicly funded clouds & one commercial cloud.

Integration in EGI-Engage was performed by using public interfacesof the supported cloud management frameworks, thus minimisingthe impact on site operations. Providers are organised into OpenStandards and OpenStack realms, each realm exposing ahomogeneous interface.

EGI-Engage also developed key software components, services andpolicies to enable federated access to multiple cloud providers viafederated identity provisioning, authentication and authorization,and to enable portability of applications and data across a hybridcloud federation.

Adoption of FAIR principles

EGI-Engage contributed to the definition and maintenance ofpolicies, best practices and tools, to make the services of thefederation compatible with the FAIR principles:

• Findable: the EGI Marketplace was designed & implementedduring the project. The EGI internal service catalogue and externalservice catalogue were also defined.

• Accessible: accessibility was improved via federated identityprovisioning (eduGAIN), and federated authentication andauthorization via the Check-in service.

• Interoperable: EGI-Engage defined guidelines for compute anddata management interoperability across multiple facilities andsuppliers. This resulted in a community-defined standardsroadmap.

• Reusable: the project produced security policies both for users &providers, including the general e-Infrastructure security policy.

9

Key Exploitable Results (1)

10

EGI Marketplace

The Marketplace service (in beta) isdesigned as an electronic market: it’sa platform where services can beadvertised and where customers caneasily order & access them.

The Marketplace will also enhancevisibility for resource and serviceproviders, raising awareness of whatthey can provide as well as helpingto promote cross-disciplinary research.

Federated service management tools

Operational tools solve commonproblems with federating opera-tions. The tools can support thecreation of new service federations,or the extension of existing ones.

For example, the Accounting andMonitoring systems can be offeredas services or used as an addedvalue to market the EGI federationto new members.

EGI Service Portfolio

The EGI service portfolio has beenimproved with service definitions &the creation of external and internalservice catalogues.

The two catalogues are a reflection ofwhat EGI is offering to the participantorganisations to enable the federationitself, and what EGI is offering collec-tively as a federation to researchersand research communities.

Key Exploitable Results (2)EGI Check-in service

The EGI Check-in service provides areliable and interoperable AAI solutionthat can be used as a service for thirdparties. Check-in enables single sign-on to services through eduGAIN andother identity providers.

Users without institutional accountscan access services through socialmedia or other external accounts,including Google, LinkedIn or ORCID.

11

EGI Applications on Demand

The Applications on Demand service(in beta) gives access to onlineapplications and application-hostingframeworks for compute-intensivedata analysis.

This service targets individual resear-chers, research groups & early-stageresearch infrastructures, especiallythose with limited access to dedica-ted computing/storage resources.

Examples:

Thematic services

Thematic services are scientificapplications, tools, environmentsintegrated with EGI’s e-Infrastructureservices & typically exposed via webportals. They are not part of the EGIService Catalogue but they rely onEGI services to run.

Thematic services are co-designedand co-developed with structuredscientific communities.

Key Exploitable Results (3)EGI Open Data Platform

The EGI Open Data Platform is asolution designed to make datadiscoverable and available in an easyway across all EGI resources.

The Open Data platform is foundedon the OneData technology and canoffer scalable data access and com-pute capabilities around scientificdatasets for scientific groups at alarge scale.

12

IMS & Certification

EGI has defined a system to plan,implement & improve the businessprocesses under the responsibility ofthe EGI Foundation, resulting in:• the implementation of anIntegrated Management System(IMS) which unifies all organisationalprocesses into one framework.• two ISO certifications: ISO9001:2015 & ISO/IEC 20000-1:2011.

EGI Federated Cloud Computing

The EGI Federated Cloud is a IaaS-type cloud, made of academicprivate clouds & virtualised resour-ces & is built on open standards.

During EGI-Engage, the FederatedCloud was expanded with new cap-bilities, now integrating commercial &public IaaS Cloud deployments & e-Infrastructures with the currentproduction infrastructures.

Key Exploitable Results (4)Policy papers on the EOSC

EGI and other leading European ini-tiatives have shared their joint visionfor the European Open Science Cloudfor Research with several elements ofsuccess that contribute to the DigitalSingle Market.

The publication "European OpenScience Cloud for Research" sets outthe partners’ vision for the EOSC'sgovernance & organisation.

13

Strategy, governance & procurementThe following progresses were madeduring EGI-Engage:

• Analysis of barriers and opportu-nities for cross-border procurement.

• Governance evolution: assessingthe suitability of the EGI governancemodel in relationship to the evolu-tion of the strategy & organisationalbusiness models.

• A strategy update for 2015-2020.

EGI Security Policies

EGI has a full set of security policiesthat accommodate the operation ofdistributed infrastructures suppor-ting international collaborations.

During the EGI-Engage project, thesesecurity policies were revised toaddress issues related to the evo-lution of EGI services and technologyand to mitigate risks identified inrecent security analyses.

Looking to the future:from EGI-Engage to EOSC-hub

During the EGI-Engage project, EGI endorsed the principles of theEOSC and advocated the European Open Science Cloud to be theinitiative addressing the needs of open access, sharing within andacross research communities, ensuring sustained funding to digitalresearch infrastructures.

Building on the achievements of EGI-Engage, the EOSC-hub projectkicked-off in January 2018 with a mission to build the Hub: a centralpoint for all European researchers and innovators to discover,access and use a broad spectrum of resources for advanced data-driven research.

The project is coordinated by the EGI Foundation and bringstogether 100 beneficiaries and linked third parties includingresearch infrastructures, e-Infrastructure providers, SMEs andacademic institutions.

14

www.eosc-hub.eu

Colophon

15

We would like to thank all the participants of EGI-Engage for

making the project happen.

The content of this publication is correct as of November 2017.

This publication was prepared by the EGI Foundation

Communications Team.

Copyright: EGI-Engage Consortium, Creative Commons

Attribution 4.0 International License.

The EGI-Engage project was co-funded by the European Union

(EU) Horizon 2020 program under grant number 654142.

For more technical information about the EGI-Engage project,

please see: go.egi.eu/egi-engage

EGI-Engage’s impact is observable at all scales of the EuropeanResearch Area: from individual research, to large collaborations

and businesses, promoting digital innovation and theimplementation of the EOSC vision and FAIR principles.

Advanced Computing for Researchwww.egi.eu