Data Analysis as a Service at ESRF

Vision, Data as a Service, Collaborations

Data Analysis as a Service at ESRF

Vision, Data as a Service, Collaborations

Vision

VisionThe next step for photon sources is to provide users with a remote online service to reduce and analyse the data they take at the source(s).

This implies :

1. Keeping data at the source (data policy, providing data as a service)2. Moving analysis programs to the web either via a portal or web UI3. Providing sufficient resources for remote users (hybrid cloud)4. Facilitate the transfer of data to public clouds (export service)

We have a lot of work to provide remote access services + redefine our mission

Progress since 2014

● Came to terms with the outcome of the PANDAAS proposal● Participated in new proposals to work on DAAS

○ NFFA (accepted), Calypso+ (decision soon), Panoramic (decision in 2016)

● Defined the official Data Policy ● Attended meetings

○ EOSC meeting (Brussels), HNSciCloud (Heidelberg), DLSR (Hamburg)

● Learned how to use Docker to package applications● Installed a private cloud based on OpenStack (DevOps team)● Joined the Helix Nebula consortium to buy public cloud resources● Started to evaluate using Amazon HPC for cpu intensive applications● Installed ownCloud as file sharing service for ESRF

Calypsoplus - JRA2

Partners : PSI, ESRF, DESY, CELLS, ELETTRA, DLS, SOLEIL, (XFEL)

Task 24.1 Design a platform which satisfies the requirements for Remote Data Analysis as a Service. (18M)

Task 24.2 Collect and compare offline (post experiment) data analysis requirements and existing software solutions from users of at least three different sites e.g. SAXS, Tomography, Diffraction (including MX) (8M)

Task 24.3 Implement DaaS on the local platform at each site with remote access to compute and storage resources using dedicated hardware resources supplied by each site (23M)

Task 24.4 Design and implement Remote Data Analysis as a Service portal demonstrator (16M)

Task 24.5 Package and deploy data analysis packages for at least two Use Cases / site (20M)

Task 24.6 Extend and deploy the Umbrella authentication system as the standard authentication mechanism. (18M)

Task 24.7 Test data analysis software for Use Cases with test data and users (6M)

Data as a Service● Defined the ESRF Data Policy

○ Archive raw (reduced) for 10 years○ Define, collect + store metadata in icat○ Embargo period of 3 years (extendible)○ Open Data after embargo period○ DOIs linking to datasets or subsets○ Endorsed by Council 1 December 2015

● Meetings ○ RDA (Paris), Active DMP (CERN)

● Hired a Data Manager (Alex) and started to define + collect generic and technique specific metadata

● Planned for 2016 - 10+ beamlines collecting metadata in icat + archiving data, generating DOIs, upgraded portal for downloading

A healthcare data demo at IBM Watson headquarters in New York City. (Jon Simon/Feature Photo Service for IBM)

Web applicationsNext step (starting this year) is to move applications to the web

● Candidates are :○ Simple calculations e.g. fisxserver.esrf.fr○ Strain calculator (xsocs/kmap)○ Ab initio calculations○ Data browser+viewer○ Python notebook

● External programs○ Spotsuite (Berkeley)

HPC computing1. ESRF HPC cluster has been oversubscribed since 12 months creating

competition between ab initio calculations (modelling) and data analysis (imaging)

2. This is being alleviated by doubling the cluster capacity but will happen again as modelling calculations and data volumes continue to increase

Problem: how to increase compute capacity at short notice and right price

Solution: use the public cloud ...

HPC in the public cloud1. ESRF is participating in HNSciCloud already but the process is slow (12m)2. Decision was taken to try out public cloud to gain experience3. With the help of a broker we chose Amazon (market leader)4. Amazon has an HPC offer since 2 years for HPC5. cfncluster create my_cluster → 15 minutes later ... cluster with sge+mpi6. ESRF is evaluating running cpu intensive mpi jobs on amazon7. First result yesterday running FDMNES test : 6h (32 cores) = 1.6 US$8. Other candidates are Quantum Espresso, ORCA, Quanty,

Feff, GEANT4, OCEAN

Questions : how to optimise storage+cpu capacity+ price, how to make it easy to use for scientists, how to scale, how to pay ...

HPC - FDMNES* running on Amazon EC2

● 2 x c4.xlarge(4 vCPU+7.5 GB)

● Next step usec4.8xlarge(32 vCPU + 60 GB)(10 Gbit/s network)

*FDMNES = solves Schrödinger's equation using Finite Difference Method to calculate Near Absorption Spectra

Cloud at the ESRF

Virtualisation, openstack and docker

OpenStack Pilot➔ Discover/Learn the openstack product➔ Play with several tools/components to see

which one fits better our needs:◆ Kolla: docker based tool for

deployments◆ Neutron: use of distributed network to

avoid all traffic to be routed through a single virtual router.

◆ Ceph setup◆ REST openstack API in order to build a

custom web interface.

Amazon cloud AWS➔ Evaluate a public cloud.

◆ Can we manage software deployments/custom images ?

◆ Can we process data there ?◆ Is it worth the cost ?◆ How easy is it for final users ?

Docker ➔ Already used in production:

◆ For owncloud v9◆ MariaDB/galera◆ Wordpress◆ ISpyB Frontend (and soon full JBoss/MariaDB app)◆ Webservers / load balancers.

➔ Docker 1.12 is used:◆ Macvlan + custom scripts to assign IPs to containers.◆ It really helps devops work ! Get people to use it.

Open Questions, or Topics under investigation

➔ Openstack:◆ Software Defined Network with real switches. (Extreme Networks, or cumulus/DELL)◆ How to manage resources:

● Should we mix it with our existing compute cluster: use of batch scheduler to extend OS cluster ?

➔ Docker:◆ Allow the use of containers in our batch scheduler.◆ High availability◆ Orchestration tool: swarm or other.◆ Live migration feature !

Ideas for CollaborationWe would like to collaborate on at least the following topics:

1. Using Public clouds for HPC2. Setting up private clouds with OpenStack3. Packaging standard codes to clouds (images, containers, packages)4. Porting data analysis applications to the web5. Developing a web portal for DAAS for hybrid clouds

Goal: We would like to build PANDAAS as a disruptive technology together with those who share our vision + can commit some resources.