open science and geoss: the cloud sandbox enablers

THEME[ENV.2011.4.1.3-1]: Inter-operable integration of shared Earth Observation in the

Global Context Duration: Sept. 1, 2011 – Aug. 31, 2014

Total EC funding: 6,399,098.00 € Project Web Site: www.geowow.eu

EC Grant Agreement no. 282915

GEOSS interoperability for Weather, Ocean and Water

Hervé Caumont Terradue

[email protected]

Open Science & GEOSS: the Cloud Sandbox enablers

GEO-X Plenary Geneva, January 14th, 2014

Digital Earth Communities

A long-term vision for GEOSS - GCI evolution … …considering feedback from all the stakeholders___

to engage with more user categories

data providers, data specialists, scientists, decision makers

within a more flexible architecture community components, resource enablers, cloud services

the GEOWOW Vision oo ooo

2 GEO-X Plenary


14/01/2014 3 GEO-X Plenary

CREATING THE

CONDITIONS


The concept of Cloud Sandbox enablers

4

Connect web resources into an experiment apparatus

Data assembly APIs in the Cloud

Data usage rights globally registered for scientific use

Exchange and reuse of a scientist’s workspace

14/01/2014 GEO-X Plenary

http://en.wikipedia.org/wiki/ATLAS_experiment


The present situation for most GEO partners

5

A discovery and download modus operandi Dataset file selections To reach out a multitude of fragmented project environments



A look at the Future through Cloud Sandboxes

6

Repeatable environments for reuse of scientific work Data as a Service within federated environments, with usage metrics Shared resources across Cloud Computing clusters



Matching the Open Science goals

Open Source

Open Data

Open Access

Open Notebook

Transparency in experimental approach & collection of observations

Public availability & reusability of scientific data

Public accessibility of peer-reviewed scientific communication

Shared versioning environment to facilitate progress in science




LOOKING FOR

WHAT’S INSIDE ?


Sandbox instance

Tools (Python, Libraries, ...)

Cloud Sandbox Enablers

•  Cloud Appliances Marketplace: a ‘VM Store’ to manage user Sandboxes •  Platform as a Service (PaaS): an algorithms integration environment •  Data staging tools: access and manage dataset slices required by applications

Sandboxes


Apache Hadoop Streaming programming model

Scale up MapReduce Jobs from single servers to thousands of computing nodes

Automate failure handling at the application layer

The PaaS environment


Cloud Sandbox enablers

Cloud Sandbox UI dashboard -  VM Information, App Descriptor, App runs workflow

status, VM monitoring, Run invocation, Support tools

Cloud Applications linked to GitHub repositories –  Automated Code Versioning & Collaborative

developments –  A new paradigm: a distributed service for dissemination

Data Casting enablers for data preparation -  Get the atomic, scalable, data slice units -  Validate a processing job in Sandbox simulation mode -  Scale out on a cluster (e.g. by time slices)

11 14/01/2014 GEO-X Plenary


Tutorials & Training material



THEY DARED

TO TEST IT


Developers on Cloud Sandboxes

On-boarding GEOSS partners: Dec’12: UNESCO Apr’13: INPE May’13: ECMWF Sep’13: ESA



Catalyzing resources

Support researchers to compute indicators for policy makers

Feed resources to Computing Clusters to run global & regional marine ecosystems assessments

Take-up for future commercial applications (under ad-hoc conditions)

Go through research oriented, not for profit, uses of TIGGE-LAM data in order to spread innovation

15

Expand uses of ESA satellites data

Development of new applications leveraging innovative uses of earth observations

Improve natural resources management and evolve policies

Handle large Earth Observation Temporal Series for Land Change Events Detection



Capacity Building

Ability to leverage ESGF’s CMIP5 Climate projections data, slice it and process it

Support reproducibility of scientific experiments & open science

Explore and visualize ECMWF data from integrated Cloud appliances

Improve accessibility of key TIGGE data for a wide user community and better support ECMWF partner users

16

Experiment with ENVISAT SCIAMACHY, global measuring of trace gases in the troposphere and in the stratosphere

Towards integration of Earth Explorers missions: catalogs & processors on Cloud Sandboxes

Streamline the processing of large EO data Temporal Series

Experiment flexible access to compute-intensive resources



Data Sharing

Ocean Biogeographical Information System (OBIS)

Transboundary Waters Assessment Programme (TWAP) indices and indicators

TIGGE and TIGGE LAM, THORPEX Interactive Grand Global Ensemble Limited Area Model

ERA Re-Analysis archives

17

ESA Missions data (archived and Earth Explorer)

Earth System Grid Federation – Model Intercomparison Project Phase 5 (CMIP5) data



PERSPECTIVES Developer Cloud Sandboxes – What’s next

18

IN THE NEWS


What we can have for GEOSS

Smooth deployments of Cloud environments for users.

Lean on-boarding process for custom user needs.

Data casting services for data intensive computing needs.

EO data processing applications shared and reused.



Bringing in the lean approach

Build Monitor Learn


Coming next from GEOWOW

Open Science ready


Best practices Success stories

Hands-on tutorials

Getting users from all GEOSS communities on Cloud Sandboxes

Online demos

THEME[ENV.2011.4.1.3-1]: Inter-operable integration of shared Earth Observation in the

Global Context Duration: Sept. 1, 2011 – Aug. 31, 2014

Total EC funding: 6,399,098.00 € Project Web Site: www.geowow.eu

EC Grant Agreement no. 282915

GEOSS interoperability for Weather, Ocean and Water

Open Science & GEOSS: the Cloud Sandbox enablers

GEO-X Plenary Geneva, January 14th, 2014

open science and geoss: the cloud sandbox enablers

Technology

earth explorer14012014tigge

ecmwf data

data specialists

data preparation

cloud data usage rights

cloud servicesgeox plenary2

tiggelam data

geox plenary6