CEMS: The Facility for Climate and Environmental Monitoring from Space

Victoria Bennett, ISIC/CEDA/NCEO RAL Space

CEMS – what is it

A joint academic-industrial facility for climate and environmental data services

Based in ISIC, Harwell Alongside Visualisation Centre, EO Hub, SRU

CEMS – what is it

CEMS will provide: Access to large-volume climate and EO datasets, alongside processing

capability; Commercial and scientific applications and services, hosted alongside

key datasets; Data quality, integrity and visualisation tools alongside advice and

consultancy;

Initial partners:

CEMS Overview

CEMS Infrastructure

Layered architecture High performance storage (1.7 PB)

Panasas parallel filesystem: resilience, scalability, fast performance, eliminating I/O bottlenecks

Processing hardware (20 nodes, each with 12 cores) Managed through cloud-based environment

Virtualisation layer based on VMware Allows computing resources to be used by third parties via a

cloud Platform to host applications and services for academic and

commercial user communities

Dual site Deployed across neighbouring academic and commercial sites on the

Harwell campus

5

Data

6

3.5PB Storage

STFC SCARF HPC (2000 CPUs)

JASMIN

10 x compute nodes

0.7PB Storage

CEMS Commercial CEMS Academic

10GB

it link IS

IC - S

TF

C

20 x compute nodes

Electron Building, ISIC R89 Building, RAL STFC

1100 blades, fast storage connected into low latency network

ATLAS Tape Store (3.5PB)

10 x compute nodes

1.1PB Storage

Initial CEMS funding has enabled the creation of new state of the art infrastructure to host data, and services to process it

EO datasets are being transferred from UK’s NEODC (NERC EO Data Centre) to CEMS infrastructure (a huge undertaking)

New datasets to follow

CEMS also integrates with other UK academic activities

JASMIN: access to CMIP5 for intercomparison with model data

STFC Scientific Computing: SCARF processing cluster and tape back-up

Harwell Site, Oxfordshire

Data sharing and integration of services between academic and commercial sites

CEMS Concept – multi-level approach

Exploiting Cloud Computing for CEMS

CEMS provides a flexible resource for processing and storage of EO data, using a cloud computing model

Different groups can be allocated portions of the storage, network and compute infrastructure

tailored to their needs without the need for the upfront capital for

hardware purchase.

We’ve built links with the user community, including NCEO and the ESA CCI programme

First users are accessing the system

Share of Cloud Resource

Organisation AOrganisation BInternal ServicesOrganisation C

Abstraction of Physical Resources

ComputeNetworkStorage

Core System

The core system provides the underlying functionality for CEMS to operate

It builds on top of hardware and cloud virtualisation layers

Key use cases were prioritised User management and support services Data discovery and access Virtual machine access

This is a baseline upon which to build

Applications and Services

Demonstrator applications were developed in parallel with the deployment of the infrastructure

These proto-CEMS apps illustrate and showcase the possibilities for use of CEMS Interactive (web based) applications EO Data Processing

Aimed to demonstrate one of more of the following: Complex or demanding processing of large

datasets Bringing multiple datasets together Visualisation

ISIC video wall showing Model Data (Met Office Hadley Centre HADGEM2 model, near-surface air temperature) and NASA Blue Marble on interactive spinning globes

OceanDIVA demonstrator developed by Reading e-Science Centre with data and scientific advice provided by University of Edinburgh

Applications and Services: OceanDIVA

Quick-look tool for comparing model/assimilation with in-situ data

CEMS Evolution

EngagementCEMS Components Timeline

Tech

nica

l im

plem

enta

tion

Hardware Data IntegrityDemonstrators

Est. CEMS Board and Integrated Project Team, groups: e.g. outreach, business modelling

CEMS Vision

Identifying the goalsEstablishing initial collaboration

August 2011: User Requirements Analysis

Nov 2011: £3m funding from UKSA

Dec 2011: H/W purchase

Core System

Virtualisation and Cloud

Mar 2012: H/W delivery + demonstration S/W

Service providers build apps to host on CEMSFirst hosted processing for research groups and industry

Aug 2012: core software services + virtualisation

ApplicationsSupport services,

consultancy

An eco-system of applications and support services forEnd user communities

Sept 2012+: 1st cloud users, growing user community

Who will use CEMS, and how?

Bids/Proposals – CEMS usage in projects (ESA, EC, UK)

Private Use – use of CEMS to support internal use e.g. data, compute or core services to support product development

Application Hosting – infrastructure for hosting and deployment, as well as a shop window to reach a large customer base.

Academic Usage – scientific community processing and storage of long time series of data

Community Data Hosting – we are looking to build up the data offering

Capability Development – tools or core services (e.g. to manipulate, inter-compare or visualise data sets) developed and supplied by users/customers

Divide up processing across mul-tiple cores Processing

Core 1

Processing Core 2

Processing Core 3

Processing Core 4

Current Status and Future Developments

CEMS is open for business: http://isic-space.com/cems/ Contact cemsinfo@isic-space.com

Future developments of the system:1. Storage and hardware: expand to support user

community 2. Cloud: collaboration with external partners –

GSCB, European Grid Infrastructure, Open Source clouds

3. Core system: Earth System Grid Federation – a federated infrastructure for climate data, suitable for EO also

4. Apps and services: Hosted processing – OGC WPS Hosted interactive development

environments – iPython Notebook

Individual user exploitation of cloud

Virtualisation

IPython Notebook interactive environment

15

Questions