SpaceGRID and EGSO

Satu Keski-Jaskari

Maria Vappula

Parallal Computing – Seminar

01.12.2003

Index

• SpaceGRID– overview– project– infrastructure– involved domains– in the future

• EGSO– overviwe– workpackages– system architecture

• roles

– data model

SpaceGRID

• ESA funded project (general study programme)

• The project aims:– How GRID technology can serve requirements

across a large variety of space disciplines– Sketch the design of an ESA-wide GRID

infrastructure – Foster collaboration and enable shared efforts

across space applications. – Proof of concept through prototyping– Keep Europe up with GRID efforts!

SpaceGRID - timetable

• Kick off: September 26, 2001• User Requirements and applications to be

prototyped selected: April 4, 2002• Infrastructure analysed and specified: July 4, 2003• ESA Grid Infrastructure set-up for demonstrations:

March, 2003• Prototypes evaluated/accepted: March 20 – June

20, 2003• Final Workshop: May 21-22, 2003• SpaceGRID study final report: 19.09.2003

SpaceGRID - Goals

• Assess how GRID technology can serve requirements across a large variety of space disciplines

• Foster collaboration and enable shared efforts across space applications

• Sketch the design of an ESA-wide (and common) infrastructure

• Proof of concept through prototyping• Involve both industry and research centres• Keep Europe up with GRID efforts!!

SpaceGRID ProjectSpaceGRID

Project

Requirements ESA Analysis Prototyping Future Analysis

Earth Observ.

Space Research

Solar System R.

Mechanical Eng.

Earth Observ.

Space Research

Solar System R.

Earth Observ.

Space Research

Solar System R.

Mechanical Eng.

Earth Observ.

Project Management

Study Teams (involved domains of SpaceGRID)

Study Team

Spacecraft Plasma InteractionsQinetiQ

Earth ObservationDATAMAT

Radiation TransportQinetiQ

Space WeatherCS-SI

Mechanical EngineeringAlcatel Space

Solar System ResearchRAL

Earth Observation

Radiation Transport

Spacecraft Plasma Interactions

Solar System Research

Mechanical Engineering

Space Weather Simulation

Space Weather Simulation Scenario

EGSO (European Grid of Solar

Observations) • EGSO is a Grid test-bed

• EGSO is funded by the Europea Commission

• The role of the EGSO is to provide the infrastructure, based on concepts such as grid sharing of resources and peer to peer networking

EGSO - Timetable

• Project was started in March 2002 • A first, live demonstration of basic

querying capabilities was given early in October 2003

• Implementation of major system components is under way with incremental releases that are planned for December 2003 and February 2004

EGSO - Goals

• Construct a virtual observatory • The main products are a common

metadata standard for solar physics and a common set of FITS keywords for future missions

• Enable more efficient identification and utilization of solar data from a broad range of sources.

Why EGSO is made

• Distributed data sources– Combining multiple data sources is often

necessary

• Large Data Volumes – Current heliosesimological datasets already

contain tens of terabytes

• Data Mining – Locating the combination of data from multiple

instruments most suitable for addressing a given problem is increasingly problematic

EGSO - Workpackages

• Six workpackages – WP0– WP1– WP2– WP3– WP4– WP5

• All workpackages involve multiple project institutions

Workpackage 0

• responsible for overall project management, communication and dissemination.

• monitors, assesses and co-ordinates the activities of the other workpackages

Workpackage 1

• is responsible for the definition of community and scientific requirements for EGSO, and the development of the system architecture for EGSO

• plans and facilitates the integration of the outputs of other rorkpackages, ensuring interoperability

Workpackake 2

• produce a working version of EGSO, initially taking the form of a 'trial federation' of some of solar data archives held by EGSO partner institutions

Workpackage 3

• responsible for producing a set of advanced and intuitive search and visualisation tools

• create a series of customisable interfaces tailored to the needs of different users accessing.

Workpackage 4

• create a unified catalogue of all solar observations, providing a single point of access to data from dozens of solar instruments

Workpackage 5

• responsible for the design of advanced image recognition techniques to identify individual features in the solar atmosphere

EGSO – System architecture

Grid Infrastructure

RD RI RD RI

RD RIRD RI

Brokers

User Interface

Role-specific Infrastructure

Connectors and Adapters

Internal Resources

Providers Consumers

EG

SO

Roles

• The system architecture for EGSO makes use of three high level abstractions (roles) to describe and partition system functionality – Provider– Broker– Consumer

• EGSO has adopted a generic architecture for all roles that is based on the popular "n-tier" style

Provider

• is the source of all that is handled by EGSO to satisfy user request

• is the crucial interface between existing data archives and the rest of the EGSO system

• may offer one or more of the following types of resources – Metadata, obtained descriptions of observations – Data, the actual data files – Services, is there being any capabilities

Broker

• inserts itself as an intermediary between the provider and the consumer

• will maintain a list of available providers, a resource registry, based on information submitted by the providers themselves, as well as information shared from others Brokers.

• will also maintain a catalog, at course granularity, containing a summary of all the observation catalogs that have been incorporated into EGSO

Consumer

• provides the means for a user of the system to actually visualize, select and manipulate the metadata and data obtained from multiple resources

• will be able to store the results provided by the Broker, both in memory or, for a longer term persistence, in a disk cache managed by the Consumer

• will also offer the possibility to use or define workflows, sequences of operations to be performed upon appropriate datasets of interest.

Data model

• A Data Model has been designed so that incorporates all the relevant entities in the system

• Framework will include various types of metadata that describe the objects to be included in the EGSO system.– Observational Metadata describes the contents of the

stored datasets – Administrative Metadata describes the organizations

and resources known to EGSO and the means of accessing them.

– Derived Metadata describes additional information extracted from observational data.