progress and future plans for pl-grid development · domain-oriented services and resources of...

Domain-oriented services and resources of Polish Infrastructure for Supporting Computational Science in the European Research Space

1

Progress and Future Plansfor PL-Grid Development

KU KDM 2014, Zakopane, 12-14.3.2014

ACK Cyfronet AGH, Kraków, PolandPL-Grid Consortium

J. Kitowski, K. Wiatr, Ł. Dutka, T. Szepieniec, M. Sterzeland R. Pająk

2Outline

Motivation and Background

Consortium

Family of PL-Grid Projects

Domain-specific solutions and services

Overview

Use-cases

Future Plans

Conclusions

3Motivation and Background

Addressing Computational Science Problems

PL-Grid Consortium (2007) – 5 Polish ComputerCentres

Continuous development based on:

Projects funded by the European Regional Development Fund as part of the Innovative Economy Program

Polish Optical Internet (PIONIER) (2x10++ Gb/s)

Close international collaboration

EGEE, EGI InSPIRE, EMI, PRACE

ESFRI projects, e.g., CTA, EPOS, ERIC cases…..)

RoadMaps for e-Infrastructure developments

Previous / Current projects (5FP, 6FP, 7FP, EDA…)

Integration with international platforms

5

PLGrid PLUS Project (2011–2015)(run by PL-Grid Consortium)

Budget: total ca.18 M€, from EU: ca.15 M€

Number of people involved: ca. 120

IT scientists

Computational scientists

Field experts

Close collaboration between Partners

Homogeneous project view

Family of PL-Grid Projectscoordinated by Cyfronet

PL-Grid Project (2009–2012)(run by PL-Grid Consortium)

Budget: total 21 M€, from EU 17M€

Number of people involved: ca. 80(total, from different Polish Centres)

Outcome: Common base infrastructure

National Grid Infrastructure (NGI_PL)

Resources: 230 Tflops, 3.6 PB

Domain specific solutions: 13 domains(Specific computing environments)

QoS by SLM

Extension of resources and services by:

500 Tflops, 4.4 PB

Keeping diversity for users

Clusters (thin and thick nodes, GPU), SMP, vSMP, Clouds

Expected outcome:

Focus on users

6Family of PL-Grid Projectscoordinated by Cyfronet, con’t.

PLGrid NG Project (2014–2015)(run by PL-Grid Consortium)

Budget: total ca. 3,6 M€

Expected outcome:

Optimization of resources usage, training

Increase of scientific level

Extension of domain specific solutionsby 14 add’l domains

Extension of resources and services by:

ca. 8 Tflops, some PB

77

Computer centresHardware/Software

User friendlyServices

DomainExperts

Real Users

Focus on Users

Help Desk QoS/SLM

Grants

88TOP500 Polish Sites

Rank Site System CoresRmax

(TFlop/s)

Rpeak

(TFlop/s)

81

106

113

145

Cyfronet

Poland

Zeus - Cluster Platform

SL390/BL2x220, Xeon X5650 6C,

2.660GHz, Infiniband QDR, NVIDIA

2090

Hewlett-Packard

11,694

23,932

25,468

25,468

104.8

234.3

266.9

266.9

124.4

357.5

373.9

373.9

-----

143

170

221

ICM

Warsaw

Poland

BlueGene/Q, Power BQC 16C

1.600GHz, Custom Interconnect

IBM

-----

16,384

16,384

16,384

-----

172.7

189.0

189.0

-----

209.7

209.7

209.7

163

-----

-----

-----

TASK

Gdańsk

GALERA PLUS -- Action Xeon HP

BL 2x220/BL490

E5345/L5640 Infiniband

ACTION

10,384

-----

-----

-----

65.6

-----

-----

-----

97.8

-----

-----

-----

194

-----

-----

-----

WCSS

Wroclaw

Cluster Platform 3000

BL2x220, X56xx, 2.66 GHz, Infiniband

Hewlett-Packard

6,348

-----

-----

-----

57.4

-----

-----

-----

67.5

-----

-----

-----

-----

375

-----

-----

PCSS

Poland

Rackable C1103-G15, Opteron 6234

12C 2.40 GHz, Infiniband QDR

SGI

-----

9,498

-----

-----

-----

89.8

-----

-----

-----

211.1

-----

-----

June 2011Nov. 2012June 2013Nov. 2013

99Cyfronet at TOP500 list

List Rank System Vendors CoresRmax

(GFlop/s)

Rpeak

(GFlop/s)

11/2013 145Zeus - Cluster Platform SL390/BL2x220, Xeon

X5650 6C 2.660GHz, Infiniband QDR, NVIDIA

2090Hewlett-Packard 25932 266900 373900



2090Hewlett-Packard 25932 266900 373900



2090Hewlett-Packard 23932 234000 357500



2050/2090Hewlett-Packard 13944 185316 271113

11/2011 88Zeus - Cluster Platform 3000 BL 2x220, Xeon

X5650 6C 2.66 GHz, InfinibandHewlett-Packard 15264 128790 162409.0

06/2011 81Zeus - Cluster Platform 3000 BL2x220, L56xx

2.26 Ghz, InfinibandHewlett-Packard 11694 104765.1 124424.2


2.26 Ghz, InfinibandHewlett-Packard 9840 88050.7 104697.6

06/2010 161Cluster Platform 3000 BL2x220, L56xx 2.26

Ghz, InfinibandHewlett-Packard 6144 39934.5 55541.8


2.5 Ghz, InfinibandHewlett-Packard 2048 16179 20480

11/1996 408 SPP1600/XA-32Hewlett-Packard

(Convex) 32 5.5 7.7

06/1996 408 SPP1200/XA-32Hewlett-Packard

(Convex) 32 4.0 7.7

1010

Almost 8 mln jobs 21,000+ daily

80 mln CPU hours 9130 years

800+ users (ca. 2000)

100PB+ usage of scratch (350 PB)

The longest job: 76 days

The biggest job: 576 cores (1024)

Ca. 50% CPU time for multicore jobs

ZEUS Statistics 2012 (2013)

Xeon, 23 TB, 169 TFlops

Opteron, 26 TB, 61 TFlops

Xeon, 3,6 TB, 136 TFlops

Xeon, 6 TB,8 TFlops

Users needs takeninto account

Supercomputer Zeus

1212PLGrid Plus:Activities in general

Integration ServicesNational and International levels

Dedicated Portals and Environments

Unification of distributed Databases

Virtual Laboratories

Remote Visualization

Service value = utility + warranty

SLA management

Computing IntensiveSolutions

Specific Computing Environments

Adoption of suitable algorithms and solutions

Workflows

Cloud computing

Porting Scientific Packages

Data Intensive ComputingAccess to distributed Scientific Databases

Homogeneous access to distributed data

Big Data – Data discovery, process, visualization, validation….

4th Paradigm of scientific research

Instruments in GridRemote Transparent Access to instruments

Sensor networks

OrganizationalOrganizational backbone

Professional support for specific disciplines and topics

Rich international collaboration(EGI.eu, EGI INSPIRE, EMI, PRACE…..

1313PLGrid Plus:Domain Grids

Program for strategic science domains and important topics of Polish/European Science

Access to the software packages is provided by:

gLite (usage 2011-2013: ca. 3000 cores/month)

Unicore (usage 2013: ca. 60-100 cores/month)

QCG (usage H2 2013: ca. 2300 cores/month)

local

Already identified 13 communities/scientific topics:

Astrophysics

High Energy Physics

Life Sciences

Quantum Chemistry and Molecular Physics

Synchrotron Radiation

Energy Sector

Metallurgy

Nanotechnology

Acoustics

Ecology

Bioinformatics

Health

Material Science

1414PLGrid NG – New GenerationActivities in general

Additional/new groups of experts involved

Security on new applications, audits

In the development stage

Before deployment

During exploitation

Optimization of resource usage -- IT experts

Operation Center

Optimization of application porting

User support

First-line support

Helpdesk

Domain experts

Training

1515PLGrid NG:Domain Grids

Identified 14 communities/scientific topics:

Medicineskładowanie, bezpieczeństwo i analiza danych, diagnostyka

OpenOxidesdane i badanie struktur tlenkowych, spintronika

Mathematicsdata mining, data analytics, analiza statystyczna także dla potrzeb zarządzania zasobami

Biologyanaliza i wykorzystanie danych NGS, integracja aplikacji z systemem Kepler

Hydrologywsparcie prognoz hydrologicznych przez prognozy pogody w wysokiej rozdzielczości

Geoinformaticsintegracja danych przestrzennych

Meteorologyprognozy pogody i jakości powietrza w wysokiej rozdzielczości

Complex Networksinteligencja obliczeniowa, ekstrakcja wiedzy, udostępnienie danych (referencyjnych)

eBaltic-Gridmodelowanie lodu i oceanu dla Bałtyku

UNRESgruboziarnista symulacja dynamiki i zwijania białek w skali ms, z pomocą MD, MM, MC

VPHmedycyna spersonalizowana, biologia molekularna, bio-banki nowej generacji

Computational Chemistryintegracja danych eksperymentalnych, zaawansowane metody obliczeniowe

Nuclear Powerwypalenie paliwa (MC), symulacja maszyn rotodynamicznych

Metal Processingmodelowanie, badania wariantowe

1616

Examples of domain specificsolutions and services

(PLGrid PLUS use cases)

1818Deployed domain specific servicesAcoustics

Project Database − the service is a web portal designed to provide PL-Grid users with the numerical database (simulation results) and experimental database (measurements results).

Noise maps – the service is an application usedto generate noise maps in urban environments,based on data provided by the user.

Animation: calculated dynamic noise map

1919AcousticsNoise Map service

Animation: the fragment of the calculated dynamic noise map of the city of Gdansk, Poland

The “Noise Map” service for creating maps of noise threats for roads, railways andindustrial sources.

Integration of the software service with the network of distributed sensors brings apossibility of making automatic updates of noise maps for a specified time period.

Illustration of an application of the developed solution is the urban area noise mapping.

The map can be updated completelywithin relatively short period of time,employing the PL-Grid Infrastructure.

Operations are performed employinga dedicated noise prediction model,optimized for a computer cluster. Inaddition, predicted maps may beadjusted, using real noise levelmeasurements.

2222EcologyAutomatic Phenology Observations Service (ApheS)

Automatic phenology observations – infrastructure

The ApheS service is based on the KIWI RemoteInstrumentation Platform - a framework for buildingremote instrumentation systems for equipment andmeteorological sensors

Devices and sensors produce different output data.Most of the sensors produce numerical data, whichis to be visualized by the PLGrid Plus Ecogrid WebPortal

Example of photo taken by KIWI Eye observation

tool

2323EcologyKIWI – universal monitoring platform

Achievements:

KNOW-HOW associated with the implementation of the observational sets for the WLIN project

Observation eq uipment, hardware, software, materials, technologies, problems and their solutions

Video: Prototype testing and the first results of an observation operating set

Video: System run and its visual effects

2525MetallurgySimulations of extrusion process in 3D – deployed

Main Objectives:

Optimization of the metallurgical process of profiles extrusion. Optimization includes:

shape of foramera,

channel position on a die,

calibration stripes,

extrusion velocity, ingot temperatures, tools.

The proposed grid-based softwaresimulates extrusion of thin profilesand rods of special alloys ofmagnesium, containing calciumsupplements.

These alloys are characterized byextremely low technologicalplasticity during metal forming. TheFEM mathematical modeldeveloped.

2626MetallurgyDeployed domain specific services

SSRVE − the service for generation a Statistically Representative Volume Element(called SSRVE) for the microstructures of materials. Representation of microstructure of two-phase metallic materials

ModFemNet – the service for weldingsimulations.

2929ChemistryDeployed domain specific service

InSilicoLab for Chemistry − theservice that aims to support the launch of complex computational quantum chemistry experiments in the PL-Grid Infrastructure.Experiments of this service make it easy to plan sequential computation schemes that require the preparation of series of data files, based on a common schema.

Animation showing the service run

3030

AstroGrid-PL core sevicesPolish Virtual Observatory and Workflow Environment for Astronomy

Whole-community astronomical grid

Main Polish astronomical institutes involved CAMK PAN

(coordinator), CA UMK, OA UJ and a few others

The Virtual Observatory (VObs) is an international astronomical community-based initiative aiming to provide transparent and distributed access to data available worldwide.

Service goals:

Setup National VObs Data Center

integration of Polish data and join international efforts

Workflow Environment

Provide astronomers with workflow environment

Kepler environment is being tested...

Universal Fluid Dynamics code Piernik

InSilicoLab for Astrophysics − the service that aims to support the launch of complex astrophysical computational experiments in the PL-Grid Infrastructure.

3333Deployed domain specific servicesSynchroGrid

Elegant − the service for those involved in the design and operation of Synchrotron. The service consists in:

provision of the elegant (ELEctron Generation ANd Tracking) application in the parallel version on a cluster,

configuring the Matlab software to read output files produced by this application in a Self Describing Data Sets (SDDS) format and to generate the final results in the form of drawings.

Animation: simulationof the Synchrotron run

Elegant is a fully 6D accelerator simulationprogram that now does much more thangeneration of particle distributions and trackingthem

34Energy SectorπESA - Platform for Integrated Energy System AnalysisService to be deployed in 2014

Video: Atmosphere transport of air pollution

πESA – narzędzie do budowy modeli systemów energetycznych TIMES,

system modelowania jakości powietrza Polyphemus oraz

model do oceny ich oddziaływania na środowisko i zdrowie ludzkie MAEH.

OptiMINE

analiza wariantów prowadzenia robót górniczych i wybór najlepszego z nich pod względem planowanego wydobycia węgla kamiennego w kopalni.

Algorytm obliczeniowy opiera się na selekcji klonalnej i innych wybranych elementach sztucznych systemów immunologicznych.

Wartości postępów w poszczególnych wyrobiskach i rozłożenie robót w czasie, których dotrzymanie zapewni osiągnięcie planowanego wydobycia.

ModWELL

Analiza funkcjonowania sektora wytwarzania energii elektrycznej w horyzoncie krótkoterminowym w wysokiej rozdzielczości. Z pomocą Wirtualnego Laboratorium GridSpace 2

3535Deployed domain specific servicesHEPGrid (High Energy Physics)

CVMFS − the service that provides catalogs of software and data needed to reconstruct and analyse data in the HEP experiments. The service operates on a dedicated server installation on a read-only virtual file system CERNVM-FS, installed by the FUSE module in the local user space. With this service, there areimmediately available all versions of the software and any modifications made atthe central servers, at economical consumption of the local storageresources.

3636Future Development

New Computer Ecosystem

Problems (to be) tackled

Technologies for OpenScience

Interactive Data processing

Transparent access to data

Platform for HPC with Workflows

Data-Farming

PaaS for scientists

MapReduce environment

Support for strategic scientific projects (ESFRI, etc.)

Unified Portal

SSO HelpdeskCommunity Portal Community Portal Community Portal Accounting

Unified Acess Portals

CYFRONET ICM PCSS TASK WCSS

Geographically Distributed Infrastructure

Cloud ServicesLarge-scale databasesLarge-scale file systems Computational Grids

Computation and Data Processing Services

3737Conclusions

Further developement needed, as identified currently, mainly on Domain Specific Grids

Request from the users’ communities

Capacity for organization of future development according to

Expertise and experience

Strong scientific potential of the users’ communitiesbeing represented by PL-Grid Consortium

Wide international cooperation concerning the Consortium and individual Partners, good recognition worldwide

Good managerial capacity

Please visit our Web pages:

http://www.plgrid.pl/en

http://www.plgrid.pl

Credits

3838Credits

ACC Cyfronet AGH

Michał Turała

Marian Bubak

Krzysztof Zieliński

Karol Krawentek

Agnieszka Szymańska

Maciej Twardy

Teresa Ozga

Angelika Zaleska-Walterbach

Andrzej Oziębło

Zofia Mosurska

Marcin Radecki

Renata Słota

Tomasz Gubała

Darin Nikolow

Aleksandra Pałuk

Patryk Lasoń

Marek Magryś

Łukasz Flis

ICM

Marek Niezgódka

Piotr Bała

Maciej Filocha

PCSS

Maciej Stroiński

Norbert Meyer

Krzysztof Kurowski

Bartek Palak

Tomasz Piontek

Dawid Szejnfeld

Paweł Wolniewicz

WCSS

Józef Janyszek

Mateusz Tykierko

Paweł Dziekoński

Bartłomiej Balcerek

TASK

Rafał Tylman

Mścislaw Nakonieczny

Jarosław Rybicki

… and many others

domain experts ….

progress and future plans for pl-grid development · domain-oriented services and resources of...

Documents