Australian Partnership forAdvanced Computing

Partners:

• Australian Centre for Advanced Computing and Communications (ac3) in New South Wales

• CSIRO

• Queensland Parallel Supercomputing Foundation (QPSF)

• iVEC – the Hub of Advanced Computing in Western Australia

• South Australian Partnership for Advanced Computing (SAPAC)

• The Australian National University (ANU) ACT

• The University of Tasmania (TPAC)

• Victorian Partnership for Advanced Computing (VPAC)

“providing advanced computing, information andgrid infrastructure for eResearch”

Australian Partnership forAdvanced Computing

“providing advanced computing, information andgrid infrastructure for eResearch”

• APAC 1 (…2000 – 2003…)– National Facility– Education, Outreach, Training

• APAC 2 (…2004 – 2006…)– National Facility– Grid– Education, Outreach, Training

• APAC 3 (…2007 – 2011…)– National Grid– National Facility– Training

APAC’s National Infrastructure Role

• Advanced Computing Infrastructure– peak computing system (‘capability’ computing)

• Information Infrastructure– management of community-based data collections– large-scale, distributed, nationally significant (reference) data

• Grid Infrastructure– seamless access to the national computing and information

infrastructure• access to federated computing and information systems

– advanced collaborative services for research groups• collaborative visualisation• computational steering• tele-presence• virtual organisation support

– support Australian participation in international research programs• eg, astronomy, high-energy physics, earth systems, geosciences

APAC National Grid Services

Other Grids:

InstitutionalNational

International

Other Grids:

InstitutionalNational

International

Data Centres

Data Centres

Instruments

SensorNetworks

Research Teams

portals and workflowdistributed computationfederated data accessremote visualisation

collaboration services

APAC National Grid - Status

• Systems coverage– Grid users can access ALL systems in APAC Partnership– About 4000 processors and 100’s of Terabytes of disk– More than 3PB of disk cached HSM systems

• Institutional and regional coverage– Resources and team members are supported in all capital

cities (+Townsville!)– Requests for service are spreading to multiple sites in

some regions (leading to the need for an affiliate model):• Clayton in addition to the city in Victoria• UWA in addition to ARRC in W.A.• ANSTO and Newcastle in addition to ac3 in NSW• JCU and UQ as part of QPSF in Queensland

APAC National Grid - Status• Nearing operational status

– some applications close to ‘production’ mode– not all core services are fully operational everywhere

• Undertaking a re-organisation– Moving out of independent development projects– Moving towards three layers: user support, middleware

deployment team and grid operations centre

• Focus on production status of services– Eg. CA and myproxy at production status, VOMRS soon– Not all site gateway servers support all services– Most services/protocols are in stable state on some sites

Starting Point: Projects

Grid Infrastructure Computing Infrastructure

• Globus middleware• certificate authority• system monitoring and management (grid

operation centre)

Information Infrastructure• resource broker (SRB)• metadata management support

(Intellectual Property control)• resource discovery

User Interfaces and Visualisation Infrastructure

• portals to application software• workflow engines• visualisation tools

Grid Applications

Astronomy

High-Energy Physics

Bioinformatics

Computational Chemistry

Geosciences

Earth Systems Science

Organisation Chart

Strategic Management

Middleware Deployment

Research Applications

Systems Management

Program Manager

Rhys Francis

Project Leader S/C Chair Astronomy Gravitational Wave Susan Scott Rachael Webster Astrophysics portal Matthew Bailes Australian Virtual Observatory Katherine Manson Genome annotation Matthew Bellgard Mark Ragan Molecular docking Rajkumar Buyya Chemistry workflow Andrey Bliznyuk Brian Yates Earth Systems Science workflow Glenn Hyland Andy Pitman Geosciences workflow Robert Woodcock Scott McTaggart EarthBytes Dietmar Muller Experimental high energy physics Glenn Moloney Tony Williams Theoretical high energy physics Paul Coddington Remote instrument and sensors Chris Willing <tbd>

Project Leader Services Compute Infrastructure David Bannon CA VOMS/VOMRS Gram2/4 Information Infrastructure Ben Evans SRB GridFTP MDS2/4 UI&VI Rajesh Chhabra Gridsphere Myproxy Collaboration Services Chris Willing A/G

Name Partner Name Partner Youzhen Cheng ac3 David Baldwin ANU Bob Smart CSIRO Darran Carey iVEC Martin Nicholls QPSF/UQ Grant Ward SAPAC John Dalton TPAC Chris Samuel VPAC Associated grid nodes David Green Griffith Ian Atkinson JCU Ashley Wright QUT Marco La Rosa UoM

Gateway Servers

David Bannon

Services Architect Markus Buchhorn

LCG VM Marco La Rosa

Infrastructure Support

(Middleware)

Application Support

Infrastructure Support (Systems)

Examples of Grid Applications

• Earth System Sciences (ESS) – example of community based data access

• Geosciences – example of research focussed data access and compute scheduling

• High Energy Physics – example of middleware interoperation, data and compute

• Basic APAC Grid model• Services available to support applications

ESS – OPeNDAP Services

AC3 Facility (Sydney)Land surface datasets

APAC NF (Canberra)International IPCC model results

TPAC 1/8 degree ocean simulations

Met Bureau Research Centre (Melbourne)Near real-time LAPS analyses products

Sea- and sub-surface temperature products

CSIRO HPSC (Melbourne)IPCC CSIRO Mk3 model results

CSIRO Marine Research (Hobart)Ocean colour products & climatologies

Satellite altimetry dataSea-surface temperature product

TPAC & ACE CRC (Hobart)NCEP2, WOCE3 Global, Antarctic

AWS, Climate modellingSea-ice simulations

ESS – Workflow Vision

Discovery Visualisation

Digital Library

OPeNDAP

AP

AC

NF

VP

AC

AC

3

SA

PA

C

IVE

C

Job/Data Management

Analysis Toolkit

Crawler

Status

ESS – Good News Developments

• Australian Bureau of Meteorology keeps its data in MARS

• The BoM has decided to build an OPenDAP interface to its MARS storage system

• OPeNDAP developers are working with the BoM and APAC Grid to support GSI authentication

• We hope to have all available data published into the grid environment

APAC Grid Geoscience• Conceptual models • Databases• Modeling codes• Mesh generators• Visualization

packages• People• High Performance

Computers• Mass Storage

Facilities Core

Deep Mantle

UpperMantle

Oceanic Lithosphere

UpperCrust

SedimentsOceanicCrust

Oceans

Biosphere

Atmosphere

lower Crust

Subcontinentallithosphere

weathering

Workflows and services

Resource Registry

Service Registry

Results Archive

Data Management Service

HPC Repository

LoginJob

MonitorRun

SimulationEdit Problem Description

Local Repository

Archive Search

Geology S.A

Geology W.ARock Prop.

N.S.W

Rock Prop. W.A

AAA Job Management

Service

Snark ServiceEarthBytes

Service

UserStatus: iVEC siteiVEC & HPSCSites (SRB)

Good News Developments

• Project achieved common portal access to Australian exploration data during 2005

• A ‘production’ status SRB federation is operating across the continent providing sharing for ‘model’ data

• Job submission using web services interface to Globus Toolkit 4 in operation at iVEC node

• Job submission to multiple ‘east’ coast grid sites undergoing testing as we speak

• Expect to be our first application making use of the VOMRS authorisation services (May)

Belle Experiment

Belle Experiment• K.E.K. B-factory detector (Tsukuba, Japan)

– Matter/Anti-matter investigations– 45 Institutions, 400 users worldwide

• On-line data from experiments

• Locally simulated collisions or events– used to predict what we’ll see

(features of data)– essential to support design of

systems– essential for analysis

• 2 million lines of code

Belle simulations

• Computationally intensive

– simulate beam particle collisions, interactions, decays

– all components and materials : 10x10x20 m to 100 µm

– tracking and energy deposition through all components

– all electronics effects (signal shapes, thresholds, noise,

cross-talk)

– data acquisition system (DAQ)

• Need 3 times as many simulations as real events to

reduce statistical fluctuations

Belle status

• Apparatus at KEK in Japan, research done world wide

• Data shared using an SRB federation: KEK, ANU, VPAC,

Korea, Taiwan, Krakow, Beijing

• Previous job flow based on scripts

• Project has now deployed LCG middleware for job

management at University of Melbourne

• APAC National Grid deployment provides job execution (at 3

sites) and SRB data management (at 2 sites) with data flow

using international SRB federation

• Good example of inter-grid middleware interoperation

Our most important design decision

V-LAN

Gateway Server

Cluster

Datastore

Cluster

Gateway Server

Cluster

Datastore

Cluster

Installing Gateway Servers at all grid sites, using VM technology to

support multiple grid stacks

Gateways will support, GT2, GT4, LCG/EGEE, Data grid (SRB etc),

Production Portals, development portals, experimental grid stacks

High bandwidth, dedicated private networking between grid

sites

QPSF

ANU

VPAC

ac3

TPAC

CSIRO

Network:GrangeNetAPAC VPN (AARNet)

Security: APAC CAMyProxyVOMRS

National Grid InfrastructurePortal Tools:GridSphere

Workflow Tools:Kepler?

IVEC

SAPAC

APACNational Facility

a virtualsystem of

computing, data storage

and visualisation

facilities

Systems:GatewaysPartners’ Facilities

QPSF(JCU)

QPSF

ANU

VPAC

ac3

TPAC

CSIRO

Job Monitoring:ScopeMonaLisa?

Job Management:Globus, PBSNimrodLCG

Job Submission:Command LinePortals

Computing Systems: Peak Mid-range Special

IVEC

SAPAC

APACNational Facility

APAC National GridComputing Grid Infrastructure

Resource Discovery:APAC Software RegistryMDSINCA? QPSF

(JCU)

QPSF

ANU

VPAC

ac3

TPAC

CSIRO

Data Transfer: RFTGridFTPGlobal File System

Data Management:GlobusSRBSRM

Data Access:OGSA-DAIWeb servicesOPenDAP et al

Mass Data Storage Systems: Tape – based (silos) Disc-based

IVEC

SAPAC

APACNational Facility

APAC National GridData Management Infrastructure

QPSF(JCU)

QPSF

ANU

VPAC

ac3

TPAC

CSIROFacilities: Access Grids Virtual Reality Systems

Collaboration Tools: AG Whiteboard

Visualisation Services:Prism and VisServerVisualisation Software

IVECSAPAC

APACNational Facility

APAC National GridCollaboration Support

Infrastructure

QPSF(JCU)

Providing Advanced Computing

and Grid Infrastructurefor eResearch

Dr Rhys FrancisAPAC Grid Program

Manager

www.apac.edu.au

Thankyou !