Lecture 2Grid Applications

Dr Gabrielle Allen

Center for Computation & Technology

Department of Computer Science

Louisiana State University

gallen@cct.lsu.edu

Grid Summer Workshop June 26-30, 2006

Some CCT Grid Application Areas! Computational Chemistry

" GridChem: Building community deployment of chemistry and othercodes (www.gridchem.org)

! Petroleum Engineering

" UCoMS: Grid-enabling reservoir simulations, drilling technologies,integration with sensors and wireless networks, dynamic data drivenscenarios (www.ucoms.org)

! Black Hole Simulations

" Large scale simulations, novel scenarios for task farming, job migration,remote visualization and steering

! Coastal & Environmental Modeling

" SCOOP & DynaCode: Data workflows with coupled models, dynamicdata driven scenarios, metadata. (scoop.sura.org)

! High Speed Network Applications

" EnLIGHTened & LUCIFER: remote visualization, adaptive networks,co-scheduling, video conferencing.

Grand Challenge Problems

in Astrophysics

! Cosmology

! Black Hole andNeutron StarModels

! Supernovae

! AstronomicalDatabases

! GravitationalWave DataAnalysis

! Drive HEC &Grids

Gravitational Wave Physics

Observations Models

Analysis & Insight

Complex Simulations

! Requires incredible mix of technologies & expertise!

! Many scientific/engineering components

" Physics, astrophysics, CFD, engineering,...

! Many numerical algorithm components

" Finite difference? Finite volume? Finite elements?

" Elliptic equations: multigrid, Krylov subspace,...

" Mesh refinement

! Many different computational components

" Parallelism (HPF, MPI, PVM, ???)

" Multipatch

" Architecture (MPP, DSM, Vector, PC Clusters, FPGA, ???)

" I/O (generate TBs/simulation, checkpointing…)

" Visualization of all that comes out!

! New technologies

" Grid computing

" Steering, data archives

! Such work cuts across many disciplines, areas of CS…

Computational Science Needs

! Freely available, modular, portable and

manageable environment for collaboratively

developing parallel, high-performance multi-

dimensional simulations

! Developed for Numerical Relativity, but now

general framework for parallel computing

(CFD, astrophysics, climate modeling,

chemical eng, quantum gravity, …)

! Finite difference, AMR (Carpet, Samrai,

Grace), soon FE/FV, multipatch

! Active user and developer communities, main

development now at LSU and AEI.

! Open source, documentation, etc

Cactus Code

NSF Black Hole Grand

Challenge

! 8 US Institutions

! 5 years

! Attack colliding black holeproblem

Examples of Future of Science & Engineering

! Require Large Scale Simulations, beyondreach of any machine

! Require Large Geo-distributed Cross-Disciplinary Collaborations

! Require Grid Technologies, but not yetusing them!

NASA Neutron Star

Grand Challenge

! 5 US sites

! 3 years

! Colliding neutronstar problem

EU Astrophysics

Network

! 10 EU sites

! 3 years

! Continuing theseproblems

Grand Challenge Collaborations

! Community Driven" Distributed communities share resources

" Video Conferencing

" Virtual Collaborative Environments

! Data Driven" Remote access of huge data, data mining

" Eg. Gravitational wave analysis, particlephysics, astronomy

! Process/Simulation Driven" Complex Simulations of Science and

Engineering

" Task farming, resource brokering,distributed computations, workflow

! Remote visualization, steering,interaction

Typical scenario:Find remoteresources (task farm,distribute)

Launch jobs (static)

Visualize, collectresults

Prototypes and demos:need to move to:

Fault tolerance

Robustness

Scaling

Easy to use

Complete solutions

Current Grid Application Types

! Addressing large, complex, multidisciplinary

problems with collaborative teams of varied

researchers ...

! Code/User/Infrastructure should be aware of

environment

" Discover and monitor resources available NOW

" What is my allocation on these resources?

" What is bandwidth/latency Code/User/Infrastructure

should make decisions

" Slow part of simulation can run independently …

spawn it off!

" New powerful resources just became available …

migrate there!

" Machine went down … reconfigure and recover!

" Need more memory (or less!), get by adding

(dropping) machines!

Dynamically provisionand use new highend resources and

networks

New Paradigms for Dynamic Grids

S1 S2

P1

P2

S1S2

P2P1

S

We see something,

but too weak.Please simulate

to enhance signal!

Future Dynamic Grid Computing

Found a black hole,Load new component

Look forhorizon

Calculate/OutputGrav. Waves

Calculate/OutputInvariants

Find bestresources

Free CPUs!!

NCSA

SDSC

RZG

LRZArchive data

SDSC

Add more resources

Clone job withsteered parameter

Queue time over, find new machine

FurtherCalculations

AEI

Archive to LIGOexperiment

Future Dynamic Grid Computing

New Grid Scenarios! Intelligent Parameter Surveys, speculative computing, monte carlo

! Dynamic Staging: move to faster/cheaper/bigger machine

! Multiple Universe: create clone to investigate steered parameter

! Automatic Component Loading: needs of process change,

discover/load/execute new calc. component on approp. machine

! Automatic Convergence Testing

! Look Ahead: spawn off and run coarser resolution to predict likely future

! Spawn Independent/Asynchronous Tasks: send to cheaper machine, main

simulation carries on

! Routine Profiling: best machine/queue, choose resolution parameters

based on queue

! Dynamic Load Balancing: inhomogeneous loads, multiple grids

! Inject dynamically acquired data

! Abstract programming

interface between

applications and Grid

services

! Designed for applications

(move file, run remote

task, migrate, write to

remote file)

! Led to GGF Simple API

for Grid Applications

(SAGA)

www.gridlab.org/GAT

Main resultfrom GridLab

project

Scp, DRMAA, Condor, SGE, SRB, Curl, RFT.Under Develop

Basic functionality, will work on single isolatedmachine (e.g. cp, fork/exec)

Default Adaptors

GRMS, Mercury, Delphoi, iGridGridLab Adaptors

Core Globus functionality: GRAM, MDS, GT-RLS,GridFTP

Globus Adaptors

Grid Application Toolkit (GAT)

Distributed Computation! Issues

" Bandwidth (increasing fasterthan CPU)

" Latency

" Communication needs,Topology

" Communication/computation

! Techniques to be developed" Overlapping

communication/computation

" Extra ghost zones to reducelatency

" Compression

" Algorithms to do this forscientist

Harnessing MultipleComputers

Why do this?Capacity: computers can’t keep upwith needsThroughput: combine resources

HTTP

Streaming HDF5Autodownsample

Any Viz Client:

LCA Vision, OpenDX

Changing steerableparameters• Parameters• Physics, algorithms• Performance

Remote Viz & Steering

Cactus Worm (SC2000)

! Cactus simulation starts, launched from

portal

! Migrates itself to another site

" Grid technologies

! Registers new location

! User tracks/steers, using HTTP, streaming

data, etc…

! Continues around Europe…

User only has to invokeCactus “Spawner” thorn…

Appropriate analysis tasksspawned automatically tofree resources worldwide

Task Spawning (SC2001)Cactus “Spawner” thorn automatically

prepares analysis tasks for spawning

Grid technologies find resources,

manage tasks, collect data

Intelligence to decide when to spawn

SC2001: resources of GGTC testbed.

Main Cactus BHsimulation starts

here

! 5 continents and over14 countries.

! Around 70 machines,7500+ processors

! Many hardware types,including PS2, IA32,IA64, MIPS,

! Many OSs, includingLinux, Irix, AIX, OSF,True64, Solaris, Hitachi

! Many organizations:DOE, NSF, MPG,universities, vendors

! All ran same Gridinfrastructure, and usedfor differentapplications

Cactus black holesimulations spawnedapparent horizon

finding tasks acrossthe grid.

Supercomputing 2001

Prizes for mostheterogeneous and most

distributed testbed

Global Grid Testbed Collaboration

Main Cactus BHSimulation startedin California

Dozens of low resolutionjobs test corotationparameter

Huge jobgeneratesremote datavisualized inBaltimore

Error measurereturned

Black holeserver controlstasks and steersmain job

Black Hole Task Farming (SC2002)

Job MigrationGridLab demonstration

SC2003

GridSpherePortal

SMSServer

Mail Server

“The Grid”

ReplicaCatalog

User details,notification prefs andsimulation information

IM Server

Notification & Information

Computational Chemistry Grid (CCG)

! NSF funded (LSU, UK,

OSC, TACC, NCSA)

! Aim to provide access to

computational chemistry

software, hardware, and

services using well

established Grid

technologies

! For expert, novice and non-

traditional users

! Gaussian, GAMESS,

NWChem, MolPro, + …

Terminology

! Computational Chemistry Grid: A virtual

organization that will provide access to distributed

high performance resources for the computational

chemistry community with distributed support

and services, intuitive interfaces, and a

measurable quality of service.

! GridChem: Java based desktop client

! Grid Middleware Server: Interfaces between

client and HPC resource, Handles authentication

and certificate management

! Coordinated use of compute and network resources

! Define the network through the set of services that it can offer

• MCNC (lead),

LSU, RENCI,

NCSU, Cisco,

AT&T, Calient

Networks

• 10Gig capacity

testbed running

over LONI and

NLR

iGrid 2005

Distributed & collaborative

visualization using optical

networks

! High-definition video

conference connecting

multiple sites

! Central visualization

server

! Interaction from all

participating sites

! Data on remote machines

! “Ubiquitous Computing &Monitoring System for Discovery &Management of Energy Resources”

! DOE/Louisiana BOR funded

" Petroleum engineering

" Wireless sensor networks

" Grid technologies

! Applications

" Reservoir simulation! Uncertainty analysis, sensitivity

studies, history matching

" Real-time well surveillance

" Drilling performance analysis withhigh-rate data

“UCOMS”

Oil Industry in Louisiana

Major oil producingstate in US:" 5th in production

" 8th in reserves

" Home to 2 of 4 strategicpetroleum reserves

" 17 petroleum refineries(capacity 2.8Mbarrels/day)

" Ports receive ultra largeoil tankers

" 20,000 oil producingwells, around 4Koffshore.

Reservoir Studies

! Assessments and predictions of oil/gas reservoir

performance, depending on

" Geological heterogeneity

" Engineering choices

! Used for development and operational decisions

… models assess different production scenarios.

! Applications:

" Well placement & performance prediction

" Sensitivity analysis & uncertainty assessment

" History matching (model verification and tuning)

Reservoir Simulation

! Mathematical model for fluid flow in a reservoir involvesdensity, permeability (K), mobility, pressure (P), productionrate (q), porosity & saturation, where m denotes either oil,water or gas.

! Many geological parameters cannot bemeasured or modeled and are unknowns.

! We are using UTChem (3D, multiphase,multicomponent, compositional, variabletemperature, FD simulator)

Core Computational NeedsCompute:

! Large-scale computation: Seismic inversion, flow numericalsimulations, Ggeostatistics, analysis, …

Data:

! Large data sets (TB-PB): Seismic, Geological & Geophysical(G&G), Well logging, Simulation results, …

Security:

! Commercial benefits lead to high security for all data, proprietarycodes, etc.

Workflow:

! Parameter selection, model construction, data movement, modeldeployment, results analysis etc.

Move towards dynamic, data driven scenarios, including direct inputfrom sensor data.

Current Workflow

! Example case: Eleven geological factors e.g. initial pressure+ three engineering factors e.g. tubing diameter with either 3or 4 levels.

! Factorial design:" 4^6 x 3^8 = 26,873,856 reservoir simulations

" 100 days on 1024 proc cluster (at 6 mins per run)

! Even with experimental design many runs needed

Motivation for ResGrid Infrastructure

! Reservoir simulation is one of the largest users ofcomputing power

" Large, complex, uncertain models

" High risks and rewards

! Can be performed more efficiently if

" Moderate-sized jobs can be farmed out onto aheterogeneous, underutilized grid

" Large jobs can be run in parallel on a grid

! Efficiency gains can be used to assess risks,estimate parameters, run larger and more complexmodels, and optimize developments

ResGrid Usage Scenario

! Five key components:

" Portal

" Reservoir modeling

" Large scale distributedreservoir simulation

" Analysis

" Visualization

! Light weight client to

locate and replicate data

files

Data Replication Tool: getdata

! SuraGrid is a beyond-regionalinitiative in support of regionalstrategy." Grid Infrastructure connecting states in

South-Eastern US.

" conceived to enable seamless sharingof resources between variousinstitutions to further scientificresearch and Grid development.

! Looking for applications !!!

! Partnering with IBM and othervendors to procure HPC Resourcesacross various SURA sites to build astrong computational resourcebackbone.

! Many sites connected via high speedoptical networks like NLR and LONI.

! http://www.sura.org/suragrid

SURAgrid

Louisiana Coastal Area (LCA)! 1927 flood, levees, loss of wetlands,

growing crisis, social impact

" 25% lost wetlands in last century, futurepredictions dire; increases flooding, surgerisk

! Hurricanes Katrina, Rita, Wilma (2005)

" 1.4M FEMA aid applications, 35K > 1000miles away, 33K evacuations by coastguard alone (6x 2004). Years to rebuild

! Important problems: ecological,hurricane, algal bloom/salinityforecasting, restoration, evacuation,emergency response strategies

! Rich dynamic environment for modeling:coupled models, multi-scale, realtimedata (sensors, satellites)

! Important role for HPC, Models, Grids,Community

Holly Beach (Hurricane Rita)

Holly Beach (Hurricane Rita)

North Breton Island, La.

looking Northwest.

South and East of Mouth

of Mississippi River

After Katrina

Beyond Katrina: Land Loss

Katrina Modeling: Tracks

Katrina Modeling: Storm Surges

! Storm surges: the worst part ~ 25ft for Katrina, kill far more peoplethan winds

! ADCIRC: Joannes Westerink,Rick Luettich, Randy Kolar, ClintDawson

! Input 2D Unstructured Mesh, wind,pressure

" 314K nodes, 85% near LA Coast

" 50km resolution in deep ocean,100m resolution

! 128 processor supercomputer: 1hour to forecast 1 day

! Want: run dozens of simulations

" Vary inputs, paths, strength

Source: http://www.nd.edu/~adcirc/

Computational domain withbathymetry (m)

Unstructured grid of the entiredomain.

ADCIRC Computational Mesh

Model-Model-Data Coupling

SURA Coastal Ocean Observing Program (SCOOP)

! Integrating data from regional observingsystems for realtime coastal forecasts in SE

! Coastal modelers working closely withcomputer scientists to couple models, providedata solutions, deploy ensembles of modelson the Grid, assemble realtime results withGIS technologies.

WAVCIS:Wave-Current-Surge Information System for Coastal Louisiana

! Program Director: Greg Stone, CSI, LSU

! Provide wave information (sea state)including wave height, period, direction ofpropagation, water level, surge, nearsurface current speed and direction andmeteorological conditions on a real timebasis around the entire Louisiana coast.

DynaCode: DDDAS framework for coast and

environmental modeling

! Builds on SCOOP modeling expertise

! LSU (CCT, CS, LHC, CSI, WBI), Notre

Dame (+ EU DROIDS)

! Two complimentary scenarios:

– Coupling ocean circulation, storm surge,

wave generation models for the Gulf

– Coupling ecological, hydrodynamic,

sediment transport models of the Mississippi

River Delta

! Infrastructure & algorithms

– Couple multi-models, data, external inputs

from sensors, wind & databases

– Optimize complex workflows on grids,

invoking appropriate models, meshes,

algorithms, depending on conditions.

DynaCode: DDDAS framework for coast and

environmental modeling

! New capabilities:

" dynamically invoke more accuratemodels and algorithms as hurricaneapproaches coast,

" choose appropriate computingresources for needed confidencelevels

" compare model results withobservations to feedback into runningsimulations

" realtime data assimilation

" adaptive multi-scale simulations

" dynamic component recomposition

" simulation needs steer sensors anddata input

Cyberinfrastructure

High End Computing

Computer Science

Applied Math

Application Domains

DynaCode! Focus on scenarios:

– Hurricane ensemble modeling

• Coupling ocean circulation, storm surge,wave generation models for the Gulf

• Notifications from NHC trigger customizedensemble hurricane models(surge/wind/wave), sensors verify, guidedynamic ensembles

• Event driven, dynamic componentframework with algorithm selection,optimization tools, workflow, dataassimilation, result validation withsensor/satellite.

– Ecological restoration and control

• Breton Sound diversion, control structureto allow Mississippi to flow into wetlands

• Coupled models (hydrodynamic, salinity,geomorphic, sediment) control diversion,sensors/wind fields inject real time data.

! SC05 Streaming

HD Video

" Starlight/Chicago to

SC showfloor

" CaveWave/NLR

optical network

" Data size - 21G

" Bandwidth -

500Mbps

" Frame-rate - 12fpsHigh speed networks critical;

Collaborative technology brings

experts together across state & world

Want Real Time! LONI! NLR!

Emergency Response

! Run many simulations if possible

" High res, high throughput critical: HPC + Grids!

" Presently 6 hours to convert NHC forecast to surge

! 240 processors per run

" Bracket official forecasts with different tracks

" Create “MEOW” map

! Notify officials: FEMA, State, Governor’s Office

" email, web, phone: can do better!

" evacuation orders: based on this work!

! Continue forecast to inform emergency workers

Emergency Forecast

Emergency Forecast

Very accurate forecasts

are possible;

reliability can be improved

Used to notify officials,

FEMA, State,

Governor. Email, web,

phone: can do better!

Components

! Infrastructure & algorithms to couple models, to eachother and to external inputs from sensors, wind &databases to optimize execution of complex workflows ongrids, invoking appropriate models, meshes, andalgorithms, depending on current conditions." Applications: control algorithms and coupling interfaces for

coastal/eco-codes

" Maths: Model errors for scenarios and implement basic statstoolkit in Cactus to drive ensembles

" Systems: Enhance dynamical capabilities of Cactus/Triana,decision making infrastructure, dynamic recomposition. Addlegacy ensemble modeling to Cactus. Think about tracking dataflow/data sensitivity through Cactus.

" Measurements: Integrate data to scenarios from existing sensors,plan out interfaces for sensor control.

! $500M to restore 250 sq. miles since 1986

– 12 sq. miles/year, but losing 2x this per year!

! How to catch up? $14B dollars later...

– floods? diversions? pipelines of mud? many ideas!

! Complex processes: comprehensive approach needed to understand

competing forces. But there is a quantitative answer!

Wetland Restoration

Changing How Science is Done

Source: Building a Cyberinfrastructure for the Biological Sciences, NSF Report 2003

Cyberinfrastructure

Applications• Environmental sciences• Computational fluid dynamics• Bioinformatics• …

Hardware• Compute resources• Networks• Storage• …

Programming & Developmenttools & libraries

Services & Middleware

Domain specific software tools

Sharedsoftware tools

Distributed hardware

}

Disco

very

& I

nnov

ation

Educ

ation

& T

raining

Integrated Cyberinfrastructure

Finally …

! Any questions … ask Archit over the next days

! Links from my web pages

http://www.cct.lsu.edu/~gallen or CCT Web

pages http://www.cct.lsu.edu

! Email me … gallen@cct.lsu.edu

! Very interested in talking to people interested in

using Cactus, GridSphere, GAT/SAGA, working

with SURAgrid

! Visitor program at CCT, setting up student

internships.