cyberinfrastructure for the geosciences a community modeling environment: geodynamic integration of...

www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

A community modeling environment: A community modeling environment: geodynamic integration of multi-scale geodynamic integration of multi-scale

geoscience datageoscience data

Mian LiuMian Liu11, Huai Zhang, Huai Zhang1,21,2, ,

Youqing YangYouqing Yang11, Qingsong Li, Qingsong Li11, Yaolin Shi, Yaolin Shi22

1-University of Missouri-Columbia

2-Computational Geodynamic Lab, CAS, China


Motivation 1:

Exponentially increase of multi-scale observational data that need to be integrated and interpreted within a self-consistent geodynamic framework


EarthScope Instruments


EarthScope Annual Data Volume

Data volumes over next 10 years

GPS: ~7.7 TB

BSM/LSM: ~10.5 TB

Seismic: ~120 TB


Multi-timescales of geoscience data


Motivation 2: • Advance of computer hardware

(especially PC clusters and grid computers) and software engineering have provided unprecedented computing power;

• Data infrastructure have made integrating multiscale data both easy and necessary.


So we built the data So we built the data cyberinfrastructures, now cyberinfrastructures, now

what? what?

Internet

Data Grid

Physical model

HPCC

GEON


Internet

Data Grid

Physical model

HPCC

Free scientists from coding Free scientists from coding to do science, or whatever to do science, or whatever

they do bestthey do best


• Earth Simulator-GeoFEM project • Geoframwork• QuakeSim• SCEC Community Modeling Environment • CIG (Computational Infrastructure for

Geodynamics )

Some current efforts on geodynamic Some current efforts on geodynamic computationscomputations


More than one way to do it …

• Develop specific type of models (e.g., mantle convection);

• Use plug-in modules in a general system to generate specific type of models (wave, fluid, structure, etc.)


Multi-Purpose/Multi-physics Parallel FE Simulator/Platform for Solid Earth

Different finite element model can be plugged into this system

http://geofem.tokyo.rist.or.jp

Example: Earth Simulator- GeoFEM project


Won’t it be nice if we can have a general, flexible

community modeling system? • Not all geological needs can fit into the

pigeonholes;

• Need integration with data CI;

• Scalable for parallel and grid computation

Wouldn’t it be nice if all (or most of) these can be automated?


Examples of commercial FE code generation systems

• PED2D (http://members.aol.com/pde2d)

• FEPG (Finite Element Program Generator) (http://www.fegensoft.com/english/index.htm)


The devil is in the details …The devil is in the details …

PDE2FEMsystem

FEM meshes generator

Geometric modeling of real application

Graph partition (Metis4.0, Pmetis3.0) and Data partition

GES,PDE,CDE, SDE etc. element subroutine generators

GCN, NFE etc. nonlinear algorithms generators

Libs for PDEs, shape functions and other software packages

Theoretical and application documents for users

Server-Client communication system

PFEPG Server administration tools

Common Gateway Interface via www browsers for users

User level communication protocol

MPI

Data structure and optimization

Dynamic load-balancing of each node in parallel computer

Sequential solvers

Krylov subspace iterative solvers and preconditioners

Blas,blacs,pblas,splib,fepglib,

AZsolv, MUMPSArpack, SuperLU

LMDDM and LMDDA algorithm kernel subroutines


disp u vcoor x yfunc funa funb func shap %1 %2gaus %3mass %1load = fu fv $c6 pe = prmt(1)$c6 pv = prmt(2)$c6 fu = prmt(3)$c6 fv = prmt(4)$c6 fact = pe/(1.+pv)/(1.-2.*pv)funcfuna=+[u/x]funb=+[v/y]func=+[u/y]+[v/x]stifdist =+[funa;funa]*fact*(1.-pv)+[funa;funb]*fact*(pv)+[funb;funa]*fact*(pv)+[funb;funb]*fact*(1.-pv)+[func;func]*fact*(0.5-pv)

*es,em,ef,Estifn,Estifv,

*es(k,k),em(k),ef(k),Estifn(k,k),Estifv(kk),

goto (1,2), ityp1 call seuq4g2(r,coef,prmt,es,em,ec,ef,ne) goto 32 call seugl2g2(r,coef,prmt,es,em,ec,ef,ne) goto 33 continue

DO J=1,NMATEPRMT(J) = EMATE((IMATE-1)*NMATE+J)End doPRMT(NMATE+1)=TIMEPRMT(NMATE+2)=DTprmt(nmate+3)=imateprmt(nmate+4)=num

Other element matrix computing SubsPDE expressionContains information of the physical model, such as variables and equations for generating element stiffness matrix.

Fortran Segmentscodes that realize the physical model at element level.

variables

equation

Automated Code Generator

Step 1: From PDE expression to Fortran segments

Segment 1

Segment 2

Segment 3

Segment 4


Step 2: From algorithm expression to Fortran segments

do i=1,k do j=1,k estifn(i,j)=0.0 end do end do do i=1,k estifn(i,i)=estifn(i,i) do j=1,k estifn(i,j)=estifn(i,j)+es(i,j) end do end do

U(IDGF,NODI)=U(IDGF,NODI) *+ef(i)

defistif Smass Mload Ftype emdty lstep 0

equationmatrix = [S]FORC=[F]

SOLUTION Uwrite(s,unod) U

end

Algorithm ExpressionContains information for forming global

stiffness matrix for the model.

Fortran Segmentscodes that realize the physical model at global level.

Stiffness matrix Segment 5

Segment 6


SUBROUTINE ETSUB(KNODE,KDGOF,IT,KCOOR,KELEM,K,KK, *NUMEL,ITYP,NCOOR,NUM,TIME,DT,NODVAR,COOR,NODE,#SUBET.sub *U) implicit double precision (a-h,o-z) DIMENSION NODVAR(KDGOF,KNODE),COOR(KCOOR,KNODE), *U(KDGOF,KNODE),EMATE(300),#SUBDIM.sub *R(500),PRMT(500),COEF(500),LM(500)#SUBFORT.sub#ELEM.subC WRITE(*,*) 'ES EM EF ='C WRITE(*,18) (EF(I),I=1,K)#MATRIX.sub L=0 M=0 I=0 DO 700 INOD=1,NNE ……… U(IDGF,NODI)=U(IDGF,NODI)#LVL.sub DO 500 JNOD=1,NNE ………500 CONTINUE700 CONTINUE ……… return end

Program StencilFortran Segments generated

Step 3: Plug Fortran segments into a stencil, forming final FE program

Segment 1

Segment 2

Segment 4

Segment 3

Segment 5

Segment 6

…………..


Examples

• Western US tectonics

• Deformation of Asian continent

• Stress evolution and strain localization in the San Andreas Fault


A Preliminary Finite Element Model of Active A Preliminary Finite Element Model of Active Crustal Deformation in the Western USCrustal Deformation in the Western US


The Power of GEON Cluster NodeThe Power of GEON Cluster NodeOriginal series model (single CPU)

•Less than 3000 elements

•Three layers in R-direction

•2 min for per time step

Preliminary parallel model (16-nodes, 32 CPUs)

•More than 800,000 unstructured elements

•Major Faults and more deformation zones

•Subduction of Juan de Fuca slab

•21 layers in R-direction

(x 40vertical topographic exaggeration)


Automatic domain decomposition Automatic domain decomposition for parallel computingfor parallel computing


The model now allows simulation of large scale The model now allows simulation of large scale continental deformation with unprecedented detailcontinental deformation with unprecedented detail


Ongoing Effort:Ongoing Effort: Toward a new 3D model of Toward a new 3D model of continental deformation in Asiacontinental deformation in Asia


Predicted surface shear stress


Loading the San Andreas Fault by relative PA-NA motion

•Fully 3D•Dynamic•Plastic-viscoelastic•Co-seismic/interseismic cyclesfrom seconds to 104 years•Parallel computing on PC clusters


Comparison of predicted surface velocity and GPS data


Predicted maximum shear stress


Predicted rate of plastic strain energy release outside the SAF


Dream on…

• Integrating the community modeling environment with the geoscience data cyberinfrastructure;

• Grid computation and data integration;

• Automated (optimized?) work flow management (the Kepler system?)


0

0

0

zzzyzxz

yyzyyxy

xxzxyxx

fzyx

fzyx

fzyx

Automatic sourcecode generator

funcfuna=+[u/x] ………funf=+[u/y]+[v/x]

………dist =+[funa;funa]*d(1,1)+[funa;funb]*d(1,2)+[funa;func]*d(1,3)+[funb;funa]*d(2,1)+[funb;funb]*d(2,2)+[funb;func]*d(2,3)+[func;funa]*d(3,1)+[func;funb]*d(3,2)+[func;func]*d(3,3)+[fund;fund]*d(4,4)+[fune;fune]*d(5,5)+[funf;funf]*d(6,6) load = +[u]*fu+[v]*fv+[w]*fw-[funa]*f(1)-[funb]*f(2)-[func]*f(3)-[fund]*f(4)-[fune]*f(5)-[funf]*f(6)

PDEs

Complete source code

FEM Modeling Language

Data Grid (GEON and others)

Physical model

Model results

HPCC

Data =>???

SWF

SWF


Thank you!Thank you!

cyberinfrastructure for the geosciences a community modeling environment: geodynamic integration of...

Documents

tb slide

best slide

multiscale data

data cyberinfrastructures

earthscope instruments

data partition ges

multiscale observational

data ci scalable