eavl extreme-scale analysis and visualization library

EAVL

EXTREME-SCALE ANALYSIS AND VISUALIZATION LIBRARY

Jeremy Meredith

SDAV Next-Gen Library MeetingSeptember, 2012

History

• Originally ORNL LDRD– Jeremy Meredith, Sean Ahern, Dave Pugmire– plus Rob Sisneros joined as a postdoc

• Many hours sitting in conference rooms arguing over things like “what does it mean to have one of your dimensions be unstructured?”– then determine what to do that’s practical without

falling off the data modeling deep end . . . .

• Exascale focus

Approaching the Exascale Problems

• Update traditional data model to handle modern simulation codes and a wider range of data.

• Investigate how an updated data and execution model can achieve the necessary computational, I/O, and memory efficiency.

• Explore methods for visualization algorithm developers to achieve these efficiency gains and better support exascale architectures.

DATA MODELING CHALLENGES

Connectivity3D Point CoordinatesCell FieldsPoint Fields

Dimensions3D Point CoordinatesCell FieldsPoint Fields

Dimensions3D Axis CoordinatesCell FieldsPoint Fields

A Traditional Data Set Model

Data Set

Rectilinear Structured Unstructured

Challenge: Non-Physical Data Analysis

• Graph Data– topologically 0D vertices, 1D edges– non-spatial; storing X/Y/Z values is wasted space

• Pure Parameter Studies– e.g. reaction rate of combustion• FOUR “spatial” dimensions

– e.g. methane concentration vs oxygen concentration vs temperature vs pressure

• more complex reaction higher dimensionality

oxyg

en

met

hane

temperature

pressure

Challenge: Molecular Data(e.g., LAMMPS, VASP)

• To represent using vtkPolyData or vtkUnstructuredGrid:– VTK_VERTEX cells for the atoms– VTK_LINE cells for the bonds

• Any field data must exist on both element types– Not only inefficient:

• dummy bond strengths on the atoms?• dummy atomic numbers on the bonds?

– But also incorrect:• e.g. average(BondStrength) uses dummy values from atoms?

H

C

H

C

H

H

BondStr11112

AtomicNum661111

Challenge: Side Sets(e.g. Exodus, flux surfaces)

• The flow from A to B is defined on a set of faces• The flux variable is defined only on those faces– do you combine them into a single mesh?• waste space on dummy values, potentially introducing errors

– or create a separate mesh and lose the mapping info?• horribly expensive and error-prone to recalculate mapping

A Bflux surface lives inside the volumetric mesh

Challenge: Dimensionality, Refinement(e.g. GenASiS)

• (a) seven (or eight) dimensional mesh– f(x,y,z,ϴ,ϕ,λ,F)=E, plus time

• (b) refinement occurs on a per-cell basis– can’t assume per-block refinement– sometimes referred to as “unstructured AMR”

www.vacet.org

Challenge: Unique Mesh Topologies(e.g. MADNESS)

• MADNESS does not have a traditional mesh– Just a quad-tree with polynomial coefficients– Up to 30 refinement levels / tree depth

31 2 4

5

root

1

2

345

spatial structure internal tree representation

www.vacet.org

Challenge: Very High Order Fields(e.g. MADNESS)

• Legendre polynomial series at each tree node– Each tree node has Kdim coefficients– K can be up to approx. 20• i.e. 400 coeffs per tree node in 2D, 8000 in 3D

(example with K=3, dim=2)

0.834 0.592 0.0030.592 0.003 0.0100.003 0.010 0.007

THE EAVL DATA MODEL

Connectivity3D Point CoordinatesCell FieldsPoint Fields

Dimensions3D Point CoordinatesCell FieldsPoint Fields

Dimensions3D Axis CoordinatesCell FieldsPoint Fields

A Traditional Data Set Model (again)

Data Set

Rectilinear Structured Unstructured

TreeConnectivity Dimensions

FieldNameComponent

NameAssociationValues

Cells[]Points[]Fields[]

The EAVL Data Set Model

Data Set

CellSet

Explicit Structured

CoordsField

QuadTreeCellList

Subset

Connectivity: (a bunch of cells)

FieldName: “c” “c” “c”Component: 0 1 2

Name: “c”Association: PointsValues[3*npts]

Cells[1]Points[1]Fields[1]

Example: An Unstructured Grid(with interleaved coordinates)

eavlDataSet

eavlExplicitCellSet

eavlCoordinates

eavlField


FieldName: “x” “y” “z”Component: 0 0 0

Name: “x”Association: PointsValues[npts]


Example: An Unstructured Grid(with separated coordinates)

eavlDataSet

eavlExplicitCellSet

eavlCoordinates

eavlField #0Name: “y”Association: PointsValues[npts]

eavlField #1Name: “z”Association: PointsValues[npts]

eavlField #2

RegularStructure: 30 40 30


Name: “x”Association: PointsValues[npts]


Example: A Curvilinear Grid

eavlDataSet

eavlStructuredCellSet

eavlCoordinates

eavlField #0Name: “y”Association: PointsValues[npts]

eavlField #1Name: “z”Association: PointsValues[npts]

eavlField #2



Name: “x”Association: LogicalDim0Values[ni]


Example: A Rectilinear Grid

eavlDataSet


eavlCoordinates

eavlField #0Name: “y”Association: LogicalDim1Values[nj]

eavlField #1Name: “z”Association: LogicalDim2Values[nk]

eavlField #2

RegularStructure: 30 40 30 4 360

FieldName: “x” “y” “z” “μ” “ϴ”Component: 0 0 0 0 0

Name: “x”Association: LogicalDim0Values[ni]


Example: High-Dimensional Grid

eavlDataSet


eavlCoordinates

eavlField #0Name: “y”Association: LogicalDim1Values[nj]

eavlField #1Name: “z”Association: LogicalDim2Values[nk]

eavlField #2Name: “μ”Association: LogicalDim3Values[nμ]

eavlField #3Name: “ϴ”Association: LogicalDim4Values[nϴ]

eavlField #4

RegularStructure: 30 40

FieldName: “lat” “lon”Component: 0 0

Name: “lat”Association: LogicalDim0Values[ni]


Example: Geospatial Data

eavlDataSet


eavlCoordinates

eavlField #0Name: “lon”Association: LogicalDim1Values[nj]

eavlField #1Name: “c”Association: PointsValues[3*npts]

eavlField #2


eavlCoordinates

Example: Molecular Data

Connectivity: the atoms


Name:”atomic number”Association: Cell Set #0Values[ncells #0]


eavlDataSet

eavlExplicitCellSet #0

eavlCoordinates

eavlField #1

Connectivity: the bonds

eavlExplicitCellSet #1


eavlField #0Name: “bond strength”Association: Cell Set #1Values[ncells #1]

eavlField #2

Example: Face-centered Data

Connectivity: volumetric



eavlDataSet

eavlExplicitCellSet

eavlCoordinates

Parent: ( )

eavlAllFacesOfExplicit


eavlField #0Name: “facevariable”Association: Cell Set #2Values[nfaces]

eavlField #1

FILTERING IN EAVL

Data flow networks in EAVL (or not)

• A “Filter” is a stage in a data flow network– Creates a new data set from an old one

• Many operations do not change a mesh structure (assuming data model is sufficiently descriptive)– Arithmetic expressions: only modifies fields– External facelist: points and structure remain– Feature edges: just a new cell set with old points– Smooth, displace, elevate: only modify coordinates

• So: eavlMutator is an alternative to eavlFilter– Modifies a data set in-place

eavlMutator• In-place data set modification• Support for destructive in-place operation– free memory as you go

• Execute multiple mutators simultaneously on the same data set (barring conflicts)– e.g. displace (coords) + threshold (cells) concurrently

• How about data flow network support?– encapsulate an eavlMutator through a

eavlFilterFromMutator facade • Of course, some operations are natively eavlFilters– can facade through eavlMutatorFromFilter (?)


• Explicit cells can be combined with structured coordinates.

Example: Thresholding an RGrid (a)

eavlCoordinates

Name: “x”

Association: LogicalDim0Values[ni]

eavlField#0Name: “y”

Association: LogicalDim1Values[nj]

eavlField#1Name: “z”

Association: LogicalDim2Values[nk]

eavlField#2




eavlCoordinates

Name: “x”






eavlField#2


eavlExplicitCellSet

Cells: (…)

Parent: ( )

• A second Cell Set can be added which refers to the first one

Example: Thresholding an RGrid (b)


eavlStructuredCellSet eavlSubset


eavlCoordinates

Name: “x”






eavlField#2



Name: “x”






eavlField#2


eavlCoordinates

30x30 or 30x40

eavlStructSubset

30x30 or 30x40

eavlStructSubset

• Add six new subset-cell sets to original mesh

Example: Structured External Facelist


eavlCoordinates

Name: “x”

Association: PointsValues[npts]





eavlField#2


eavlCoordinates

Name: “x”






eavlField#2


eavlStructuredCellSet30 40 30

eavlStructCellSet30x30 or 30x40

eavlStructSubset

x6

• No problem-sized data modifications.– Interleaved and separated coordinates can be used simultaneously.

Example: Elevating a Structured Grid

FieldName: “c” “c”Component: 0 1

eavlCoordinates

Name: “c”Association:PointsValue[2*npts]

eavlField#0Name: “val”Association:PointsValues[npts]

eavlField#1



FieldName: “c” “c” “val”Component: 0 1 0

eavlCoordinates



Name: “c”Association:PointsValue[2*npts]

eavlField#0Name: “val”Association:PointsValues[npts]

eavlField#1

• No problem-sized data modifications.– Some axes on logical dims, with others on the points.

Example: Elevating a Regular Grid

FieldName: “x” “y”Component: 0 0

eavlCoordinates

Name: “x”




eavlField#1Name: “val”


eavlField#2



Name: “x”




eavlField#1Name: “val”


eavlField#2

FieldName: “x” “y” “val”Component: 0 0 0

eavlCoordinates



DEALING WITH CONCURRENCY

Concurrency at Multiple Levels

• Distributed Parallelism– Message passing still works well– Avoid global communication• local domain interconnectivity information

– Hybrid (e.g. spatiotemporal) parallelism• Task Parallelism– Fine-grain dependency tracking• e.g. displace (coords) + threshold (cells) concurrently

– eavlMutator helps– single eavlDataSet container class helps

• Thread Parallelism– Fine-grain data parallelism; CUDA, OpenMP

Data Parallelism for Developers

• Functor + iterator paradigm

• Iteration patterns for mesh topologies

• CUDA + OpenMP execution back-ends

A Simple Data-Parallel Operation

void CellToCellDivide(Field &a, Field &b, Field &c){ for_each(i) c[i] = a[i] / b[i];}

void CalculateDensity(...){ //... CellToCellDivide(mass, volume, density);}

Internal Library API Provides ThisAlgorithm Developer Writes This

Functor + Iterator Approach

void CalculateDensity(...){ //... CellToCellBinaryOp(mass, volume, density, Divide());}

template <class T> void CellToCellBinaryOp<T>(Field &a, Field &b, Field &c T &f){ for_each(i) f(a[i],b[i],c[i]);}

struct Divide{ void operator()(float &a, float &b, float &c) { c = a / b; }};

Internal Library API Provides ThisAlgorithm Developer Writes This

Custom Functor

void CalculateDensity(...){ //... CellToCellBinaryOp(mass, volume, density, MyFunctor());}

template <class T> void CellToCellBinaryOp<T>(Field &a, Field &b, Field &c T &f){ for_each(i) f(a[i],b[i],c[i]);}

struct MyFunctor{ void operator()(float &a, float &b, float &c) { c = a + 2*log(b); }};

Algorithm DeveloperWrites These

Internal Library API Provides This

Functor Efficiency on CPU and GPU

• Data: noise.silo• Surface normal

0 μs

100 μs

200 μs

300 μs

400 μs

500 μs

600 μs

unoptimized optimized unoptimized optimized

Intel Xeon L5420 NVIDIA GeForce 8800GTS

Functor Inline

Binding Values to Functorsstruct ScaleByConst{ float scale; ScaleByConst(float s) : scale(s) { } void operator()(float &a, float &b) { b = a * scale; }};

void CalculateDensity(...){ //... cell_volume = mesh_volume / mesh_numcells; CellToCellUnaryOp(mass, density, ScaleByConst(1.0/cell_volume));}

DATA PARALLELISM BASICS

Map with 1 input, 1 output

Simplest data-parallel operation. Each result item can be calculated from its corresponding input item alone.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

6 14 0 2 8 0 0 8 10 6 2 0

struct f { float operator()(float x) { return x*2; }};

result

Map with 2 inputs, 1 output

With two input arrays, the functor takes two inputs. You can also have multiple outputs.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

5 11 2 2 12 3 9 9 10 4 3 1

struct f { float operator()(float a, float b) { return a+b; }};

result

y 2 4 2 1 8 3 9 5 5 1 2 1

Scatter with 1 input (and thus 1 output)

Possibly inefficient, risks of race conditions and uninitialized results. (Can also scatter to larger array if desired.)Often used in a scatter_if–type construct.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

0 1 3 0

No functor

result

indices

4

2 4 1 5 5 0 4 2 1 2 1 4

Gather with 1 input (and thus 1 output)

Unlike scatter, no risk of uninitialized data or race condition. Plus, parallelization is over a shorter indices array, and caching helps more, so can be more efficient.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

7 3 0 3 1

No functor

result

indices 1 9 6 9 3

Reduction with 1 input (and thus 1 output)

Example: max-reduction. Sum is also common.Often a fat-tree-based implementation.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

7result

struct f { float operator()(float a, float b) { return a>b ? a : b; }};

Inclusive Prefix Sum (a.k.a. Scan)with 1 input/output

Value at result[i] is sum of values x[0]..x[i].Surprisingly efficient parallel implementation.Basis for many more complex algorithms.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

3 10 10 11 15 15 15 19 24 27 28 28

No functor.

result

+ + + + + + + + + + +

Exclusive Prefix Sum (a.k.a. Scan)with 1 input/output

Initialize with zero, value is sum of only up to x[i-1].May be more commonly used than inclusive scan.

x 3 7 0 1 4 0 0 4 5 3 1 0

0 1 2 3 4 5 6 7 8 9 10 11

0 10 10 11 15 15 15 19 24 27 28

No functor.

result 3

+ + + + + + + + + + +0

DATA PARALLELISM ON MESHES

Example: Surface Normal

• For each 2D cell(i.e. each polygon):– Get three adjacent points– Pair-wise vector subtract– Cross product

• Data-parallel:– Repeat for all cells

Example: Surface Normal

• INPUT:– 3-dimensional

coordinates array on the mesh NODES

– example: length = 9

• OUTPUT:– 3-component surface

normals array onthe mesh CELLS

– example: length = 4

Under the Covers: Node-to-Cell on CPUvoid NodeToCellOp3::ExecuteCPU(){#pragma omp parallel for for (int i=0; i<input->NumCells(); i++) { // get cell node indices

int nNodes, nodeIds[8]; float nodeValues[3][8]; conn.GetCellNodes(index, nNodes, nodeIds);

// get coordinates for nodes for (int i=0; i<nNodes; i++) { nodeValues[0][i] = array0[nodeIds[i]]; nodeValues[1][i] = array1[nodeIds[i]]; nodeValues[2][i] = array2[nodeIds[i]]; }

// call functor functor(nodeValues[0], nodeValues[1], nodeValues[2], &out0[i], &out1[i], &out2[i]); } }

Under the Covers: Node-to-Cell on GPU

void NodeToCellOp3::ExecuteGPU(){ float *d_arr0 = (float*)array0->GetCUDAArray(); float *d_arr1 = (float*)array1->GetCUDAArray(); float *d_arr2 = (float*)array2->GetCUDAArray(); float *d_out = (float*)output0->GetCUDAArray();

// calculate CUDA thread grid num blocks & threads // based on input->NumCells()

nodeToCellKernel3<<<nb,nt>>>(d_arr0, d_arr1, d_arr2, d_out0, d_out1, d_out2, conn, functor);}

Under the Covers: The CUDA Kerneltemplate <class F>__global__ void NodeToCellKernel3(float *array0, float *array1, float *array2, float *out0, float *out1, float *out2, ExplicitConnectivity conn, F functor){ const int index = blockIdx.x * blockDim.x + threadIdx.x; // get cell node indices int nNodes, nodeIds[8]; float nodeValues[3][8]; conn.GetCellNodes(index, nNodes, nodeIds); // get coordinates for nodes for (int i=0; i<nNodes; i++) { nodeValues[0][i] = array0[nodeIds[i]]; nodeValues[1][i] = array1[nodeIds[i]]; nodeValues[2][i] = array2[nodeIds[i]]; } // call functor functor(nodeValues[0], nodeValues[1], nodeValues[2], &out0[i], &out1[i], &out2[i]);}

WRITING ALGORITHMS IN EAVL

Face Surface Normal Functor+Iteratorstruct PolyNormalFunctor{ void operator()(int nvals, int shapetype, float x[], float y[], float z[], float *nx, float *ny, float *nz) { // get two adjacent edge vectors float ax = x[1]-x[0], ay = y[1]-y[0], az = z[1]-z[0]; float bx = x[2]-x[1], by = y[2]-y[1], bz = z[2]-z[1]; // calculate their cross product *nx = ay*bz - az*by; *ny = az*bx - ax*bz; *nz = ax*by - ay*bx; }};

void CalculateFaceNormals(...){ eavlExecutor::AddOperation( new eavlTopologyMap_3_3(inputcells, EAVL_NODES_OF_CELLS, xcoord, ycoord, zcoord, xnormal, ynormal, xnormal, PolyNormalFunctor()));}

Making it Easy for Developers

• Single functor code works on either CPU or GPU• Iteration patterns have multiple back-ends• CPU vs GPU execution choice is made at runtime• EAVL array class automatically supports

heterogeneous memory spaces for GPUs• Internal APIs support CUDA if developers want to

write their own CUDA kernels

Algorithms are Sequences of Operations(a) calculate face surface normals

with node-to-cell operation

(b) average to point normals via cell-to-node operation

EXAMPLE: THRESHOLD

47 52 63

32 49

31

Starting Mesh

We want to threshold a mesh based on its density values (shown here).

43 47 52 63

32 38 42 49

31 37 41 38

density 43 47 52 63 32 38 42 49 31 37 41 38

0 1 2 3 4 5 6 7 8 9 10 11

If we threshold 35 < density < 45, we want this result:

43

38 42

37 41 38

Which Cells to Include?

Evaluate a Map operation with this functor:struct InRange { float lo, hi; InRange(float l, float h) : lo(l), hi(h) { } int operator()(float x) { return x>lo && x<hi; }}

1 0 0 0

0 1 1 0

0 1 1 1

density 43 47 52 63 32 38 42 49 31 37 41 38

0 1 2 3 4 5 6 7 8 9 10 11

1 0 0 0 0 1 1 0 0 1 1 1inrange

InRange()

How Many Cells in Output?

Evaluate a Reduce operation using the Add<> functor.We can use this to create output cell length arrays.

1 0 0 0

0 1 1 0

0 1 1 1

0 1 2 3 4 5 6 7 8 9 10 11

1 0 0 0 0 1 1 0 0 1 1 1inrange

6result

plus

Where Do the Output Cells Go?

Inputindices

Outputindices

0 1 2 3

4 5 6 7

8 9 10 11

0 1 2 3 4 5 6 7 8 9 10 11

0 1 2 3 4 5output cell

0

1 2

3 4 5

input cell

How do we create this mapping?

Create Input-to-Output Indexing?

Exclusive Scan (exclusive prefix sum) gives us the output index positions.

0

1 2

3 4 5

0 1 2 3 4 5 6 7 8 9 10 11

1 0 0 0 0 1 1 0 0 1 1 1inrange

0 1 1 1 1 1 2 3 3 3 4 5startidx

+ + + + + + + + + + +0

startidx

Create Output-to-Input Indexing?

We want to work in the shorter output-length arrays and use gathers. A specialized scatter in EAVL creates this reverse index.

0 1 2 3 4 5 6 7 8 9 10 11

0 5 6 9 10 11revindex

0 1 1 1 1 1 2 3 3 3 4 5

0

5 6

9 10 11

density

Gather Input Mesh Arrays to Output?

We can now use simple gathers to pull input arrays (density, pressure) into the output mesh.

0 1 2 3 4 5 6 7 8 9 10 11

43 47 52 63 32 38 42 49 31 37 41 38

0 5 6 9 10 11revindex

43 38 42 37 41 38

43

38 42

37 41 38

output_density

PLANS AND STATUS

Plans and Status• Now Open Source (BSD “2-clause”)– Project website at http://ft.ornl.gov/eavl – Source code, docs at https://github.com/jsmeredith/EAVL

• Data Model– Most infrastructure in place– Still a few areas not fully fleshed out– The client-facing API needs work

• Execution Model– Currently imperative style– Exploring pipeline paths, contract/metadata needs

• Productization Status– More iterators– More optimization– Lots of internal cleanup

eavl extreme-scale analysis and visualization library

Documents

updated data

wider range of data

bondsany field data

tree nodeeach tree node

dummy values

traditional meshjust

d vertices

execution model