Realize Your Product Promise®

ANSYS Fluent HPC 16.0 Update

ANSYS Fluids HPC Team:Rongguang JiaPavan MutnuriSunil SatheBassem GirgisKingshuk HaldarShriram Jagannathan

12/18/2014

• Why HPC is important?

• Solver performance improvement

• IO/Mesh improvement

• Partition and load balancing improvement

• Some more benchmark cases

• Status update on GPU

• Summary and next steps

Outline

© 2015 ANSYS, Inc.

Why HPC Is Important?

© 2015 ANSYS, Inc.

Time steps done in one day with 16.0.0oil_rig_7m Intel Haswell

Faster• Reduce turn around time• Consider more design variants

Larger

• Assess larger, more detailed models• Consider more complex physics• From single component to system simulations

Easier• Put powerful computation resources at users’ fingertips• Efficient decisions earlier in the product development cycle

Solver Improvement

5 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

• A continuous software development focus on HPC enabling parallel improvements made release by release - also at R16.0.

• Fluent is highly optimized to run fast and deliver outstanding parallel scaling at an increasing scale of parallelism!

• As we are committed to taking simulations to new levels of software scalability, we are having strong technology partnerships with hardware vendors, and supercomputer centers (e.g. NCSA, HLRS).

Deliver Outstanding Parallel Performance & Scaling

Supercomputing Milestone

6 © 2015 ANSYS, Inc. July 1, 2015

Improved Solver Performance & Scaling

Improved parallel scalability:

• Also observed for more typical model sizes at more regular core counts!

• At extreme core countso 86% efficiency for 830M cell case at 36K cores,

with species transport

o 80% efficiency for 91M cell case at 16K cores, with complex physics

7 © 2015 ANSYS, Inc. July 1, 2015

Improvements of moving mesh cases•Wall distance calculation by 6X speedup•Sliding interface computation by 2X speedup

Case Details:

• 4-stroke spray guided Gasoline Direct Injection

• Number of cells: 1.7 Million

• Cell Type: Mixed

• Models used: Standard K-ε turbulence

• DPM

• Solver: Pressure based segregated, Species Partially Premixed Combustion, dynamic mesh, Green-Gauss node based, unsteady

Improved MDM Performance & Scaling

8 © 2015 ANSYS, Inc. July 1, 2015

•General solver scalability improvements

Case Details:

• Vehicle exhaust model

• Number of cells: 33 Million

• Cell Type: Mixed

• Models used: SST K-omega turbulence

• Solver: Pressure based coupled, Least Squares cell based, steady

Improved Solver Performance & Scaling

9 © 2015 ANSYS, Inc. July 1, 2015

Improved Parallel Robustness• Parallel check to inspect system usage, interconnect speeds and load balance

• Enhanced automatic handling of AMG divergence

• Not default, can be turned on by (rpsetvar ‘amg/protective-enhanced? #t)

• M_MMAP_THRESHOLD to 256KB as default

• To resolve OS (glibc) related issues causing memory build up

• Cache flush

• Automatically checks for file cache buildup and warns users of potential performance impact

• New flag “-cflush” added to perform cache flush and restore the memory

• Results in more consistent and improved performance in affected cases

IO/Meshing Improvements

Wall Distance Calculation Enhancement

• Wall Distance Computation is improved:o Optimized communication algorithmso Improved with multithreaded

calculations

• 6~25+ times faster

• Benefits moving mesh, and cases with SST turbulence or enhanced wall treatments

© 2015 ANSYS, Inc. 11

Introducing HDF5-Based Case/Data IO

© 2015 ANSYS, Inc. 12

• HDF5-based file IO is introduced in R16.0Fluent recognizes the .h5 suffixread-case box.cas.h5read-data box.dat.h5write-case box.cas.h5write-data box.dat.h5

• Fully supports parallel IO for case/data read/write

• Sectioned data compression is implemented natively through HDF5

/file/hdfio-options>compression-level

• Selectable IO Modes /file/hdfio-options>io-mode

Low level code shared with AIM Fluids

HDF5 IO Performance

© 2015 ANSYS, Inc. 13

• It generally improved performance for all stages

• Case read/write got most of the boost

• File size (uncompressed):

o Case is about 10% smaller

o Data is about the same

• Removed parallel bottleneck for large meshes

• Improved parallel performance of optimization step

• Option to preserve prismatic extrusion layer during conversion

• Added TUI option to skip migration and reordering after conversion

Release Poly Conversion time

15.0 14.3 h

16.0 1.7 h

Example: 8X time reduction for polyhedral mesh conversion on 64 CPUs for 100M+ cell

case of nacelle pylon

Polyhedra Conversion Performance Improved

Main work done by solver meshing, Parallel team helping

15 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

Partition and Load Balance Improvements

16 © 2015 ANSYS, Inc. July 1, 2015

Partitioning based on grouping by Laplace coefficients

• This avoids partition interfaces through high cell aspect ratio areas

• It improves convergence for cases with highly stretched cells

Laplace Partitioning

17 © 2015 ANSYS, Inc. July 1, 2015

Laplace coarsening based partitions – IC• Original partition cuts through high

aspect-ratio regions

• Convergence issues cause mass conservation problems

• Use laplace partitioning to avoid going along high aspect ratio area

• Convergence improved significantly

• Mass conservation is much improved

18 © 2015 ANSYS, Inc. July 1, 2015

Improved Multi-layer Shell Conduction Performance & Scaling

• Uses the enhanced partition algorithm for encapsulation of shell zoneso Faster speedo Better overall load balance and qualityo Now default in R160.

• Significantly improveso Case read time (up to 5X)o Solver time (up to 6X)

• Example CAT case with DPM + Shell (with 32M cells)

Parallel team in cooperation with shell/radiation team

19 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

Some More Benchmark Cases

20 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

New Benchmark Suite

21 © 2015 ANSYS, Inc. July 1, 2015

Case Details:

• External flow over aircraft landing gear

• Number of cells: 15 Million

• Cell Type: Mixed

• Models used: LES + Acoustics

• Solver: Pressure based coupled, Least Square cell based, Unsteady

Improved Parallel Performance & Scaling

22 © 2015 ANSYS, Inc. July 1, 2015

Case Details:

• Wave loading on Oil Rig

• Number of cells: 7 Million

• Cell Type: Mixed

• Models used: SST K-omega turbulence

• Solver: Pressure based segregated, VOF, Green-Gauss cell based, unsteady

Improved Parallel Performance & Scaling

23 © 2015 ANSYS, Inc. July 1, 2015

Case Details:

• Single-stage Transonic axial-flow Fan Stator Row

• Number of cells: 2.5 Million

• Cell Type: Hexahedral

• Models used: SST K-omega turbulence

• Solver: Pressure based coupled, Least Squares cell based, pseudo transient, sliding interfaces

Ref: NASA-103800

Improved Parallel Performance & Scaling

24 © 2015 ANSYS, Inc. July 1, 2015

Case Details:

• Cavity flow in a centrifugal pump

• Number of cells: 2 Million

• Model used: Realizable K-ε turbulence

• Solver: Pressure based segregated, mixture multiphase, cell based gradient, steady

Improved Parallel Performance & Scaling

25 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

GPGPU Update

• Utilities for radiation s2s

o Viewfac

o Raytrace

• Core solver

o Coupling with AmgX

26 © 2015 ANSYS, Inc. July 1, 2015

Fluent + AmgX Performance

• Fluent loads AmgX to run on nVidia GPUs to solver linear systems

• 9.6M Tet Cells, Pressure based coupled, Internal Flow, Double Precision

• CPU: Sandy Bridge, E5-2667, 12 = 2 Sockets x 6 Cores, 2.9 GHz, 128 GB

• GPU: Kepler, Tesla K20Xm, 2688 CUDA Cores, 0.73 GHz, 6.14 GB

• Much more beneficial to add a GPU to low CPU process count Time on 4 nodes, the lower the better

CPU Cores per node

27 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

Coupled PBNS, Double precision, Steady, Run until converged by nVidia. This is different than the standard benchmarks

• CPU: Intel Xeon E5-2680, 8 cores per node• GPU: Tesla K40, 2 per node• 1 node

• CPU: Intel Xeon E5-2667, 12 cores per node• GPU: Tesla K40, 4 per node• 12 nodes

Fluent + AmgX Performance

28 © 2015 ANSYS, Inc. July 1, 2015

Fluent AmgX Options

Per equation user controls for AMG-GPGPU parameters

29 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

• HPC Improvements have been in the area of IO, meshing, partitioning, and solver scalability to 36K+ cores. All these would bring great benefits to our customers

• We are working on the following areas for 17.01. Enhance solver mesh operation scalability, especially for MDM

& sliding mesh cases

2. Improve core solver scalability beyond 50K cores, with further optimization of partitioning and AMG solver

3. Optimize load balancing algorithms to have a more scalable multi-physics solver

4. HDF and I/O Consolidation and enhancements

5. Introduce an object oriented domain framework that can be used for GPU/CPU hybrid computing

6. DPM parallel performance enhancements/extensions

Summary & Next Steps

30 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential https://sites.google.com/a/ansys.com/products/ansys-hpc

31 © 2014 ANSYS, Inc. July 1, 2015 ANSYS Confidential

Easier Presentation Thanks to Applications Classification

31

and Easier Access Via Xchange

Realize Your Product Promise®

THANK YOU!