March 12, 2008

A Parallel Algorithm for Optimization-Based

Smoothing of Unstructured 3-D Meshes

byVincent C. Betro

Outline

• Introduction

• Objectives

• Serial Optimization-Based Smoothing

• Parallel Implementation

• Results• Conclusions• Future Work

IntroductionParallel Hierarchical Unstructured Cartesian Mesh Generation

• Using P_HUGG, developed with Steve L. Karman, Jr. as part of an NRA with NASA Langley• Uses hexahedral, Cartesian elements • Body conforming through general cutting, yielding the four basic element types or polyhedral elements• Can cause small, high aspect ratio elements to be formed• Allows user to determine refinement based on geometry spacing

IntroductionWhy do we need optimization-based smoothing?

< BEFORE

AFTER >

Introduction

Terminology• Ghost nodes: Nodes that exist and will be worked on by a

neighboring processor but that are part of elements on the given processor. They allow information about movement to be shared with the neighboring processors fluidly and yet only allow one processor to move them in the smoothing process in order to preserve uniqueness.

• Threshold: User-defined value that determines the level of the cost function (between 0 and 1 for non-inverted elements) that will be allowed to exist as a maximum in the final mesh. Usual values for a very smooth mesh are between 0.1 and 0.3. Also, low thresholds cause most, if not all, nodes in the mesh to be worked on, making the METIS partitioning more accurate and relevant.

Objectives

• To determine that one can smooth a mesh in parallel using ghost nodes on processor borders and obtain the same results in a comparable number of iterations / time

• To determine if weighting based on nodes being on the geometric boundaries (needing to be re-projected), nodes being completely internal (being perturbed in more directions), or no weighting at all would yield the most benefit in determining the METIS partition

• To determine if the algorithm would only allow larger meshes to be smoothed or if it would also garner speed-up on smaller meshes, if only to a point

Serial Optimization-Based Mesh Smoothing

In order to remove high aspect ratio elements (sliver cells) and get improved results from the flow solver, optimization-based smoothing is performed on the mesh.

• Each node is perturbed based on a cost function calculated using Jacobians and condition numbers of the surrounding elements.

Serial Optimization-Based Mesh Smoothing

First, the surrounding triangles’ cost function is computed:

Then, the node’s cost function is computed using the max and average:

The blending function is used to avoid the average cost being overshadowed:

Serial Optimization-Based Mesh Smoothing

• If the perturbation improves the cost function for the node, the node is moved permanently to the new position

– Along direction of opposite face normal

– Along direction of included edge

– Along cardinal x, y, or z coordinate axis

Note: A node is always perturbed by a pre-determined epsilon, based on a fraction of the length

of the smallest edge

Serial Optimization-Based Mesh Smoothing

• The mesh is moved until:

– eventually all perturbations cannot improve the cost function or

– the threshold value supplied by the user is obtained or

– the maximum number of perturbations supplied by the user is attained.

Experience will tell the user when to stop working on the boundaries and/or internal points and/or change threshold values or maximum iterations.

Parallel Implementation

• Using the METIS libraries, one can partition the mesh to be distributed to multiple processors on which the optimization-based smoothing can then be carried out

• Available at www.cs.umn.edu/~metis

• METIS finds optimal configuration to assist in load balancing and decrease surface area for communication

• Using OPENMPI for the message-passing protocol

Parallel Implementation

• Similar implementation except:– Ghost nodes processed first to allow bias to propagate as

normal– Communication of changes required between each

iteration using processor-index triple• Processor index• Owning processor local index• Global index

– Global communication to discern global threshold, action taken based on global value; processors already attaining value continue to improve smoothness and adjust for neighbors who have not attained threshold of cost function

Results: DTMB 5415 Surface Ship

• Geometry extracted from existing tetrahedral mesh.

• 382,481 nodes• 904,452 elements

– 391,828 tetrahedra– 187,274 hexahedra– 325,350 pyramids

• 1 to 64 processors on Dell Linux Cluster– 325 nodes, 1300 processors (4

per node)– 4 Gbytes per node– Dual-core Intel EM64T 3.0 GHz Xeon

Results: DTMB 5415 Surface Ship• Weighting proved not to

assist in time used or smoothness of final mesh

• Due to similar numbers of different operations needing done whichever type the node was

• Also, the connectivity of each node and type of opposite face come to bear on workload.

• Cases run with high thresholds tended to benefit less from the parallelization and the addition of extra processors.

Results: DPW III

• Surface geometry created in Gridgen– High-aspect ratio quads on wing

converted to triangles• 1,739,455 points• 4,382,046 elements

– 2,326,329 tetrahedra– 899,786 hexahedra– 1,155,931 pyramids

• 1 to 64 processors on Dell Linux Cluster– 325 nodes, 1300 processors (4

per node)– 4 Gbytes per node– Dual-core Intel EM64T 3.0 GHz

Xeon

Surface definition

Symmetry planemesh

Results: DPW III• Weighting proved not

to assist in time used or smoothness of final mesh

• Due to similar numbers of different operations needing done whichever type the node was

• Also, the connectivity of each node and type of opposite face come to bear on workload.

• Cases run with high thresholds tended to benefit less from the parallelization and the addition of extra processors.

Results: DPW IIIDPW volume mesh before and after optimization-based

smoothing is applied to airfoil.

< BEFORE

AFTER >

Results: DPW IIIDPW volume mesh after extra, internal-only optimization-based smoothing is

applied to the airfoil with threshold of 0.01 for 50 iterations.

Conclusions

• An effective optimization-based smoothing algorithm can be easily parallelized using OPENMPI

• Using low threshold values on both large and small meshes showed the most gains due to the fact that nearly all nodes in the mesh are being worked on the whole time to improve the cost function

• Large threshold values showed some gains with large meshes but generally showed no improvement past four processors for small meshes

• Moreover, regardless of the threshold values, very large meshes that could not fit on a serial machine were smoothed, such as one based on USGS data for Yosemite National Park

• Weighting boundary and non-boundary nodes differently has almost no affect on speed-up or smoothness, and can actually adversely affect both of the above

• The bias that might be attributable to ghost nodes was successfully damped by smoothing them first

Future Work

• Add in ParMETIS functionality to allow for pre-partitioned mesh files, which allows the decomposition step to be skipped and greater load balancing and decreased surface area to be obtained through the use of ParMETIS

• Allow the smoothing to work on general polyhedral elements, not only the four basic element types, by creating a cost function that can handle the corners of any type of polyhderal element

• Use this simpler optimization as a template for parallelizing linear-elastic smoothing and viscous layer insertion codes

• Use optimization-based and/or linear-elastic smoothing in conjunction with solution and feature based adaptation in parallel using the octree-based P_HUGG to re-mesh between solution steps for moving, body-conforming, viscous meshes on the fly without intermediate processing