fall 2001 navo msrc navigator journal · 4 a proteomics approach to a malaria vaccine 7 metvr:...

Fall is arriving after what has been a monumentallybusy and tragic summer. A record audience ofapproximately 400 folks attended the 2001 UsersGroup Conference in Biloxi, Mississippi in June. TheMajor Shared Resource Center (MSRC) TechnologyInsertion for FY01 (TI-01) planning and approvalactivities were completed, with substantial computingupgrades to be delivered shortly to the AeronauticalSystems Center and Engineer Research andDevelopment Center MSRCs. The MSRC TI-02 plan-ning and approval activities have subsequentlybegun, with major upgrades to be delivered to theApplied Research Laboratory and NAVO MSRCs byJune of 2002. The follow-on contract award for theDefense Research and Engineering Network (DREN)has been delayed slightly, but final award is anticipat-ed for this fall. The new DREN capability promises tobe much more capable and cost-effective—hats off toMr. Rodger Johnson of the High PerformanceComputing Modernization Office (HPCMO) and theentire DREN team who are making it happen. Andfinally, the long-awaited follow-on contract for thenew Programming Environment and Training (PET)program was awarded this summer and will be fullyimplemented at the start of FY02. Dr. Leslie Perkinsof the HPCMO and her PET technical advisory teamhave done an outstanding job of taking the successes

and lessons learned from the initial five-year PETprogram and incorporating them as the buildingblocks for an expanded HPC Modernization Program-wide PET effort.

Finally, no words can adequately describe the anguishand sorrow we all feel as a result of the tragic eventsof September 11 in New York, Washington, D.C., andPennsylvania. The United States Department ofDefense (DoD) will play a major role in the responseto this tragedy—it is my fervent hope that the pasteight years of support by the DoD HPCModernization Program will, in meaningful ways,make DoD's mission more effective, much quicker,and substantially safer for those who must go inharm's way in response to these attacks.

2 FALL 2001 NAVO MSRC NAVIGATOR

The Director’s Corner

About the Cover:

Virtual environment built by the NAVO MSRC Visualization Center for the Concurrent Computing Laboratory forMaterials Simulation at Louisiana State University. This application allows the researchers to visualize a millionatom simulation of an indentor puncturing a block of gallium arsenide. See pages 14-15 (Centerfold) for additional information.

Steve Adamec, NAVO MSRC Director

Moving Ahead at the NAVO MSRC

3FALL 2001NAVO MSRC NAVIGATOR

The Naval Oceanographic Office (NAVO)

Major Shared Resource Center (MSRC):

Delivering Science to the Warfighter

The NAVO MSRC provides Department ofDefense (DoD) scientists and engineers with highperformance computing (HPC) resources, includ-ing leading edge computational systems, large-scale data storage and archiving, scientific visuali -zation resources and training, and expertise in spe-cific computational technology areas (CTAs).These CTAs include Computational FluidDynamics (CFD), Climate/Weather/OceanModeling and Simulation (CWO), EnvironmentalQuality Modeling and Simulation (EQM),Computational Electromagnetics and Acoustics(CEA), and Signal/Image Processing (SIP).

NAVO MSRC

Code N7

1002 Balch Boulevard

Stennis Space Center, MS 39522

1-800-993-7677 or

[email protected]

NAVO MSRC Navigator

www.navo.hpc.mil/navigator

NAVO MSRC Navigator is a biannual technicalpublication designed to inform users of the news,events, people, accomplishments, and activities ofthe Center. For a free subscription or to makeaddress changes, contact NAVO MSRC at theaddress above.

EDITOR:

Gioia Furness Petro, [email protected]

DESIGNERS:

Patti Geistfeld, [email protected]

Kerry Townson, [email protected]

Lynn Yott, [email protected]

Any opinions, conclusions, or recommendations inthis publication are those of the author(s) and donot necessarily reflect those of the Navy or NAVOMSRC. All brand names and product names aretrademarks or registered trademarks of theirrespective holders. These names are for informa-tion purposes only and do not imply endorsementby the Navy or NAVO MSRC.

Approved for Public Release

Distribution Unlimited

ContentsThe Director’s Corner

2 Moving Ahead at the NAVO MSRC

Feature Articles

4 A Proteomics Approach to a Malaria Vaccine

7 MetVR: Using Virtual Reality for Meteorological Visualization

10 Towards Prediction of Aircraft Spin

13 OpenMP Parallel Implementation of SWAN

Scientific Visualization

14 CCLMS Challenge Project

18 Updates from NAVO MSRC

19 Enhancing NAVO MSRC Visualization Capabilities

Programming Environment and Training

20 DoD User Group Conference 2001 Sights

22 NAVO MSRC PET Update

Navigator Tools and Tips

23 Running X Windows with Parallel Codes onHABU Under LoadLeveler

The Porthole

24 A Look Inside NAVO MSRC

Upcoming Events

27 Conference Listings


A Proteomics Approach to a Malaria Vaccine Giri Chukkapalli, Amitava Majumdar, and Robert SinkovitsScientific Computing Department, San Diego Supercomputer Center,La Jolla, CACapt Daniel Carucci, MC, USNMalaria Program, Naval Medical Research Center, Silver Spring, MDJohn Yates and Laurence FlorensThe Scripps Research Institute, La Jolla, CAAccording to the World Health Organization, malaria cur-rently affects 300-500 million people worldwide. More than90 percent of these cases occur in sub-Saharan Africa andare responsible for over one million deaths each year. Thebattle against malaria has been hampered by the emer-gence of drug-resistant strains of Plasmodium falciparum,the parasite responsible for the majority of malaria infec-tions. Further complicating the use of antimalarial drugs isthe fact that most cases are concentrated in the world'spoorest countries, thereby imposing the additional require-ment that newly developed treatments be easily affordable.

As an alternative to antimalarial drugs, work is activelybeing done on the development of a malaria vaccine. Dueto the complicated multistage life cycle of P. falciparum (seeFigure 1), this is a difficult task. Using the recently complet-ed draft of the P. falciparum genome, researchers at theNaval Medical Research Center (NMRC) in the laboratory

of Daniel Carucci aresystematicallyapproaching this prob-lem by studying theproteins that areexpressed at various stages of the parasite's develop-ment.1,2 The study uses a new approach to the analysis ofprotein mixtures that proteolytically digest the proteinspresent at each stage of the life cycle and then analyzes theresulting peptides to determine the proteins present, asshown in Figure 2.

Tandem Mass Spectrometry (MS/MS) is the primary experi-mental tool used for the protein identification. The firststage is used to isolate peptides having a selected mass tocharge ratio. The isolated peptide ion is then subjected tocollisionally activated dissociation, and the second stage ofthe MS/MS is used to measure the masses of the fragmentsyielding a characteristic sequence ion fingerprint as shownin Figure 3. The Sequest program,3 developed in the labo-ratory of John Yates while at the University of Washingtonand marketed by ThermoFinnigan, is used to identify theproteins by comparing the MS/MS output to the appropri-ate protein or DNA database (Figure 4). It is this analysisstep that is the current bottleneck in the analysis of the P.falciparum parasite.

NAVO MSRC PET Tiger Team collaboration between theSan Diego Supercomputer Center (SDSC), The ScrippsResearch Institute, and NMRC had been initiated earlier thisyear to reduce or eliminate this computational bottleneck.Currently, the analysis of the P. falciparum proteome on asingle-processor machine takes about 30 days of computertime. By taking a two-pronged approach—single processoroptimization and code parallelization—we expect to be ableto reduce the time to solution on the largest NAVO MSRCmachines to less than 30 minutes. Sequest is the most wide-ly used software for the analysis of MS/MS data and is esti-mated to be used by more than 500 laboratories.Improvements to Sequest should be expected to not onlyaccelerate the malaria vaccine project, but also have theFigure 1. Life cycle of the malaria parasite P. falciparum.

Anopheline mosquito responsible fortransmission of malaria

T CellsCytkinines

Antibodies

T CellsCytkinines

Free Radicals


potential to impact a much broader pro-teomics community.

SINGLE-PROCESSOR OPTIMIZATION

The goal of single-processor optimiza-tion is to reduce the time to solutionfor a calculation carried out on a singleprocessor. Since the serial code is usedas the starting point for the parallelapplication, this also boosts the per-formance of the parallel version of thecode. A number of general and specialtechniques were applied as describedbelow.

As would be expected with a pro-teomics application, a significantamount of time is spent performingstring operations. One of the codemodifications that had a large impacton performance involved the replace-ment of calls to the C string library rou-tines with logical tests on pre-computedarrays of integers. Normally, library func-tions give much better performance than user-developedroutines, but in this case the full functionality of the libraryroutine was not required. The C strchr () function returnsthe location of the first occurrence of a character (typicallya residue of the peptide sequence) in a string (for example,a list of protein cleavage sites). Since the test strings arenot modified after the initialization stage of the program, agreat deal of computational overhead can be avoided bytesting the strings for the occurrence of all characters in theamino acid alphabet (normally 20 letters for the naturallyoccurring amino acids, plus optionally additional charac-ters for unknown or nonstandard amino acids) at the startof the calculation.

Another optimization that had a large impact on the codeperformance was the precalculation of loop-invariantquantities. (A loop invariant is an expression that appearswithin a loop, but whose value remains constant acrossiterations of the loop.) While most compilers are quiteeffective at recognizing loop-invariant quantities, the pres-ence of function calls within the loop may force the com-piler to make more conservative choices. These choicesmay be required because the functions may have the sideeffect of modifying the variables that are used in theexpression. Other optimizations such as the elimination ofredundant logical expressions, loop pealing, force reduc-tions, and replication of key loops for special cases furtherreduced run time. Timing comparisons for the original andoptimized versions of the code are given in Table 1.

CODE PARALLELIZATION

While the benefits of single-processor optimization areundeniable, the biggest potential impact on reducing thetime to solution arises as a result of code parallelization.Since the analysis of the P. Falciparum proteome involvesthe independent processing of roughly 100,000 input files,it is expected that a parallel version of the code should becapable of linear scaling up to thousands of processors.

The optimized version of the serial Sequest code was usedas the starting point for a parallel implementation devel-oped using the Message Passing Interface (MPI). In theparallel code, input files are handed out in a round robinfashion to processors rather than dividing up the list ofinput files at the start of the simulation. This makes no

Figure 2. Experimental setup used to determine proteins expressed by P. falciparum.The Sequest program is used in the analysis of the MS/MS spectrum.

Article Continues Page 6...

Table 1. Performance of original and optimized scalar version ofSequest code. The five test cases stress various parts of theSequest code and are representative of the types of calculationsmost frequently performed using Sequest. Test 5 is based onparameters most relevant to the malaria vaccine project. All tim-ings were obtained on a single 375-MHz IBM Power3 processorusing aggressive optimization (-O3).

Original Optimized Speedup

Test 1 83 45 1.84

Test 2 137 63 2.17

Test 3 202 111 1.82

Test 4 381 173 2.20

Test 5 269 139 1.93


difference in the case where the time to process each inputfile is identical, but avoids load-balancing problems whenthere is significant variation in the processing times.Benchmarks carried out on the IBM SP at SDSC showthat it takes about 25 sec to process 17 input files on 8CPUs and about 300 sec to process 1700 input files on 64CPUs. These tests show that speedup is nearly linear forthe parallel Sequest code.

In addition to problem decomposition, attention was alsofocused on efficient I/O. The original version of the coderead in the complete protein or genome database for eachinput file. For the analysis of the P. falciparum, the data-base is 30 MB in size, and there are on the order of100,000 input files. The total I/O requirements for this runwould be 3 TB, enough to potentially saturate the I/O net-work and slow down the simulation. By reading in thedatabase just once per MPI task, the amount of I/O for thiscase run on 1000 processors could be reduced by twoorders of magnitude. We are currently testing a scheme inwhich the database would is read in once by a singleprocessor and then broadcast to all the other processors.

IMPACT ON RESEARCH

We expect that the combination ofcode parallelization and single-processor optimization should lead toa reduction in time to solution from30 days to less than 30 minutes. Thisdramatic improvement in code per-formance should accelerate theprogress of the malaria researchersand allow them to easily performnew analyses of the P. falciparumgenome as improved versions of theprotein or genome databases becomeavailable. Since Sequest is a general-purpose tool in use by hundreds ofMS/MS labs around the world, the

work that we have done has thepotential to dramatically impact anentire proteomics community.

AcknowledgementsThe research described was funded by Naval Medical Research Center Work Units 61102AA0101BFX and 611102A0101BCX and the U.S. ArmyMedical Research and Materiel Command Contract (DAMD17-98-2-8005). Sequest optimization and parallelization was made possible through sup-port by the NAVO MSRC PET program. The opinions expressed are those of the authors and do not reflect the official policy of the Department ofthe Navy, Department of Defense, or the U.S. Government.

References1. Carucci, D.J., “Malaria research in the post-genomic Era” [In Process Citation], Parasitol Today, 2000, 16:434-8.

2. Hoffman, S.L., Rogers, W.O., Carucci, D.J., Venter, J.C., “From Genomics to Vaccines: Malaria as a Model System” [Comment], Nat. Med.,1998; 4:1351-3.

3. Eng, J.K., McCormack, A.L., Yates III, J.R., J. Am. Soc. Mass Spectrom. 1994, 5, 976-989.

Figure 4. Flowchart of Sequest operation. The key step isthe comparison of the sequence ion fingerprint to the DNAor protein database for protein identification.

Figure 3. Typical mass spectrum originating from the second stage of MS/MS.


Traditional methods for displaying weather products aregenerally two-dimensional (2D). It is difficult for forecastersto get the entire picture of the atmosphere using thesemethods, as the atmosphere isthree-dimensional (3D).

The problems apparent in2D with comparing and cor-relating multiple layers areovercome by adding adimension. However, simplyusing a 3D approach is notenough. Visualization in 2Dhas a capacity for analysisof small-scale but importantfeatures. This capacity is lostwhen transitioning to 3D. We propose that 3D's advan-tages can be incorporated with2D's small-scale analysis byusing an immersive virtual envi-ronment, also known as virtual reality. Currently, we aredeveloping such an application that we call MetVR.

THE PROBLEM

TRADITIONAL 2D VISUALIZATION

Traditional 2D visualization of multilayer, time-series datamakes it difficult to see all layers and time steps in a singleimage. Animation is the most obvious solution to visualiz-ing the time-series aspect of the data. However, we still

have the problem that inherently 3D data, which capturesthe essence of atmospheric behavior, is being reduced to2D.

The figure at the bottom ofthe page illustrates howmultiple layers are usuallydisplayed in 2D. To beable to analyze phenome-na that span several lay-ers, one would have toimagine each imagesuperimposed over theother. This could be diffi-cult to imagine for manylayers and would be fur-ther complicated withtime-series data.

TRADITIONAL 3D VISUALIZATION

Using 3D visualization andanimation, we can easily view multilayer, time-series datasets in a unified manner. Interactive 3D visualizationexploits the mind's ability to grasp complex environments.For example, instability of atmosphere is crucial for thedevelopment of storms. The stability of the atmosphere isdependent upon vertical structure. Three-dimensionalvisualization techniques can provide a visual understand-ing of vertical structure in any part of the domain.

Traditional "desktop 3D" visualization loses an importantfeature of 2D: the ability to closely examine small-scale,but important, phenomena within the data set. This is an

MetVR: Using Virtual Reality for MeteorologicalVisualizationSean Ziegeler and Robert J. Moorhead, ERC, Mississippi State UniversityPaul J. Croft and Duanjun Lu, Department of Physics, Jackson State University

Three layers from an MM5 model output data set: surface layer, 250 mb, and 300 mb. Each displays wind vectors with asecond variable shown as isolines.

A view from the VR Juggler simulator of a simulation ofHurricane Dennis. Rainfall is shown on the terrain. The iso-surfaces show potential energy.


easy task in 2D. We simply zoom into the area of an imagein which we are interested. If we were to zoom into thecenter of an image, for example, the surrounding datawould not occlude our view of that area. Also, if we want-ed to compare that area to another area, zooming out andpanning around the image is easy. However, with desktop3D, we are generally viewing a volume of data from out-side the volume. Although it is possibleto implement 3D navigation suchthat the user is placed inside thedata, it is still difficult for the user toanalyze the data as if inside the dataset.1

THE SOLUTION

Our solution is to implement animmersive Virtual Environment (VE)that we call MetVR. Besides our pri-mary goal, there are additionaladvantages to applying VE technolo-gy to meteorological visualization:

? Improved human-computerinteraction2

? More degrees-of-freedom fornavigating the data 2

? A "visual paradigm" more closelyrelated to the real world3

? Stereoscopy for improved depthperception2

? Wider field-of-view2

The application itself is built upon the VRJuggler libraries with OpenGL as the graph-ics Application Programming Interface. Wechose VR Juggler because it is open sourceand more portable than other competing VElibraries. VR Juggler is also written in C++,and we are using an object-oriented designstrategy.

IMPROVEMENTS UPON PREVIOUS VES

Two other VE systems, Cave5D and vGeo,can cache data on disk, but MetVR is opti-mized for multivariable, time-series data sets.Thus it streams in the data as needed.MetVR is faster and more flexible. We alsofeel that the visuals are much richer. Finally,MetVR is free and user extendible. To usevGeo, one must purchase both the CaveLibs

and vGeo itself.

THE RESULTS

With MetVR at a usable phase in its development, wecould compare its effectiveness with traditional 2D and 3Dapplications. The evaluation involved meteorologist testingthe application in the Computer-Assisted VirtualEnvironment (CAVE) to determine its effectiveness andusefulness.


This view is a simulation of the view inside the CAVE. Once again, the variablesare the same as in previous figures. The left, center, and right images show theleft, center, and right walls of the CAVE, respectively. The bottom image is thefloor of the CAVE.

A view from the VR Juggler simulator. The particles indicate snow (white) &ice (blue). Rainfall is shown on the terrain, and clouds as white isosurfaces.


The evaluators had considerable experience with common2D and 3D visualization software. We allowed them toexplore the data set and noted any comments and ques-tions. When the users were finished evaluating, we asked aseries of evaluation questions.

SMALL-SCALE ANALYSIS

With their experience in traditional 2D and 3D visualiza-tion, the users did indeed find that small-scale features wereeasier to see and analyze in the VE. They could identifyregions of interest from a distance, and then simply navi-gate to the area for further inspection. One specific advan-tage that a user noted was that the VE shows the tops andbottoms of cloud layers when he navigated to a specificregion. This is an important feature in storm analysis sincethe location and shape of clouds indicate if a storm is orwill be present (i.e., low altitude, thin clouds usually mean astorm).

ADDITIONAL ADVANTAGES

The users found the navigation more natural than with adesktop system. They could simply point to the locationwith a wand and press a button to fly there. The usersthought that they could easily complete common analysistasks using the given interface.

As expected, the users found that immersion in the meteo-rological data in the VE was an excellent metaphor foractual weather. They found the stereoscopy to be very use-ful in discriminating between near and distant features—amuch more difficult task using desktop 3D. Also, the widerfield-of-view allowed them to see more data simultaneously.

CONCLUSIONS AND FUTURE WORK

Using an immersive VE, we were able to combine 3D'sgreater ability for multilayer analysis with 2D's capacity forsmall-scale analysis. We could qualitatively identify how ourimmersive VE achieved this goal. We believe that such anadvantage, in addition to the other listed advantages inusing immersion, could lead to improved interpretation ofmeteorological data. Such an improvement would bringabout better weather-related products and increased capabil-ity in decisionmaking. Our conclusion from this study is thata VE could be properly tailored for atmospheric analysis,and further development of such a VE would be valuable.

For future work, we plan to do a comprehensive studycomparing the abilities to see small-scale features so thatwe can quantify the improvements gained by immersion.We also plan to implement improvements to MetVR basedon users' recommendations. To help indicate the user'slocation within the dataset, we will add a "you-are-here"overview map with an icon indicating the user's locationinside the dataset.

For the particle density fields, spheres instead of pointsmay be desirable. However, this is unrealistic, consideringthe rendering time necessary to render the required num-ber of spheres. Instead, we propose using a single bill-boarded polygon and texture mapping the polygon withthe image of a sphere. Additional features will include newtools and enhancements to tools.

References

1. Sarathy, S., Shujace, K., Cannon, K., “Visualization of Large Complex Datasets Using Virtual Reality,” IEEE Information Technology: Coding and Computing, 522-526, March 2000.

2. Youngblut, C., Johnson, R., Nash, S., Wienclaw, R., Will, C., “Review of Virtual Environment Interface Technology,” IDA Paper P-3186, Institute for Defense Analyses, 1801 N. Beauregard Street, Alexandria, VA 22311-1772,March 1996.

3. Cruz-Neira, C., “Applied Virtual Reality,” Course 14 Notes, In Proceedings of SIGGRAPH '98 (Orlando, FL, July1988), ACM Press.

Acknowledgements

This work was done as part of the High Performance Visualization Center Initiative funded by the Department of Defense High Performance Computing Modernization Program and administered by the NAVO MSRC.

Most of the flow fields encountered in Department ofDefense applications occur within and around complexdevices and at speeds for which the underlying state ofthe fluid motion is turbulent. While Computational FluidDynamics (CFD) is gaining increased prominence as auseful approach to analyze and ultimately design config-urations, efficient and accurate solutions require substan-tial effort and expertise in several areas. Geometrydescription and grid generation, numerical solution ofthe Navier-Stokes equations, and efficient postprocessingare all key elements.

While advances have taken place in areas such as gridgeneration and fast algorithms for solutionof systems of equations, CFDhas remainedlimited

as a reliable tool for prediction of inherently unsteadyflows at flight Reynolds numbers. Current engineeringapproaches to prediction of unsteady flows are based onsolution of the Reynolds-averaged Navier-Stokes (RANS)equations. The turbulence models employed in RANSmethods necessarily model the entire spectrum of turbu-lent motions. While often adequate in steady flows withno regions of reversed flow, or possibly exhibiting shal-low separations, it appears inevitable that RANS turbu-lence models are unable to accurately predict phenome-na dominating flows characterized by massive separa-tions. Unsteady, massively separated flows are character-ized by geometry-dependent and three-dimensional (3D) turbulent eddies. These eddies, arguably, are what

defeat RANS turbulence models of any complexity.

To overcome the deficiencies of RANS modelsfor predicting massively separated flows, Dr.Philippe Spalart proposed Detached-EddySimulation (DES) with the objective of devel-oping a numerically feasible and accurateapproach combining the most favorableelements of RANS models and Large EddySimulation (LES). The primary advantageof DES is that it can be applied at highReynolds numbers, as can Reynolds-aver-aged techniques, but also resolves geome-try-dependent, unsteady 3D turbulentmotions as in LES.

Computations were performed of theflow over an F-15E at angle of attackof 65º and zero sideslip. Boeing pro-vided the authors with a stability andcontrol database for the F-15E thatwas developed from a comprehen-

sive spin-flight test program. Two sta-ble spin conditions were detailed,

including data for symmetric and asym-metric fuel loads. The aircraft with sym-

metric loading maintains a stable spin at 65º

angle of attack. Prior to computing the actual spinusing rigid body moving grids, the performance of thecomputational model was investigated at the same fixed


Towards Prediction of Aircraft SpinKyle D. Squires, Arizona State UniversityJames R. Forsythe, United States Air Force AcademyKenneth E. Wurtzler, William Z. Strang, Robert F. Tomaro, Cobalt Solutions, LLCPhilippe R. Spalart, Boeing Commercial Airplanes

Figure 1. Comparison forthe F-15E of RANS (SA) andDES at angle of attack = 65º.Isosurface of vorticity colored by pressure.

angle of attack as for the stable spins. All computationswere made matching the flight test conditions at a Machnumber of 0.3 and standard day 30,000 feet. This result-ed in a chord-based Reynolds number of 13.6 million.

Simulations were performed at the NAVO MSRC CrayT3E. The F-15E runs (5.9 million cells) were computedusing up to 512 processors and required 1 to 2 days forcomplete turnaround. For the DES runs, the F-15E gridwas mirrored about the symmetry plane, resulting in11.8 million cells. These runs required approximatelyfour days for turnaround on 256 processors and werecomputed using a non-dimensional time-step(using the chord andfreestream velocity) of 0.01.

Side by side comparisonsof DES and RANS predic-tions across the symmetryplane are shown in Figure1. The isosurfaces of vortic-ity illustrate the capabilityof DES in "LES mode"resolving the unsteady,geometry-dependent flowfeatures. Force histories ofCL, CD, and CM for the twosimulation techniques arenext compared to the Boeingdatabase in Figures 2, 3, and 4.The largest discrepancy for the

RANS result is in themoment coefficient, whichis overpredicted (in a neg-ative sense) by 12%. Thelift and drag coefficientsare overpredicted by 9%and 7%, respectively. DESpredictions of these quan-tities are all within 4% forlift and drag and 7% formoment for both timesteps. The expected accu-racy of the Boeing data-base for this angle ofattack is anticipated to be

around 5%.

To examine the source of thedifferences between theRANS and DES results, the

pressure coefficient from the (steady-state) RANS is com-pared to the time average of the DES predictions inFigure 5. As evident from the figure, DES yields a rela-tively flat profile in Cp that is the norm on a wing withseparation, while the RANS predicts a more varied pres-sure distribution. This lends confidence to the accuracyof the DES results in the absence of experimental pres-sure profiles. In the current study, the computational costof the time-accurate calculations is roughly an order ofmagnitude greater than a steady-state run. The unsteady


Figure 3. Time and iteration histories of CD on the F-15E.

Figure 2. Time and iteration histories of CL on the F-15E.

Figure 4. Time and iteration histories of CM on the F-15E.



Figure 6. Comparison of DES method (left image) and standard RANS method (right image) applied to sideslip flow past a tractor-trailertruck model.

RANS calculations carry the same cost as DES if the gridand time step are the same and if the sampling periodfor calculation of averaged quantities is fixed. The cur-rent calculations, although not yet demonstrating grid-convergent solutions, suggest that DES exhibits improvedpredictions on a grid originallydesigned for RANS. Althougha factor of ten increase in costmay seem large, if Moore's lawcontinues to hold, this factorof ten will be surpassed in lessthan three and a half years.Thus, facilities that can cur-rently support steady RANScalculations on full aircraftshould be able to accommo-date DES runs on at leastcoarse grids.

For the F-15E, the presentDES calculations are probablythe first applications of a tur-bulence-resolving technique tofull aircraft at flight Reynoldsnumbers in which turbulentboundary layers on the vehiclewere represented without recourseto wall functions (i.e., with gridspacings within one vis-cous unit at the wall).This technique is beingapplied to drag analysisof semitruck tractor-trailers. The sharpedges of the truckgeometry creates a defi-nite separation line,producing large-scale

unsteadiness. The DES solution captures this regionwhile the RANS solution dampens the unsteadiness(Figure 6).

Combining the time for grid generation and solution, aDES calculation could be made within two weeks, given

the performance of theHPC machines. In sixyears, if Moore's lawcontinues, HPC will seeits capacity increase byapproximately 16 times,allowing rapid solutionsfor grids nearing 200million cells. As gridsare made more dense,the fidelity of the DESsolutions will improvefor the current simula-tion (static aircraft).Alternatively, increasedcomputing capacity willenable the incorpora-tion of additional physi-

cal effects into the model-ing. The ability of DES topredict unsteady flow phe-

nomenon at highReynolds numbersopens the possibili-ty to handle morecomplex problemsthat require anunsteady solutionsuch as aerolastici-ty and aeroa-coustics.

Acknowledgements

This work is partially supported by AFOSR Grant F49620-00-1-0050(Program Manager: Dr. Tom Beutner). The authors are also grateful forthe assistance of Dr. Glen Peters, Dr. Ken Walck, and Dr. WaltLabozzetta of Boeing Military, who provided the stability and controldatabase and the F-15E geometry. Mr. Matthieu Chapelet and CadetKory Klismith assisted with grid generation and postprocessing. CadetWarren Carroll and Cadet Alan VantLand aided in postprocessing the F-15E results.

Figure 5. Comparison of the pressure coefficient from the(steady-state) RANS case and the time average of theDES case.

This article describes a recent collaboration between NAVOMSRC Programming Environment and Training (PET)analysts and Naval Research Lavoratory (NRL) researchersto realize a parallel implementation of an important wavemodeling code called Simulating WAves Nearshore(SWAN). A similar success-ful collaboration involvinga 3-D finite element circu-lation model was describedin the Spring 2001 NAVOMSRC Navigator. The goalof this project was to mod-ernize the memory man-agement used in SWANand then produce a paral-lel code that involved mini-mal changes in the algo-rithms used by the modeland no changes to the userinterface or configurationfiles. With these goals inmind, we chose to port themodel using the OpenMPmultithreading directives.Since OpenMP directivesare seen as comments bynon-OpenMP compilers,with careful planning, thisalso allowed us to producea code that was equallysuitable for serial architec-tures as well shared memo-ry architectures.Consequently, the samecode can be maintainedand distributed to theSWAN community at largeregardless of their choiceof architecture.

MODEL DESCRIPTION

SWAN1 is a serially coded wind-wave model (WAM)designed to overcome the traditional difficulties of applyingpredecessor models (e.g., WAM,2 presently used at NAVO-CEANO) at relatively small scales (e.g., less than 500-mgrid spacing). Specifically, it uses a semi-implicit, uncondi-tionally stable numerical scheme for geographic propaga-

tion, so that high geographic resolution does not dictate anexcessively small (and therefore expensive) time step. Inaddition to a nonstationary mode, similar to WAM, SWANcan be used in a stationary mode, similar to the STationaryWAVE (STWAVE) model used at NAVOCEANO. SWAN

includes a more completedescription of the physicsthan does STWAVE, whichresults in more computationand longer turnaround time.This, coupled with the lackof a parallel version, repre-sents a significant obstaclein model development andtransitioning SWAN from aresearch code to an opera-tional code.

The SWAN code consists ofapproximately 30,000 logi-cal lines of code with over200 subroutines writtenmostly in Fortran 77 with asmall number of Fortran 90constructs added in recentyears. The model solves thespectral action balanceequation using finite differ-ence techniques in two geo-graphic dimensions (X andY), in spectral space (fre-quency and direction), andtime. In geographic space,an upwind finite differenceis applied, resulting in thestate at each grid pointbeing dependent only onthe state in the upwave (asdefined by the direction ofpropagation) grid points.This permits the spectral

space to be decomposed intofour directional quadrants. The whole solution processinvolves iteratively computing a sequence of four forward-marching sweeps, as illustrated in Figure 1, across the geo-graphical grid, with each sweep utilizing only boundary


OpenMP Parallel Implementation of SWANJohn Cazes, NAVO MSRC Programming Environment and TrainingTim Campbell, NAVO MSRC Programming Environment and TrainingErick Rogers, Oceanography Division, Naval Research Laboratory


OpenMP is a parallel programming model forshared memory and distributed shared memorymultiprocessors that works with either standardFortran or C/C++. OpenMP consists of compilerdirectives, which take the form of source code com-ments, that describe the parallelism in the sourcecode. A supporting library of subroutines is alsoavailable to applications. The OpenMP specificationand related material can be found at the OpenMPweb site: http://www.openmp.org. Online training inOpenMP is part of the NAVO MSRC PET distancelearning (http://www.navo.hpc.mil/pet/Video), andlinks to other online training material can be foundat the NAVO PET Parallel Computing Portal(http://www.navo.hpc.mil/Tools/pcomp.html).

In Fortran, OpenMP compiler directives are struc-tured as comments, written as C$OMP or !$OMP.This allows for the design of OpenMP codes thatcan be compiled for serial as well as parallel archi-tectures with no code changes. An OpenMP pro-gram begins as a single process, called the masterthread. When a parallel region, which is precededby either a PARALLEL or PARALLEL DO construct,is encountered, threads are forked, on separateprocessors if available, to execute the statementsenclosed within the parallel construct. At the end ofthe parallel region, the threads synchronize, andonly the master thread remains to continue execu-tion of the program. The PARALLEL DO constructis commonly used to parallelize computationallyintensive loops within a program.

Under Department of Defense High Performance Computing Management Office (HPCMO) support researchers at the Concurrent ComputingLaboratory for Materials Simulations (CCLMS), Louisiana State University, are applying expertise in materials simulations and high performance computing to

study high temperature materials (HTMs). The purpose of the project is to improve understanding of how atomic level processes determine macroscopic properties such asstructure, adhesion, and fracture toughness of HTMs and to identify new ways in which the performance of HTMs can be improved under extreme operating conditions.

The ability to interactively visualize large-scale atomic systems is critical to molecular dynamics simulations and understanding the atomic processes that occur. Together with the NAVOMSRC Visualization Center staff, CCLMS researchers are working to develop techniques that allow interactive visualization of large-scale atomic systems. Interactive walkthrough of

multimillion atom systems have been achieved using fast visibility culling based on octree data structures and multiresolution rendering. Currently, the use of parallel processing is being explored to achieve billion-atom walkthroughs.

The images shown here are snapshots of molecular dynamics simulations performed on the NAVO MSRC IBM SP to model nanoindentation of a thin film ofgallium arsenide (GaAs). The indenter, shown in gray, is slowly pressed into the GaAs surface, causing plastic deformation. In these images, only the

surface atoms and the atoms near the indenter region are shown. The color scale represents the centrosymmetry parameter. Centrosymmetry isa parameter used to measure cumulative non-affine deformation in crystals, and thus is useful for studying dislocations.

Indentation and microindentation are both classic techniques used by materials engineers to determine the hardness of materials. Nanoindentation allows scientists to similarly test the hardness of extremely thin films. Experimentally, these tests

are done using an atomic force microscope (AFM). The nanoindenter itself is created with micro- or nanofabrication techniques of a hard material such as diamond or silicon nitride. The AFM apparatus with the

nanoindenter tip is used to perform the indentation and then again to examine and image thedamage caused by the tip.

0(A)2

ment Office (HPCMO) support researchers at the Concurrent Computinge applying expertise in materials simulations and high performance computing tounderstanding of how atomic level processes determine macroscopic properties such as

he performance of HTMs can be improved under extreme operating conditions.s simulations and understanding the atomic processes that occur. Together with the NAVOhat allow interactive visualization of large-scale atomic systems. Interactive walkthrough ofree data structures and multiresolution rendering. Currently, the use of parallel process-

formed on the NAVO MSRC IBM SP to model nanoindentation of a thin film ofto the GaAs surface, causing plastic deformation. In these images, only the color scale represents the centrosymmetry parameter. Centrosymmetry isn crystals, and thus is useful for studying dislocations.

used by materials engineers to determine the hardness of materi-hardness of extremely thin films. Experimentally, these testse nanoindenter itself is created with micro- or nanofabri-

ond or silicon nitride. The AFM apparatus with theation and then again to examine and image the

Top. View from above that shows the indenter and the ejectedmaterial on the surface of the GaAs film. The structure of theejected material is consistent with that of amorphous GaAs, indi-cating that a phase transformation, solid-state amorphization, isoccurring underneath the indenter during the indentation. Left. View from the side of the film that shows the indenter sur-rounded by the damaged region of the film. The bulk of the plasticdeformation occurs around the indenter. However, in this picturewe see plastic deformation beginning to appear some distancefrom the film-indenter contact region (lower center).

conditions or previouslycalculated points in theupwind directions. Thisresults in a semiimplicitmethod that is inherentlystable.

APPROACH

Due to the data dependen-cies in the sweeping tech-nique employed, a simpledomain decomposition ofthe spatial grid is not possi-ble without completely alter-ing the numerical algorithm.Instead, we opted for a shared memory pipelined parallelapproach using OpenMP. This approach represents a non-traditional, i.e., not loop-level, way of using OpenMP toport an application to shared memory platforms. Thisallowed us to introduce parallelism at a fairly high level,requiring changes in only a few subroutines and leavingthe large majority of the code untouched.Since the numerical technique uses an upwind methodthat is dependent on the sweep direction, each spatial gridpoint has at most a data dependency in two directions.For example, in the first sweep, execution proceeds in thepositive X direction then the positive Y direction (seeFigure 1). Therefore, the data dependencies for each gridcell lie in the negative X and negative Y directions. Bydecomposing along the Y direction, each thread can beginexecution as its data dependency in the negative Y direc-tion is satisfied. The dependency in the negative X direc-tion will already have been satisfied because the sweep inthe X direction is serial.The changes required in the SWAN code to implement thepipelined parallel approach for the sweeping loops arestraightforward as illustrated with the following code:

The OpenMP PARALLEL DO directive distributes the Yloop among the available threads. The SCHEDULE clausedirects each thread to execute specific iterations of the Yloop with a chunk size of 1. For example, if the loop runsfrom 1 to 9 and there are 3 threads, thread 1 executes IY= 1,4,7, thread 2 executes IY = 2, 5, 8, and thread 3executes IY = 3, 6, 9. The STATIC keyword informs thecompiler that the IY values assigned to each thread willnot change during the execution of the PARALLEL DO.The LLOCK array, which is reset prior to beginning eachsweep, is a logical array that ensures that each thread can-not proceed to the next grid point until that grid point'sdata dependency in the Y direction has been satisfied.After the state at a grid point (IX,IY) is computed by thecall to the subroutine SWOMPU, the thread processingthat grid point signals it is finished by settingLLOCK(IX,IY) to FALSE.

The code illustration given above appears very simple,and may in fact be somewhat deceiving. What is notshown is the large number of OpenMP directives used todescribe the nature of the arrays passed to the SWOMPUsubroutine and arrays accessed through COMMONblocks. Each of these data structures must be declared asshared (all threads access the same data) or private (eachthread has its own private copy of the data). Data-dependency analysis of this sort for a code as large andcomplex as SWAN is not trivial.

RESULTS

We verified the correctness of the OpenMP version ofSWAN with four different test cases that exercised a num-ber of different options in the model such as time evolu-tion, external currents, and wave refraction. Due to theminimal changes in the SWAN code and the absence ofchanges to the numerical algorithm, we were able to


Article Continued From Page 13...

!$OMP PARALLEL DO SCHEDULE(STATIC, 1)DO IY = IY1, IY2, IDYDO IX = IX1, IX2, IDX

DO WHILE( LLOCK(IX,IY-IDY) )END DOCALL SWOMPU (IX, IY, ...)LLOCK(IX,IY) = .FALSE.

ENDDOENDDO

Figure 1. Schematic of geographic sweeping technique used in SWAN.

Sweep 1

0-90o

Sweep 2

90-180o

Sweep 3

180-270o

Sweep 4

270-360o

achieve bit-for-bit reproducibility of the parallel code withthe results from the original serial version of SWAN.Results have been verified on the NAVO MSRC SunE10000, IBM SP, andOrigin 2000.Unfortunately, we wereunable to execute thecode on the Cray SV1due to the fact that theCray Fortran 90 compil-er does not yet fully sup-port OpenMP.

The speedup of a paral-lel program on P proces-sors is defined as thesingle processor execu-tion time divided by theexecution time on Pprocessors. If we assumethat the amount of com-putational work at eachgeographic grid is exactlythe same and that there iszero overhead due tothread management, then we can express the idealspeedup, S, for a single sweep as

S = (XY)/(X*CEILING(Y/P) + (Y-1)%P) (1)

where X is the number of grid points in the X direction, Yis the number of grid points in the Y direction, and P isthe number of processors.

In Figure 2, we present the results from one test case on a48x48 spatial grid that includes external currents. Thedashed line is the ideal speedup for a single sweep, asexpressed in Eq. 1. The jogs in the ideal speedup corre-spond to points where P is a divisor of Y. The solid line isthe actual speedup per iteration of the OpenMP version ofSWAN as measured on the NAVO MSRC Sun E1000. Wesee that for P less than 16, the OpenMP version of SWAN

exhibits a jog pattern similar to the ideal and sustains a

speedup of at least 80% of the ideal. By 24 processors thespeedup saturates. A detailed look at Eq. 1 and the ideal

plotted in Figure 2 showsthat the speedup gainedby increasing the num-ber of processors pastY/2 is very small. Dueto the large amount ofwork per grid point, thistest case gave the bestperformance results.Less computationallydemanding test casesshowed lower perform-ance. Further analysis isrequired to determinethe sources of overheadand performance bottle-necks.

CONCLUSION

Most descriptions ofOpenMP implementations

focus on loop-level parallelization, an approach that webelieve would have not only been unduly difficult but alsounsuccessful with a code the size and complexity ofSWAN. A loop-level approach in SWAN would involveapplying OpenMP to the large number of loops that formthe various matrix solvers that are invoked for each spa-tial grid point. By taking into account the data dependen-cies in the solution procedure, we were able to introduceparallelism at a fairly high level using a small number ofOpenMP constructs with minimal changes to the legacycode. This allows other SWAN developers to easily addmore modules to enhance the physical model or improvethe numerical techniques without affecting the paralleliza-tion. The improvements gained from this project mayeven be instrumental in SWAN becoming approved as anoperational code.


References

1. Booij, N., R.C. Ris, and L. H. Holthuijsen, 1999, “A Third Generation Wave Model for Coastal Regions, 1, ModelDescription and Validation,” J. Geogr. Res. , 104, 7649-7666.

2. WAMDI Group, 1988, “The WAM Model - A Third Generation Ocean Wave Prediction Model,” J. Phys . Oceanogr. 18, 1775-1810.

Figure 2. Speedup of test case with external currents on a 48x48 spa-tial grid. The dashed line is the ideal speedup for a single sweep, andthe solid line is the measured speedup. These results were producedon the NAVO MSRC 64 processor Sun E10000.


Updates from NAVO MSRCSignal and Image Processing & Climate,Weather, and Ocean Modeling Forums Dr. Robert Melnik, CTA CoordinatorBrian Tabor, Training Coordinator

NAVO MSRC Remote Rendering InitiativePete Gruzinskas, NAVO MSRC Visualization CenterChristine E. Cuicchi, NAVO MSRCComputational Science and Applications LeadAs computing capabilities within the HPCMP grow, so doesthe size of the datasets that users wish to visualize.Specialized sci-vis servers and their capabilities are availableat selected Shared Resource Centers (SRCs) for use primar-ily by collocated users, but are not generally available to theoverwhelming majority of users (80% at the NAVO MSRC)who are remote to the SRCs where their large datasets maybe stored. A potentially viable approach for providing opti-mal sci-vis application support to remote users is a tech-nique called "remote rendering." With remote rendering, auser would remotely log in to an SRC Onyx2-class sci-visserver system, start an application on that system whichaccesses locally available/generated SRC data, and securelyexport and control the rendered/animated display stream toa local client display system such as those found on manyusers' desktops. This technique may dramatically increasethe availability and utility of centralized sci-vis capabilitiesavailable at the larger SRCs by providing application trans-parent access to these costly resources over high-speed(100 Mb/sec or better) TCP/IP-based network connectionssuch as DREN. "Application transparent" simply means thatthe remote rendering software runs independently from anyvisualization application and vice versa.

The NAVO MSRC is actively working with its PET andHigh Performance Visualization Center Initiative (HPVCI)university partners on remote rendering initiatives.Additionally, the NAVO MSRC has been engaged in activedialogue with other SRCs (TARDEC, ARL, and NRL-DC) toexplore opportunities for collaborative visualization testbedsthroughout the SRCs and to remote user communities fromthese central locations. The remote rendering softwarepackages VizServer (SGI) and Exceed 3D (Hummingbird)are being evaluated across various combinations of desktopplatforms and network connection speeds. This initial test-bed effort should be viewed as the first step toward a long-term goal to facilitate the establishment of a remote render-ing capability throughout the HPCMP.For more information on the remote rendering software,see:? SGI VizServer - http://www.sgi.com/software/

vizserver/overview.html? Hummingbird Exceed 3D - http://www.humming

bird.com/products/nc/exceed/3dfeatures.html

The NAVO and ARL MSRC PET programs have been co-sponsoring forums in Signal and Image Processing (SIP)annually since 1998. This year they co-sponsored andorganized the fourth annual SIP Forum (SIP2001). NAVOPET also organized the first forum in Climate, Weather,and Ocean Modeling (CWO2001). The overall objective ofthe forums is to bring together select researchers withdiverse expertise to share their views and experience in therole of High Performance Computing in each of theseComputational Technology Areas (CTA's). The forums alsoprovided an opportunity to review the status of CHSSIprograms in each of the CTA's.

SIP2001SIP2001, which was co-chaired by Dr. Rich Linderman,US Air Force Rome Laboratory and CTA lead for SIP, andDr. Keith Bromley, SPAWAR Systems Center, San Diego,was held May 15-16 in Biloxi, Mississippi, with NAVO PETserving as the local host. The specific objectives ofCWO2001 were to (1) bring together experts in the SIPcommunity to share their experience in the application ofHPC to SIP, (2) identify critical problems in SIP that canbenefit from advanced HPC technology, (3) identify exist-ing and emerging HPC technologies that can be applied tothe benefit of critical SIP problem areas, including recentaccomplishments under the CHSSI program, and (4)develop requirements for PET support of SIP. The SIP2001Forum brought together 56 researchers and managers fromthe DOD SIP and MSRC communities. Thirty-two paperswere presented in six sessions entitled: General overviews,Enabling Software technologies, Example Applications,CHSSI SIP Project Status, Hyperspectral ImagingExploitation, and Commercial SIP ProcessingTechnologies. Highlights of the program were an overviewof the new DOD HPC Modernization Program by Dr.Leslie Perkins of the High Performance ComputingModernization Office (HPCMO) and an Overview of SIPTechnology by Dr. Richard Linderman.

CWO2001The first forum in Climate, Weather, and Ocean Modeling(CWO2001), which is being chaired by Dr. GeorgeHeburn, NRL, Stennis Space Center and CTA Lead for



Enhancing NAVO MSRC Visualization CapabilitiesPete Gruzinskas, NAVO MSRC Visualization CenterSuccess in scientific visualization, like high-performancecomputing, is a matter of hitting a moving target. That isto say, the technology is constantly changing. To succeedin meeting the mission of the High PerformanceComputing Modernization Program (HPCMP), thesechanging technologies need to be seamlessly inserted oradopted by the various components within the program.These changesalmost alwaysrepresentimprovement,but changes arealways difficultto incorporateinto a work-place, evenwhen they ulti-mately makelife easier in thelong run.

The NAVO MSRCVisualization Centeris no exception. We actually welcome change, painful as itmay be, to our work process, if it can make the Center’smission easier and/or more cost efficient. Recently, theVisualization Center has undergone some major modifica-tions, which represent more than just change, but a shift-ing paradigm in the visualization industry and a modifiedstrategy to accommodate this new paradigm into ourworkflow. The good news about all these shifts andchanges is that they will ultimately benefit the Departmentof Defense (DoD) researchers and their efforts to analyzetheir data.

The images shown on thesepages represent changes tothe physical layout of ourVisualization Center, but thechanges made to the Centergo deeper than merely thephysical layout (Figure 1).The physical layout compli-ments the groundbreakingwork accomplished by theVisualization staff, by provid-ing a means of collaborationand demonstration in a com-fortable, well-equipped setting.

The new collaboration area, shown in Figure 2, is support-ed by a BARCO 1209 analog RGB projector. This projec-tor can be driven by an SGI Onyx2, a Windows-based sys-tem, a VCR, or other desktops in the Center. Beyond thephysical changes, there are enhancements to theVisualization Center in the areas of hardware, software, andnetworking, which will increase the availability of Centerassets to the remote user community (Figure 3).

The VisualizationCenter hardwarebase has beendiversified fromexclusively SGIIRIX to a combi-nation of SUNSOLARIS,LINUX, WIN-DOWS, and SGIIRIX. This diversi-fication provides a

level of compatibili -ty with the variousarchitectures

employed by the user community. Additionally, all Centerdesktops are now equipped with Gigabit Ethernet NICs,and the Onyx2 has been retrofitted with GigE and ATM-OC12. At this time, testing several software packages arebeing actively tested, which will facilitate remote renderingwithin the DoD security model. Remote rendering will sig-nificantly augment the visualization resources available tothe user community by placing large shared-memory parti-tions, state-of-the-art graphics architectures, software, mas-sive storage partitions, and expertise at their disposal.

So, while change can initiallybe painful, the recentchanges to the NAVOMSRC Visualization Centerwill only enhance its abilityto provide state-of-the-artservice to the user commu-nity.Questions concerningVisualization Center assetsor the use of these assetsshould be directed to PeteGruzinskas, 228-688-4027

or [email protected] or con-tact the NAVO MSRC HelpDesk at 228-688-7677.

Figure 3. Collaboration Area in use by senior Navymanagement (including the Deputy Oceanographer ofthe Navy) for a brief on support provided for EhimeMaru salvage/recovery operations.

Figure 1. Rendering of the newVisualization Center.

Figure 2. Rendering of the newCollaboration Area.


DoD User Group Conference 2001 Sights

The NAVO MSRC, in conjunction with the SharedResource Center Advisory Panel, hosted the recent2001 HPCMP Users Group Conference at theBeau Rivage Resort in Biloxi, MS. The focus ofthe conference was, as always, the users and highperformance computing (HPC) support which theprogram affords them. Each year this conferenceis hosted by one of the four MSRCs to gather thepeople involved in the program for an exchangeof ideas and information. This year's attendancewas close to 400—the largest conference in DoDHigh Performance Computing ModernizationProgram (HPCMP) history.Featured speakers of the conference this year wereDr. Robert Ballard of the Institute for Explorationand Dr. Arthur Hopkins of the Defense ThreatReduction Agency. The conference afforded theusers a chance to present their progress on variousprojects sponsored by the HPCMP. Due to extraor-dinary demand, an extra session was added to thetechnical program to accommodate additionaluser presentations. A variety of tutorials were alsoprovided that covered a wide range of topics suchas the Grid, Kerberos, MATLAB ProgrammingTechniques, Pthreads, Advanced FORTRAN90/95, Introduction to the Cray MTA Architecture,and Numerical Methods for Sparse Systems.For information regarding the UGC 2001 confer-ence in Biloxi, visit the conference web site at: http://www.hpcmo.hpc.mil/Htdocs/UGC/UGC01/index.html

NAVO MSRC PET UpdateEleanor Schroeder, NAVO MSRC ProgrammingEnvironment and Training Program (PET) Government Lead

tives such as the Information Environment group.Legion and Globus have banded forces to worktoward a strong metacomputing system that willone day be a common computational environment.

So many, many thanks to all our partners whosecontributions have made this program a success:Alcorn State, Center of Higher Learning/University of Southern Mississippi, DukeUniversity, Grambling State, Illinois Institute ofTechnology, Mississippi State University, MPITechnologies, Morgan State University, NorthCarolina A&T, Oregon State University, San DiegoSupercomputing Center, Syracuse University,Tennessee State, University of Minnesota,University of California at San Diego, andUniversity of Virginia. And a very grateful thankyou to the wonderful staff that I've had the privi-lege of interfacing with since 1997 when I firstcame on board: Dr. Timothy Campbell, Dr. JohnCazes, Dr. Howard Cohl, Dr. Jay Jayakumar, EruchKapadia, Shelley Clay, Luke Lonergan, Dr. RobertMelnik, Jack Morgan, Dr. John Pormann, Dr. GilRochon, Andrew Schatzle, Dr. Walter Shackelford,Margaret Simmons, Brian Tabor, Gail Van Nattan,Evan Willett, and Chuck Young.

So PET as we knew it wound down, and the new(and hopefully improved) version of PET is in highgear. NAVO's PET, as Component 1, has three pri-mary support responsibilities with respect to theDoD HPCMP community: Climate, Weather, andOcean Modeling (CWO); Environmental QualityModeling (EQM); and ComputationalEnvironments (CE).

We welcome to the new NAVO PET family Dr.George Heburn (Mississippi State University inte-grator component lead), Dr. Jay Boisseau(University of Texas in Austin, CWO POC), Dr.Mary Wheeler (University of Texas in Austin, EQMPOC), and Dr. Shirley Moore (University ofTennessee in Knoxville, CE POC).

The first round of projects have been approvedand are underway. We have retained some of thestaff from the previous PET program, includingAndrew Schatzle (Online KnowledgeCenter/Collaborative Distance LearningTechnologies Technologist), Brian Tabor(Education, Outreach, and Training Technologist),and Dr. Phu Luong (EQM onsite at EngineerResearch and Development Center (ERDC)). Atpress time, none of the CWO onsite positions(NAVO and ERDC) or the NAVO EQM onsite posi-tion, had been filled.

Please visit the PET web site (www.navo.hpc.mil/pet)to learn more about the new program and theexciting things that we will be doing.

AND A BIG THANKS….

Not enough can be said about the team of peoplethe NAVO MSRC PET were fortunate to haveworking with them for the past five years. Theirdedication, enthusiasm, and hard work epitomizethe successes of that program. The success of theTiger Team effort was such that it was adoptedunder the new PET program. Several of the usabil-ity tools (such as Web Queue Stats, ResourceAllocation Tool, and Resource AllocationExchange) are being adopted into corporate initia-


This article describes a simple method for running multi-node, multiprocessor programs "interactively" on theNAVO IBM SP3 (HABU) internal compute nodes underLoadLeveler, the IBM batch scheduling system. We makeuse of the X Window system and Kerberized ssh to secure-ly set up an X connection from the internal network usedby HABU's compute nodes back to a user's local computerrunning X Windows.

Parallel jobs on HABU, such as Message Passing Interface(MPI) codes running under the Parallel OperatingEnvironment (POE), have to run on compute nodes andalso have to be submitted as batch jobs via LoadLeveler.Since HABU's internal network is locally configured to onlytalk to HABU's two interactive login nodes and the NAVOArchive Servers, we cannot set our DISPLAY back to ourlocal computer (in the standard X Windows manner) andsend an xterm window or other X-based program from aLoadLeveler run back to our home computer. Instead wemust make use of Secure Shell (ssh) to handle the requiredsetup.This method allows us to send xterm windows from a run-ning LoadLeveler job back to our home computer to uselike normal login windows. We can then run Unix com-mands, debug programs with the IBM pedb program (aGraphic User Interface to IBM's pdbx parallel programdebugger) or run pdbx itself in command-line mode. Wecan then start multiple programs in background and rerunour code under POE from the xterm window started withinour LoadLeveler batch job.The following procedure outlines the required steps to usewhether you have ssh on your home system or not, andan example LoadLeveler script is also provided.

PROCEDURE


Navigator Tools and Tips

Running X Windows With Parallel Codes onHABU Under LoadLevelerJohn Skinner, NAVO MSRC Support Analyst


1. Connect to HABU interactively via one of theKerberized login commands Kerberized ssh, ktelnet,krlogin, or krsh (if possible, use ssh for its ease of usewith the X Windows setup):

mycomputer% ssh -l skinman habu.navo.hpc.milLast login: Tue Aug 29 15:14:33 2000 from 204.222.177.188f15n13e%

2. If using ssh from your local system to logon directly

to HABU, continue to Step 3. Otherwise, you need tomanually set up and test a valid X Window connec-tion and DISPLAY after logging in, as noted in stepsA - F below:

A. Run "xauth list" on your local workstation to list MIT "Magic Cookie" string that your X Server uses to authenticate X Clients that connect to your computer.

An xterm window or any other X-based pro-grams you want to run from within your LoadLeveler job will be the X Clients that need access to this encrypted "cookie" so they can display on your screen:mycomputer% xauth list mycomputer:0 MIT-MAGIC-COOKIE-1 057834173e477d52401161623779276aB. Use xauth with the "add" option to cut and

paste the appropriate line from your local X setup into your .Xauthority file on HABU. Be sure to use either your local system's IP number or full domain name when adding this to your .Xauthority file on HABU:

f18n13e% xauth add mycomputer.at.my.domain:0 MIT-MAGIC-COOKIE-1057834173e477d52401161623779276aNOTE: It should be necessary to execute steps 2.A and 2.B only once, since the information will be saved and valid as long as you don't logoff your current interactive login session on HABU.C. On HABU, set your DISPLAY environment

variable to either the IP number or the fully qualified domain name for your local worksta-tion (don't forget the "0.0" part):

f18n13e% setenv DISPLAY mycomputer.at.mydomain:0.0

orf18n13e% setenv DISPLAY 204.222.177.65:0.0D. Test the connection by starting an X client such

as xclock or xterm from your HABU login window and verifying that it displays on your workstation screen:

f18n13e% /usr/bin/X11/xtermE. From the same login window on the IBM,


We welcome our visitors...

A Look Inside NAVO MSRC

Left:

L-R - CAPT Tim McGee, Comanding Officer,NAVOCEANO; U.S. Representative GeneTaylor

Left:

L-R - Rich Peel, Program Manager, NationalUUV Test and Evaluation Center (NUTEC);Dave Cole, NAVOCEANO; Debbie Hisayasu,Director, NUTEC; Debbie Triplet, NUTEC;Craig Peterson, Director, NAVOCEANO, N9

Right:

L-R - CAPT Fred Shutt, CommandingOfficer, Coastal Systems Station; Dave

Cole; LCDR Angel Rivera, Staff METOC Officer,

Right:

L-R - Frank Lovato, Security Officer, NAVOCEANO; IGA Yves Desnoes,

Director, French Naval Hydrographic andOceanographic Service (SHOM),

and Paul Cooper, Head, NAVOCEANO, International Division

Left:

L-R - Cecil Mills, Mississippi Enterprise forTechnology; Dr. D.L. Durham, TechnicalDirector, COMNAVMETOCOM; SuzanneCase, Gulf Coast Office Director for U.S.Senator Thad Cochran; Dave Cole

Right:

Staffers to U.S. Senator Thad Cochran andU.S. Representative Chip Pickering


Left:

Space Campers at work

Right:

L-R - Dave Cole; John Palmer, U.S.Ambassador to Portugal

Left:

L-R - Julie McClean, NAVO MSRCChallenge User; Ludwig Goon, NAVO-CEANO Visualization Specialist

Right:

L-R - Dave Cole; CAPT John Kamp, DARPA;Craig Peterson, N9; Larry Raynor, PMS 395

Left:

L-R - LCDR Mykyta, N096; LCDR F. Swett,NAVOCEANO

Right:L-R - Stuart Holmes, Legislative Assistant

for Defense to U.S. Senator Thad Cochran;Clayton Heil, Legislative Director to U.S.

Senator Thad Cochran; Dave Cole; CAPT McPherson, Chief of Staff,

COMNAVMETOCOM

Left:

L-R - CAPT McPherson,Chief of Staff, COMNAVMETOCOM; CAPT ChrisGunderson, Deputy Oceanographer of theNavy

Right:Mississippi Students visit NAVOCEANO


SCRIPT

What follows is a sample LoadLeveler script that sets upthe correct DISPLAY and starts an interactive xterm ses-sion from which you can run any HABU commands, startup a pdbx session, or run pedb and send the debuggerwindow back to your computer. You can run in this man-ner until you reach the wallclock limit set for your batchjob and then llsubmit the job again using the same DIS-PLAY. Remember that if you kill the login ssh session thatfirst set up the DISPLAY, you will have to re-login with sshand modify your LoadLeveler script accordingly to pointto the new DISPLAY setting.

#@ environment = ENVIRONMENT=BATCH#@ shell = /bin/csh#@ output = $(jobid).$(stepid).out#@ error = $(jobid).$(stepid).error#@ network.MPI = css0,shared,US#@ job_type = parallel#@ job_name = my_debug_job#@ account_no = NA0101#@ node = 6#@ tasks_per_node = 4#@ node_usage = not_shared #@ wall_clock_limit = 1:00:00#@ class = batch#@ queu

setenv WORKDIR /scr/skinman/my_testif (! -e ${WORKDIR}) then mkdir -p $WORKDIRendifcd $WORKDIR

# Copy needed files to $WORKDIRcp $HOME/mpi.exe $WORKDIRcp $HOME/mpi-src/*.f90 $WORKDIRcp $HOME/input.dat $WORKDIR

# Set DISPLAY to value set by your ssh login to # HABU or, if logged in via ktelnet/krsh/krlogin to # HABU, set to the value created by the ssh from # HABU to itself.

setenv DISPLAY f18n13e.navo.hpc.mil:11.0

# Start an xterm from this script and send it to your # local workstation and use it like a normal login # session.

echo "trying xterm connection to $DISPLAY..."xterm

# You can also run the xterm in background, start up # the pedb debugger, and use both interactive win-# dows before quitting the LoadLeveler batch job.

xterm&; pedb

# You can now run your parallel code under the # graphical parallel debugger, pedb, and interactively # step through your program while it is executing on # multiple processors. You can rerun the job multiple # times and also run other commands, but you are # limited to no more than the 6 nodes this # LoadLeveler script requests when it is queued with # the llsubmit command.

pedb ./mpi.exe -procs 24 -labelio yes# End of sample LoadLeveler script

Article Continued from Page 23

now ssh locally from HABU to HABU itself:f18n13e% ssh habu.navo.hpc.milLast login: Thu Aug 31 10:24:40 2000 from 204.222.177.182f15n13e%F. List the new DISPLAY setting created by this

ssh login so that you can hardcode it into your LoadLeveler batch script.

f18n13e% env | grep DISPLAYDISPLAY=f18n13e.navo.hpc.mil:11.0

Notice that ssh has set your DISPLAY variable in thisnew login session to a new value, which happens tobe one of the two IBM login nodes with an additionalssh identifier at the end of the string. The ssh com-mand automatically takes care of setting DISPLAYand the xauth "magic cookie" setup needed for thisnew login and connects it back to your existingHABU login, where your DISPLAY is still set to"204.222.177.65:0.0" (or "mycomputer.at.my.domain:0.0"). The connection to the X11 DISPLAYis automatically forwarded by ssh in such a way thatX programs started from a HABU shell will now gothrough the encrypted channel, and the connectionwill be made from HABU to the X Server running onyour computer.Once you close the interactive ssh login session, thenew DISPLAY setting that ssh just set for you is nolonger valid and can't be used again. For this reason,do not abort this ssh connection until you arethrough with your pedb debugging work or any otherX-based programs that you want to run from withinyour LoadLeveler script on the IBM SP computenodes (which is where LoadLeveler jobs run).3. Add the DISPLAY information created from Step2 to your LoadLeveler script and llsubmit your job.Be sure to set the DISPLAY variable within yourscript before you execute any X commands such asXterm, pedb, or aixterm within the script itself. Youcan then use your current interactive login window torun llq and /maui/bin/showq to monitor your job untilit begins execution and sends the Xterm to yourworkstation.


35th Hawaii InternationalConference on System Sciences

(HICSS-35)January 7-10, 2002 ·Big Island, Hawaii

http://www.hicss.org/

(IPDPS 2002) (IPDPS =IPPS+SPDP)

April 15-19, 2002 · Fort Lauderdale, FLhttp://www.ipdps.org/ipdps2002/

Focus: MTS Oceans 2002October 28-31, 2002 · Biloxi, MS

http://www.mtsgulfcoast.org/oceans2002.html

Annual ACM Symposium onParallel Algorithms andArchitectures (SPAA)

Aug 10-13, 2002 ·Winnipeg, Manitoba,Canada

http://www.spaa-conference.org/

Current and Future Trends inNumerical PDEs:

Where is the field, and where is itgoing?

February 8-9, 2002 · University of Texas at Austin, Austin, TX

http://www.ticam.utexas.edu/%7Earbogast/jim.html

Upcoming Events

CWO, is to be held on September 18-20 inGulfport, Mississippi. The specific objectives forCWO 2001 are to provide a forum for the inter-change of ideas on (1) the modeling of the earth'sphysical environment using high-performance com-putational resources, and (2) the challenges ofdeveloping scalable, portable model codes that exe-cute efficiently on computational resources rangingfrom high-powered workstations and clusters tosupercomputers. At the time this article was written,38 invitees from the CWO community accepted invi-

tations to attend the forum. The Forum program,which can be viewed on the CWO2001 web site,includes 24 papers arranged in seven sessions enti-tled: General overviews, Atmospheric Modeling,DMEFS (Distributed Marine Environment ForecastSystem), CWO CHSSI Alpha Test Reviews, Gulf ofMexico, PET Support to CWO, and Ocean Modeling.

Further information on SIP2001 and CWO2001 isavailable at their respective web sites:

? http://www.navo.hpc.mil/pet/sip2001/

? http://www.navo.hpc.mil/conferences/cwo/

Downloadable speaker charts are available at theweb site for most of the SIP forum talks and will befor the CWO talks shortly after CWO2001.

Signal and Image Processing & Climate, Weather andOcean Modeling Forums Article Continued From Page 18...

fall 2001 navo msrc navigator journal · 4 a proteomics approach to a malaria vaccine 7 metvr:...

Documents