the resource, summer 2000universal character set, in both ucs-2 and utf-8 form, with full...

20
Summer 2000 U.S. ARMY ENGINEER RESEARCH AND DEVELOPMENT CENTER NEWSLETTER The Resource

Upload: others

Post on 19-Apr-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

Summer 2000

������������������ ������������� �� ������

The��������

Page 2: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������� ������������������������������������������������������������������������ ����� ��!������F

rom

the

Dir

ecto

r’s

chai

r

�"��!�!��� �#��$

����������%� � &��� ���'(((�����)��*��!���)"�+���+��,��������)������&�!�����**���'(((���-!-� ��&����

��������.���%)� ���!����- �-���.-!��!�������������%� � &��� ���'(((����!��!�.���� �%���� �))/�%���� !

!��� �.�)�!!���!��*� /��&�������������

���0��*���#-�.�'(((��������� ������0��*���#-�.�'(((1���)��������'23451�'(((1�.�����������%%��!� -!/

&���!���� !-���%��0��*�!���-0�)-0�!�!�����-!-��)���)��!��������� ��������� !�-"�!���!��!�����#� ��*� !��&

*-)-!��/�� ��"�!!)�&-�)��!��� �)�0-������-�!������ �!���)�.���)�&!�-��������)�����!!��1���%�!/�� ���������!��/

�&���&� ���&�����-� ���� ������ �)�0/�6��'(((�%-�!���������!��/��&��-7�������1����� ��!-��)��/�!�*�

� !���6�� 88�6�����!��/1�%�0��98�

The recent High Performance ComputingModernization Program’s (HPCMP’s) ProgramReview 2000 (PR2000) was held during the lastweek of March. It was a milestone event for theHPCMP. We started this program in the earlynineties but really did not formalize it with adefinitive set of participants until 1996. At thatpoint, we began to establish our business models,modes of intercommunication, working relation-ships, and friendships. PR2000 was the first timewe stopped and looked back at what we hadaccomplished. The original goals of the programwere in general to deploy high-end computers,foster the maturity of software systems, expandand train the Department of Defense (DoD) highperformance computing (HPC) user base, contrib-ute to the national HPC infrastructure, and hook itall together with a very high-speed network. TheProgram Review not only demonstrated that wehave done a great job at achieving those goals butalso that we have assembled a tremendous team ofexperts in HPC.

I look at PR2000 as a stake in the ground. It isa time when we look back and say, “Did we makeit?” Sure there are areas for improvement, butoverall I think the answer is unequivocally yes.The comments and ratings from the review boardconfirm this. It is also a time when we need tolook forward and ask ourselves, “Where do we gofrom here?” A few goals might include applica-tions that harness a larger pool of computationalresources through distributed computing tech-

nologies, the development of applications thatimplement a coupled approach to multidisciplinaryproblems, and the increased use of interactivecomputing. Through the continued coordinatedefforts of the Common High Performance Com-puting Software Support Initiative, ProgrammingEnvironment and Training (PET), Shared Re-source Centers, and Defense Research and Engi-neering Network, this program is poised to takeon the next wave of computational challenges inthe DoD.

Bradley M. ComesDirector, ERDC MSRC

Page 3: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

:��!����

���������������������������� ���� �

��%��!*� !�������������� ������������������������������������������������������������������������������������������������������������������������ ��

��������� ������������������������������������������������������������������������������������������������������������������������������������� �

���� ����� �������������������������������������������������������������������������������������������������������������������������������� �

������� ��������������������������������������������������������������������������������������������������������������������������������������� !

"����#��$�$��%������������ ������������������������� �������������������������������������������������� �&

�����'��(��� ��������������������������������������������������������������������������������������������������������������������������� ��!

�������) ������������������������������������������������������������������������������������������������������������������������������������ �!

*��)������#��$�� ���������������������������������������������������� +

�����#��'�*�����������,�"����-������

����%�� ��'��& ����������������������������������������������������������� �

Page 4: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

� �� ���� ������������������������

anno

unce

men

ts������!� �))�!-��!�%���%�����/�!�*��� �����#-�����-���!��

Dr. Art Cullati has recently joined the U.S. Army Engineer Research and Develop-ment Center (ERDC) Major Shared Resource Center (MSRC) team as the Systemsand Services Director. In this key position, he will oversee User Services, the HPCsystems, and the network support teams.

Dr. Cullati has more than 30 years of experience in the computing field and 20 yearsof experience managing HPC centers. He joins us from his position as the Director of theNational Environmental Supercomputing Center of the Environmental Protection Agency.

Dr. Cullati’s experience includes developing software for the Minuteman Missile and managingthe supercomputing installations at the Numerical Aerodynamic Simulation, National Aeronautics andSpace Administration/Ames Research Center. He began his career in systems and applications workincluding designing, developing, implementing, and supporting his own original special-purpose operat-ing system and writing a language translator that compiled structured English and generated object codein another vendor’s computer. He was also the project leader of the benchmark effort for Cray Research,Inc., in its first commercial sale to Boeing Computer Services.

We look forward to Dr. Cullati’s knowledge and experience contributing significantly to initiatives suchas a common user interface, metacomputing, shared file systems, and remote disaster recovery.

tool

tips

�.� �**� ��

���������

Recently, the ERDC MSRC added several useful commands for manipulating files on the Mass StorageFacility (MSF). These commands are available on all HPC hosts, as are manpages for each command. Ashort synopsis of each command follows.

�������� change file permissions on the MSF������ copy a file from the old MSF to the new MSF����� copy a file from the MSF��� � list the contents of a directory on the MSF������� copy multiple files from the old MSF to the new MSF������ copy multiple files from the MSF�������� create directories on the MSF������� copy multiple files to the MSF����� rename a file on the MSF������ copy a file to the MSF����� delete a file on the MSF�������� delete directories on the MSF������� check the status and availability of the MSF

Page 5: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������������������� ���� &

tool tips�����-!/��%��!�

�� �������

The ERDC MSRC has been working with severaldevelopers to provide better Windows tools toaccess the MSRC systems. Our first success inthis endeavor is the integration of KRB5 withSecurID support into Kermit-95. Many mayremember Kermit from the days of serial/dial-upaccess into the system. It is still around andgreatly enhanced.

Listed below are the major features of Kermit-95and contact information to obtain Kermit-95.Users will still need the Win32 KRB5 kit fromNaval Research Laboratories to use the newKRB5 functions in Kermit. (It is available free ofcharge; please see www.hpcmo.hpc.mil/Htdocs/SECURITY/kerberos_q.html on how to get thesoftware.) We would like to thank Jeffrey Altman,Columbia University, for all of his work to get theKRB5 with SecurID functioning.

Kermit-95 is the Windows 95/98/NT/2000 andOS/2 edition of Columbia University’s well-known Kermit communications software. Ithandles both modem and TCP/IP connectionswith equal ease and offers 38 different terminalemulations, Kermit and XYZMODEM file trans-fer protocols, international character-set conver-sion (now including Unicode), numeric andalphanumeric paging, host mode for incomingconnections (serial or network), and a scriptinglanguage to allow automation of any communica-tions task that can be done by hand. The newKermit-95+ package also includes C-Kermit 7.0,its file-transfer and client/server partner for UNIX,VMS, VOS, and other operating systems on CD-ROM, including source code and prebuilt binariesfor hundreds of platforms. Also included are on-line manuals for Kermit-95 and C-Kermit, fullsource code for C-Kermit, and dozens of samplescripts.

Compliance: Year 2000, Euro, Win32, Winsock,TAPI (Telephony Application ProgrammingInterface).

Communication methods: Direct serial, modem,TCP/IP Telnet and Rlogin, SuperLAT, DECnet.(SuperLAT and DECnet require the correspondingexternal network protocol stacks.)

Security: Kerberos IV, Kerberos V, SSL/TLS,SRP (subject to USA export restrictions).

Terminal emulations: VT320/220/102/100/52;VT320PC/VT220PC; Linux Console; ANSI,SCOANSI, AT386, Avatar; Wyse 30/50/60/160/

370; Televideo 910/925/950; DG200/210/217;IBM HFT/AIXTERM/3151; HP 2621/HPTERM;QNX; Microsoft NTVT; Siemens Nixdorf BA80and 97801; Heath-19; Hazeltine 1500; VolkerCraig 404.

Special terminal features: Colors, selectablescreen dimensions, international character-settranslation, complete built-in default keymaps foreach terminal type, flexible key and mouse-eventredefinition, keystroke macros, URL hot spots, upto 2,000,000 lines of scrollback with bookmarkand search capability, and special keyboardmodes: PCTERM, EMACS, Word Perfect, Rus-sian, Hebrew.

File transfer protocols: Kermit, XYZMODEM,ASCII. Kermit protocol includes client/servermode and international character-set translation.

Scripting: All operations can be programmed forautomatic unattended execution in a consistentway, regardless of platform or connection method,using the portable Kermit scripting language,which includes file and communications input/output, block structure, looping, variables, arrays,functions, and structured programming features.

Languages: Kermit 95’s user interface is inEnglish, but in its communication functions itsupports translation of all the character sets usedfor Western European languages (English, Italian,Portuguese, Norwegian, Icelandic, German,Spanish, French, etc.); Eastern European Lan-guages (Czech, Polish, Hungarian, Romanian,etc.); Greek; Hebrew and Yiddish; languageswritten with the Cyrillic alphabet (Russian,Ukrainian, Belorussian, Bulgarian, etc.); lan-guages written with the Hebrew alphabet (Hebrew,Yiddish, Ladino, etc.); and (in file transfer only)Japanese. New to version 1.1.20: Unicode, theUniversal Character Set, in both UCS-2 andUTF-8 form, with full translation to/from all theother character sets.

Cost: Single copies are $64.00 U.S. dollars (USD),although quantity discounts are available. Low-cost bulk-right-to-copy licenses for 100 seats ormore are available for prices beginning at $10USD per seat. Academic site licenses are $2000USD.

More information�������������� ����� ��������

Page 6: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

� �� ���� ������������������������

off-

cam

pus

��-� !-&-���-���)-;�!-� <��-)���%�/1���&!.���1��-�����-!- 0

������ ����������

������������� ������������

On February 8, 2000, three members of the ERDCMSRC Scientific Visualization Laboratory pre-sented a half-day seminar to the faculty andstudents at Jackson State University (JSU), anacademic partner to our PET Program. Theseminar was part of a 2-day workshop on scien-tific visualization.

Dr. Kent Eschenberg, Acting Lead for the Scien-tific Visualization Laboratory, discussed thephilosophy of the way visualization can supportscience and provided an introduction to visualiza-tion programming. Mr. Glen Browning reviewedsoftware packages that can essentially eliminateprogramming and offered details on two packagesregularly used at the ERDC MSRC ScientificVisualization Laboratory. Mr. David Longmirediscussed basic principles and procedures forvideo editing. A list of resources in books, jour-nals, and Web sites was also provided.

��������������������������������������� ��������� �����������!�����������������"������#$�%�& ������������'��� ����!��������������������������������(� �����������#$�%�& ���������������������������((���������

!��������!������������������������(� �������� �&�(������)����!�����������������

���*�+���*������������!����������������� ��������!�(�,�-����*�������,�-�������������������(�����������!�#$��"# ��������$�����������(���.��'�������������������!�������*��� �

#$����������!���������������������� �*������������������/��������������������

Page 7: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������������������� ���� .

off-campus

� ���������� � ������� ���-� !-&-���-���)-;�!-� �:� ��*� !�)�

�����������

In a later visit in early spring, Mr. John E. West,ERDC MSRC, gave a seminar on scientificvisualization at JSU. The seminar was designedto introduce the fundamental concepts of datavisualization to students with no prior backgroundin visualization or computer graphics. Dr. JulieBaca, JSU Computer Science Department, hosted

���,����������,�-���( ������������ ��(�����������������!����������������������(����

,�������0�������������������������������!�����������������(�������,�-�

the seminar, which was attended by graduate andundergraduate students in the Department ofComputer Science. Dr. Baca had two goals for theseminar. First, she wanted it to serve as a supple-ment to enrich the experience her students receivein regular classes by introducing a new technol-ogy area in which computer science plays acritical role. Second, some of the students attend-ing the seminar are beginning scientific visualiza-tion research on externally funded projects, andDr. Baca saw the seminar as a way to jumpstartthose students’ preparation for the efforts to come.

In all, the seminar was a big success, with follow-on sessions being planned to cover more advancedtopics. Interactions like these are a welcomereminder that training the next generation ofscientists and engineers for the HPC community isone of the most important ways in which we canhave a lasting impact on the long-term future ofour communities and our Nation.

Page 8: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

! �� ���� ������������������������

visi

tors

��%���&� 0- ���������� �)��!!� ������ �*%�!����%%)-��!-� �� )���

During the week of January 24-28, 2000,17 students from various Corps of Engineers’District offices attended the ComputerApplications for Engineers and EngineeringManagers (CAEEM) course held at theERDC Information Technology Laboratory(ITL). The course, presented annually by theITL Computer Aided Engineering Division,includes the use of computer-aided engineer-ing design techniques and applicationsavailable within the Corps of Engineers witha “hands-on” emphasis.

��������- �� ��%����%���� !�!-#��

U.S. Marine Corps representatives from installations in North Carolina, Arizona, California, Virginia, andHawaii in the United States and from two installations in Japan toured the ERDC MSRC HPC facility inFebruary 2000. These representatives were at the ERDC ITL to attend the Marine Corps GeospatialInformation Systems Users Group meeting at the CADD/GIS Technology Center for Facilities, Infrastruc-ture, and Environment. Thisgroup convenes as requiredto exchange information,promote and sustain the useof GIS for integratedinstallation management,and assists in the develop-ment of policy for dissemi-nation by Installations andLogistics, Headquarters,U.S. Marine Corps.

���1���������/��� �������������!���������������������2������������#���������

��������������������*�����������������������3+��+�����������#���������

-������������ ��� ��������������������������������3+��!�������

ERDC MSRC teammembers Mr. DavidStinson, Dr. Henry Gabb,Mr. Mike McCraney,Mr. Mike Nassour, andDr. Kent Eschenbergpresented an overview ofsupercomputing at theERDC MSRC. Topics ofdiscussion included HPCapplications, mass stor-age, networks, andscientific visualization.

Page 9: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������������������� ���� /

visitors �� �-)��&� �)� �)�

The U.S. Army Training and Doctrine Command (TRADOC) Modeling and Simulation Advisory Councilattended the Council of Colonels meeting held at the ERDC Headquarters, Vicksburg, MS, on March 15-16. This group meets quarterly to report on taskings and perform other business of the Advisory Council.Members visit varioussites to learn moreabout Army organiza-tions and their areasof expertise withinthe DoD.

Dr. Louis Turcotte,Assistant for Technol-ogy in the ERDC ITL,conducted a tour ofthe ERDC MSRCHPC facility and theScientific Visualiza-tion Center for the54 meeting attendeesrepresenting manyagencies and commands.

�-��-��-%%-��!�!��� -#���-!/�����-�� !�� ���!�&&���*"���

The president of MississippiState University (MSU),Dr. Malcolm Portera, alongwith some of the university’snewest faculty membersand a few senior adminis-trators, visited the ERDCMSRC on March 30.Mr. Tim Ables, ActingDirector, ITL, served asthe tour guide, assisted byMr. Dennis Gilman, ERDCMSRC.

The university is conduct-ing a number of bus toursaround Mississippi for newfaculty to show them

firsthand the impact thatteaching, research, and service programs have in the State of Mississippi. This effort also helps acquaintfaculty with the strengths and aspirations of Mississippi communities.

MSU is one of the ERDC MSRC’s PET partners and the host university for the National Science Founda-tion (NSF) Engineering Research Center for Computational Field Simulation.

����������������������!������( ����!���������!��������#�4����������������(��������#��������������

��(�#/������������������ �!�(�����( ��������������+�����

Page 10: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

+ �� ���� ������������������������

The HPCMP had its first program-wide review theweek of March 27-31, 2000. The review coveredall aspects of the HPCMP in a series of tracks,including the network and software initiatives aswell as the shared resource centers. The reviewwas held in Fairfax, Virginia, and included anintroduction by Dr. Delores Etter, Deputy UnderSecretary of Defense for Science and Technology.Meeting attendees included service agencyofficials, members of the HPC Advisory Panel,and external reviewers as well as members fromall facets of the HPCMP.

The four MSRCs had a significant role in theweek’s activities. The centerpiece of the MSRCs’participation in the review was the MSRC Post-Implementation Review, during which eachCenter Director highlighted the goals, characteris-tics, and processes that have contributed to theenormous success of this program. Each CenterDirector spent some time highlighting the way inwhich the shared resource centers cooperateacross mission and service agency boundaries to

������������ �����������������������������������������������

deliver the latest in mission-critical computingtechnologies and service to the entire DoDresearch, development, test and evaluationcommunity.

In addition to presenting MSRC and PET brief-ings during the Post-Implementation Review, theERDC MSRC participated in a wide range oftracks during the review. Its staff members werepart of sessions of the Mass Storage and ArchivalSystems Working Group and Networking Techni-cal Advisory Panel sessions, the Service/AgencyApproval Authority briefing and dialog sessions,the Requirements and Resource AllocationReview, and the Challenge Lessons Learnedmeeting.

Program Review 2000 was a rare opportunity forthe entire program to highlight the critical role theDoD HPCMP has contributed to the advancementof military and battlefield technologies. It wasalso an opportunity for the program to look backas well as into the future.

������(���������������������(�*������ �������������+���(��������5666�

Page 11: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������������������� ���� =

6�-�!���������!��/��&��-7�������1��� ���� 8

Page 12: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

� �� ���� ������������������������

� !�����!-�

Current-generation, massively parallel computingplatforms such as the 512-processor Cray T3E atthe ERDC MSRC allow opportunities for scien-tific inquiry that were impossible just a few yearsago. Complex nonlinear problems that offer onlyweakly nonlinear solutions through traditionalanalytic methods are being directly attacked dailythrough high performance computational technol-ogy at the Nation’s Supercomputing Centers. Theopportunities for scientists are growing, but so tooare logistical difficulties. One of these difficultiesis the subject of this article: massively paralleldisk input/output (I/O), i.e., performance tuningassociated with parallel data transfers to themassively parallel processing (MPP) disks, whichare then migrated off-line to a long-term datastorage facility. In particular, we describe a newtechnique called “Hierarchical Data Structuring”(HDS).1 HDS can speed up massively parallel I/O-limited calculations. We present an example inwhich the speedup can range from 1.4 to 2.1 ormore, depending on the number of data filesinvolved.

As a vehicle for demonstrating the proper ap-proach to massively parallel data handling on theERDC MSRC Cray T3E, we describe high-resolution turbulence simulations in support of theAir Force AirBorne Laser (ABL) Program. Thework is part of an effort of the Space VehiclesDirectorate of the Air Force Research Laboratory.The scientists conducting the simulations areDrs. Joe Werne and Dave Fritts of ColoradoResearch Associates (CoRA).2 Their computa-tions at the ERDC MSRC involve the highestresolution pseudospectral calculations of atmo-spheric turbulence processes (such as wind shearand gravity-wave breaking) currently beingconducted. (Please see the Spring 2000 issue ofthe ERDC MSRC’s The Resource newsletter orthe Spring 1999 CEWES MSRC Journal atwww.wes.hpc.mil (under Publications) for furtherinformation on the ABL project.) We expect thatthe performance enhancement exhibited by HDSwill be universally applicable to other architectures.

������!������-%!-�

The ABL turbulence solutions require massivecomputational resources and long-term disk storage.Tens of terabytes (TB) of numerical data will begenerated by the project by the end of this year.Runs currently being conducted employ 500 processingelements (PEs) and generate 410 GB of data duringone 20-hr job. Because of the large sizes of the datasets and limited high-speed storage space on theCray T3E, ABL data must be shuttled off-line inreal-time to the ERDC MSF. The MSF consists ofan SGI Origin 2000 front end to a Cray J16/4512,which migrates data to three StorageTek roboticsilos. Data are transferred between the T3E and theMSF across a High-Performance Parallel Interface(HiPPI) network, capable of 290-megabits-per-second (Mbps) sustained data transfers with bursts ashigh as 800 Mbps. Actual transfer rates vary accord-ing to the channel load, as the HiPPI channel isshared; however, the bandwidth is sufficient for thisproblem such that throughput is typically limitedby read/write speeds on the Cray T3E and MSF.To facilitate the data movement, the ERDC MSRChas developed a suite of utilities to facilitate thestorage and retrieval of large files on the MSF.The utilities are based on the SGI File TransferAgent software package and provide fetch(������������), store (��������������),status (�������), and file manipulation(������������ ������������) capabilities.

For optimal performance of the applicationprogram, I/O between Cray T3E memory and the

���������������������������������������������� ������������!���"�����

��������������� �������������������������������������������������������� !��"����#$%&��������������'��"�(����������������)�*�� !��� �(�����+!����+�� ����*�(�

tech

nica

l for

um

Cray T3E

Page 13: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������������������� ���� ��

T3E data disks must be considered carefully. Forthe ABL simulations, a poor I/O strategy can ruinan otherwise scalable algorithm. Two approachesto massively parallel disk I/O were considered:

1. Have each processor perform I/O to its own set of files.

2. Stripe one large datafile across multiple data disks, and instruct participating PEs to write to their portion of the global file.

Physically, the two solutions are the same, withseparate processors writing data to separate disks.The main difference involves specifics of theimplementations and the manner in which filenames are handled. Though the user conceives asingle file when employing (2), in reality, portionsof the global file reside on all of the disks span-ning the set of processors used. In order to stripefiles across multiple disks, the �� and ���optionsof the Cray assign command must be used. The�� option indicates which partitions to use in thefilesystem. The��� option indicates how muchdata will be placed on a single disk before movingon to the next disk. Users must be careful topreserve the intended file layout and use the Crayfdcp command when copying files from anothermachine. It is clear that parallel I/O via filestriping immediately involves machine-specificcommands, and portability can become limited.

Solution (1) presents a simpler alternative. Assur-ance that a processor’s output files reside on itsdisk is accomplished trivially by instructing eachprocessor to create its own files. Though restartdata from a previous run will not be locallyresident if file prefetching is done by anotherprocessor, presumably restart data are read onlyonce, and poor I/O performance for the initial readis tolerable. To ensure that subsequent writes tothe restart files are not hindered by the files’ initialprefetched locations, the user should ensure that a

new set of restart files is created for each run andthat each processor creates its own restart file.

Dr. Werne implemented solution (1) with direct-access, fixed-record-length binary files for theABL turbulence simulations. Output files areopened just before writing and then closed againafter writing is complete. This approach mini-mizes the number of simultaneously open filesand avoids problems with unflushed file bufferswhen a run is unexpectedly stopped.

The Cray T3E code scales linearly with thenumber of processors when I/O is turned off;however, the code’s linear scaling is adverselyimpacted when run in production mode as a resultof the massive amounts of I/O demanded by theproject, even when solution (1) is properlyimplemented. We used two improvements torestore the code’s scaling with the number ofprocessors during production runs: (a) write-behind data buffering3 and (b) HDS. The firstimprovement involves buffering output-datawrites so that computation and I/O can be over-lapped for as long as the array being written is notneeded for subsequent computation. On the T3Ethis is accomplished with an option to the Crayassign command: -F bufa:num1:num2, wherenum2 is the number of buffers used, and num1 isthe size in 4,096-byte blocks of each buffer layer.Dr. Werne uses ��������������. See on-lineman “intro_ffio” for more information on this andother assign options. The second performanceimprovement, HDS, is a technique we recentlydiscovered. It involves dividing and subdividingthe output data directory into multiple subdirectoriesand sub-subdirectories, ending with tails thatcontain no more than NCPU files (NCPU is thenumber of processors used for a given calcula-tion). For example, for three-dimensional volumefiles written at time levels 00, 01, etc., instead of adirectory named “data” with the following layout:

���� ��� �� ���� ������!�

������ �� ���������� ��� ���������� ������� ��������!!

�� ���������� ��� ���������� ������� ��������!! ����

�� ���������� ��� ���������� ������� ��������!!

�� ���������� ��� ���������� ������� ��������!! ����

$��������������������,�- � !��� �+!��.-��)�� �������"��������������� ������� � !��*������+����/���!�+��))�+���*0����("�������1�����+�)�+�� ��� �� � ����/��2��- ��)����-�,���+�����������+��2� !���� !�������"!�����/��-�������������+��) �2!��+���-+ ��/��,�������,����3�+�����+���� ������/���� !��*0��#$%����� !����2�%(�"���("��4����2��$����

technical forum

Page 14: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

�� �� ���� ������������������������

HDS uses the following layout:

The finer resolution via a directory tree ensuresthat the application program does not spend timewading through an overwhelming number of filenames. For example, when using NCPU = 500,ABL simulations generate in excess of 72,000individual files during the course of a 20-hr run.When real-time data migration to the MSF isaccomplished according to plan, all 72,000 filesdo not reside on the T3E /tmp disks at the sametime. They are collected into tar files and movedoff-line immediately after they are closed. Never-theless, a minimum of 9,000 files simultaneouslyreside on the T3E /tmp space during the run, andwith the old data layout, this is sufficient toimpact performance. For instance, when runningwith the old directory structure, ABL turbulencesimulations required 82 min to complete 50 time-steps and do the associated disk I/O. When datafrom previous runs also occupied the directory, asmuch as 2 hr or more could be required. In fact,when performing tests to diagnose the source ofthe variable execution time, we intentionallyallowed the data directory to fill with files thatwere not transferred when previous tests failed.Eventually, the directory became so full with100,000 files that 50 time-steps were not com-pleted in as many as 6 hr. Often an error“_sma_deadlock_wait” was encountered on theT3E. In contrast, now that we have implementedthe new HDS strategy, execution time no longervaries. The 50-time-step tests finish in 59 min +1 min and 20-hr dedicated runs demonstrate thesame predictable performance regardless of thenumber of files in the directory tree.

� �)��-�

We expect that the performance enhancementexhibited by HDS will be universally applicable toother architectures. Given its simplicity, webelieve it could be implemented for all MPPapplications that generate large amounts of dataand large numbers of individual files. An addi-

���������� ���� �� ���������� ��� ���������� ������� ��������!!

���������� ���� �� ���������� ��� ���������� ������� ��������!!

���������� ���� �� ���������� ��� ���������� ������� ��������!!

���������� ���� �� ���������� ��� ���������� ������� ��������!! ����

tech

nica

l for

um

tional benefit of HDS is that the user can easilynavigate the directory tree and inspect data as theyare generated by the application program. This isnot feasible with a non-HDS directory containing10,000 or more files.

Dr. Joseph Werne is a research scientistat the CoRA division of NWRA. Hehas 11 years of experience in highperformance computing on shared-memory vector and distributed-memoryparallel platforms. He has written highlyoptimized Fortran programs, including the ABLturbulence code, for both stably and unstablystratified turbulence simulations as part of NSFGrand Challenge and DoD Challenge applicationssince 1993. Dr. Werne receives support for ABLsimulations from AFRL F19628-98-C-0030and AFOSR F49620-98-C-0029. Relatedsupercomputer work is sponsored by ARODAAD191-99-C-0037.

Mr. Paul Adams recently moved from Applications Support Analyst to User Services Manager at the ERDCMSRC. He has 12 years of experience

in the computer industry, providingprogramming and user support on Cray, SGI, andIBM platforms.

Mr. David Sanders is an ApplicationsSupport Analyst at the ERDC MSRCand a Systems Support Engineer withCray, Inc. He has 16 years of program-ming and user-support experience onshared-memory, distributed-memory parallel,and distributed-shared-memory platforms. Hewrote the suite of MSF data-handling utilitiesdescribed above.

Page 15: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

���������������������������� ���� �&

interview w

ith users

������ ������ �7���/���������!�������8+����������(������

Dr. Fred Tracy, ERDC ITL, and Mr. Dave Richards,ERDC Coastal and Hydraulics Laboratory, spokewith us about using the ERDC MSRC Cray T3Efor the remarkably difficult computational chal-lenge of modeling the Everglades National Park(ENP) and the Dade County South Florida areausing the finite element program FEMWATER-123.The ENP model involves both two-dimensional(2-D) overland and 3-D groundwater flow. Themore sophisticated Dade model also includes 1-Dcanal flow. The goal is to do a 1-year simulationusing 1- to 2-hr 3-D time-steps on a parallel highperformance computing platform in a 24-hr timeperiod. Since the canal, overland, and ground-water flow equations are all separately nonlinear,the required coupled solution seemed impossibleon any computer. However, after much effort, theneeded solution ran in 16.5 hr using 40 PEs on theCray T3E. This application is being highlightedbecause of its computational complexity andbecause FEMWATER-123 is applicable to mili-tary applications where it is currently beingsuccessfully used for environmental remediationof military sites.

What are you modeling? The JacksonvilleDistrict of the U.S. Army Corps of Engineers andthe South Florida Water Management Districthave responsibilities for managing water resourcesand, as a result, the natural environment ofsouthern Florida. These includeoperating the extensive flood protec-tion works and canals that signifi-cantly affect the surface and ground-water hydrologic processes of theregion. While these flood protectionworks have served their purpose overthe years, there has been an effortrecently to revisit their design withthe hope of returning the environmentback to a more natural state. This is adifficult task since this may entailraising water levels, thereby affectinglocal flood risk. In addition, there arecompeting interests for the use of thewater. Everglades National Park

Interview with users...Dr. Fred Tracy and Mr. Dave Richards

desires redistribution of water for ecologicalreasons; agricultural interests desire certain waterlevels to develop their businesses; and municipali-ties have increasing populations that want a cleanwater supply, recreational facilities, and goodflood protection at the same time. Therefore, theflow of water in the ground, on the surface, and inthe canals is being modeled using FEMWATER-123 to aid management in making these difficultdecisions.

What are the computational challenges?Groundwater flow is characterized by manydifferent heterogeneous materials as illustrated bythe 3-D finite element mesh of the Dade model.Also, unsaturated 3-D groundwater flow by itselfis highly nonlinear and often converges veryslowly if at all. Add to that the challenges ofcreating an efficient scalable parallelizedFEMWATER code, and you are already in a toughspot. It was first thought that FEMWATER wouldnot converge at all for large problems, and if itwere possible to do a parallel version, it would notscale very well. However, a message-passageinterface version of the 3-D code was developed,along with improvements to the nonlinear solver.A parallel speedup of 1.8 when going from 64 to128 PEs and a scaled speedup of 90 percent whengoing from 8 to 64 PEs on the Cray T3E wereachieved, which is excellent.

Page 16: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

�� �� ���� ������������������������

inte

rvie

w w

ith u

sers

Added to the 3-D challenge are the 2-D overlandflow computations on the 2-D mesh on the top ofthe 3-D mesh and the 1-D canal flow computa-tions in the canal network. These are bothnonlinear and also interact with the 3-D computa-tion to form a highly nonlinear coupled system.At first, the 1-D solution used an explicit formu-lation, and it became unstable for time-stepsgreater than about 0.025 min. This resulted inthe 1-D computations alone needing over 6months of compute time using one T3E PE toaccomplish the 1-year simulation. This wasclearly unworkable. However, switching to animplicit formulation brought the computationtime down to an acceptable level. The 2-Dsolution also took too long. However, by adjust-ing solver parameters, the 2-D solution thenconverged much faster.

������ ���������(�������������������

Developing a parallel version of the FEMWATER-123 code presented other difficulties, such as howto partition the three separate meshes andparallelize the three separate program piecesefficiently. METIS was used to partition the 3-Dmesh; the 2-D mesh was assigned a PE accordingto how the 3-D mesh was partitioned; and loopparallelization was used in the 1-D piece of code.Although more efficiency can be obtained, thiswas acceptable for the first version.

Finally, the interaction between the 2-D and 3-Dsolutions would not converge at first, as thesolution oscillated and became very unstable intime. A new paradigm for boundary conditionswas then developed, and the solution became veryfast and stable. This allowed the successfulcompletion of the 1-year simulation using 40 PEs

Page 17: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

�������������������������� ���� �.

interview w

ith users

in 16.5 hr on the Cray T3E, which waswell under the needed time of 24 hr.

How large have your problems grownover the past 5 years? The complexityhas grown by a factor of 10, and the sizehas grown by a factor of 2 or 3.

Do you anticipate them growing? Yes.The parallel version of FEMWATER (3-Donly) and FEMWATER-123 will allow usto do even larger problems in the futurewith finer detail provided by largermeshes. The goal of doing a million-node3-D mesh is now attainable.

What is the impact on the DoD? Theparallel version of FEMWATER iscurrently being used to model the envi-ronmental remediation of military siteswith specific application to pump-and-treat. Helping the DoD more efficientlyclean up these military sites will result insaving millions of dollars for other use bythe warfighter. Parallel FEMWATER-123represents a significant advance in thestate of the art and will be used in thefuture to model important DoD projects.9:�������������*�!�������!���������������!�����������(�����

;:��!���������(����(����!�����������(����������(��������

Page 18: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

trai

ning

�! �� ���� ������������������������

���- - 0�������)�>

June 27-28Programming Tools

July 12-13Workshop on Handheld Supercomputing(to be held at Florida State University)

August 22-24IBM Power3 SMP Applications Programming

*Additional courses may be offered. Please check the ERDC MSRC Web page atwww.wes.hpc.mil

acro

nym

sBelow is a list of acronyms commonly used among the DoD HPC community. You will find these acro-nyms throughout the articles in this newsletter.

ABL AirBorne LaserASC Aeronautical Systems CenterAVS Application Visual Systems, Inc.CAEEM Computer Applications for Engineers and Engineering ManagersCoRA Colorado Research AssociatesDoD Department of DefenseENP Everglades National ParkERDC Engineer Research and Development CenterHDS Hierarchical Data StructuringHiPPI High-Performance Parallel InterfaceHPC High Performance ComputingHPCMP High Performance Computing Modernization ProgramI/O Input/OutputITL Information Technology LaboratoryJSU Jackson State UniversityMPP Massively Parallel ProcessingMSF Mass Storage FacilityMSRC Major Shared Resource CenterMSU Mississippi State UniversityNAVO Naval Oceanographic OfficeNSF National Science FoundationNWRA NorthWest Research AssociatesPE Processing ElementPET Programming Environment and TrainingPR2000 Program Review 2000TRADOC Training and Doctrine Command

������ ���� �.�)��*�����**� !��� ����00��!-� ����0���- 0����������� �

- #-!�����!-�)����"*-��-� ����)������� ����"*-��-� ��!��!���&�))�.- 0��3*�-)��������$

�� ���������������

����������������������������������������

����������������������������������

����������������������������������

�������������������������������

�����������������������������������

�����������������������

Page 19: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

�� ���� ��.�)�!!��

�-!���)��!�&&

Interdisciplinary CoordinatorMary L. Hampton

Chief EditorsRose DykesMary Gabb

Visual Information SpecialistBetty Watson

�� ���� � ��!�*������-�!� ��� � !����,��� �5�������� �����

%3 ���5������ ������ �����

#����!���5���3677387739:''

��������������������������������������������������������������� ������������������� ������ ����� ����

������������������������������ ����������� �� ��!���������������������������������������������

���������������������������� ��"������#��������$�����������������%�� �����&��������

'(������)��������*������ ����� ����������$�����

�������������������������+�������������������������

Page 20: The Resource, Summer 2000Universal Character Set, in both UCS-2 and UTF-8 form, with full translation to/from all the other character sets. Cost: Single copies are $64.00 U.S. dollars

�������������� ���������������������������������������������������������������������������� ����� ��!������

0�������'��)�������������%���1���#��������� �����

���23��

���"4

�5������',��6

&77�4�����8���'����1

��,�9��)���

���&7�+�!�77