adios 1.0 user's manual
TRANSCRIPT
ORNL/TM-2009/100
ADIOS 1.0 User’s Manual
November 2009
This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.
DOCUMENT AVAILABILITY Reports produced after January 1, 1996, are generally available free via the U.S. Department of Energy (DOE) Information Bridge:
Web site: http://www.osti.gov/bridge Reports produced before January 1, 1996, may be purchased by members of the public from the following source:
National Technical Information Service 5285 Port Royal Road Springfield, VA 22161 Telephone: 703-605-6000 (1-800-553-6847) TDD: 703-487-4639 Fax: 703-605-6900 E-mail: [email protected] Web site: http://www.ntis.gov/support/ordernowabout.htm
Reports are available to DOE employees, DOE contractors, Energy Technology Data Exchange (ETDE) representatives, and International Nuclear Information System (INIS) representatives from the following source:
Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831 Telephone: 865-576-8401 Fax: 865-576-5728 E-mail: [email protected] Web site: http://www.osti.gov/contact.html
ORNL/TM-2009/100
ADIOS1.0USER’SMANUAL
Prepared for the Office of Science
U.S. Department of Energy
S. Hodson, S. Klasky, Q. Liu, J. Lofstead, N. Podhorszki, F. Zheng, M. Wolf
November 2009
Prepared by
OAK RIDGE NATIONAL LABORATORY
Oak Ridge, Tennessee 37831-6070 managed by
UT-BATTELLE, LLC for the
U.S. DEPARTMENT OF ENERGY under contract DE-AC05-00OR22725
iii
ContentsAcknowledgments...........................................................................................................viii1 Introduction .................................................................................................................. 11.1 Goals .........................................................................................................................................................11.2 WhatIsADIOS? ....................................................................................................................................11.3 TheBasicADIOSGroupConcept ..................................................................................................11.4 OtherInterestingFeaturesofADIOS..........................................................................................11.5 FutureADIOS2.0Goals ....................................................................................................................2
2 Installation .................................................................................................................... 32.1 ObtainingADIOS..................................................................................................................................32.2 QuickInstallation ................................................................................................................................32.2.1 Linuxcluster.................................................................................................................................32.2.2 CrayXT4.........................................................................................................................................32.2.3 SupportforMatlab ....................................................................................................................3
2.3 ADIOSDependencies .........................................................................................................................42.3.1 Mini‐XMLparser(required)..................................................................................................42.3.2 MPIandMPI‐IO(required)....................................................................................................42.3.3 Fortran90compiler(optional).............................................................................................42.3.4 SerialNetCDF‐3(optional) ....................................................................................................42.3.5 SerialHDF5andNetCDF(optional) ...................................................................................42.3.6 PHDF5(optional) .......................................................................................................................52.3.7 Read‐onlyinstallation..............................................................................................................5
2.4 FullInstallation ....................................................................................................................................53 ADIOSWriteAPI........................................................................................................... 63.1 WriteAPIDescription .......................................................................................................................63.1.1 Introduction .................................................................................................................................63.1.2 ADIOS‐requiredfunctions ......................................................................................................63.1.3 Nonblockingfunctions.......................................................................................................... 103.1.4 Otherfunction .......................................................................................................................... 103.1.5 CreateafirstADIOSprogram ............................................................................................ 11
4 XMLConfigFileFormat ...........................................................................................124.1 Overview.............................................................................................................................................. 124.2 adios‐group......................................................................................................................................... 134.2.1 Declaration................................................................................................................................. 134.2.2 Variables ..................................................................................................................................... 134.2.3 Attributes.................................................................................................................................... 144.2.4 Gwrite/src .................................................................................................................................. 154.2.5 Globalarrays ............................................................................................................................. 154.2.6 Time‐index................................................................................................................................. 164.2.7 Declaration................................................................................................................................. 164.2.8 Methodslist ............................................................................................................................... 17
4.3 Bufferspecification ......................................................................................................................... 174.3.1 Declaration................................................................................................................................. 17
4.4 AnExampleXMLfile....................................................................................................................... 185 Transportmethods...................................................................................................195.1 Synchronousmethods ................................................................................................................... 19
iv
5.1.1 NULL ............................................................................................................................................. 195.1.2 POSIX ............................................................................................................................................ 195.1.3 MPI‐IO .......................................................................................................................................... 195.1.4 MPI‐CIO ....................................................................................................................................... 215.1.5 PHDF5 .......................................................................................................................................... 215.1.6 PNetCDF ...................................................................................................................................... 225.1.7 Othermethods.......................................................................................................................... 22
5.2 Asynchronousmethods................................................................................................................. 225.2.1 MPI‐AIO ....................................................................................................................................... 225.2.2 DataTap ....................................................................................................................................... 235.2.3 DecoupledandAsynchronousRemoteTransfers(DART).................................... 24
6 ADIOSReadAPI..........................................................................................................256.1 Introduction ....................................................................................................................................... 256.2 ReadCAPIdescription .................................................................................................................. 266.2.1 adios_errmsg/adios_errno................................................................................................ 266.2.2 adios_fopen................................................................................................................................ 266.2.3 adios_fclose................................................................................................................................ 276.2.4 adios_gopen/adios_gopen_byid...................................................................................... 276.2.5 adios_gclose............................................................................................................................... 276.2.6 adios_inq_var/adios_inq_var_byid................................................................................. 286.2.7 adios_free_varinfo................................................................................................................... 286.2.8 adios_read_var/adios_read_var_byid ........................................................................... 296.2.9 adios_get_attr/adios_get_attr_byid ............................................................................... 296.2.10 adios_type_to_string............................................................................................................ 306.2.11 adios_type_size ...................................................................................................................... 30
6.3 ReadFortranAPIdescription..................................................................................................... 306.4 Compilingandlinkingapplications.......................................................................................... 32
7 BPfileformat ..............................................................................................................337.1 Introduction ....................................................................................................................................... 337.2 Footer .................................................................................................................................................... 347.2.1 Version......................................................................................................................................... 347.2.2 Offsetsofindices ..................................................................................................................... 347.2.3 Indices.......................................................................................................................................... 34
7.3 ProcessGroups.................................................................................................................................. 367.3.1 PGheader ................................................................................................................................... 377.3.2 Varslist........................................................................................................................................ 387.3.3 Attributeslist ............................................................................................................................ 38
8 Utilities..........................................................................................................................408.1 adios_lint.............................................................................................................................................. 408.2 bpls ......................................................................................................................................................... 408.3 bpdump ................................................................................................................................................ 41
9 Converters ...................................................................................................................449.1 bp2h5 .................................................................................................................................................... 449.2 bp2ncd .................................................................................................................................................. 449.3 bp2ascii ................................................................................................................................................ 449.4 ParallelConverterTools ............................................................................................................... 45
10Groupread/writeprocess......................................................................................46
v
10.1 Gwrite/gread/read....................................................................................................................... 4610.2 Addconditionalexpression ...................................................................................................... 47 DependencyinMakefile ...................................................................................................................... 4710.3 .................................................................................................................................................................... 47
11 CProgrammingwithADIOS ...................................................................................4811.1 Non‐ADIOSProgram .................................................................................................................... 4811.2 ConstructanXMLFile ................................................................................................................. 4911.3 Generate.chfile(s)....................................................................................................................... 4911.4 POSIXtransportmethod(Pwriters,Pfiles)...................................................................... 5011.5 MPI‐IOtransportmethod(Pwriters,1file) ...................................................................... 5111.6 Readingdatafromthesamenumberofprocessors ...................................................... 5211.7 WritingtoSharedFiles(Pwriters,Nfiles) ........................................................................ 5311.8 GlobalArrays................................................................................................................................... 5411.9 WritingTime‐IndexintoaVariable....................................................................................... 56
12DeveloperManual .....................................................................................................5912.1 CreateNewTransportMethods.............................................................................................. 5912.1.1 Addthenewmethodmacrosinadios_transport_hooks.h ................................. 5912.1.2 Createadios_abc.c ................................................................................................................ 6012.1.3 Awalk‐throughexample................................................................................................... 61
12.2 ProfilingtheApplicationandADIOS..................................................................................... 6712.2.1 UseprofilingAPIinsourcecode.................................................................................... 6712.2.2 Usewrapperlibrary ............................................................................................................ 70
13Appendix ......................................................................................................................72
vi
FiguresFigure1.ADIOSprogrammingexample. ................................................................................. 11Figure2.ExampleXMLconfiguration ...................................................................................... 13Figure3.ExampleXMLfilefortimeallocation..................................................................... 18Figure4.Server‐friendlymetadataapproach:offsetthecreate/openintime ...... 20Figure5.DataTaparchitecture.................................................................................................... 23Figure6.BPfilestructure .............................................................................................................. 34Figure7.Groupindextable........................................................................................................... 35Figure8.Variablesindextable .................................................................................................... 36Figure9.Processgroupstructure.............................................................................................. 37Figure10.Attributeentrystructure ......................................................................................... 39Figure11.bplsutility....................................................................................................................... 41Figure12.bpdumputility .............................................................................................................. 43Figure13.Originalprogram(examples/C/manual/1_nonadios_example.c). ........ 49Figure14.Exampleconfig.xmlfile ............................................................................................. 49Figure15.Examplegwrite_temperature.chfile................................................................... 50Figure16.ExampleadiosprogramtowritePfilesfromPprocessors(examples/C/manual/2_adios_write.c) .................................................................................. 51Figure17.Readindatageneratedby2_adios_writeusinggread_temperature.ch(examples/C/manual/3_adios_read.c) .................................................................................... 53Figure18.Exampleofageneratedgread_temperature.chfile...................................... 53Figure19.ExampleADIOSprogramwritingNfilesfromPprocessors(N)............ 54Figure20.Config.xmlforaglobalarray(examples/C/global‐array/adios_global.xml) ................................................................................................................. 55Figure21.gwriteheaderfilegeneratedfromconfig.xml ................................................ 55Figure22.Config.xmlforaglobalarraywithtime(examples/C/global‐array‐time/adios_globaltime.xml).......................................................................................................... 57
vii
AbbreviationsADIOS AdaptiveInput/OutputSystem
API applicationprograminterface,asetofroutines,protocols,andtoolsforbuildingsoftwareapplications.AgoodAPImakesiteasiertodevelopaprogrambyprovidingallthebuildingblocks.Aprogrammerthenputstheblockstogether.
Mostoperatingenvironments,suchasMS‐Windows,provideanAPIsothatprogrammerscanwriteapplicationsconsistentwiththeoperatingenvironment.AlthoughAPIsaredesignedforprogrammers,theyareultimatelygoodforusersbecausetheyguaranteethatallprogramsusingacommonAPIwillhavesimilarinterfaces.Thismakesiteasierforuserstolearnnewprograms.
DART DecoupledandAsynchronousRemoteTransfers
GTC GyrokineticTurbulenceCode
HPC high‐performancecomputing
I/O input/output
MDS metadataserver
MPI Message‐PassingInterface
NCCS NationalCenterforComputationalSciences
ORNL OakRidgeNationalLaboratory
OS operatingsystem
PG processgroup
POSIX PortableOperatingSystemInterface
RDMA remotedirectmemoryaccess
XML ExtensibleMarkupLanguage
viii
Acknowledgments TheAdaptiveInput/Output(I/O)system(ADIOS)isajointproductoftheNationalCenterofComputationalSciences(NCCS)atOakRidgeNationalLaboratory(ORNL)andtheCenterforExperimentalResearchinComputerSystemsattheGeorgiaInstituteofTechnology.ThisworkisbeingledbyScottKlasky(ORNL);JayLofstead(GeorgiaTech,fundedfromSandiaLabs)isthemaincontributor.ADIOShasgreatlybenefitedfromtheeffortsofthefollowingORNLstaff:SteveHodson,whogavetremendousinputandguidance;ChenJin,whointegratedADIOSroutinesintomultiplescientificapplications;NorbertPodhorszki,whointegratedADIOSwiththeKeplerworkflowsystemandworkedwithQingGaryLiuonthereadAPI.ADIOSalsobenefitedfromtheeffortsoftheGeorgiaTechteam,includingProf.KarstenSchwan,Prof.MattWolf,HassanAbbasi,andFangZheng.WeiKengLiao,NorthwesternUniversity,andWangDi,SUN,havealsobeeninvaluableinourcodingeffortsofADIOS,writingseveralimportantcodeparts.Essentially,ADIOSiscomponentizationofI/Otransportmethods.Amongthesuiteoftransportmethods,DecoupledandAsynchronousRemoteTransfers(DART)wasdevelopedbyProf.ManishParasharandhisstudentCiprianDocanofRutgersUniversity.
Without a scientific application, ADIOS would not have come this far. SpecialthanksgotoStephaneEthieratthePrincetonPlasmaPhysicsLaboratory(GTS);Researcher Yong Xiao and Prof. Zhihong Lin from the University of California,Irvine (GTC); Julian Cummings at the California Institute of Technology; andSeung‐HoeandProf.C.S.ChangatNewYorkUniversity(XGC).
ThisprojectissponsoredbyORNL,GeorgiaTech,TheScientificDataManagementCenter(SDM)atLawrenceBerkeleyNationalLaboratory,and
theU.S.DepartmentofDefense.
ADIOScontributorsANL:RobRoss
GeorgiaTech:HasanAbbasi,JayLofstead,KarstenSchwan,FangZheng,MatthewWolfNCSU:XiaosongMa
NorthwesternUniversity:AlokChoudhary,WeiKengLiao,ChenJinORNL:SteveHodson,ScottKlasky,QingGaryLiu,NorbertPodhorszki,StevePoole
RutgersUniversity:CiprianDocan,ManishParasharSUN:WangDi
1
1 Introduction
1.1 GoalsAs computational power has increased dramatically with the increase in thenumber of processors, input/output (IO) performance has become one of themost significant bottlenecks in today’s high‐performance computing (HPC)applications.With this inmind,ORNLand theGeorgia InstituteofTechnology’sCenterforExperimentalResearchinComputerSystemshaveteamedtogethertodesign theAdaptive I/OSystem(ADIOS)asa componentizationof the IO layer,which is scalable,portable,andefficientondifferentclustersorsupercomputerplatforms. We are also providing easy‐to‐use, high‐level application programinterfaces(APIs)sothatapplicationscientistscaneasilyadapttheADIOSlibraryandproducesciencewithoutdivingtoodeeplyintocomputerconfigurationandskills.
1.2 WhatIsADIOS?ADIOS is a state‐of‐the‐art componentization of the IO system that hasdemonstrated impressive IO performance results on leadership classmachinesandclusters;sometimesshowinganimprovementofmorethan1000timesoverwellknownparallelfileformats.ADIOSisessentiallyanI/Ocomponentizationofdifferent I/O transport methods. This feature allows flexibility for applicationscientists to adopt the best I/O method for different computer infrastructureswithvery littlemodificationof theirscientificapplications.ADIOShasasuiteofsimple,easy‐to‐useAPIs.InsteadofbeingprovidedastheargumentsofAPIs,allthe required metadata are stored in an external Extensible Markup Language(XML) configuration file, which is readable, editable, and portable for mostmachines.
1.3 TheBasicADIOSGroupConceptTheADIOS“group”isaconceptinwhichinputvariablesaretaggedaccordingtothefunctionalityoftheirrespectiveoutputfiles.Forexample,acommonscientificapplication has checkpoint files prefixed with restart and monitoring filesprefixedwithdiagnostics. In theXMLconfiguration file, theusercandefinetwoseparategroupswithtagnamesofadios‐groupas“restart”and“diagnostic.”Eachgroupcontainsasetofvariablesandattributesthatneedtobewrittenintotheir respective output files. Each group can choose to have different I/Otransportmethods,whichcanbeoptimalfortheirI/Opatterns.
1.4 OtherInterestingFeaturesofADIOSADIOS contains a new self‐describing file format, BP. The BP file format wasspecifically designed to support delayed consistency, lightweight datacharacterization, and resilience. ADIOS also contains python scripts that allowuserstoeasilywriteentire“groups”withtheinclusionofoneincludestatementinsidetheirFortran/Ccode.AnotherinterestingfeatureofADIOSisthatitallows
2
userstousemultiple I/Omethods forasinglegroup.This isespeciallyuseful ifusers want to write data out to the file system, simultaneously capturing themetadatainadatabasemethod,andvisualizingwithavisualizationmethod.
The read API enables reading arbitrary subarrays of variables in a BP file andthusvariableswrittenoutfromNprocessorcanbereadinonarbitrarynumberofprocessors.ADIOSalsotakescareoftheendiannessproblematconvertingtothereader’sarchitectureautomaticallyatreadingtime.MatlabreaderisincludedinthereleasewhiletheVisItparallel interactivevisualizationsoftwarecanreadBPfilestoo(fromversion1.13).
ADIOS is fullysupportedonCrayXTandIBMBlueGene/PcomputersaswellasonLinuxclusters.
1.5 FutureADIOS2.0GoalsOne of the main goals for ADIOS 2.0 is to produce faster reads via indexingmethods.AnothergoalistoprovidemoreadvanceddatatypesviaXMLinADIOSsothatitwillbecompatiblewithF90/c/C++structures/objects.
WewillalsoworkonthefollowingadvancedtopicsforADIOS2.0:
• Alinktoanexternaldatabaseforprovenancerecording.• Autonomics through a feedback mechanism from the file system to
optimizeI/Operformance.Forinstance,ADIOScanbeadaptivelychangedfromasynchronoustoanasynchronousmethodordecidingwhentowriterestarttoimproveI/Operformance.
• Astagingareafordataquerying,analysis,andinsituvisualization.
3
2 Installation
2.1 ObtainingADIOSYoucandownloadthelatestversionfromthefollowingwebsite
http://www.nccs.gov/user-support/adios
2.2 QuickInstallationTo get startedwith ADIOS, the following steps can be used to configure, build,test,andinstalltheADIOSlibrary,headerfiles,andsupportprograms.
cd trunk/
./configure –prefix=<install-dir> --with-mxml=<mxml-location>
make
make install
Note: There is a runconf batch script in the trunk set up for our machines.Studying it can help you setting up the appropriate environment variables andconfigureoptionsforyoursystem.
2.2.1 LinuxclusterThe following is a snapshot of the batch scripts on Ewok, an Intel‐basedInfinibandclusterrunningLinux:
export CC=mpicc ./configure --prefix = <location for ADIOS software installation> --with-mxml=<location of mini-xml installation> --with-hdf5=<location of HDF5 installation> --with-netcdf=<location of netCDF installation>
2.2.2 CrayXT4ToinstallADIOSonaCrayXT4,therightcompilercommandsandconfigureflagsneed tobe set.The requiredcommands forADIOS installationon Jaguarareasfollows:
export CC=cc export FC=ftn ./configure --prefix = <location for ADIOS software installation> --with-mxml=<location of mini-xml installation> --with-hdf5=<location of HDF5 installation> --with-netcdf=<location of netCDF installation>
2.2.3 SupportforMatlabMatlab requires ADIOS be built with the GNU C compiler. It also requiresrelocatablecodes,soyouneedtoaddthe–fPICflagtoCFLAGSbeforeconfiguringADIOS.Thematlabreaderisnotbuiltautomaticallyatmakeandisnotinstalled
4
withADIOS.Youneed tocompile itwithMatlab’sMEXcompilerafter themakeandcopythefilesmanuallytosomewherewhereMatlabcanseethem.
cd tools/matlab make matlab
2.3 ADIOSDependencies
2.3.1 Mini‐XMLparser(required)TheMini‐XMLlibrary isusedtoparseXMLconfigurationfiles.Mini‐XMLcanbedownloadedfrom
http://www.minixml.org/software.php
2.3.2 MPIandMPI‐IO(required)MPIandMPI‐IOisrequiredfortheADIOS1.0release.
Currently, most large‐scale scientific applications rely on the Message PassingInterface (MPI) library to implement communication among processes. Forinstance, when the Portable Operating System Interface (POSIX) is used astransportmethod,therankofeachprocessorinthesamecommunicationgroup,which needs to be retrieved by the certain MPI APIs, is commonly used indefining the output files. MPI‐IO can also be considered the most generic I/Olibraryonlarge‐scaleplatforms.
2.3.3 Fortran90compiler(optional)TheFortran90interfaceandexamplecodesarecompiledonlyifthereisanf90compileravailable.Bydefaultitisrequiredbutyoucandisableitwiththeoption--disable-fortran.
2.3.4 SerialNetCDF‐3(optional)Thebp2ncdconverterutilitytoNetCDFformatisbuiltonlyifNetCDFisavailable.Currently ADIOS uses the NetCDF‐3 library. Use the option--with-netcdf=<path>orensure that theNETCDF_DIRenvironmentvariableissetbeforeconfiguringADIOS.
2.3.5 SerialHDF5andNetCDF(optional)The bp2h5 converter utility to HDF5 format is built only if a HDF5 library isavailable.CurrentlyADIOSusesthe1.6versionoftheHDF5APIbutitcanbebuiltand used with the 1.8.x version of the HDF5 library too. Use the option--with-hdf5=<path>orensurethattheHDF5_DIRenvironmentvariableissetbeforeconfiguringADIOS.
We will continue to work on the parallelized converters, which will requirePHDF5andPnetCDF.
5
2.3.6 PHDF5(optional)Thetransportmethodwriting files in theParallelHDF5format isbuiltonly ifaparallelversionoftheHDF5libraryis(also)available.Youneedtousetheoption--with-phdf5=<path>tobuildthistransportmethod.
Notethatifyoubuildthistransportmethod,ADIOSwilldependonPHDF5whenyoulinkanyapplicationwithADIOSevenifyoutheapplicationdoesnotintendtousethismethod.
2.3.7 Read‐onlyinstallationIf you just want the read API to be compiled (for example, on Mac OSX) forreadingBPfiles,usethe--disable-write option.
2.4 FullInstallationThefollowinglististhecompletesetofoptionsthatcanbeusedwithconfiguretobuildADIOSanditssupportutilities:
--help print the usage of ./configure command --with-tags[=TAGS] include additional configurations [automatic] --with-mxml=DIR Location of Mini-XML library --with-gengetopt=<path> Location of gengetopt --with-hdf5=<location of HDF5 installation> --with-hdf5-incdir=<location of HDF5 includes> --with-hdf5-libdir=<location of HDF5 library> --with-phdf5=<location of PHDF5 installation> --with-phdf5-incdir=<location of PHDF5 includes> --with-phdf5-libdir=<location of PHDF5 library> --with-netcdf=<location of NetCDF installation> --with-netcdf-incdir=<location of NetCDF includes> --with-netcdf-libdir=<location of NetCDF library>
Someinfluentialenvironmentvariablesarelistsbelow:
CC C compiler command CFLAGS C compiler flags LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir> CPPFLAGS C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir> CPP C preprocessor CXX C++ compiler command CXXFLAGS C++ compiler flags FC Fortran compiler command FCFLAGS Fortran compiler flags CXXCPP C++ preprocessor F77 Fortran 77 compiler command FFLAGS Fortran 77 compiler flags MPICC MPI C compiler command MPIFC MPI Fortran compiler command
6
3 ADIOSWriteAPIAs mentioned earlier, ADIOS writing is comprised of two parts: the XMLconfigurationfileandAPIs.Inthissection,wewillexplainthefunctionalityofthewritingAPIindetailandhowtheyareappliedintheprogram.
3.1 WriteAPIDescription
3.1.1 IntroductionADIOSprovides both Fortran andC routines. All ADIOS routines and constantsbegin with the prefix “adios_” . For the remainder of this section, only the CversionsofADIOSAPIsarepresented.TheprimarydifferencesbetweentheCandFortran routines is that error codes are returned in a separate argument forFortranasopposedtothereturnvalueforCroutines.
AuniquefeatureofADIOSisgroupimplementation,whichisconstitutedbyalistof variables and associated with individual transport methods. This flexibilityallows the applications tomake the best use of the file systemaccording to itsowndifferentI/Opatterns.
3.1.2 ADIOS‐requiredfunctionsThissectioncontainsthebasicfunctionsneededtointegrateADIOSintoscientificapplications.ADIOSisalightweightI/Olibrary,andthereareonlysevenrequiredfunctions fromwhich users canwrite scalable, portable programswith flexibleI/Oimplementationonsupportedplatforms:
adios_init—initializeADIOSandloadtheconfigurationfileadios_open—openthegroupassociatedwiththefileadios_group_size—passthegroupsizetoallocatethememoryadios_write—writethedataeithertointernalbufferordiskadios_read—associatethebufferspacefordatareadintoadios_close—commitwrite/readoperationandclosethedataadios_finalize—terminateADIOSYoucanaddfunctionstoyourworkingknowledgeincrementallywithouthavingtolearneverythingatonce.Forexample,youcanachievebetterI/Operformanceon some platforms by simply adding the asynchronous functionsadios_start_calculation, adios_end_calculation, and adios_end_iteration to yourrepertoire. These functions will be detailed below in addition to the sevenindispensablefunctions.The following provides the detailed descriptions of required APIs when usersapplyADIOSintheFortranorCapplications.
3.1.2.1 adios_initThisAPIisrequiredonlyonceintheprogram.ItloadsXMLconfigurationfileandestablishes the execution environment. Before any ADIOS operation starts,
7
adios_init is required to be called to create internal representations ofvariousdatatypesandtodefinethetransportmethodsusedforwriting.
intadios_init(constchar*xml_fname)Input:
xml_fname–stringcontainingthenameoftheXMLconfigurationfile
Fortranexample:calladios_init("config.xml",ierr)
3.1.2.2 adios_openThis API is called whenever a new output file is opened. adios_open,corresponding to fopen (not surprisingly), opens an adios‐group given bygroup_nameandassociatesitwithoneoralistoftransportmethods,whichcanbeidentifiedinfutureoperationsbytheFilestructurewhosepointerisreturnedasfd_p.ThegroupnameshouldmatchtheonedefinedintheXMLfile.TheI/Ohandlefd_ppreparesthedatatypesforthesubsequentcallstowritedatausingthe io_handle. The third argument, file_name, is a string representing thenameof the file.As the last argument,mode is a string containing a file accessmode.Itcanbeanyofthesethreemodespecifiers:“r,” “w,” or“a.” Currently,ADIOSsupportsthreeaccessmodes:“writeorcreateiffiledoesnotexist,”“read,”and“appendfile.”Thecallopensthefileonlyifnocoordinationisneededamongprocessesfortransportmethodsthattheusershavechosenforthisadios_group,such as POSIX method. Otherwise, the actual file will be opened inadios_group_sizebasedontheprovidedargumentcomm,whichwillbeexaminedin Sect. 4.1.2.3. As the last argument, we pass the pointer of coordinationcommunicatordowntothetransportmethodlayerinADIOS.ThiscommunicatorisrequiredinMPI‐IO–basedmethodssuchascollectiveandindependentMPI‐IO.
intadios_open(int64_t*fd_p,constchar*group_name ,constchar*file_name,constchar*mode,void *comm)Input:fd_p—pointertotheinternalfilestructuregroup_name—stringcontainingthenameofthegroupfile_name—stringcontainingthenameofthefiletobeopenedmode—stringcontainingafileaccessmodecomm—communicatorformulti‐processcoordination
Fortranexample:calladios_open(handle,“restart”,“restart.bp”,"w",comm,ierr)
3.1.2.3 adios_group_sizeThis function passes the size of the group to the internal ADIOS transportstructuretofacilitatetheinternalbuffermanagementandtoconstructthegroupindextable.Thefirstargumentisthefilehandle.Thesecondargumentisthesizeofthepayloadforthegroupopenedintheadios_openroutine.Thisvaluecanbecalculated manually or through our python script. It does not affect read
8
operationbecause the size of thedata canbe retrieved from the file itself. Thethird argument is the returned value for the total size of this group, includingpayloadsizeandthemetadataoverhead.Thevaluecanbeusedforperformancebenchmarks,suchasI/Ospeed.
int adios_group_size (int64_t * fd_p, uint64_t group_size, uint64_t *total_size)Input:
fd_p—pointertotheinternalfilestructuregroup_size—sizeofdatapayloadinbytestobewrittenout.Ifthereisaninteger2×3array,thepayloadsizeis4*2*3(4isthesizeofinteger)output:total_size—the total sum of payload and overhead, which includesname,datatype,dimensionsandothermetadata)
Fortranexample:calladios_group_size(handle,groupsize,totalsize,ierr)
3.1.2.4 adios_writeTheadios_writeroutinesubmitsadataelementvar forwritingandassociatesitwiththegivenvar_name,whichhasbeendefinedintheadiosgroupopenedbyadios_open. If theADIOSbuffer isbigenoughtoholdall thedata that theadiosgroup needs to write, this API only copies the data to buffer. Otherwise,adios_writewillwritetodiskwithoutbuffering.Currently,adios_writesupportsonlytheaddressofthecontiguousblockofmemorytobewritten.Inthecaseofanoncontiguousarraycomprisingaseriesof subcontiguousmemoryblocks,var shouldbegivenseparatelyforeachpiece.
InthenextXMLsection,wewill furtherexplainthatvar_name isthevalueofattribute “name”whilevar is thevalueofattribute “gwrite,”bothofwhicharedefined in the corresponding<var>element insideadios_group in theXML file.Bydefault, itwillbethesameasthevalueofattribute“name” if “gwrite” isnotdefined.
intadios_write(int64_tfd_p,constchar*var_name,void*var)Input:
fd_p—pointertotheinternalfilestructurevar_name—stringcontainingtheannotationnameofscalarorvectorin
theXMLfilevar—theaddressofthedataelementdefinedneedtobewritten
Fortranexample: calladios_write(handle,"myvar",v,ierr)
9
3.1.2.5 adios_readThewriteAPI containsa read function (historically, the firstone) thatuses thesametransportmethodandthexmlconfig file toreadindata. Itworksonlyonthe same number of processes as the data was written out. Typically,checkpoint/restartfilesarewrittenandreadonthesamenumberofprocessorsandthisfunctionisthesimplestwaytoreadindata.However,ifyouneedtoreadinonadifferentnumberofprocessors,oryoudonotwanttocarrythexmlconfigfilewiththereadingapplication,youshouldusethenewerandmoregenericreadAPIdiscussedinSection6.
Similartoadios_write,adios_readsubmitsabufferspacevar forreadingadataelementinto.ThisdoesNOTactuallyperformtheread.Actualpopulationofthebufferspacewillhappenonthecalltoadios_close.Inotherwords,thevalue(s)ofvar can only be utilized after adios_close is performed. Here, var_name corresponds to the value of attribute “gread“ in the <var> element declarationwhilevar ismapped to the value of attribute “name.” By default, itwill be assameasthevalueofattribute“name”if“gread”isnotdefined.
intadios_read(int64_tfd_p,constchar*var_name,uint64_tread_size,void* var
)Input:fd_p–pointertotheinternalfilestructurevar_name–thenameofvariablerecordedinthefilevar–theaddressofvariabledefinedinsourcecoderead_size–sizeinbytesofthedatatobereadinFortranexample: calladios_read(handle,“myvar”,8,v,ierr)
3.1.2.6 adios_closeThe adios_close routine commits thewriting buffer to disk, closes the file, andreleases the handle. At that point, all of the data that have been copied duringadios_writewillbesentas‐isdownstream.Ifthehandlewereopenedforread,itwould fetch thedata,parse it, andpopulate it into theprovidedbuffers.This iscurrentlyhard‐codedtouseposixI/Ocalls.
intadios_close(int64_t*fd_p);Input:fd_p–pointertotheinternalfilestructureFortranexample: calladios_close(handle,ierr)
10
3.1.2.7 adios_finalizeThe adios_finalize routine releases all the resources allocatedbyADIOSandguaranteesthatallremainingADIOSoperationsarefinishedbeforethecodeexits. The ADIOS execution environment is terminated once the routine isfulfilled.Theproc_id parameterprovidesuserstheopportunitytocustomizespecialoperationonproc_id—usuallytheIDoftheheadnode.
intadios_finalize(intproc_id)
Input:proc_id – the rank of the processe in the communicator or the user‐definedcoordinationvariable
Fortranexample: calladios_finalize(rank,ierr)
3.1.3 Nonblockingfunctions
3.1.3.1 adios_end_iterationTheadios_end_iterationprovidesthepacingindicator.BasedontheentryintheXML file, it will tell the transport method how much time has elapsed in atransfer.
3.1.3.2 adios_start_calculation/adios_end_calculationTogether, adios_start_calculation and adios_end_calculation indicate to thescientific code when nonblocking methods should focus on engaging their I/Ocommunicationeffortsbecausetheprocessismainlyperformingintense,stand‐alone computation. Otherwise, the code is deemed likely to be communicatingheavilyforcomputationcoordination.Anyattemptstowriteorreadduringthosetimes will negatively impact both the asynchronous I/O performance and theinterprocessmessaging.
3.1.4 OtherfunctionOneofourdesigngoalsistokeepADIOSAPIsassimpleaspossible.InadditiontothebasicI/Ofunctions,weprovideanotherroutinelistedbelow.
3.1.4.1 adios_get_write_bufferThe adios_get_write_buffer function returns thebuffer space allocated from theADIOS buffer domain. In other words, instead of allocating memory from freememory space, users candirectlyuse the allocatedADIOSbuffer area and thusavoidcopyingmemoryfromtheADIOSbuffertoauser‐definedbuffer.
intadios_get_write_buffer(int64_tfd_p,constchar*var_name,uint64_t*size,void**buffer)Input:fd_p–pointertotheinternalFilestructurevar_name–nameofthevariablethatwillberead
11
size–sizeofthebuffertorequestoutput:buffer–initialaddressofread‐inbufferforstoringthedataofvar_name
3.1.5 CreateafirstADIOSprogramFigure 1 is a programming example that illustrates how to write a double‐precision arrayt and a double‐precision array with size ofNX into file called“test.bp,” which is organized in BP, our native tagged binary file format. Thisformatallowsuserstoincluderichmetadataassociatedwiththeblockofbinarydataaswelltheindexingmechanismfordifferentblocksofdata(seeChap.5).
/*example of parallel MPI write into a single file */ #include <stdio.h> // ADIOS header file required #include ”adios.h” int main (int argc, char *argv[]) { int i, rank, NX; double t [NX]; // ADIOS variables declaration int64_t handle; uint_64 total_size; MPI_Comm comm = MPI_COMM_WORLD; MPI_Init ( &argc, &argv); MPI_Comm_rank (comm, &rank); // data initialization for ( i=0; i<NX; i++) t [i] = i * (rank+1) + 0.1; // ADIOS routines adios_init ( “config.xml”); adios_open (&handle, “temperature”, “data.bp”, “w”,&comm); adios_group_size (handle, 4, total_size); adios_write (handle, ”NX”, &NX); adios_write (handle, ”temperature”, t); adios_close (handle); adios_finalize (rank);
MPI_Finalize(); return 0; }
Figure1.ADIOSprogrammingexample.
12
4 XMLConfigFileFormat
4.1 OverviewXML is designed to allow users to store as much metadata as they can in anexternal configuration file.Thus thescientificapplicationsare lesspollutedandrequirelessefforttobeverifiedagain.
First,wepresenttheXMLtemplate.Second,wedemonstratehowtoconstructtheXML file from the user’s own source code. Third,we note how to troubleshootanddebugtheerrorsinthefile.
Abstractingmetadata, data type, and dimensions from the source code into anXML file gives users more flexibility to annotate the arrays or variables andcentralizesthedescriptionofallthedatastructures,whichinreturn,allowsI/Ocomponentization for different implementation of transport methods. Bycataloguing the data types externally, we have an additional documentationsourceaswellasawaytoeasilyvalidatethewritecallscomparedwiththereadcallswithout having to decipher the data reorganization or selection code thatmaybeinterspersedwiththewritecalls.ItisusefulthattheXMLnameattributesarejuststrings.Theonlyrestrictionsfortheircontentarethatiftheitemistobeused in a dataset dimension, itmust not contain commas andmust contain atleastonenon‐numericcharacter.This isuseful for incorporatingexpressionsasvarious arraydimensions elements. Figure2 illustrates the correspondingXMLconfigurationfortheexamplewedemonstratedinFigure1.
At a minimum, a configuration document must declare an adios-configelement.Itservesasacontainerforotherelements;assuch,itMUSTbeusedasthe rootelement.Theexpectedchildren inanyorderwouldbeadios-group,method,andbuffer.Themainelementsofthexmlfileformatareoftheformat
<element-name attr1 attr2 …>
<adios‐config>
<adios‐group><var/>……<attribute/>……
</adios‐group>…<method>……<buffer>
</adios‐config>
13
Figure2.ExampleXMLconfiguration
4.2 adios‐groupTheadios‐groupelementrepresentsacontainerforalistofvariablesthatsharethecommonI/OpatternasstatedinthebasicconceptsofADIOSinfirstchapter.In this case, the group domain division logically corresponds to the differentfunctions of output in scientific applications, such as restart, diagnosis, andsnapshot.Dependingonthedifferentapplications,adios‐groupcanoccurasmanytimesasisneeded.
4.2.1 DeclarationThe followingexample illustrateshow todeclareanadiosgroup insideanXMLfile.Firstwestartwithadios‐groupasourtagname,whichiscaseinsensitive.Ithasanindispensableattributecalled“name,”whosevalueisusuallydefinedasadescriptivestring indicating the functionof thegroup. In thiscase, thestring iscalled “restart,” because the files into which this group is written are used ascheckpoints. The second attribute “host‐language” indicates the language inwhich this group’s I/O operations are written. The value of attribute“coordination‐communicator” is used to coordinate the operations on a sharedfile accessed by multiple processes in the same communicator domain.“Coordination‐var” provides the ability to use the user‐defined variable, forexamplemype,ratherthananMPIcommunicatorforfilecoordination.
<adiosgroupname=“restart”host‐language=“C”coordination‐communicator=”comm”coordination‐var=”mype”time‐index=”iter”/>
Required:
•name—containingadescriptivestringtonamethegroup
Optional:
•host‐language—languageinwhichthesourcecodeforgroupiswritten
•coordination‐communicator—MPI‐IOwritingtoasharedfile
•coordination‐var—coordinationvariablesfornon‐MPImethods,suchasDatatapmethod
•time‐index—time attribute variable
4.2.2 VariablesThenestedvariableelement“var”foradios_group,whichcanbeeitheranarrayoraprimitivedatatype,isdeterminedbythedimensionattributeprovided.
14
4.2.2.1 DeclarationThefollowingisanexampleshowinghowtodefineavariableintheXMLfile.
<varname=“z‐planeionparticles”gwrite=“zion”gread=”zion_read”type=”adios_real”dimensions=”7,mimax”read=”yes”/>
4.2.2.2 AttributelistTheattributesassociatedwithvarelementasfollows:
Required:
•name–thestringnameofvariablestoredintheoutputfile
•type–thedatatypeofthevariable
Optional:
•gwrite–thevaluewillbeusedinthepythonscriptstogenerateadios_writeroutines;thedefaultvaluewillbethesameasattributenameifgwriteisnotdefined.
•gread–thevaluewillbeusedinthepythonscriptstogenerateadios_readroutines’thedefaultvaluewillbethesameasattributenameifgreadisnotdefined.
•path‐HDF‐5‐stylepathfortheelementorpathtotheHDF‐5groupordataitemtowhichthisattributeisattached.Thedefaultvalueis“/”.
•dimensions‐acomma‐separatedlistofnumbersand/ornamesthatcorrespondtointegervarelementsdeterminethesizeofthisitem.Ifnotspecified,thevariableisscalar.
•read–valueiseitheryesorno;inthecaseofno,theadios_readroutinewillnotbegeneratedforthisvarentry.Ifundefined,thedefaultvaluewillbeyes.
4.2.3 AttributesThe attribute element for adios_group provides the users with the ability tospecifymoredescriptiveinformationaboutthevariablesorgroup.Theattributescanbedefinedinbothstaticordynamicfashions.
4.2.3.1 DeclarationThestatictypeofattributescanbedefinedasfollows:
15
<attributename=“experimentaldate”path=“/zion”value=”Sep‐19‐2008”type=”adios_real”/>
Ifanattributehasdynamicvaluethatisdeterminedbytheruntimeexecutionoftheprogram,itcanbespecifiedasfollows:<attributename=“experimentaldate”
path=“/zion”var=”time”/>
wherevar“time”needtobedefinedinthesameadios‐group.
4.2.3.2 AttributelistRequired:
•name‐nameoftheattribute
•path–hierarchicalpathinsidethefilefortheattribute
•value–attributehasstaticvalueoftheattribute,mutuallyexclusivewiththeattributevar
•type–stringornumerictype,pairedwithattributevalue,inotherwords,,mutuallyexclusivewiththeattributevaralso
•var–attributehasdynamicvaluethatisdefinedbyavariableinvar
4.2.4 Gwrite/srcTheelement<Gwrite/src>isunlike<var>or<attribute>,whichareparsedandstoredintheinternalfilestructureinADIOS.Theelement<gwrite>onlyaffectstheexecutionofpythonscripts(seeChap.10).Anycontent(usuallycomments,conditionalstatements,orloopstatements)inthevalueofattribute“src”iscopiedidenticallyintogeneratedpre‐processingfiles.Declaration
<gwritesrc=””/>
Required:
•src‐anystatementthatneedstobeaddedintothesourcecode.Thiscodemustwillbeinsertedintothesourcecode,andmustbeabletobecompiledinthehostlanguage,CorFortran.
4.2.5 GlobalarraysGlobal‐bounds is an optional nested element for adios‐group. It specifies theglobalspaceandoffsetswithinthatspacefortheenclosedvariableelements.Inthecaseofwritingtoasharedfile,theglobal‐boundsinformationisrecordedinBP file and can be interpreted by converters or other postprocessing tools or
16
used towrite out eitherHDF5orNetCDF filesbyusingPHDF5or thePnetCDFmethod.
4.2.6 Time‐indexADIOS allows a dataset to be expanded in the space domain given by globalboundsandintimedomain.Itisverycommonforscientificapplicationstowriteoutamonitoringfileatregularintervals.Thefileusuallycontainsagroupoftime‐based variables that have undetermined dimensional value on the time axis.ADIOSisSimilartoNetCDFinthatitaccumulatesthetime‐indexintermsofthenumberofrecords,whichtheoreticallycanbeaddedtoinfinity.
Ifanyofvariablesinanadiosgrouparetimebased,theycanbemarkedoutbyaddingthetime‐indexvariableasanotherdimensionvalue.
4.2.6.1 Declaration<globalboundsdimensions=“nx_g,ny_g”offsets=“nx_o,0”/> …variabledeclarations…</globalbounds>
Required:
•dimensions‐thedimensionofglobalspace
•offsets–theoffsetofthedatasetinglobalspace
Any variables used in the global‐bounds element for dimensions or offsetsdeclaration need to be defined in the same adios‐group as either variables orattributes.
Fordetailedglobalarraysuse,seetheexampleillustratedinSection11.8.
ChangingI/OWithoutChangingSource:Themethodelementprovidesthehookbetween the adios‐group and the transport methods. The user employs adifferent transport method simply by changing the method attribute of themethodelement.Ifmorethanonemethodelementisprovidedforagivengroup,eachelementwillbe invokedintheorderspecified.Thisneatlygivestriggeringopportunitiesforworkflows.Totriggeraworkflowoncetheanalysisdatasethasbeenwrittentodisk,theusermakestwoelemententriesfortheanalysisadios‐group. The first indicates how to write to disk, and the second performs thetrigger for theworkflowsystem.Norecompilation, relinking,oranyothercodechangesarerequiredforanyofthesechangestotheXMLfile.
4.2.7 DeclarationThetransportelementisusedtospecifythemappingofanI/Otransportmethod,includingoptionalinitializationparameters,totherespectiveadios‐group.Therearetwomajorattributesrequiredforthemethodelement:
<transportgroup=“restart”method=“MPI”
17
priority=”1”iteration=”100”/>
Required:
•group‐correspondstoanadios‐groupspecifiedearlierinthefile.
•method–astringindicatingatransportmethodtousewiththeassociatedadios‐group
Optional:
•priority–anumericpriorityfortheI/Omethodtobetterschedulethiswritewithothersthatmaybependingcurrently
•base‐path–therootdirectorytousewhenwritingtodiskorsimilarpurposes
•iterations–anumberofiterationsbetweenwritesofthisgroupusedtogaugehowquicklythisdatashouldbeevacuatedfromthecomputenode
4.2.8 MethodslistAsthecomponentizationoftheIOsubstrate,ADIOSsupportsalistoftransportmethods,describedinSection5:
• NULL• POSIX• MPI‐IO• MPI‐CIO• MPI‐STRIPE• PHDF5• ADAPTIVE(asresearchmethod,notpublishedin1.0)• DATATAP(asresearchmethod,notpublishedin1.0)
4.3 BufferspecificationThe buffer element defines the attributes for internal buffer size and creatingtimethatapplytothewholeapplication(Figure3).Theattributeallocate‐timeisidentifiedasbeingeither “now”or“oncall”toindicatewhenthebuffershouldbeallocated. An “oncall” attribute waits until the programmer decides that allmemoryneeded for calculation has been allocated. It then calls uponADIOS toallocatebuffer.Therearetwoalternativeattributesforuserstodefinethebuffersize:MBandfree‐memory‐percentage.
4.3.1 Declaration<buffersizeMB=“100”
allocatetime=“now”/>
Required:
18
•size‐MB–theuser‐definedsizeofbufferinmegabytes.ADIOScanatmostallocatefromcomputenodes.Itisexclusivewithfree‐memory‐percentage.
•free‐memorypercentage–theuser‐definedpercentagefrom0to100%offreememoryavailableonthemachine.Itisexclusivewithsize‐MB.
•allocate‐time–indicateswhenthebuffershouldbeallocated
4.4 AnExampleXMLfile
<adios-config host-language="C"> <adios-group name="temperature" coordination-communicator="comm"> <var name="NX" type="integer"/> <var name="t" type="double" dimensions="NX"/> <attribute name="recorded date" path="/" value="Sep 19, 2008" type="string"/> </adios-group> <method group=" temperature " method="MPI"/> <buffer size-MB="1" allocate-time="now"/> </adios-config>
Figure3.ExampleXMLfilefortimeallocation.
19
5 TransportmethodsBecauseof the time itcan take tomovedata fromoneprocess toanotheror towriteand readdata toand fromadisk, it isoftenadvantageous toarrange theprogramsothatsomeworkcanbedonewhilethemessagesareintransit.Sofar,wehaveusednon‐blockingoperationstoavoidwaiting.Herewedescribesomedetails for arranging a program so that computation and I/O can take placesimultaneously.
5.1 Synchronousmethods
5.1.1 NULLTheADIOSNULLmethodallowsuserstoquicklycommentoutanADIOSgroupbychangingthetransportmethodto“NULL,”userscantestthespeedoftheroutinebytimingtheoutputagainstnoI/O.Thisisespeciallyusefulwhenworkingwithasynchronousmethods,which take an indeterminate amount of time. AnotherusefulfeatureofthisI/OisthatitquicklyallowsuserstotestoutthesystemanddeterminewhetherbugsarecausedbytheI/Osystemorbyotherplacesinthecodes.
5.1.2 POSIXThesimplestmethodprovided inADIOS justdoesbinaryPOSIX I/Ooperations.Currently, it does not support shared file writing or reading and has limitedadditionalfunctionality.ThemainpurposeforthePOSIXI/Omethodistoprovideasimplewaytomigrateaone‐file‐per‐processI/OroutinetoADIOSandtotestthe results without introducing any complexity from MPI‐IO or other I/Omethods.Performancegainsjustbyusingthistransportmethodarelikelyduetoour aggressive buffering for better streaming performance to storage. ThebufferingmethodwritesoutfilesinBPformat,whichisacompact,self‐describingformat.
Additional features may be added to the ADIOS POSIX transport method overtime.anewtransportmethodwitharelatedname,suchasPOSIX‐ASCII,maybeprovided to perform I/O with additional features. The POSIX‐ASCII examplewouldwrite out a text version of the data formatted nicely according to someparametersprovidedintheXMLfile.
5.1.3 MPI‐IOManylarge‐scalescientificsimulationsgeneratealargeamountofdata,spanningthousandsoffilesordatasets.TheuseofMPI‐IOreducestheamountoffilesandthusishelpfulfordatamanagement,storage,andaccess.
The original MPI‐IO method was developed based on our experiments withgeneratingthebetterMPI‐IOperformanceontheORNLJaguarmachine.ManyoftheinsightshavehelpedusachieveexcellentperformanceonboththeJaguarXT4machine and on the other clusters. Some of the key insights we have takenadvantage of include artificially serialized MPI_File_open calls and additional
20
timing delays that can achieve reduced delays due to metadata server (MDS)conflictsontheattachedLustrestoragesystem.
The adapted code takes full advantage of NxM grouping through thecoordination‐communicator. This grouping generates one file per coordination‐communicatorwiththedatastoredsequentiallybasedontheprocessrankwithinthecommunicator.Figure4presentsintheexampleofGTCcode,32processesinthesameToroidalzonewritetooneintegratedfile.AdditionalserializationoftheMPI_File_open calls is done using this communicator as well because eachprocessmayhaveadifferentsizedatapayload.Rank0calculatesthesizethatitwillwrite,callsMPI_File_open,andthensendsitssizetorank1.Rank1listensfortheoffsettostartfrom,addsitscalculatedsize,doesanMPI_File_open,andsendsthe new offset to rank 2. This continues for all processes within thecommunicator. Additional delays for performance based on the number ofprocessesinthecommunicatorandtheprojectedloadontheLustreMDScanbeused to introduce some additional artificial delays that ultimately reduce theamount of time theMPI_File_open calls take by reducing the bottleneck at theMDS.AnimportantfacttobenotedisthatindividualfilepointersareretrievedbyMPI_File_opensothateachprocesshasitsownfilepointerforfileseekandotherI/Ooperations.
Figure4.Server‐friendlymetadataapproach:offsetthecreate/openintime
Webuilt theMPI‐IOtransportmethod,mainlywithLustre inmindbecauseit isthe primary parallel storage service we have available. However, other file ‐system ‐specific tunings are certainly possible and fully planned as part of thistransport method system. For each new file system we encounter, a newtransportmethodimplementationtunedforthatfilesystem,andpotentiallythatplatform,canbedevelopedwithoutimpactinganyofthescientificcode.
TheMPI‐IO transportmethod forLustre is themostmature, fully featured,andwelltested.Werecommendtoanyonecreatinganewtransportmethodthatthey
21
study itasamodelof full functionalityandsomeof theadvantages thatcanbemadethroughcarefulmanagementofthestorageresources.
5.1.4 MPI‐CIOMPI‐IO defines a set of portable programming interfaces that enable multipleprocesses tohaveconcurrentaccess toshared files [1]. It isoftenused tostoreandretrievestructureddataintheircanonicalorder.Theinterfacesaresplitintotwo types: collective I/O and independent I/O. Collective functions require allprocesses to participate. Independent I/O, in contrast, requires no processsynchronization.
CollectiveI/OenablesprocesscollaborationtorearrangeI/Orequestsforbetterperformance[2,3].ThecollectiveI/OmethodinADIOSfirstdefinesMPIfileviewsforallprocessesbasedonthedatapartitioninginformationprovidedintheXMLconfigurationfile.ADIOSalsogeneratesMPI‐IOhints,suchasdatasievingandI/Oaggregators, based on the access pattern and underlying file systemconfiguration. The hints are supplied to the MPI‐IO library for furtherperformanceenhancement.Thesyntaxtodescribethedata‐partitioningpatternin theXML file uses the <global‐bounds dimensions offsets> tag,which definestheglobalarraysizeandtheoffsetsoflocalsubarraysintheglobalspace.
The global‐bounds element contains one or more nested var elements, eachspecifying a local array that existswithin the described dimensions and offset.Multiple global‐bounds elements are permitted, and strictly local arrays canbespecifiedoutsidethecontextoftheglobal‐boundselement.
Aswithotherdataelements,eachoftheattributesoftheglobal‐boundselementis providedby the adios_write call. Thedimensions attribute is specified by allparticipatingprocessesanddefineshowbigthetotalglobalspaceis. Thisvaluemustagree forallnodes.Theoffsetattributespecifiestheoffset intothisglobalspacetowhichthelocalvaluesareaddressed.Theactualsizeofthelocalelementis specified in the nested var element(s). For example, if the global boundsdimension were 50 and the offset were 10, then the var(s) nested within theglobal‐boundswouldallbedeclared inaglobalarrayof50elementswitheachlocalarraystartingatanoffsetof10fromthestartofthearray.Ifmorethanonevarisnestedwithintheglobal‐bounds,theysharethedeclarationoftheboundsbutaretreatedindividuallyandindependentlyfordatastoragepurposes.
Currentlythismethodisunfinishedatthetimeofwriting,butwillbereleasedinthenextminorreleaseinQ12010.
5.1.5 PHDF5HDF5,asahierarchicalFilestructure,hasbeenwidelyadoptedfordatastorageinvarious scientific research fields. Parallel HDF5 (PHDF5) provides a series ofAPIs toperform the I/Ooperations inparallel frommultipleprocessors,whichdramatically improves the I/O performance of the sequential approach toread/write anHDF5 file. In order tomake the difference in transportmethods
22
and file formats transparent to the end users, we provide a mechanism thatwrite/read an HDF5 file with the same schema by keeping the same commonadiosroutineswithonlyoneentrychangeintheXMLfile.thismethodprovidesusers with the capability to write out exactly the same HDF5 files as thosegeneratedbytheiroriginalPHDF5routines.Doingsoallowsforthesameanalysistoolchaintoanalyzethedata.
Currently, HDF5 supports two I/O modes: independent and Collective read orwrite,whichcanuseeithertheMPIorthePOSIXdriverbyspecifyingthedatasettransfer property list in H5Dwrite function calls. In this release, only the MPIdriver is supported in ADIOS; later on, both I/O drivers will be supported bychangingtheattributeinformationforPHDF5methodelementsinXML.
5.1.6 PNetCDFAnotherwidelyacceptedstandardfileformatiscalledNetCDF,whichisthemostfrequentlyusedfileformatintheclimateandweatherresearchcommunities.InADIOS2.0,thismethodwillbesupported.
5.1.7 OthermethodsADIOS provides an easy plug‐in mechanism for users or developers to designtheir own transportmethod.A step‐by‐step instruction for inserting anew I/OmethodisgiveninSection12.1.Usersarelikelytochoosethebestmethodfromamong the supported or customized methods for the running their platforms,thus avoiding theneed to verify their source codesdue to the switchingof I/Omethods.
5.2 Asynchronousmethods
5.2.1 MPI‐AIOThe initial implementation of the asynchronous MPI‐IO method (MPI‐AIO) ispatterned after the MPI‐IO method. Scheduled metadata commands areperformed with the same serialization of MPI_Open calls as given in Error!Referencesourcenotfound..
The degree of I/O synchronicity depends on several factors. First, the ADIOSlibrarymustbebuiltwithversionsofMPI thatarebuiltwithasynchronousI/Osupport through the MPI_File_iwrite, MPI_File_iread, and MPI_Wait calls. IfasynchronousI/Oisnotavailable,thecallsreverttosynchronous(readblocking)behavioridenticaltotheMPI‐IOmethoddescribedintheprevioussection.
Another important factor is the amount of availableADIOSbuffer space. In theMPI‐IOmethod,dataaretransportedandADIOSbufferallocationisreclaimedforsubsequent usewith calls to adios_close (). In theMPI‐AIOmethod, the “close”processcanbedeferreduntilbufferallocationisneededfornewdata.However,ifthe buffer allocation is exceeded, the data must be synchronously transportedbeforetheapplicationcanproceed.
23
ThedeferralofdatatransportiskeytoeffectivelyschedulingasynchronousI/Owithacomputation(tobeimplementedinversion2.0).InADIOSversion1.0,theapplication explicitly signals that data transport must be complete withintelligent placement of the adios_close () call to indicate when I/O must becomplete. Later versions of ADIOS will perform I/O betweenadios_begin_calculation and adios_end_calculation calls, and complete I/O onadios_end_iterationcalls.
ThisresearchmoduleisnotreleasedinADIOS1.0.
5.2.2 DataTapDataTap is an asynchronous data transport method built to ensure very highlevels of scalability through server‐directed I/O [7,8]. It is implemented as arequest‐read service designed to bridge the order‐of‐magnitude differencebetween available memories on the I/O partition compared with the computepartition. We assume the existence of a large number of compute nodesproducingdata(werefertothemas“DataTapclients”)andasmallernumberofI/Onodesreceivingthedata(werefertothemas“DataTapservers”)(seeFigure5).
Figure5.DataTaparchitecture
Uponapplicationrequest,thecomputenodemarksupthedatainPBIO[9]formatand issues a request for a data transfer to the server. The server queues therequestuntilsufficientreceivebufferspaceisavailable.Themajorcostassociatedwithsettingupthetransfer is thecostofallocatingthedatabufferandcopyingthe data. However, this overhead is small enough to have little impact on theoverallapplicationruntime.Whentheserverhassufficientbufferspace,aremotedirect memory access (RDMA) read request is issued to the client to read theremote data into a local buffer. The data are then written out to disk ortransmittedoverthenetworkasinputforfurtherprocessingintheI/OGraph.
We used the Gyrokinetic Turbulence Code (GTC) as an experimental tested fortheDataTaptransport.GTCisaparticle‐in‐cellcodeforsimulatingfusionwithintokamaks,anditisabletoscaletomultiplethousandsofprocessors.InitsdefaultI/Opattern, thedominant I/Ocost is fromeachprocessorwritingout the localparticle array into a file. Asynchronous I/O reduces this cost to just a localmemorycopy,therebyreducingtheoverheadofI/Ointheapplication.
24
ThisresearchmoduleisnotreleasedinADIOS1.0.
5.2.3 DecoupledandAsynchronousRemoteTransfers(DART)DART is an asynchronous I/O transfermethodwithinADIOS that enables low‐overhead, high‐throughput data extraction from a running simulation. DARTconsists of two main components: (1) a DARTClient module and (2) aDARTServer module. Internally, the DART system uses RDMA to implementcommunication, coordination, and data transport between the DARTClient andtheDARTServermodules.
TheDARTClientmoduleisalightlibrarythatimplementstheasynchronousI/OAPI. It integrates with the ADIOS layer by extending the generic ADIOS datatransporthooks. ItusestheADIOS layer featurestocollectandencodethedatawrittenbytheapplicationintoalocaltransportbuffer.Onceithascollecteddatafrom a simulation,DARTClient notifies theDARTServer through a coordinationchannelthatithasdataavailabletosendout.DARTClientthenreturnsandallowsthe application to continue its computations while data are asynchronouslyextractedbytheDARTServer.
TheDARTServermoduleisastand‐aloneservicethatrunsindependentlyofthesimulation.IttypicallyrunsondedicatedI/Onodes,andtransfersdatafromtheDARTClients and to remote sites (e.g., a remote a storage system such as theLuster file system. One instance of the DARTServer can service multipleDARTClients instances in parallel. Further, the server can run in cooperativemode (i.e., multiple instances of the server cooperate to service the clients inparallel and to balance load). The DARTServer receives notification messagesfromtheclients,schedulestherequests,andinitiatesthedatatransfersfromtheclients inparallel.Theserverschedulesandprioritizes thedata transferswhilethesimulationiscomputinginordertooverlapdatatransferswithcomputations,tomaximizedatathroughput,andtominimizetheoverheadonthesimulation.
ThisresearchmoduleisnotreleasedinADIOS1.0.
25
6 ADIOSReadAPI
6.1 IntroductionWecanreadinanyvariableandanysub‐arrayofavariablewiththereadAPIaswellastheattributes.TherewerethreedesignchoiceswhencreatingthisAPI:
1.GroupsintheBPfilesarehandledseparately
MostBPfilescontainasinglegroupandthevariablesandattributesinthatgrouphavetheirpathssoitlooksliketheyareorganizedintoahierarchy.IfaBPfilecontainsmorethanonegroups,thesecondgroupcanhaveavariablewiththesamepathandnameasavariableinthefirstgroup.Wechoosenotto add the name of the groups to the root of all paths because that isinconvenientforthemajorityoftheBPfilescontainingasinglegroup.
2.DimensionsofarraysarereporteddifferentlyforCandFortran
When reading from a different language thanwriting, the storage order ofthe dimensions is the opposite. Instead of transposing multidimensionalarrays in memory to order the data correctly at read time, simply thedimensionsarereportedreversed.
3. The C API returns structures filled with information while the Fortran APIreturnsinformationinindividualarguments
Since theBP file format ismetadata rich, and themetadata is immediatelyaccessible inthefooterofthefile,wecanhaveaneasytouseAPIwithfewfunctions.Theopenfunctionreturnsinformationonthenumberofelementsandtimestepsandthe listofgroups inthe file.Thegroupopenreturnsthelistofvariablesandattributes.Theinquiryofavariablereturnsnotjustthetype and dimensionality of a variable but also the global minimum andmaximumofitwithoutreadinginthecontentofthevariablefromthefile.
ThereadAPIlibraryhastwoversions.TheMPIversionshouldbeusedinparallelprograms.Onlytherank=0processreadsthefooterofthefileandbroadcastsittothe other processes in adios_fopen(). File access is handled through MPI‐IOfunctions.Sequentialprogramscanuseanyofthetwoversionsbutifyoudonotwant dependency onMPI, link your programwith the non‐MPI version,whichusesPOSIX I/O functions. In this case, youneed to compile your codewith the‐D_NOMPI option. There is nodifference in performanceor functionality in thetwoversions(insequentialapplications).
NotethatthewriteAPIcontainstheadios_read()function,whichisusefultoreadindata fromthesamenumberofprocessorsas thedatawaswritten from, likehandling checkpoint/restartdata (seeSection3.1.2.5.).However, if youneed toread in fromadifferentnumberofprocessorsor to read inonlya subsetofanarrayvariable,youneedtousethisreadAPI.
26
6.2 ReadCAPIdescriptionThesequenceofreadinginavariablefromtheBPfileis
‐ openfile‐ openagroup‐ inquirythevariabletogettypeanddimensions‐ allocatememoryforthevariable‐ readinvariable(wholeorpartofit)‐ freevarinfodatastructure‐ closegroup‐ closefile
ExamplecodesusingtheCAPIare
‐examples/C/read_all/read_all.c‐examples/C/global‐array/adios_read_global
6.2.1 adios_errmsg/adios_errnoint adios_errno char * adios_errmsg()
If an error occurrs during the call of a C api function, it either returns NULL(insteadofapointertoanallocatedstructure)oranegativenumber.Italsosetstheintegeradios_errnovariable(thenegativereturnvalueisactually‐1timesthe errno value).Moreover, it prints the errormessage into an internal buffer,whichcanberetrievedbyadios_errmsg().
Note that adios_errmsg() returns the pointer to the internal buffer instead ofduplicatingthestring,sorefrainfromwritinganythingintoit.Also,onlythelasterrormessageisavailableatanytime.
6.2.2 adios_fopenADIOS_FILE * adios_fopen (const char * fname, MPI_Comm comm)
ADIOSFILEisastructof
uint64_t fh; Filehandlerint groups_count; Numberofadiosgroupsinfile
int vars_count; Numberofvariablesinallgroups
int attrs_count; Numberofattributesinallgroupsint tidx_start; Firsttimestepinfile,usually1
int ntimesteps; Numberoftimestepsinfile. Thereisalwaysatleastonetimestep
int version; ADIOSBPversionoffileformat
uint64_t file_size; Sizeoffileinbytes
27
int endianness; 0:littleendian,1:bigendian Youdonotneedtocareaboutthis.char** group_namelist; Namesoftheadiosgroupsinthefile (cf.groups_count)
Thearrayforthe listofgroupnames isallocated inthe functionandis freedintheclosefunction.
If you use theMPI version of the library, pass the communicator,which is thecommunicator of all processes that call the open function. Rank=0 processbroadcaststhemetadatatotheotherprocessessothatweavoidopeningthefilefrommanyprocessesatonce.Ifyouusethenon‐MPIversionofthelibrary,justpassonanarbitraryintegervalue,whichisnotusedatall.
6.2.3 adios_fcloseint adios_fclose (ADIOS_FILE *fp)
Youareexpectedtocloseafilewhenyoudonotneeditanymore.Thisfunctionreleasesalotofinternalmemorystructures.
6.2.4 adios_gopen/adios_gopen_byidADIOS_GROUP * adios_gopen (ADIOS_FILE *fp, const char * grpname) ADIOS_GROUP * adios_gopen_byid (ADIOS_FILE *fp, int grpid)
Youneed toopena group to get access to its variables andattributes. You canopen a group either by its name returned in the ADIOS_FILE struct’sgroup_namelistlistofstringsorbyitsindex,whichistheindexofitsnameinthislistofnames.
Youcanhaveseveralgroupsopenatthesametime.
ADIOS_GROUPisastructof
uint64_t gh; Grouphandler
int grpid; groupindex(0..ADIOS_FILE.groups_count‐1)int vars_count; Numberofvariablesinthisadiosgroup
char** var_namelist; Variablenamesinachar*array
int attrs_count; Numberofattributesinthisadiosgroupchar** attr_namelist; Attributenamesinachar*array
ADIOS_FILE*fp; pointertotheparentADIOS_FILEstructThearraysforthelistofvariablenamesandattributenameareallocatedinthefunctionandarefreedinthegroupclosefunction.
6.2.5 adios_gcloseint adios_gclose (ADIOS_GROUP *gp)
28
Youneedtoclosethegroupwhenyoudonotneeditanymore.
6.2.6 adios_inq_var/adios_inq_var_byidADIOS_VARINFO * adios_inq_var (ADIOS_GROUP *gp, const char * varname) ADIOS_VARINFO * adios_inq_var_byid (ADIOS_GROUP *gp, int varid)
Thisfunctionshouldbeusedifyouwanttodiscoverthetypeanddimensionalityofavariableorwanttogettheminimum/maximumvalueswithoutreadinginthedata.Youcanrefertothevariablewithitsname(fullpath)intheADIOS_GROUPstruct’svar_namelistorbytheindexinthatlist.
ADIOS_VARINFO structure is allocated in the function but there is nocorrespondingclosing function, thereforeuserhas to free theADIOS_VARINFO*pointer yourself when you do not need it anymore by using theadios_free_varinfo()function.
ADIOS_VARINFOisastructof
uint64_t gh; Grouphandlerint grpid; groupindex(0..ADIOS_FILE.groups_count‐1)int varid; variableindex(0..ADIOS_GROUP.var_count‐1)enumADIOS_DATATYPEStype; typeofvariableint ndim; numberofdimensions,0forscalarsuint64_t* dims; sizeofeachdimensionint timedim; ‐1:variablehasnotimestepsinfile, >=0:whichdimensionistimevoid* value; valueofascalarvariable,NULLforarray.void* gmin; minimumvalueinanarrayvariable. Forscalars,gmin=value.void* gmax; maximumvalueofanarrayvariable (overalltimesteps)void* mins; minimumpereachtimestep (arrayoftimestepelements) always=NULLin1.0void* maxs; maximumpereachtimestep always=NULLin1.0
Note that you can modify the ADIOS_GROUP’s namelists because they are notused in thediscoveryof thevariables.However, the indexof thevariable is theindexofthevariable’spositionintheoriginalorderofthelist.Ifyousortthislistfororderedprintouts,youneedtoremembertheoriginalindicesofthevariablesortoidentifythevariablesbyname.
6.2.7 adios_free_varinfovoid adios_free_varinfo (ADIOS_VARINFO *cp)
29
FreesuptheADIOS_VARINFO*structurereturnedbyadios_inq_var().
6.2.8 adios_read_var/adios_read_var_byidint64_t adios_read_var (ADIOS_GROUP * gp, const char * varname, const uint64_t * start, const uint64_t * count, void * data) int64_t adios_read_var_byid (ADIOS_GROUP * gp, int varid, const uint64_t * start, const uint64_t * count, void * data)
This function is used to read in the content of a variable, or a subset of it. Youneed toallocatememory for receiving thedatabeforecalling this function.Thesubset(ortheentireset)isdefinedbythestartandcountineachdimension.Thestart and count arrays must have as many elements as many dimensions thevariable has. Start contains the starting offsets for each dimension and countcontainsthenumberofelementstoreadinagivendimension.Ifyouwanttoreadintheentirevariable,startshouldbeanarrayofzerosandcountshouldequaltothedimensionsofthevariable.
Notethatstartandcountisrelatedtothenumberofelementsineachdimension,not the number of bytes needed for storage. When allocating the data array,multiplythetotalnumberofelementswiththesizeofoneelement.Ifyouneedtobegenericinthiscalculation,youcanusetheadios_type_size()functiontogetthesizeofoneelementofagiventype.
6.2.9 adios_get_attr/adios_get_attr_byidint adios_get_attr (ADIOS_GROUP * gp, const char * attrname, enum ADIOS_DATATYPES * type, int * size, void ** data) int adios_get_attr_byid (ADIOS_GROUP * gp, int attrid, enum ADIOS_DATATYPES * type, int * size, void ** data)
Thisfunctionretrievesanattributeincludingitstype,memorysizeanditsvalue.An attribute can only be a scalar value or a string.Memory is allocated in thefunctiontostorethevalue.Theallocatedsizeisreturnedinthesizeargument.
Thisfunctiondoesnotreadthefileusually.Theattribute’svalueisstoredinthefooterandisalreadyinthememoryafterthefileisopened.However,anattribute
30
can refer to a scalar (or string) variable too. In this case, this function callsadios_read_varinternally,sothefilewillbeaccessedtoreadinthatscalar.
6.2.10 adios_type_to_stringconst char * adios_type_to_string (enum ADIOS_DATATYPES type)
Thisfunctionreturnsthenameofagiventype.
6.2.11 adios_type_sizeint adios_type_size(enum ADIOS_DATATYPES type, void *data)
Thisfunctionreturnsthememorysizeofonedataelementofanadiostype.Ifthetype is adios_string, and the second argument is the string itself, it returnsstrlen(data)+1.Forothertypes,dataisnotusedandthefunctionreturnsthesizeoccupiedbyoneelement.
6.3 ReadFortranAPIdescriptionTheFortranAPIdoesnotdealwiththestructuresoftheCapiratheritrequiresseveralargumentsinthefunctioncalls.TheyareallimplementedassubroutineslikethewriteFortranAPIandthelastargumentisanintegervariabletostoretheerrorcodeoutputofeachfunction(0meaningsuccessfuloperation).
An example code can be found in the source distribution astests/bp_read/bp_read_f.F90.
Themostimportantthingtonoteisthatsomefunctionsneedinteger*8(scalarorarray)arguments.Passinganinteger*4arrayfromyourcodeleadstofatalerrors.Please,doublechecktheargumentsofthefunctioncalls.
Due to the lack of structures and because the Fortran API does not allocatememoryforthem,youhavetoinquirythefileafteropeningitandtoinquirythegroup after opening it. You also have to inquiry an attribute to determine thememorysizeandallocatespaceforitbeforeretrievingit.
WheretheAPIfunctionreturnsalistofnames(inquiryfileorinquirygroup),youhave to provide enough space for them using the counts returned by theprecedingopencall.
HereisthelistoftheFortransubroutines.TheGENERICwordindicatesthatyoucan use that function with any data type at the indicated argument. SinceFortran90doesnotallowdefiningfunctionsthatcantakeanytypeofargument,wedonotprovideanF90moduleforthisAPI.ThefunctionsareactuallydefinedinCandduetothelackofcompilerchecking,youcanpassanytypeofarrayorvariablewhereaGENERICarrayisdenoted.
subroutine adios_errmsg (msg) character(*), intent(out) :: msg
end subroutine
31
subroutine adios_fopen (fp, fname, comm, groups_count, err) integer*8, intent(out) :: fp character(*), intent(in) :: fname integer, intent(in) :: comm integer, intent(out) :: groups_count integer, intent(out) :: err
end subroutine subroutine adios_fclose (fp, err)
integer*8, intent(in) :: fp integer, intent(out) :: err
end subroutine subroutine adios_inq_file (fp, vars_count, attrs_count, tstart, ntsteps, gnamelist, err)
integer*8, intent(in) :: fp integer, intent(out) :: vars_count integer, intent(out) :: attrs_count integer, intent(out) :: tstart integer, intent(out) :: ntsteps character(*), dimension(*), intent(inout) :: gnamelist integer, intent(out) :: err
end subroutine subroutine adios_gopen (fp, gp, grpname, vars_count, attrs_count, err)
integer*8, intent(in) :: fp integer*8, intent(out) :: gp character(*), intent(in) :: grpname integer, intent(out) :: vars_count integer, intent(out) :: attrs_count integer, intent(out) :: err
end subroutine subroutine adios_gclose (gp, err)
integer*8, intent(in) :: gp integer, intent(out) :: err
end subroutine subroutine adios_inq_group (gp, vnamelist, anamelist, err)
integer*8, intent(in) :: gp character(*), dimension(*), intent(inout) :: vnamelist character(*), dimension(*), intent(inout) :: anamelist integer, intent(out) :: err
end subroutine subroutine adios_inq_var (gp, varname, vartype, ndim, dims, timedim, err)
integer*8, intent(in) :: gp character(*), intent(in) :: varname integer, intent(out) :: vartype
32
integer, intent(out) :: ndim integer*8, dimension(*), intent(out) :: dims integer, intent(out) :: timedim integer, intent(out) :: err
end subroutine subroutine adios_read_var (gp, varname, start, count, data, read_bytes)
integer*8, intent(in) :: gp character(*), intent(in) :: varname integer*8, dimension(*), intent(in) :: start integer*8, dimension(*), intent(in) :: count GENERIC, dimension(*), intent(inout) :: data integer*8, intent(in) :: read_bytes
end subroutine subroutine adios_get_varminmax (gp, varname, value, gmin, gmax, mins, maxs, err)
integer*8, intent(in) :: gp character(*), intent(in) :: varname GENERIC, intent(out) :: value GENERIC, intent(out) :: gmin GENERIC, intent(out) :: gmax GENERIC, dimension(*), intent(inout) :: mins GENERIC, dimension(*), intent(inout) :: maxs integer, intent(out) :: err
end subroutine subroutine adios_inq_attr (gp, attrname, attrtype, attrsize, err)
integer*8, intent(in) :: gp character(*), intent(in) :: attrname integer, intent(out) :: attrtype integer, intent(out) :: attrsize integer, intent(out) :: err
end subroutine subroutine adios_get_attr_int1 (gp, attrname, attr, err)
integer*8, intent(in) :: gp character(*), intent(in) :: attrname GENERIC, dimension(*), intent(inout) :: attr integer, intent(out) :: err
end subroutine
6.4 CompilingandlinkingapplicationsInaCcode,includetheadios_read.hheaderfile.
In a Fortran 90 code, you do not need to include anything. It is stronglyrecommended to double check the integer parameters because the read API
33
expectsinteger*8argumentsatseveralplacesandprovidinganintegeronlycanbreakyourcodeandthendebuggingitprovestobeverydifficult.
If youwant touse theMPIversionof the library, then linkyour (CorFortran)applicationwith‐ladiosread.
Ifyouwanttousethenon‐MPIversionofthelibrary,youneedtocompileyourcode with the –D_NOMPI option and link your application with‐ladiosread_nompi.
7 BPfileformat
7.1 IntroductionThischapterdescribesthefilestructureofBP,whichistheADIOSnativebinaryfile format, to aid in understanding ADIOS performance issues and how filesconvertfromBPfilestootherscientificfileformats,suchasnetCDFandHDF5.
To avoid the file size limitation of 2 gigabytes by using a signed 32‐bit offsetwithin its internalstructure,BP formatusesanunsigned64‐bitdatatypeas thefile offset. Therefore, it is possible towrite BP files that exceed 2 gigabytes onplatformsthathavelargefilesupport.
ByadaptingADIOS read routinesbasedon theendianness indication in the filefooter,BPfilescanbeeasilyportableacrossdifferentmachines(e.g.,betweentheCray‐XT4andBlueGene).
Toaidindataselection,wehavealow‐overheadconceptofdatacharacteristicstoprovideanefficient, inexpensivesetofattributesthatcanbeusedto identifydatasetswithoutanalyzinglargedatacontent.
AsshowninFigure6,theBPformatcomprisesaseriesofprocessgroupsandthefilefooter.Theremainderofthischapterdescribeseachcomponentindetailandhelpstheusertobetterunderstand(1)whyBPisaself‐describingandmetadata‐rich file format and (2) why it can achieve high I/O performance on differentmachineinfrastructures.
34
Figure6.BPfilestructure
7.2 FooterOneknownlimitationoftheNetCDFformatisthatthefilecontentsarestoredinaheader that is exactly big enough for the information provided at file creation.Any changes to the length of that datawill requiremoving data. To avoid thiscost,we choose toemploya foot index instead.Weplaceourversion identifierand the offset to the beginning of the index as the last few bytes of our file,makingitsimpletofindtheindexinformationandtoaddnewanddifferentdatatoourfileswithoutaffectinganydataalreadywritten.
7.2.1 VersionWe reserve 4 bytes for the file version, in which the highest bit indicatesendianness. Because ADIOS uses a fixed‐size type for data, there is no need tostoretypesizeinformationinthefooter.
7.2.2 OffsetsofindicesIn BP format, we store three 8‐byte file offsets right before the version word,which allows users or developers to quickly seek any of the index tables forprocessgroups,variables,orattributes.
7.2.3 Indices
7.2.3.1 CharacteristicsBeforewe dive into the structures of the three index tablesmentioned earlier,let’sfirsttakealookwhatcharacteristicmeansintermsofBPfileformat.Tobeabletomakeasummaryinspectionofthedatatodeterminewhetheritcontainsthefeatureofgreatestinterest,wedevelopedtheideaofdatacharacteristics.Theideaofdatacharacteristicsistocollectsomesimplestatisticaland/oranalyticaldataduringtheoutputoperationor later foruse in identifyingthedesireddatasets.Simplestatisticslikearrayminimumandmaximumvaluescanbecollectednearly for free as part of the I/O operation. Other more complex analyticalmeasures like standard deviations or specialized measures particular to the
35
sciencebeingperformancebyrequiremoreprocessing.AspartofourBPformat,westorethesevaluesnotonlyaspartofdatapayload,butalsoinourindex.
7.2.3.2 PGIndextableAsshowninFigure7,theprocessgroup(PG)indextableencompassesthecountandthetotal lengthofall thePGsasthe first twoentries.Therestof thetablescontain a set of information for each PG, which contains the group nameinformation,processID,andtimeindex.TheProcessIDspecifieswhichprocessagroup iswrittenby.Thatprocesswillbe therankvalue in thecommunicator iftheMPImethodisused.Mostimportantly,thereisafile‐offsetentryforeachPG,allowingafastskipofthefileintheunitoftheprocessgroup.
Figure7.Groupindextable
7.2.3.3 VariablesindextableThevariablesindextableiscomposedofthetotalcountofvariablesintheBPfile,the size of variables index table, and a list of variable records. Each recordcontainsthesizeoftherecordandthebasicmetadatatodescribethevariable.AsshowninFigure8,themetadataincludethenameofthevariable,thenameofthegroupthevariableisassociatedwith,thedatatypeofthevariable,andaseriesofcharacteristic features. The structure of each characteristic entry contains anoffsetvalue,whichisaddressedtothecertainoccurrenceofthevariableintheBPfile.For instance, ifnprocesseswriteout thevariable “d”per timestep,andmiterations have been completed during thewhole simulation, then the variablewillbewritten(m×n) times in theBP file that isproduced.Accordingly, therewillbethesamenumberofelementsinthelistofcharacteristics.Inthisway,wecanquicklyretrievethesingledatasetforalltimestepsoranyotherselectionof
36
time steps. This flexibility and efficiency also apply to a scenario in which aportionofrecordsneedstobecollectedfromacertaingroupofprocesses.
Figure8.Variablesindextable
7.2.3.4 AttributesindextableSinceanattributecanbeconsideredtobeaspecialtypeofvariable,itsindextableinBPformatisorganizedinthesamewayasavariablesindextableandthereforesupportsthesametypesoffeaturesmentionedintheprevioussections.
7.3 ProcessGroupsOneofthemajorconcepts inBPformat iswhat iscalled“processgroup”orPG.TheBPfileformatencompassesaseriesofPGentriesandtheBPfilefooter.Eachprocess group is the entire self‐contained output from a single process and iswritten out independently into a contiguous disk space. In that way, we canenhance parallelism and reduce coordination among processes in the samecommunication group. The data diagram in Figure 9 illustrates the detailedcontentineachPG.
37
Figure9.Processgroupstructure
7.3.1 PGheader
7.3.1.1 UnlimiteddimensionBPformatallowsuserstodefineanunlimiteddimension,whichwillbespecifiedasthetime‐indexintheXMLfile.Userscandefinevariableshavingadimensionwithundefinedlength,forwhichthevariablecangrowalongthatdimension.PGisaself‐contained,independentdatastructure;thedatasetinthelocalspacepereachtimestepisnotreconstructedatthewritingoperationsacrosstheprocessesorattimesteps.Theoretically,PGscanbeappendedtoinfinity;theycanbeaddedoneafteranothernomatterhowmanyprocessesortimestepstakeplaceduringthesimulation.ThusADIOSisabletoachievehighI/Operformance.
7.3.1.2 TransportmethodsOneoftheadvantagesoforganizingoutputintermsofgroupsistocategorizeallthe variables based on their I/O patterns and logical relationships. It providesflexibilityforeachgrouptochoosetheoptimizedtransportmethodaccordingtothe simulation environment and underlying hardware configuration or thetransport methods used for a performance study without even changing thesourcecode.InPGheaderstructure,eachentryinthemethodlisthasamethodID and method parameters, such as system‐tuning parameters or underneathdriverselection.
38
7.3.2 Varslist
7.3.2.1 Varheader
7.3.2.1.1 Dimensions structure Internal to bp is sufficient information to recreate any global structure and toplacethelocaldataintothestructure.Inthecaseofaglobalarray,eachprocesswritesthesizeoftheglobalarraydimensions,specifiesthelocaloffsetsintoeach,andthenwritesthelocaldata,notingthesizeineachdimension.Onconversiontoanotherformat,suchasHDF5,thisinformationisusedtocreatehyperslabsforwritingthedataintothesingle,contiguousspace.Otherwise,itisjustreadbackin and used to note where the data came from. In this way, we can enhanceparallelism and reduce coordination. All of our parallel writes occurindependently unless the underlying transport specifically requires collectiveoperations.Eveninthosecases,thecollectivecallsareonlyforafullbufferwrite(assuming the transportwaswritten appropriately) unless there is insufficientbufferspace.
AsshowninFigure9,thedimensionstructurecontainsatimeindexflag,whichindicateswhetherthisvariablehasanunlimitedtimedimension.Var_idisusedtoretrieve thedimensionvalue if thedimension isdefinedasvariable in theXMLfile;otherwise,therankvalueistakenasthearraydimension.
7.3.2.2 PayloadBasicstatisticalcharacteristicsgiveuserstheadvantageforquickdatainspectionandanalysis. InFigure9, redundant informationabout characteristics is storedalongwithvariablepayloadsothatifthecharacteristicspartinthefilefootergetscorrupted,itcanstillberecoveredquickly.Currently,onlysimplestatisticaltraitsaresavedinthefile,butthecharacteristicsstructurewillbeeasilyexpandedormodifiedaccordingtotherequirementsofscientificapplicationsortheanalysistools.
7.3.3 AttributeslistThe layout of the attributes list (see Figure 10) is very similar to that of thevariables. However, instead of containing dimensional structures and physicaldataload,theattributeheaderhasanis_varflag,whichindicateseitherthatthevalueoftheattributeisreferencedfromavariablebylookingupthevar_idinthesamegrouporthatitisastaticvaluedefinedintheXMLfile.
39
Figure10.Attributeentrystructure
40
8 Utilities
8.1 adios_lintWeprovideaverificationtool,calledadios_lint,whichcomeswithADIOS1.0. Itcan help users to eliminate unnecessary semantic errors and to verify theintegrity of the XML file. Use of adios_lint is very straightforward; enter theadios_lintcommandfollowedbytheconfigfilename.
8.2 bplsThe bpls utility is used to list the contents of a BP file or to dump arbitrarysubarrays of a variable. Bydefault it lists the variables in the file including thetype, name anddimensionality. Amore detailed listing (‐l option) provides theglobalminimumandmaximumofeacharrayandthevalueofeachscalar.Notethat the detailed listing does not have extra overhead of processing since thisinformation isavailable in the footerof theBP file.Attributescanalsobe listedbesides the variables with the –a option, and the full listing can be sorted bynameswith–toption.Namemaskstolistonlyasubsetofthevariables/attributescanbegivenlikewiththelscommandorasregularexpressions(with–eoption).
Since bpls iswritten in C, the order of dimensions is reportedwith row‐majorordering,i.e., ifFortranapplicationwroteanNxM2Dvariable,bplsreportsitasanMxNvariable.
Variables can be dumped with the –d option. A subset of a variable can bedumped by using start and count values for each dimension with –s and –coption, e.g. –s “10,20,30” –c “10,10,10” reads in a 10x10x10 sub‐array of avariablestartingfromthe(10,20,30)element.Indicesstartfrom0.Asinpython,‐1denotesthelastelementofanarrayandnegativevaluesarehandledascountsfrombackward. Thus, ‐s “‐1,‐1” –c “1,1” reads in the very last element of a 2Darray,or–s“0,0”–c“1,‐1”reads inonerowofa2Darray.Or–s“1,1”–c“‐2,‐2”readsinthevariablewithouttheedgeelements(row0,colum0,lastrowandlastcolumn).
Timeishandledasanadditionaldimension,i.e.,ifa2Dvariableiswrittenseveraltimes into the sameBP file, bpls lists it as a 3Darraywith the timedimensionbeingthefirst(slowestchanging)dimension.
In the examplebelow, a4process applicationwrote a4x4array (eachprocesswrotea2x2subset)withvalues from0 to15onceunder thename/var/int_xyand3timesunderthename/var/int_xyt.
$ bpls -latv g_2x2_2x2_t3.bp File info: of groups: 1 of variables: 11 of attributes: 7 time steps: 3 starting from 1
41
file size: 779 KB bp version: 1 endianness: Little Endian Group genarray: integer /dimensions/X scalar = 4 integer /dimensions/Y scalar = 4 integer /info/nproc scalar = 4 string /info/nproc/description attr = "Number of writers" integer /info/npx scalar = 2 string /info/npx/description attr = "Number of processors in x dimension" integer /info/npy scalar = 2 string /info/npy/description attr = "Number of processors in y dimension" integer /var/int_xy {4, 4} = 0 / 15 string /var/int_xy/description attr = "2D array with 2D decomposition" integer /var/int_xyt {3, 4, 4} = 0 / 15 string /var/int_xyt/description attr = "3D array with 2D decomposition with time in 3rd dimension"
Figure11.bplsutility
Thecontentof/var/int_xycanbedumpedwith
$ bpls g_2x2_2x2_t3.bp -d -n 4 var/int_xy integer /var/int_xy {4, 4} (0,0) 0 1 2 3 (1,0) 4 5 6 7 (2,0) 8 9 10 11 (3,0) 12 13 14 15
The“central”2x2subsetof/var/int_xycanbedumpedwith
$ bpls g_2x2_2x2_t3.bp -d -s "1,1" -c "2,2" -n 2 var/int_xy integer /var/int_xy {4, 4} slice (1:2, 1:2) (1,1) 5 6 (2,1) 9 10
Thelastelementof/var/int_xytforeachtimestepcanbedumpedwith
$ bpls g_2x2_2x2_t3.bp -d -s "0,-1,-1" -c "-1,1,1" -n 1 var/int_xyt integer /var/int_xyt {3, 4, 4} slice (0:2, 3:3, 3:3) (0,3,3) 15 (1,3,3) 15 (2,3,3) 15
8.3 bpdumpThe bpdump utility enables users to examine the contents of a bp file morecloselytotheactualBPformatthanwithbplsandtodisplayall thecontentsor
42
selected variables in the format on the standard output. Eachwriting process’outputisprintedseparately.
It dumps the bp file content, including the indexes for all the process groups,variables, and attributes, followed by the variables and attributes list ofindividualprocessgroups(seeFigure12).
bpdump[‐dvar|‐‐dumpvar]<filename>
========================================================ProcessGroupsIndex:Group:temperatureProcessID:0TimeName:Time:1OffsetinFile:0========================================================VarsIndex:Var(Group)[ID]:/NX(temperature)[1]Datatype:integerVarsCharacteristics:20Offset(46)Value(10)Var(Group)[ID]:/size(temperature)[2]Datatype:integerVarsCharacteristics:20Offset(77)Value(20)…Var(Group)[ID]:/rank(temperature)[3]Datatype:integerVarsCharacteristics:20Offset(110)Value(0)…Var(Group)[ID]:/temperature(temperature)[4]Datatype:doubleVarsCharacteristics:20Offset(143)Min(1.000000e‐01)Max(9.100000e+00)Dims(l:g:o):(1:20:0,10:10:0)…========================================================AttributesIndex:Attribute(Group)[ID]:/recorded‐date(temperature)[5]Datatype:stringAttributeCharacteristics:20Offset(363)Value(Sep‐19‐2008)…
43
Figure12.bpdumputility
44
9 ConvertersTomakeBPfilescompatiblewiththepopularfileformats,weprovideaseriesofconverterstoconvertBPfiles toHDF5,NETCDF,orASCII.As longasusersgivethe required schema via the configuration file, the different converter toolscurrently in ADIOS have the features to translate intermediate BP files to theexpectedHDF5,NetCDF,orASCIIformats.
9.1 bp2h5This converter, as indicated by its name, can convert BP files into HDF5 files.Therefore,thesamepostprocessingtoolscanbeusedtoanalyzeorvisualizetheconvertedHDF5files,whichhavethesamedataschemaastheoriginalones.Theconvertercanmatchtherow‐basedorcolumn‐basedmemorylayoutfordatasetsinside the file basedonwhich language the source codes arewritten in. If theXML file specifies global‐bounds information, the individual sub‐blocks of thedatasetfromdifferentprocessgroupswillbemergedintooneglobalthedatasetinHDFfile.
9.2 bp2ncdThebp2ncdconverterisusedtotranslatebpfilesintoNetCDFfiles.InChap.5,wedescribe the time‐index as an attribute for adios‐group. If the variable is time‐based, one of its dimensions needs to be specified by this time‐index variable,which is defined as an unlimited dimension in the file into which it is to beconverted.aNetCDFdimensionhasanameandalength.Iftheconstantvalueisdeclared as a dimension value, the dimension in NetCDF will be namedvarname_n, in which varname is the name of the variable and n is the nthdimension for that variable. To make the name for the dimension value moremeaningful,theuserscanalsodeclarethedimensionvalueasanattributewhosenamecanbepickedupbytheconverterandusedasthedimensionname.
BasedonthegivenglobalboundsinformationinaBPfile,theconvertercanalsoreconstructtheindividualpiecesfromeachprocessgroupandcreatetheglobalspacearray inNetCDF.A finalwordabouteditingtheXMLfile: thenamestringcancontainonlyletters,numbersorunderscores(“_”).Therefore,theattributeorvariablenameshouldconformtothisrule.
9.3 bp2asciiSometimes,scientistswantonevariablewithallthetimestepsorwanttoextracttwo variables at the same time steps to and store the resulting data in ASCIIformat.TheBp2asciiconvertertoolallowsuserstoaccomplishthosetasks.
Bp2asciibp_filename–vx1…xn[–c/‐r]–tm,n
‐v–specifythevariablesneedtobeprintedoutinASCIIfile
‐c–printvariablevaluesforallthetimestepsincolumn
45
‐r–printvariablevaluesforallthetimestepsinrow
‐t–printvariablevaluesfortimestepmton,ifnotdefined,allthetimestepswillbeprintedout.
9.4 ParallelConverterToolsCurrently,allof theconvertersmentionedabovecanonlysequentiallyparsebpfiles.Wewill work on developing parallel versions of all of the converters forimproved performance of ADIOS 2.0 . As a result, the extra conversion cost totranslatebpintotheexpectedfileformatcanbeunnoticeablecomparedwiththefiletransfertime.
46
10 Groupread/writeprocess InADIOS1.0,weprovideapythonscript,whichtakesaconfigurationfilenameasaninputargumentandproducesaseriesofpreprocessingfilescorrespondingtothe individual adios‐group in the XML file.Depending onwhich language (C orFORTRAN) is specified in XML, the python script either generates filesgwrite_groupname.ch and gread_groupname.ch for C or fileswith extension .fhfor Fortran. These files contain the size calculation for the group andautomatically print adios_write calls for all the variables defined inside adios‐group. Using only one “#include filename.ch/filename.fh” statement in sourcecodebetweenthepairofadios_openandadios_close.
UserseithertypethefollowingcommandlineorincorporateitintoMakefie:
python gpp.py <config_fname>
10.1 Gwrite/gread/readBelowareafewexampleofthemappingfromvarelementtoadios_write/read:
Inadios‐group“weather”,wehaveavariabledeclaredinthefollowingforms:
1) <varname=”temperature”gwrite=”t”gread=”t_read”type=”adios_double”dimensions=”NX”/>
Whenthepythoncommandisexecuted,twofilesareproduced,gwrite_weather.chandgread_weather.ch.Thegwrite_weather.chcommandcontains
adios_write (adios_handle, “temperature”, t);
whilegread_weather.chcontains
adios_read (adios_handle, “temperature”, t_read).
2) <varname=”temperature”gwrite=”t”gread=”t_read”type=”adios_double”dimensions=”NX”read=”no”/>
Inthiscase,onlytheadios_writestatementisgeneratedingwrite_weather.ch.Theadios_readstatementisnotgeneratedbecausethevalueofattributereadissettono.
3) <varname=”temperature”gread=”t_read”type=”adios_double”dimensions=”NX”/>
adios_write (adios_handle, “temperature”, temperature)
adios_read (adios_handle, “temperature”, t_read).
4) <varname=”temperature”gwrite=”t”type=”adios_double”dimensions=”NX”/>
47
adios_write (adios_handle, “temperature”, t)
adios_read (adios_handle, “temperature”, temperature)
10.2 AddconditionalexpressionSometimes, the adios_write routines are not perfectly written out one afteranother. Theremight be some conditional expressions or loop statements. Thefollowing example will show you how to address this type of issue via XMLediting.
<gwritesrc=”if(rank==0){”/>
<varname=”temperature”gwrite=”t”gread=”t_read”type=”adios_double”dimensions=”NX”read=”no”/>
<gwritesrc=”}”/>
Rerun the python command; the following statements will be generated ingwrite_weather.ch,
if (mype==0) {
adios_write (adios_handle, “temperature”, t)
}
gread_weather.chhassameconditionexpressionadded.
10.3 DependencyinMakefileSinceweincludetheheaderfilesinthesource,theusersneedtoincludetheheaderfilesasapartofdependencyrulesintheMakefile.
48
11 CProgrammingwithADIOSThischapterfocusesonhowtointegrateADIOSintotheusers’sourcecodeinCandhowtowriteintoseparatefilesorasharedfilefrommultipleprocessesinthesame communication domain. These examples can be found in the sourcedistributionundertheexamples/C/manualdirectory.
InthefollowingstepswewillcreateprogramsthatuseADIOStowrite
‐ ametadata‐richBPfileperprocess‐ onelargeBPfilewiththearraysfromallprocesses‐ NfilesfromPprocesses,whereN<<P‐ thedataofallprocessesasoneglobalarrayintoonefile‐ aglobal‐arrayoverseveraltimestepsintoonefile
The strength of the componentization of I/O in ADIOS allows us to switchbetween the first two modes by selecting a different transport method in aconfigurationfileandruntheprogramwithoutrecompilingit.
11.1 Non‐ADIOSProgramThe starting programming example, shown in Figure 13, writes a double‐precision arrayt with size ofNX into a separate file per process (the array isuninitializedintheexamples).
#include<stdio.h>#include"mpi.h"#include"adios.h"intmain(intargc,char**argv){
charfilename[256];intrank;intNX=10;doublet[NX];FILE*fp;MPI_Init(&argc,&argv);MPI_Comm_rank(MPI_COMM_WORLD,&rank);sprintf(filename,"restart_%5.5d.dat",rank);fp=open(filename,"w");fwrite(&NX,sizeof(int),1,fp);fwrite(t,sizeof(double),NX,fp);fclose(fp);MPI_Finalize();return0;
49
}
Figure13.Originalprogram(examples/C/manual/1_nonadios_example.c).
$ mpirun -np 4 1_nonadios_example $ ls restart_* restart_00000.dat restart_00001.dat restart_00002.dat restart_00003.dat
11.2 ConstructanXMLFileIn the example above, theprogram isdesigned towrite a file for eachprocess.There is a double‐precision one‐dimensional array called “t”. We also need todeclare and write all variables that are used for dimensions (i.e. NX in ourexample). Therefore, our configuration file is constructed as shown in Error!Referencesourcenotfound.Figure14.
/*config.xml*/<?xmlversion="1.0"?><adios‐confighost‐language="C"><adios‐groupname="temperature"coordination‐communicator="comm"><varname="NX"type="integer"/><varname="temperature"gwrite="t"type="double"dimensions="NX"/><attributename="description"path="/temperature"type=”string”value="Temperaturearray"/></adios‐group><methodgroup="temperature"method="POSIX"/><buffersize‐MB="1"allocate‐time="now"/></adios‐config>
Figure14.Exampleconfig.xmlfile
11.3 Generate.chfile(s)The adios_group_size function and a set of adios_write functions can beautomatically generated in gwrite_temperature.ch file by using the followingpythoncommand(seeChap.10):gpp.py config.xml Thegeneratedgwrite_temperature.chfileisgiveninError!Referencesourcenotfound.Figure15.
/*gwrite_temperature.ch*/
50
adios_groupsize=4\+8*(NX);adios_group_size(adios_handle,adios_groupsize,&adios_totalsize);adios_write(adios_handle,"NX",&NX);adios_write(adios_handle,"temperature",t);
Figure15.Examplegwrite_temperature.chfile
11.4 POSIXtransportmethod(Pwriters,Pfiles)Forourfirstprogram,wesimplytranslatetheprogramofFigure13,sothatalloftheI/OoperationsaredonewithADIOSroutines.ThePOSIXmethodcanbeusedtowrite out separate files for each processor inError! Reference source notfound.Figure16.Thechangestotheoriginalexamplearehighlighted.WeusetheMPI_COMM_WORLD as a communicator only for the sake of the next example.ThePOSIXmethoddoesnotcommunicatebetweentheprocesses.
/*writeSeparatefileforeachprocessbyusingPOSIX*/#include<stdio.h>#include"mpi.h"#include"adios.h"intmain(intargc,char**argv){
charfilename[256];intrank;intNX=10;doublet[NX];/*ADIOSvariablesdeclarationsformatchinggwrite_temperature.ch*/intadios_err;uint64_tadios_groupsize,adios_totalsize;int64_tadios_handle;MPI_Comm*comm=MPI_COMM_WORLD;MPI_Init(&argc,&argv);MPI_Comm_rank(MPI_COMM_WORLD,&rank);sprintf(filename,"restart.bp");adios_init("config.xml");adios_open(&adios_handle,"temperature",filename,"w",&comm);#include"gwrite_temperature.ch"adios_close(adios_handle);adios_finalize(rank);MPI_Finalize();return0;
}
51
Figure16.ExampleadiosprogramtowritePfilesfromPprocessors(examples/C/manual/2_adios_write.c)
ThePOSIXmethodappendstherankoftheprocess(accordingtoMPI_COMM_WORLD)tothenameofthefile(hererestart.bp),soforexampleprocess2willwriteafilerestart.bp.2.
$ mpirun -np 4 2_adios_write $ ls restart.bp.* restart.bp.0 restart.bp.1 restart.bp.2 restart.bp.3 $ bpls -lad restart.bp.2 -n 10 integer /NX scalar = 10 double /temperature {10} = 20 / 29 (0) 20 21 22 23 24 25 26 27 28 29 string /temperature/description attr = "Temperature array"
11.5 MPI‐IOtransportmethod(Pwriters,1file)Basedonthesamegroupdescriptionintheconfigurefileandtheheaderfile(.ch)generated by python script, we can switch among different transportmethodswithoutchangingorrecompilingthesourcecode.
Oneentrychangeintheconfig.xmlfilecanswitchfromPOSIXtoMPI:<methodgroup=”temperature”method=”MPI”/>
TheMPI communicator ispassedas anargumentof adios_open().Because it isdefinedasMPI_COMM_WORLD in theposix examplealready, theprogramdoesnotneedtobemodifiedorrecompiled.
$ mpirun -np 4 2_adios_write $ ls restart.bp restart.bp $ bpls -l restart.bp Group temperature: integer /NX scalar = 10 double /temperature {10} = 0 / 39
Thereareseveralwaystoverifythebinaryresults.Wecaneitherchoosebpdumptodisplaythecontentofthefileoruseoneoftheconverters(bp2ncd,bp2h5,orbp2ascii), to produce the user’s preferred file format (NetCDF, HDF5 or ASCII,respectively) and use its dump utility to output the content in the standardoutput. Bpls cannot list the individual arrayswritten by the processes becausethegenericreadAPIitusesdoesnotsupportthis(itcanseeonlyoneofthemasthesizeof/temperaturesuggestinthelistingabove).Itissuggestedtouseglobalarrays(seeexamplebelow)topresentthedatawrittenbymanyprocessesasoneglobalarray,whichthencanbelistedandanysliceofitcanberead/dumped.
52
This example, however, can be used for checkpoint/restart files where theapplicationwouldonlyreadindatafromthesamenumberofprocessesasitwaswritten(seenextexample).ThetransparentswitchbetweenthePOSIXandMPImethods allows the user choose the better performingmethod for a particularsystemwithoutchangingthesourcecode.
11.6 ReadingdatafromthesamenumberofprocessorsNow let’s move to examples of how to read the data from BP or other files.Assuming that we still use the same configure file shown in Figure 14, thefollowing steps illustrate how to easily change the code and xml file to read avariable.
1. addanothervariableadios_buf_sizespecifyingthesizeforread.
2. calladios_openwith“r”(readonly)mode.
3. Insert#include“gread_temperature.ch”
/*Readindataonsamenumberofprocessors*/#include<stdio.h>#include"mpi.h"#include"adios.h"intmain(intargc,char**argv){
charfilename[256];intrank;intNX=10;doublet[NX];/*ADIOSvariablesdeclarationsformatchinggread_temperature.ch*/intadios_err;uint64_tadios_groupsize,adios_totalsize,adios_buf_size;int64_tadios_handle;MPI_Commcomm=MPI_COMM_WORLD;MPI_Init(&argc,&argv);MPI_Comm_rank(MPI_COMM_WORLD,&rank);sprintf(filename,"restart.bp");adios_init("config.xml");adios_open(&adios_handle,"temperature",filename,"r",&comm);#include"gread_temperature.ch"adios_close(adios_handle);adios_finalize(rank);MPI_Finalize();return0;
}
53
Figure17.Readindatageneratedby2_adios_writeusinggread_temperature.ch(examples/C/manual/3_adios_read.c)
Thegread_temperature.chfilegeneratedbygpp.pyisthefollowing:
/*gread_temperature.ch*/adios_group_size(adios_handle,adios_groupsize,&adios_totalsize);adios_buf_size=4;adios_read(adios_handle,"NX",&NX,adios_buf_size);adios_buf_size=NX;adios_read(adios_handle,"temperature",t,adios_buf_size);
Figure18.Exampleofageneratedgread_temperature.chfile
11.7 WritingtoSharedFiles(Pwriters,Nfiles)As the number of processes increases to tens or hundreds of thousands, theamountoffileswillincreasebythesamemagnitudeifweusethePOSIXmethodorasinglesharedfilemaybetoolargeifweusetheMPImethod.Inthisexamplewe address a scenario in which multiple processes write to N files. In thefollowing example (Figure 19), we write out N files from P processes. This isachieved by creating a separate communicator for N subsets of the processesusingMPI_Comm_split().
#include<stdio.h>#include"mpi.h"#include"adios.h"intmain(intargc,char**argv){
charfilename[256];intrank,size;intNX=10;intN=3;doublet[NX];/*ADIOSvariablesdeclarationsformatchinggwrite_temperature.ch*/intadios_err;uint64_tadios_groupsize,adios_totalsize;int64_tadios_handle;MPI_Commcomm;/*intcolor,key;MPI_Init(&argc,&argv);MPI_Comm_rank(MPI_COMM_WORLD,&rank);
54
MPI_Comm_size(MPI_COMM_WORLD,&size);/*MPI_Comm_splitpartitionstheworldgroupintoNdisjointedsubgroups,*theprocessesarerankedintermsoftheargumentkey*anewcommunicatorcommisreturnedforthisspecificgridconfiguration*/color=rank%N;key=rank/N;MPI_Comm_split(MPI_COMM_WORLD,color,key,&comm);/*everyP/Nprocesseswriteintothesamefile*thereareNfilesgenerated.*/sprintf(filename,"restart_%5.5d.bp",color);adios_init("config.xml");adios_open(&adios_handle,"temperature",filename,"w",&comm);#include"gwrite_temperature.ch"adios_close(adios_handle);adios_finalize(rank);MPI_Finalize();return0;
}
Figure19.ExampleADIOSprogramwritingNfilesfromPprocessors(N)
The reconstructed MPI communicator comm is passed as an argument of theadios_open()call.Therefore,inthisexample,eachfileiswrittenbytheprocessesinthesamecommunicationdomain.
ThereisnoneedtochangetheXMLfileinthiscasebecausewearestillusingtheMPImethod.
11.8 GlobalArraysIf each process writes out a sub‐array that belongs to the same global space,ADIOSprovidesthewaytowriteoutglobalinformationsothegenericreadAPIcanseeasingleglobalarray(andalsotheHDF5orNetCDFfilewhenusingourconverters). This example demonstrates how towrite global arrays,where thenumberofprocessesbecomesaseparatedimension.EachprocessiswritingtheonedimensionaltemperaturearrayofsizeNXandtheresultisatwodimensionalarrayofsizePxNX.Figure20showshowtodefineaglobalarrayintheXMLfile.
<?xmlversion="1.0"?><adios‐confighost‐language="C"><adios‐groupname="temperature"coordination‐communicator="comm"><varname="NX"type="integer"/>
55
<varname="size"type="integer"/><varname="rank"type="integer"/><global‐boundsdimensions="size,NX"offsets="rank,0"><varname="temperature"gwrite="t"type="double"dimensions="1,NX"/></global‐bounds><attributename="description"path="/temperature"value="Globalarraywrittenfrom'size'processes"type="string"/></adios‐group><methodgroup="temperature"method="MPI"/><buffersize‐MB="2"allocate‐time="now"/></adios‐config>
Figure20.Config.xmlforaglobalarray(examples/C/global‐array/adios_global.xml)
Thevariable is inserted intoa<global‐bounds>…</global‐bounds> section.Theglobalarray’sglobaldimensionisdefinedbythevariablessizeandNX,availableinallprocessesandallwiththesamevalue.Theoffsetofalocalarraywrittenbyaprocessisdefinedusingtherankvariable,whichisdifferentoneveryprocess.Thevariableitselfisdefinedasan1xNXtwodimensionalarray,althoughintheCcodeitisstillaonedimensionalarray.Thegwriteheaderfilegeneratedbygpp.pyisthefollowing:
/*gwrite_temperature.ch*/adios_groupsize=4\+4\+4\+8*(1)*(NX);adios_group_size(adios_handle,adios_groupsize,&adios_totalsize);adios_write(adios_handle,"NX",&NX);adios_write(adios_handle,"size",&size);adios_write(adios_handle,"rank",&rank);adios_write(adios_handle,"temperature",t);
Figure21.gwriteheaderfilegeneratedfromconfig.xml
Theprogramcodeisnotverydifferentfromtheoneusedintheaboveexample.Itneeds to have the size and rank variables in the code defined (seeexamples/C/global‐array/adios_global.c)
$ mpirun -np 4 ./adios_global
56
$ ls adios_global.bp adios_global.bp $ bpls -latd adios_global.bp -n 10 integer /NX scalar = 10 integer /rank scalar = 0 integer /size scalar = 4 double /temperature {4, 10} = 0 / 39 (0,0) 0 1 2 3 4 5 6 7 8 9 (1,0) 10 11 12 13 14 15 16 17 18 19 (2,0) 20 21 22 23 24 25 26 27 28 29 (3,0) 30 31 32 33 34 35 36 37 38 39 string /temperature/description attr = "Global array written from 'size' processes"
Thebp2ncdutilitycanbeusedtoconvertthebpfiletoanNetCDFfile:
$ bp2ncd adios_global.bp $ ncdump adios_global.nc netcdf adios_global { dimensions: NX = 10 ; size = 4 ; rank = 1 ; variables: double temperature(size, NX) ; temperature:description = "Global array written from \'size\' processes" ; data: temperature = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39 ; }
Theexamples/C/global‐array/adios_read_global.cprogramshowshowtousethegenericreadAPItoreadintheglobalarrayfromarbitrarynumberofprocesses.
11.9 WritingTime‐IndexintoaVariableThetime‐indexallowstheusertodefineavariablewithanunlimiteddimension,alongwhichthevariablecangrowintime.Let’ssupposetheuserwantstowriteout temperature after a certain number of iterations. First, we add the “time‐index” attribute to the adios‐groupwith an arbitraryname, e.g. “iter”.Next,wefind the (global) variable temperature in the adios‐group and add “iter” as anextradimension for it; therecordnumber for thatvariablewillbestoredeverytimeitgetswrittenout.Notethatwedonotneedtochangethedimensionsand
57
offsetsintheglobalbounds,onlytheindividualvariable.Alsonote,thatthetimedimensionmustbetheslowestchangingdimension,i.e.inC,thefirstoneandinFortran,itmustbethelastone.
/*config.xml*/<adios‐confighost‐language="C"><adios‐groupname="temperature"coordination‐communicator="comm"time‐index=”iter”><varname="NX"type="integer"/><varname="size"type="integer"/><varname="key"type="integer"/><global‐boundsdimensions="size,NX"offsets="key,0"><varname="temperature"gwrite="t"type="double"dimensions="iter,1,NX"/>(Note,forFortran,“iter”needstobeputintheend,i.e.,dimension=”NX,1,iter”)</global‐bounds><attributename="description"path="/temperature"value="Globalarraywrittenfrom'size'processesoverseveraltimesteps"type="string"/></adios‐group><methodgroup="temperature"method="MPI"/><buffersize‐MB="1"allocate‐time="now"/></adios‐config>
Figure22.Config.xmlforaglobalarraywithtime(examples/C/global‐array‐time/adios_globaltime.xml)
The examples/C/global‐array‐time/adios_globaltime.c is similar to thepreviousexample adios_global.c code.Theonlydifference is that it has an iteration loopwhereeachprocesswritesoutthedataineachofits13iterations. $ mpirun -np 2 ./adios_read_globaltime $ bpls -la adios_globaltime.bp Group temperature: integer /NX scalar = 10 integer /size scalar = 4 integer /rank scalar = 0 double /temperature {13, 4, 10} = 0 / 1239 string /temperature/description attr = "Global array written from 'size' processes over several timesteps"
Asliceoftwotimesteps(6thand7th),dumpedwithbpls:
58
$ bpls adios_globaltime.bp -s "5,0,0" -c "2,-1,-1" -n 10 -d temperature double /temperature {13, 4, 10} slice (5:6, 0:3, 0:9) (5,0,0) 500 501 502 503 504 505 506 507 508 509 (5,1,0) 510 511 512 513 514 515 516 517 518 519 (5,2,0) 520 521 522 523 524 525 526 527 528 529 (5,3,0) 530 531 532 533 534 535 536 537 538 539 (6,0,0) 600 601 602 603 604 605 606 607 608 609 (6,1,0) 610 611 612 613 614 615 616 617 618 619 (6,2,0) 620 621 622 623 624 625 626 627 628 629 (6,3,0) 630 631 632 633 634 635 636 637 638 639
59
12 DeveloperManual
12.1 CreateNewTransportMethodsOneofADIOS’simportantfeaturesisthecomponentizationoftransportmethods.Userscanswitchamongthetypicalmethodsthatwesupportorevencreatetheirown methods, which can be easily plugged into our library. The followingsections provide the procedures for adding the new transport method called“abc” into the ADIOS library. In this version of ADIOS, all the source files arelocatedin/trunk/src/.
12.1.1 Addthenewmethodmacrosinadios_transport_hooks.hThe first file users need to examine is adios_transport_hooks.h,which basicallydefines all the transport methods and interface functions between detailedtransport implementation and user APIs. In the file, we first find the line thatdefinestheenumerationtypeAdios_IO_methods_datatypeaddthedeclarationofmethod ID ADIOS_METHOD_ABC, and, because we add a new method, updatetotalnumberoftransportmethodsADIOS_METHOD_COUNTfrom9to10.
1.enumAdios_IO_methodsdatatype
enumADIOS_IO_METHOD{ADIOS_METHOD_UNKNOWN=‐2,ADIOS_METHOD_NULL=‐1,ADIOS_METHOD_MPI=0……,ADIOS_METHOD_PHDF5=8ADIOS_METHOD_ABC=9,ADIOS_METHOD_COUNT=9ADIOS_METHOD_COUNT=10};2. Next, we need to declare the transport APIs for method “abc,” includinginit/finalize, open/close, should_buffer, and read/write. Similar to the othermethods,weneedtoadd
FORWARD_DECLARE(abc)
3. Then, we add the mapping of the string name “abc” of the new transportmethodtothemethodID‐ADIOS_METHOD_ABC,whichhasbeenalreadydefinedinenumerationtypeAdios_IO_methods_datatype.Asthelastparameter,“1”heremeansthemethodrequirescommunications,or“0”ifnot.
MATCH_STRING_TO_METHOD("abc",ADIOS_METHOD_ABC,1)
60
4. Lastly, we add the mapping of the string name needed in the initializationfunctions to the method ID, which will be used by adios_transport_structvariablesdefinedinadios_internals.h.
ASSIGN_FNS(abc,ADIOS_METHOD_ABC)
12.1.2 Createadios_abc.cIn this section,we demonstrate how to implement different transport APIs formethod“abc.”Inadios_abc.c,weneedtoimplementatleast11requiredroutines:
1. “adios_abc_init”allocates themethod_data field inadios_method_struct to theuser‐defined transport data structure, such as adios_abc_data_struct, andinitializesthisdatastructure.Beforethefunctionreturns,theinitializationstatuscanbesetbystatement“adios_abc_initialized=1.”
2.“adios_abc_open”opensafileifthereisonlyoneprocessorwritingtothefile.Otherwise,thisfunctiondoesnothing;instead,weuseadios_abc_should_buffertocoordinatethefileopenoperations.
3. “adios_abc_should_buffer,” called by the “common_adios_group_size” functioninadios.c,needstoincludecoordinationofopenoperationsifmultipleprocessesarewritingtothesamefile.
4.“adios_abc_write”,inthecaseofnobufferingoroverflow,writesdatadirectlytodisk.Otherwise,itverifieswhethertheinternallyrecordedmemorypointerisconsistentwith the vector variable’s address passed in the function parameterandfreesthatblockofmemoryifitisnotneededanymore.
5.“adios_abc_read”associatestheinternaldatastructure’saddresstothevariablespecifiedinthefunctionparameter.
6.“adios_abc_close”closesthefileifnobufferingschemeisused.Otherwise,thisfunction needs extra effort to perform the actual diskwriting/reading to/fromthe filebyoneormoreprocessors in thesamecommunicatordomainandthenclosethefile.
7.“adios_abc_finalize”resetstheinitializationstatusbackto0ifithasbeensetto1byadios_abc_init.
Ifyouaredevelopingasynchronousmethods,thefollowingfunctionsneedtobeimplementedaswell;otherwiseyoucanleavethemasemptyimplementation.
8.adios_abc_get_write_buffer,
9.“adios_abc_end_iteration“isatickcounterfortheI/Oroutinestotimehowfasttheyareemptyingthebuffers.
10.“adios_abc_start_calculation”indicatesthatitisnowanidealtimetodobulkdatatransfersbecausethecodewillnotbeperformingI/Oforawhile.
61
11. “adios_abc_stop_calculation“ indicates that bulk data transfers should ceasebecausethecodeisabouttostartcommunicatingwithothernodes.
ThefollowingisOneofthemostimportantthingsthatneedstobenoted:
fd‐>shared_buffer=adios_flag_no,
whichmeansthatthemethodsdonotneedabufferingscheme,suchasPHDF5,andthatdatawriteoutoccursimmediatelyonceadios_writereturns.
If fd‐>shared_buffer = adios_flag_yes, the users can employ the self‐definedbufferingschemetoimproveI/Operformance.
12.1.3 Awalk‐throughexampleNow let’s lookat an exampleof addinganunbufferedPOSIXmethod toADIOS.According to the steps described above, we first open the header file ‐‐“adios_transport_hooks.h,”andaddthefollowingstatements:
• enumADIOS_IO_METHOD{
ADIOS_METHOD_UNKNOWN=‐2,ADIOS_METHOD_NULL=‐1,ADIOS_METHOD_MPI=0…,ADIOS_METHOD_PROVENANCE=8//methodIDforbinarytransportmethod,ADIOS_METHOD_POSIX_ASCII_NB=9//totalmethodnumber,ADIOS_METHOD_COUNT=10};
• FORWARD_DECLARE(posix_ascii_nb);
• MATCH_STRING_TO_METHOD("posix_ascii_nb"
,ADIOS_METHOD_POSIX_ASCII_NB,0)
• ASSIGN_FNS(binary,ADIOS_METHOD_POSIX_ASCII_NB)
Next, we must create adios_posix_ascii_nb,c, which defines all the requiredroutines listed in Sect. 12.1.2 The blue highlights below mark out the datastructures and required functions that developers need to implement in thesourcecode.
staticintadios_posix_ascii_nb_initialized=0;
structadios_POSIX_ASCII_UNBUFFERED_data_struct{
62
FILE*f;uint64_tfile_size;};
voidadios_posix_ascii_nb_init(constchar*parameters,structadios_method_struct*method){structadios_POSIX_ASCII_UNBUFFERED_data_struct*md;if(!adios_posix_ascii_nb_initialized){adios_posix_ascii_nb_initialized=1;}method‐>method_data=malloc(sizeof(structadios_POSIX_ASCII_UNBUFFERED_data_struct));md=(structadios_POSIX_ASCII_UNBUFFERED_data_struct*)method‐>method_data;md‐>f=0;md‐>file_size=0;}
intadios_posix_ascii_nb_open(structadios_file_struct*fd,structadios_method_struct*method){char*name;structadios_POSIX_ASCII_UNBUFFERED_data_struct*p;structstats;p=(structadios_POSIX_ASCII_UNBUFFERED_data_struct*)method‐>method_data;name=malloc(strlen(method‐>base_path)+strlen(fd‐>name)+1);sprintf(name,"%s%s",method‐>base_path,fd‐>name);if(stat(name,&s)==0)p‐>file_size=s.st_size;switch(fd‐>mode){caseadios_mode_read:{p‐>f=fopen(name,"r");if(p‐>f<=0){fprintf(stderr,"ADIOSPOSIXASCIIUNBUFFERED:""filenotfound:%s\n",fd‐>name);free(name);return0;}
63
break;}caseadios_mode_write:{p‐>f=fopen(name,"w");if(p‐>f<=0){fprintf(stderr,"adios_posix_ascii_nb_open""failedforbase_path%s,name%s\n",method‐>base_path,fd‐>name);free(name);return0;}break;}caseadios_mode_append:{intold_file=1;p‐>f=fopen(name,"a");if(p‐>f<=0){fprintf(stderr,"adios_posix_ascii_nb_open""failedforbase_path%s,name%s\n",method‐>base_path,fd‐>name);free(name);return0;}break;}default:{fprintf(stderr,"Unknownfilemode:%d\n",fd‐>mode);free(name);return0;}}free(name);return0;}enumADIOS_FLAGadios_posix_ascii_nb_should_buffer(structadios_file_struct*fd,structadios_method_struct*method,void*comm){
64
//inthiscase,wedon’tuseshared_bufferreturnadios_flag_no;}voidadios_abc_write(structadios_file_struct*fd,structadios_var_struct*v,void*data,structadios_method_struct*method){structadios_POSIX_ASCII_UNBUFFERED_data_struct*p;p=(structadios_POSIX_ASCII_UNBUFFERED_data_struct*)method‐>method_data;if(!v‐>dimensions){switch(v‐>type){caseadios_byte:caseadios_unsigned_byte:fprintf(p‐>f,"%c\n",*((char*)data));break;caseadios_short:caseadios_integer:caseadios_unsigned_short:caseadios_unsigned_integer:fprintf(p‐>f,"%d\n",*((int*)data));break;caseadios_real:caseadios_double:caseadios_long_double:fprintf(p‐>f,"%f\n",*((double*)data));break;caseadios_string:fprintf(p‐>f,"%s\n",(char*)data);break;caseadios_complex:fprintf(p‐>f,"%f+%fi\n",*((float*)data),*((float*)(data+4)));break;caseadios_double_complex:fprintf(p‐>f,"%f+%fi\n",*((double*)data),*((double*)(data+8)));break; default:break; }}else{uint64_tj;intelement_size=adios_get_type_size(v‐>type,v‐>data);
65
printf("element_size:%d\n",element_size);uint64_tvar_size=adios_get_var_size(v,fd‐>group,v‐>data)/element_size;switch(v‐>type){caseadios_byte:caseadios_unsigned_byte:for(j=0;j<var_size;j++)fprintf(p‐>f,"%c",*((char*)(data+j)));printf("\n");break;caseadios_short:caseadios_integer:caseadios_unsigned_short:caseadios_unsigned_integer:for(j=0;j<var_size;j++)fprintf(p‐>f,"%d",*((int*)(data+element_size*j)));printf("\n");break;caseadios_real:caseadios_double:caseadios_long_double:for(j=0;j<var_size;j++)fprintf(p‐>f,"%f",*((double*)(data+element_size*j)));printf("\n");break;caseadios_string:for(j=0;j<var_size;j++)fprintf(p‐>f,"%s",(char*)data);printf("\n");break;caseadios_complex:for(j=0;j<var_size;j++)fprintf(p‐>f,"%f+%fi",*((float*)(data+element_size*j)),*((float*)(data+4+element_size*j)));printf("\n");break;caseadios_double_complex:for(j=0;j<var_size;j++)fprintf(p‐>f,"%f+%fi",*((double*)(data+element_size*j)),*((double*)(data+element_size*j+8)));printf("\n");break;default:break;}
66
}}voidadios_posix_ascii_nb_get_write_buffer(structadios_file_struct*fd,structadios_var_struct*v,uint64_t*size,void**buffer,structadios_method_struct*method){*buffer=0;}
voidadios_posix_ascii_nb_read(structadios_file_struct*fd,structadios_var_struct*v,void*buffer,uint64_tbuffer_size,structadios_method_struct*method){v‐>data=buffer;
v‐>data_size=buffer_size;}
intadios_posix_ascii_nb_close(structadios_file_struct*fd,structadios_method_struct*method){structadios_POSIX_ASCII_UNBUFFERED_data_struct*p;p=(structadios_POSIX_ASCII_UNBUFFERED_data_struct*)method‐>method_data;if(p‐>f<=0){fclose(p‐>f);}p‐>f=0;p‐>file_size=0;}
voidadios_posix_finalize(intmype,structadios_method_struct*method){if(adios_posix_ascii_nb_initialized)adios_posix_ascii_nb_initialized=0;}
Thebinarytransportmethodblocksmethodsforsimplicity.Therefore,nospecialimplementation for the three functions below is necessary and their functionbodiescanbeleftempty:
67
adios_abc_end_iteration(structadios_method_struct*method){}adios_abc_start_calculation(structadios_method_struct*method){}adios_abc_stop_calculation(structadios_method_struct*method){}Untilnow,wehaveimplementedthePOSIX_ASCIItransportmethod.WhenusersspecifyPOSIX_ASCII_NBmethodinxml file, theusers’applicationswillgenerateASCIIfilesbyusingcommonADIOSAPIs.However,inordertoachievebetterI/Operformance,abufferingschemeneedstobeincludedintothisexample.
12.2 ProfilingtheApplicationandADIOSThere are twoways to get profiling information of ADIOS I/O operations. OnewayisfortheusertoexplicitlyinsertasetofprofilingAPIcallsaroundADIOSAPIcalls in thesourcecode.Theotherway is to link theusercodewitharenamedADIOSlibraryandanADIOSAPIwrapperlibrary.
12.2.1 UseprofilingAPIinsourcecodeThe profiling library called libadios_timing.a implements a set of profiling APIcalls.TheusercanusetheseAPIcallstowraptheADIOSAPIcallsinthesourcecodetogetprofilinginformation.
The adios‐timing.h header file contains the declarations of those profilingfunctions.
/**initializeprofiling**Fortraninterface*/intinit_prof_all_(char*prof_file_name,intprof_file_name_size);/**recordopenstarttimeforspecifiedgroup**Fortraninterface*/voidopen_start_for_group_(int64_t*gp_prof_handle,char*group_name,int*cycle,int*gp_name_size);/**recordopenendtimeforspecifiedgroup**Fortraninterface*/voidopen_end_for_group_(int64_t*gp_prof_handle,int*cycle);/*
68
*recordwritestarttimeforspecifiedgroup**Fortraninterface*/voidwrite_start_for_group_(int64_t*gp_prof_handle,int*cycle);/**recordwriteendtimeforspecifiedgroup**Fortraninterface*/voidwrite_end_for_group_(int64_t*gp_prof_handle,int*cycle);/**recordclosestarttimeforspecifiedgroup**Fortraninterface*/voidclose_start_for_group_(int64_t*gp_prof_handle,int*cycle);/**recordcloseendtimeforspecifiedgroup**Fortraninterface*/voidclose_end_for_group_(int64_t*gp_prof_handle,int*cycle);/**Reporttiminginfoforallgroups**Fortraninterface*/intfinalize_prof_all_();/**recordstarttimeofasimulationcycle**Fortraninterface*/voidcycle_start_(int*cycle);/**recordendtimeofasimulationcycle**Fortraninterface
69
*/voidcycle_end_(int*cycle);
Anexampleofusingthesefunctionsisgivenbelow.
…!initializationADIOSCALLadios_init("config.xml"//char(0))!initializeprofilinglibrary;theparameterspecifiesthefilewhereprofilinginformationiswrittenCALLinit_prof_all("log"//char(0))…CALLMPI_Barrier(toroidal_comm,error)!recordstarttimeofopen!group_prof_handleisanOUTparameterholdingthehandleforthegroup‘output3d.0’!istepisiterationno.CALLopen_start_for_group(group_prof_handle,"output3d.0"//char(0),istep)CALLadios_open(adios_handle,"output3d.0"//char(0),“w”//char(0))!recordendtimeofopenCALLopen_end_for_group(group_prof_handle,istep)!recordstarttimeofwriteCALLwrite_start_for_group(group_prof_handle,istep)#include"gwrite_output3d.0.fh"!recordendtimeofwriteCALLwrite_end_for_group(group_prof_handle,istep)!recordstarttimeofcloseCALLcose_start_for_group(group_prof_handle,istep)CALLadios_close(adios_handle,adios_err)!recordendtimeofcloseCALLclose_end_for_group(group_prof_handle,istep)…CALLadios_finalize(myid)
70
!finalize;profilinginformationaregatheredandmin/max/mean/vararecalculatedforeachIOdumpCALLfinalize_prof()CALLMPI_FINALIZE(error)
Whenthecodeisrun,profilinginformationwillbesavedtothefile”./log”(specifiedininit_prof_all()).Belowisanexample.
FriAug2215:42:04EDT2008I/OTimingresultsOperations:minmaxmeanvarcycleno3iocount0#Open:0.1076710.1082450.1080320.000124#Openstart:1219434228.8661441219434230.7752681219434229.7486140.588501#Openend:1219434228.9742251219434230.8833351219434229.8566460.588486#Write:0.0001700.0001900.0001790.000005#Writestart:1219434228.9742261219434230.8833361219434229.8566470.588486#Writeend:1219434228.9744051219434230.8835141219434229.8568260.588484#Close:0.0016080.0017430.0016560.000036#Closestart:1219434228.9744051219434230.8835141219434229.8568260.588484#Closeend:1219434228.9760401219434230.8852111219434229.8584820.588489#Total:0.1094840.1100490.1098680.000137cycleno6iocount1#Open:0.0000070.0000110.0000090.000001#Openstart:1219434240.0984441219434242.0079511219434240.9810750.588556#Openend:1219434240.0984521219434242.0079621219434240.9810830.588556#Write:0.0001750.0001960.0001800.000004#Writestart:1219434240.0984521219434242.0079621219434240.9810830.588557#Writeend:1219434240.0986311219434242.0081581219434240.9812640.588558#Close:0.0009470.0036030.0012340.000466#Closestart:1219434240.0986311219434242.0081581219434240.9812640.588558#Closeend:1219434240.0996651219434242.0096201219434240.9824980.588447#Total:0.0011320.0037890.0014230.000466
The script “post_script.sh” extracts open time,write time, close time, and totaltimefromtherawprofilingresultsandsavestheminseparatefiles:open,write,close,andtotal,respectively.
To compile the code, one should link the code with the –ladios_timing ladiosoption.
12.2.2 UsewrapperlibraryAnother way to do profiling is to link the source code with a renamed ADIOSlibraryandawrapperlibrary.
The renamed ADIOS library implements “real” ADIOS routines, but all ADIOSpublic functions are renamed with a prefix “P”. For example, adios_open() isrenamedasPadios_open().Theroutineforparsingconfig.xmlfileisalsochangedtoparseextraflagsinconfig.xmlfiletoturnprofilingonoroff.
71
The wrapper library implements all adios pubic functions (e.g., adios_open,adios_write, adios_close) within each function. It calls the “real” function(Padios_xxx())andmeasurethestartandendtimeofthefunctioncall.
There is an examplewrapper library called libadios_profiling.a.Developers canimplementtheirownwrapperlibrarytocustomizetheprofiling.
Tousethewrapperlibrary,theusercodeshouldbelinkedwith–ladios_profiling–ladios.thewrapperlibraryshouldprecedethe“real”ADIOSlibrary.Thereisnoneed toput additional profilingAPI calls in the source code.Theuser can turnprofilingonoroffforeachADIOSgroupbysettingaflagintheconfig.xmlfile.
<adios‐groupname="restart.model"profiling=“yes|no">...</adios‐group>
72
13 Appendix
DatatypesusedintheADIOSXMLfile
size Signedtype Unsignedtype
1 byte,interger*1 unsignedbyte,unsignedinteger*1
2 short,integer*2 unsignedshort,unsignedinteger*2
4 integer,integer*4,real,real*4,float unsignedinteger,unsignedinteger*4
8 long,integer*8,real*8,double,longfloat,complex,complex*8
16 real*16,longdouble,doublecomplex,complex*16
string
ADIOSAPIsList
Function
Purpose
adios_init
Load the XML configuration file creating internal representations of the various data types and defining the methods used for writing.
adios_finalize
Cleanup anything remaining before exiting the code
adios_open
Prepare a data type for subsequent calls to write data using the io_handle. Mode is one of “r” (read), “w” (write) and “a” (append).
adios_close
Commit all the write to disk, close the file and release adios file handle
adios_group_size Passing the required buffer size to the transport layer and returned the total size back to the source code
adios_write
Submit a data element for writing. This does NOT actually perform the write in buffered mode. In the overflow case, this call writes to buffer directly.
adios_read
Submit a buffer space (var) for reading a data element into. This does NOT actually perform the read. Actual population of the buffer space will happen on the call to adios_close
73
adios_set_path
Set the HDF5-style path for all variables in a adios-group. This will reset whatever is specified in the XML file.
adios_set_path_var
Set the HDF-5-style path for the specified var in the group. This will reset whatever is specified in the XML file.
adios_get_write_buffer For the given field, get a buffer that will be used at the transport level for it of the given size. If size == 0, then auto calculate the size based on what is known from the datatype in the XML file and any provided additional elements (such as array dimension elements). To return this buffer, just do a normal call to adios_write using the same io_handle, field_name, and the returned buffer.
adios_start_calculation An indicator that it is now an ideal time to do bulk data transfers as the code will not be performing IO for a while.
adios_end_ calculation
An indicator that it is no longer a good time to do bulk data transfers as the code is about to start doing communication with other nodes causing possible conflicts
adios_end_iteration A tick counter for the IO routines to time how fast they are emptying the buffers.