introduction to hydra, · hydra basics & specifications the hydra hpc cluster is made of 80...

IntroductiontoHydra,theIFT-UAM/CSICclusterMANUELTRASHORRAS,IFT/UAM-CSIC

SCHOOLOFCOSMOLOGYTOOLS, MARCH 2017

Contents

1. WhatisHYDRA?

2. Hydrabasics&specifications

3. Login&changepasswordsinHYDRA

4. WhatisSLURM?

5. UsingSLURMandsubmittingjobs

6. Example:CompilingandrunningCosmoMC

So,whatisHYDRA?Notthis…

So,whatisHYDRA?But this.

HYDRAbasics&specifications

TheHydraHPCclusterismadeof80dualprocessorsof8or12corespernode.

• batch –>34nodestimelimit:140hnodes hydra[1-34]

• batch2 –>18nodestimelimit:140hnodes hydra[35-52]

• batch3 –>18nodestimelimit:70hnodeshydra[53-70]


TheHydraHPCclusterismadeof80dualprocessorsof8or12corespernode.

• batch –>8cores/node34nodeswithadualprocessorIntel® Xeon® E5540at2,53GHz,24GBDDR3RAM&120GBSolidStateDiskharddrive.• batch2 –>12cores/node18nodeswithadualprocessorIntel® Xeon® E5645at2.4GHz,24GBDDR3RAM&120GBSolidStateDiskharddrive.• batch3 –>12cores/node18nodeswithadualprocessorIntel® Xeon® E5-2640at2.5GHz,64GBDDR3RAM128GBSolidStateDiskharddrive.


Systemsoftwareandcompilers:◦ GNUcompiler5.2◦ Intelcompiler16◦ MKLsoftware16◦ OpenMPI software1.8.8

Defaultenvironment:GNU+autotools +OpenMPI

Softwareisaccessiblethroughamodulesystem,thatallowsuserstodynamicallymodifytheuser'senvironment,loadingandunloadingthedifferentmodules.

Thefollowingcommandareuseful◦ moduleavail –> showsthelistofavailablemodules◦ module list –>showsthelistofuser'sloadedmodules◦ module loadMODULENAME –>loadsaspecificmodule◦ modulehelp –>showshelp

HYDRAbasics&specificationsLOADEDMODULES

In/opt/ohpc/pub/moduledeps/gnu:◦ openmpi/1.8.8 (L)

In/opt/ohpc/pub/modulefiles:◦ autotools (L) gnu/5.2.0(L)◦ ohpc (L) prun/1.0 (L)

SOMEUNLOADEDMODULES

In /opt/ohpc/pub/moduledeps/gnu-openmpi:◦ fftw/3.3.4

In/opt/ohpc/pub/moduledeps/gnu:◦ getdist/0.2.7 gsl/1.16◦ hdf5/1.8.15 healpy/1.10.3◦ impi/5.1.1.109 mkl/16.0.0.109◦ mpi4py/2.0.0 mvapich2/2.1◦ numpy/1.9.2

In/opt/ohpc/pub/modulefiles:◦ intel/16.0.0.109

Login&changepasswordsinHYDRA

Tologin:

ssh –[email protected]

Yourhomedirectoryis:

/home/USERNAME

Smalljobscanbecompiled&runinhydra0interactively.

Butforlargerjobs,alwaysusethesubmittingsystem,SLURM!

Login&changepasswordsinHYDRA

Tochangepasswords:

ssh –[email protected]

>passwd

>Typeoldpassword

>Typenewpassword

Yourhomedirectoryis:

/home/USERNAME

Smalljobscanbecompiled&runinhydra0interactively.

Butforlargerjobs,alwaysusethesubmittingsystem,SLURM!

So,whatisSLURM?Notthis…

So,whatisSLURM?But this.

Source:https://slurm.schedmd.com/overview.html

UsingSLURM&submittingjobs

Tocheckthestatusofthecluster: sinfo


Tocheckthequeues: squeue


Tocheckthestatusofthecluster: sinfo

Tocheckthequeues: squeue

Tosubmitajob: sbatch JOBFILE example:sbatch script.sh

Tocancelajob: scancel JOBID example:scancel 123456

Tocheckallyourjobs: squeue –uUSERNAME example:squeue –umtrashorras

Tocancelallyourjobs: scancel –uUSERNAME example:squeue –umtrashorras

Tocheckaspecificjob: squeue –jJOBID example:squeue –j123456

UsingSLURM&submittingjobsEXAMPLESOFACOSMOMC JOB

cosmomc.sh

#!/bin/bash

#SBATCH-JJOBNAME jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”

(prun)YOURCODEHERE executethecode

cosmomc.sh

#!/bin/bash

#SBATCH-JJOBNAME jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”

(prun)YOURCODEHERE executethecode

Example:CompilingCosmoMCDOWNLOADINGCOSMOMC

1. Gotohttp://cosmologist.info/cosmomc/submit.html

2. Fillinyourdata(name,surnameande-mailaddress),andsubmit.

3. Openthee-mailyouhavereceived,andclickthelinkinit.

4. DownloadthelatestversionofCosmoMC:cosmomc_Jul15_1.tar.gz

Example:CompilingCosmoMCDOWNLOADINGTHEPLANCKLIKELIHOOD

1. Gotohttp://pla.esac.esa.int/pla/#cosmology&clickSIGNIN andREGISTER.

2. Createyourselfanaccount.Fillinyourname,surname,e-mailandsubmit.

3. Openthee-mailyouhavereceived,andcopytheauthenticationcode.

4. Openanothermailyouhavereceived,andcreateyourselfapassword.

5. Signinwithyournewaccount,anddownload:Thelikelihood code:COM_Likelihood_Code-v2.0_R2.00.tar.bz2Thedatafiles:COM_Likelihood_Data-baseline_R2.00.tar.gz(containshigh_l, low_l,lensing)

Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD

*Exampleofworkingdirectory:/home/USERNAME/1. CopythePlancklikelihood code,Plancklikelihood dataandCosmoMC toyourpath.◦ COM_Likelihood_Code-v2.0_R2.00.tar.bz2◦ COM_Likelihood_Data-baseline_R2.00.tar.gz◦ cosmomc_Jul15_1.tar.gz

2. Uncompress thePlancklikelihood code,Plancklikelihood dataandCosmoMC.1. cd/home/USERNAME/2. tar-xvf COM_Likelihood_Code-v2.0_R2.00.tar.bz2 –––>whichcreatesafoldernamedplc-2.03. tar-xvf COM_Likelihood_Data-baseline_R2.00.tar.gz –––>whichcreatesafoldernamedplc_2.04. tar-xvf cosmomc_Jul15_1.tar.gz –––>whichcreatesafoldernamedcosmomc


*CompilingthePlanklikelihood:

3. Addthistoyour.bashr fileorexecutebeforeyoucompile/runCosmoMC1. source/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/mpi/intel64/bin/mpivars.sh2. source/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/bin/compilervars.sh intel64

4. InstallthePlancklikelihood1. cd/home/USERNAME/plc-2.02. ./waf configure--install_all_deps --lapack_mkl_version=10.3

--lapack_mkl=/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/mkl./waf install

5. PutasymboliclinktothePlanckdatainCosmoMC data:1. cd/2. ln-s/home/USERNAME/plc_2.0/home/USERNAME/cosmomc/data/clik


*CompilingCosmoMC:6. ModifytheCosmoMC Makefile

3. cd/home/USERNAME/cosmomc/source/4. viMakefile replaceinline7: MPIF90C?=mpif90–––> MPIF90C?=mpif90-f90=ifort

7. CompileCosmoMC3. makeclean4. make

8. Runatoychain(leaveitrunningforaminute,this isnotaparallelrun).Done!😎3. cd..4. vitest.ini replaceinline37: action=0 (test) –––> action=4(fullMCMC)5. ./cosmomc test.ini

Example:SendingaCosmoMC jobEXAMPLESOFACOSMOMC JOB

cosmomc.sh

#!/bin/bash

#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”

./cosmomc test.ini executethecode

cosmomc.sh

#!/bin/bash

#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”

./cosmomc test.ini executethecode

Example:SendingaCosmoMC jobEXAMPLEOFAPARALELLJOB

cosmomc.sh

#!/bin/bash

#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”export$OMP_NUM_THREADS=8

prun–n1./cosmomc test.ini executethecode

cosmomc.sh

#!/bin/bash

#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”export$OMP_NUM_THREADS=16

prun2./cosmomc test.ini executethecode

Example:CosmoMC outputOutput files:test_1.txt,test_2.txt,…,test.inputparams,test.ranges,test.paramnames,test.likelihoods.

UsefullinksSLURM Guide: https://slurm.schedmd.com/quickstart.html

ModulesGuide: http://modules.sourceforge.net/

Runjobsinteractively: www.ift.uam-csic.es/hydra/quickstart_uHydra_interactive.pdf

Runjobsinparalell: www.ift.uam-csic.es/hydra/quickstart_uHydra_batch.pdf

CosmoCoffee wiki: http://cosmocoffee.info/

CosmoMCReadme: http://cosmologist.info/cosmomc/readme.html

PlanckReadme: http://cosmologist.info/cosmomc/readme_planck.html

PythonReadme: http://cosmologist.info/cosmomc/readme_python.html

GetDist GUIReadme: http://cosmologist.info/cosmomc/readme_gui.html

introduction to hydra, · hydra basics & specifications the hydra hpc cluster is made of 80...

Documents