Linux Clusters InstituteLmod A Modern

Environment Module SystemRobert McLay Manager Software Tools TACC

1

Outlinebull What are Environment Modulesbull Kinds of Module Tools Tmod Cmod Lmod hellipbull What is Lmodbull Modulefile Architecturebull Advanced Topicsbull Where to go for help

May 2017 2

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 3

Outlinebull What are Environment Modulesbull Kinds of Module Tools Tmod Cmod Lmod hellipbull What is Lmodbull Modulefile Architecturebull Advanced Topicsbull Where to go for help

May 2017 2

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 3

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 3

What are Modulesbull Modules are the way sites provide optional software MPI Boost bull Modules add to PATH and set other env vars for each packagebull Modules can be unloadedbull Modulefiles are in one file not one for each shell (eg Compiler init scripts)

May 2017 4

What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles

May 2017 5

Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the

same compilerbull Similarly MPI Applications must build and link with same MPICompiler

pairing when using pre-built MPI libraries

May 2017 6

Example of Lmod Environment Modules (I)

$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------

petsc31 (default) petsc31-debug pmetis40 tau2203

------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143

boost1460 mpich2132 openmpi151

boost1461 (default) mpich214 (default) openmpi153 (default)

-------------------------- optappsmodulefilesCore ----------------------

StdEnv intel111 papi414

adminadmin-10 intel120 (default) scite228

ddtddt lmodlmod tex2010

dmallocdmalloc locallocal (default) unixunix (default)

fdepend12 mklmkl visitvisit

gcc44 noweb211b

gcc45 (default)

May 2017 7

Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31

$ module unload gccInactive Modules1) mpich2 2) petsc

$ module load intelActivating Modules1) mpich2 2) petsc

$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc

May 2017 8

Why You Might Want To Use Lmod

bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready

May 2017 9

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles

May 2017 5

Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the

same compilerbull Similarly MPI Applications must build and link with same MPICompiler

pairing when using pre-built MPI libraries

May 2017 6

Example of Lmod Environment Modules (I)

$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------

petsc31 (default) petsc31-debug pmetis40 tau2203

------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143

boost1460 mpich2132 openmpi151

boost1461 (default) mpich214 (default) openmpi153 (default)

-------------------------- optappsmodulefilesCore ----------------------

StdEnv intel111 papi414

adminadmin-10 intel120 (default) scite228

ddtddt lmodlmod tex2010

dmallocdmalloc locallocal (default) unixunix (default)

fdepend12 mklmkl visitvisit

gcc44 noweb211b

gcc45 (default)

May 2017 7

Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31

$ module unload gccInactive Modules1) mpich2 2) petsc

$ module load intelActivating Modules1) mpich2 2) petsc

$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc

May 2017 8

Why You Might Want To Use Lmod

bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready

May 2017 9

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the

same compilerbull Similarly MPI Applications must build and link with same MPICompiler

pairing when using pre-built MPI libraries

May 2017 6

Example of Lmod Environment Modules (I)

$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------

petsc31 (default) petsc31-debug pmetis40 tau2203

------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143

boost1460 mpich2132 openmpi151

boost1461 (default) mpich214 (default) openmpi153 (default)

-------------------------- optappsmodulefilesCore ----------------------

StdEnv intel111 papi414

adminadmin-10 intel120 (default) scite228

ddtddt lmodlmod tex2010

dmallocdmalloc locallocal (default) unixunix (default)

fdepend12 mklmkl visitvisit

gcc44 noweb211b

gcc45 (default)

May 2017 7

Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31

$ module unload gccInactive Modules1) mpich2 2) petsc

$ module load intelActivating Modules1) mpich2 2) petsc

$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc

May 2017 8

Why You Might Want To Use Lmod

bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready

May 2017 9

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Example of Lmod Environment Modules (I)

$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------

petsc31 (default) petsc31-debug pmetis40 tau2203

------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143

boost1460 mpich2132 openmpi151

boost1461 (default) mpich214 (default) openmpi153 (default)

-------------------------- optappsmodulefilesCore ----------------------

StdEnv intel111 papi414

adminadmin-10 intel120 (default) scite228

ddtddt lmodlmod tex2010

dmallocdmalloc locallocal (default) unixunix (default)

fdepend12 mklmkl visitvisit

gcc44 noweb211b

gcc45 (default)

May 2017 7

Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31

$ module unload gccInactive Modules1) mpich2 2) petsc

$ module load intelActivating Modules1) mpich2 2) petsc

$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc

May 2017 8

Why You Might Want To Use Lmod

bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready

May 2017 9

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31

$ module unload gccInactive Modules1) mpich2 2) petsc

$ module load intelActivating Modules1) mpich2 2) petsc

$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc

May 2017 8

Why You Might Want To Use Lmod

bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready

May 2017 9

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Why You Might Want To Use Lmod

bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready

May 2017 9

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Lmod Features

bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag

May 2017 10

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Tmod vs Lmod

bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same

May 2017 11

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden

May 2017 12

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril

May 2017 13

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash

May 2017 14

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system

May 2017 15

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme

May 2017 16

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package

bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170

May 2017 17

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Problems w Flat naming scheme

bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170

bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles

May 2017 18

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack

May 2017 19

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories

bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3

May 2017 20

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

MODULEPATH

bull MODULEPATH is a colon separated list of directories containing directories and module files

bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH

bull Users can load compiler dependent modulesbull This includes MPI implementations modules

bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing

bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details

May 2017 21

Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN

bull TACC_HDF5_BINbull TACC_HDF5_INC

May 2017 22

Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua

bull load(gccmvapich2)

May 2017 23

A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh

if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore

elsemodule refresh

fibull Why is the module refresh there

May 2017 24

User Collections with SaveRestore

bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it

existsbull Users can create other collections

bull $ module save name to save it bull $ module restore name to restore it

bull Note that a collection does a module purge before restoring collection

May 2017 25

Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset

May 2017 26

Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)

May 2017 27

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua

bull load(gccmvapich2)

May 2017 23

A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh

if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore

elsemodule refresh

fibull Why is the module refresh there

May 2017 24

User Collections with SaveRestore

bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it

existsbull Users can create other collections

bull $ module save name to save it bull $ module restore name to restore it

bull Note that a collection does a module purge before restoring collection

May 2017 25

Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset

May 2017 26

Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)

May 2017 27

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh

if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore

elsemodule refresh

fibull Why is the module refresh there

May 2017 24

User Collections with SaveRestore

bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it

existsbull Users can create other collections

bull $ module save name to save it bull $ module restore name to restore it

bull Note that a collection does a module purge before restoring collection

May 2017 25

Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset

May 2017 26

Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)

May 2017 27

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

User Collections with SaveRestore

bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it

existsbull Users can create other collections

bull $ module save name to save it bull $ module restore name to restore it

bull Note that a collection does a module purge before restoring collection

May 2017 25

Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset

May 2017 26

Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)

May 2017 27

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset

May 2017 26

Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)

May 2017 27

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)

May 2017 27

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Module Naming Conventions (II)

bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries

May 2017 28

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Bash issues

bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive

shellsbull MPI jobs start an interactive shell

May 2017 29

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Bash Issues (II)

bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc

May 2017 30

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Bash Repair Choices

bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade

May 2017 31

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Keeping Software for Life of Cluster or Not

bull It is possible with a shared disk approachbull You might want to hide older modules

May 2017 32

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Hidden modules

bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc

May 2017 33

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Using System MODULERC to hide modules

bull In $MODULERCFILE or applmodetcrcModulehide-version foo32

May 2017 34

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Tracking Module Usage

bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details

May 2017 35

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Usage counts

$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19

May 2017 36

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Distinct Users

$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5

May 2017 37

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Why does Lmod work at all

bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all

May 2017 38

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

The trick is

bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)

May 2017 39

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Why is this important

bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt

May 2017 40

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Tracing Lmod

bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work

May 2017 41

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

How to trace Lmod startup behavior

bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on

LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead

May 2017 42

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

SHELL STARTUP DEBUG Package

bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc

May 2017 43

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

SHELL STARTUP DEBUG (II)

bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output

etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338

Time = 03156

May 2017 44

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

SHELL STARTUP DEBUG and LMOD_TRACING

bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues

running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)

May 2017 45

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Advanced Topics

bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks

May 2017 46

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

ml

bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider

May 2017 47

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Spider Cache

bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for

details

May 2017 48

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Spider Cache Advantages

bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties

May 2017 49

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Spider Cache Disadvantages

bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo

May 2017 50

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Module Properties

bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)

2) intel130 4) mklmkl () 6) petsc32 8) StdEnv

Where

(m) module is build natively for MIC

() module is build natively for MIC and offload to the MIC

add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl

May 2017 51

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Module Properties (II) Sticky

bull A module can be stickybull It requires ``--force to unload or purge

add_property(lmodsticky)

May 2017 52

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

pushenv()

bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will

setenv() pushenv()

$ module load gcc -gt CC=gcc CC=gcc

$ module load mpich -gt CC=mpicc CC=mpicc

$ module unload mpich -gt CC is unset CC=gcc

$ module unload gcc -gt CC is unset CC is unset

May 2017 53

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

SitePackagelua

bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples

May 2017 54

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Remote Debugging

bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as

bull module --config 2gt configlogbull module -D avail 2gt availlog

May 2017 55

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56

Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users

May 2017 56