VIMS HPC Tutorial

September 16th 2015

Eric J. Walter

● Understanding the available hardware and software● How to edit files / which text editor● Basic linux and shell usage● Overview of file systems and file permissions● How to use the batch system to run jobs● Compilers and creation of programs

Available HPC hardware and software

http://www.hpc.wm.edu/SciClone/Home - Wiki homepage

http://www.hpc.wm.edu/SciClone/Software - Software list (out of date)http://www.hpc.wm.edu/SciCloneTutorials/ModuleBasics - Software modules

http://www.hpc.wm.edu/SciCloneTutorials/Home - Tutorials pagehttp://www.hpc.wm.edu/SciCloneUserGuide/Home - User's guide http://www.hpc.wm.edu/SciClone/Hardware - Hardware components

http://www.hpc.wm.edu/SciClone/presentations/ClusterOverview_permissions.pdf

Recent hardware overview:

SciCloneTyphoon

typhoo

n

4 cores / node72 nodes -> 288 cores2 or 6 GB mem / coreOpteron Santa-RosaDDR Infiniband

Vortex

vortex

12 cores / node36 nodes -> 432 cores2.7 or 10.7 GB mem / coreOpteron SeoulFDR Infiniband

● SciClone has 1232 cores total● 2 Intel & 2 Opteron clusters● Typhoon uses SLES 10 – rest RHEL 6.2● Total usage for May 2015 : 74%● hurricane cluster has 2xM2075 GPUs (Fermi) ● change all passwords on Hurricane

TwisterGulfstream

Tornado Gale

Other servers for filesystems / backup etc.

Hurricane

hurrican

e

whirlwind

8 cores / node52 nodes -> 416 cores8 or 24 GB mem / coreIntel WestmereQDR Infiniband

8 cores / node12 nodes -> 96 cores6 GB mem / coreIntel WestmereQDR Infiniband

.sciclone.wm.edu

48 cores / 96 GB 32 cores / 64 GB

ice01 ice02

Opteron Magny-CoursQDR Infiniband

snow8 cores / node15 nodes -> 120 cores2 GB mem / core Opteron ShanghaiDDR/QDR Infiniband

Storm

wind

16 cores / node26 nodes -> 416 cores2 GB mem / coreOpteron Magny-CoursQDR Infiniband

Storm

● Storm has 1080 cores total● All Opteron● RHEL 6.2● Total usage for May 2015 : 67%● Has Lustre filesystem

gust

Other servers for filesystems / backup etc.

4 cores / node98 nodes -> 392 cores2-8 GB mem / coreOpteron Santa-RosaDDR Infiniband

rain

hail8 cores / node9 nodes -> 72 cores2-8 GB mem / coreOpteron ShanghaiQDR Infiniband

.hpc.wm.edu

Chesapeake

Chesapeake

● Chesapeake has 380 cores total● All Opteron● RHEL 6.2● Total usage for May 2015 : 32% ● Still need to incorporate Indian nodes

Chickahominy

York

Rappahannock

Other servers for filesystems / backup etc.

potomac

12 cores / node30 nodes -> 360 cores2.7 GB mem / coreOpteron SeoulQDR Infiniband

in1 in2

.hpc.vims.edu

Wiki

URL: www.hpc.wm.edu/SciClone/Home

Software

FACT: Software is requested and installed more quickly than documentation can be writtenSubmit a ticket for any software requests. Please make sure you need the software.

Linux Text Editors

Popular text editors: emacs or vi/vim

emacs: huge, bloated, not installed by default, but the champion!

vi/vim: tiny, always available, some users love it

nano: editor with training wheels, very easy to use, not very powerful

http://www.gnu.org/software/emacs/tour/http://www.jesshamrick.com/2012/09/10/absolute-beginners-guide-to-emacs/http://www2.lib.uchicago.edu/keith/tcl-course/emacs-tutorial.html

Emacs Tutorials

http://www.vim.org/http://vim.wikia.com/wiki/Tutorialhttp://linuxconfig.org/vim-tutorial

Vim Tutorials

http://www.nano-editor.org/Nano homepage

Linux Shell Usage

Tutorials page – Subcluster Essentials pages ; Using the XX cluster Guides to running on a particular subcluster

http://www.hpc.wm.edu/SciCloneTutorials/LinuxUnix - page of linux tutorials

Really two goals for linux understanding

1) learning linux commands and concepts2) learning how to configure your evironment

Questions to ask yourself:

- what sofware do I want loaded by default?- what should my default umask be?- what ISA am I compiling for?- where can I store files?- where should I run my jobs?

HPC Filesystems / Backup146 [hurricane] df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/VolGroup00-LogVol00 7.9G 4.8G 2.8G 64% /tmpfs 7.8G 76K 7.8G 1% /dev/shm/dev/sda1 485M 76M 384M 17% /boot/dev/mapper/VolGroup00-LogVol03 7.9G 147M 7.4G 2% /tmp/dev/mapper/VolGroup00-LogVol02 14G 5.5G 7.7G 42% /usr/dev/mapper/VolGroup00-LogVol04 7.9G 1.7G 5.9G 23% /var/dev/mapper/VolGroup00-LogVol06 413G 7.0G 385G 2% /sciclone/scr01/dev/mapper/VolGroup30-LogVol31 917G 482G 390G 56% /sciclone/home00tn00:/usr/local 134G 113G 15G 89% /usr/localtn00:/export 46G 15G 30G 33% /importmh00:/var/spool/mail 7.9G 4.3G 3.3G 57% /var/spool/mailgfs00:/sciclone/home04 591G 429G 157G 74% /sciclone/home04ty00:/sciclone/scr02 273G 81M 273G 1% /sciclone/scr02tn00:/sciclone/scr10 7.9G 2.3G 5.3G 31% /sciclone/scr10tn00:/sciclone/scr30 17T 13T 4.7T 72% /sciclone/scr30gfs00:/sciclone/data10 16T 15T 1.9T 89% /sciclone/data10gfs00:/sciclone/vims20 26T 19T 6.7T 74% /sciclone/vims20tw00-i8:/sciclone/data20 73T 57T 16T 79% /sciclone/data20/dev/md1 8.1T 5.0T 2.8T 65% /sciclone/scr20vx00:/sciclone/home10 2.7T 202M 2.6T 1% /sciclone/home10vx00:/sciclone/scr00 318G 4.8G 297G 2% /sciclone/scr00

home – nightly backup

data – weekly backup

scr – not backed up

Sharing Files & Folders : Permissions Isee

https://www.nersc.gov/users/storage-and-file-systems/unix-file-permissionsfor more information

35 [hurricane] ls -l resultstotal 3-rw------- 1 ewalter hpcf 194 Jun 16 14:37 ww.dat-rw------- 1 ewalter hpcf 194 Jun 16 14:37 yy.dat-rw------- 1 ewalter hpcf 194 Jun 16 14:37 zz.dat

list files in directory

user group other

r-readw-writex-execute

**directories need to be ‘x’ to be entered/passed through

directo

ry

associateduser

associatedgroup

print working directory

33 [hurricane] pwd/sciclone/home04/ewalter

list directory34 [hurricane] ls -ld resultsdrwx------ 2 ewalter hpcf 512 Jun 16 14:37 results

http://www.hpc.wm.edu/SciClone/presentations/Permissions.pdf

Sharing Files & Folders : Permissions III want to share my results directory and files with those in the seadas group:

Am I in the seadas group?:

Change the group for the directory and all files below:

51 [hurricane] chmod g+rX -R results/

52 [hurricane] ls -ld results/drwxr-x--- 2 ewalter seadas 512 Jun 16 14:37 results/53 [hurricane] ls -l results/total 3-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 zz.dat

Make: files group readable folders group readable and executable

46 [hurricane] groups ewalterewalter : hpcf wmall hpcstaff www seadas vasp wm sysadmin hpcadmin hpsmh

47 [hurricane] chgrp -R seadas results/

49 [hurricane] ls -ld results/drwx------ 2 ewalter seadas 512 Jun 16 14:37 results/

50 [hurricane] ls -l results/total 3-rw------- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw------- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw------- 1 ewalter seadas 194 Jun 16 14:37 zz.dat

Sharing Files & Folders : Permissions IIIWhat happens when a new file is created?

78 [hurricane] cd results/

80 [hurricane] touch newfile81 [hurricane] ls -ltotal 3-rw------- 1 ewalter hpcf 0 Jun 16 15:13 newfile-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 zz.dat

This is bad since the new file isn’t in the seadas group!

88 [hurricane] chmod g+s results/89 [hurricane] cd results/

91 [hurricane] touch newfile92 [hurricane] ls -ltotal 3-rw------- 1 ewalter seadas 0 Jun 16 15:14 newfile-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 zz.dat

add “setgid” to folder only

file group inherits the group from the folder it is in

Sharing Files & Folders : Permissions IVThe user’s umask controls what permissions files and folders are given when created:

93 [hurricane] umask77

HPC default umask is 077Can be changed in your startup file (.cshrc etc.)

135 [hurricane] touch file077136 [hurricane] ls -l file077 -rw------- 1 ewalter seadas 0 Jun 16 15:24 file077

137 [hurricane] umask 022138 [hurricane] touch file022139 [hurricane] ls -l file0*-rw-r--r-- 1 ewalter seadas 0 Jun 16 15:24 file022-rw------- 1 ewalter seadas 0 Jun 16 15:24 file077

140 [hurricane] umask 027141 [hurricane] touch file027142 [hurricane] ls -l file0*-rw-r--r-- 1 ewalter seadas 0 Jun 16 15:24 file022-rw-r----- 1 ewalter seadas 0 Jun 16 15:24 file027-rw------- 1 ewalter seadas 0 Jun 16 15:24 file077

Common Permissions Tasks IChange the permissions of a directory chmod:

Change the permissions of a directory and everything under it chmod -R:

24 [hurricane] ls -ld VASPdrwx------ 4 ewalter hpcf 512 Apr 4 2014 VASP25 [hurricane] chmod go+rX VASP26 [hurricane] ls -ld VASPdrwxr-xr-x 4 ewalter hpcf 512 Apr 4 2014 VASP27 [hurricane] chmod o-rX VASP28 [hurricane] ls -ld VASPdrwxr-x--- 4 ewalter hpcf 512 Apr 4 2014 VASP29 [hurricane] chmod g-rX VASP30 [hurricane] ls -ld VASPdrwx------ 4 ewalter hpcf 512 Apr 4 2014 VASP

32 [hurricane] ls -ld VASPdrwx------ 4 ewalter hpcf 512 Apr 4 2014 VASP33 [hurricane] ls -l VASPtotal 52457-rw-r----- 1 ewalter hpcf 22932904 Apr 4 2014 potpaw_LDA.52.tar.gz-rw-r----- 1 ewalter hpcf 25958479 Apr 4 2014 potpaw_PBE.52.tar.gzdrwxr-x--- 2 ewalter hpcf 15360 Aug 6 00:01 vasp.5.3

34 [hurricane] chmod -R g+rX VASP

35 [hurricane] ls -ld VASPdrwxr-x--- 4 ewalter hpcf 512 Apr 4 2014 VASP36 [hurricane] ls -l VASPtotal 52457-rw-r----- 1 ewalter hpcf 22932904 Apr 4 2014 potpaw_LDA.52.tar.gz-rw-r----- 1 ewalter hpcf 25958479 Apr 4 2014 potpaw_PBE.52.tar.gzdrwxr-x--- 2 ewalter hpcf 15360 Aug 6 00:01 vasp.5.3

Use this command if you want to allow group access to your home, scrXX, and dataXX directories

HPC default umask is 077

How do I change the initial permissions that files and folders are given when created:

You need to edit your .cshrc file in your home directory and add: umaskumask 077 files get “-rw-------” folders get “drwx------” umask 027 files get “-rw-r-----” folders get “drwxr-x---” umask 022 files get “-rw-r--r--” folders get “drwxr-xr-x”

What groups are I in?: groups52 [hurricane] groups ewalterewalter : hpcf wmall hpcstaff www seadas vasp sysadmin wm hpcadmin wheel hpsmh

My primary (default) group is hpcf and the rest are secondary

Change a group associated with a file or directory: chgrp 54 [hurricane] ls -ld projectdrwx------ 2 ewalter hpcf 512 Aug 18 20:47 project55 [hurricane] chgrp hpcstaff project56 [hurricane] ls -ld projectdrwx------ 2 ewalter hpcstaff 512 Aug 18 20:47 project

Common Permissions Tasks II

Using the Batch Systemhttp://www.hpc.wm.edu/SciCloneTutorials/LaunchJobs

Concepts:Interactive vs. Batch jobSubmitting / checking / deleting Forwarding X

Special VIMS commands:qsubdep – submit a new job with a dependency on all current jobsqpri – arrange a list of jobs to run in a particular order, one at a time

13 [chesapeake] qstat ­u ewalter          (qsu)

Job id                    Name             User            Time Use S Queue                        ­­­­­­­­­­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­ ­­­­­­­­ ­ ­­­­­                        1000.cp00                Job1              ewalter         00:00:29 R qx             1001.cp00                Job2              ewalter         31:56:22 R qx             1002.cp00                Job3              ewalter                0 Q qx

14 [chesapeake] qsubdep runscript

Job id                    Name             User            Time Use S Queue                        ­­­­­­­­­­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­ ­­­­­­­­ ­ ­­­­­                        1000.cp00                Job1              ewalter         00:00:29 R qx             1001.cp00                Job2              ewalter         31:56:22 R qx             1002.cp00                Job3              ewalter                0 Q qx1003.cp00                Job4              ewalter                0 H qx                       

Using the Batch System13 [chesapeake] qsub ­h run1          13 [chesapeake] qsub ­h run2          13 [chesapeake] qsub ­h run3          

13 [chesapeake] qstat ­u ewalter          

Job id                    Name             User            Time Use S Queue                    ­­­­­­­­­­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­ ­­­­­­­­ ­ ­­­­­                1000.cp00                Job1              ewalter                0 H qx             1001.cp00                Job2              ewalter                0 H qx             1002.cp00                Job3              ewalter                0 H qx

14 [chesapeake] qpri ­r 1000 1001 1002

Job id                    Name             User            Time Use S Queue                    ­­­­­­­­­­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­­ ­­­­­­­­­­­­­­­ ­­­­­­­­ ­ ­­­­­                    1000.cp00                Job1              ewalter         00:00:00 R qx             1001.cp00                Job2              ewalter                0 H qx             1002.cp00                Job3              ewalter                0 H qx            

qpri - hold each job until the previous finishes the -r flag removes the user hold on the jobs if you skip -r, you need to use qrls to release the hold qrls <jobid>

CompilersAll platforms have GNU and Portland C/C++/FORTRAN compilersCheck the Tutorials - Subcluster Essentials / Using XX pages for what flags are suggested for optimization.

For Chesapeake / Potomac:

Important to know what the isa of the target processor is. All potomac nodes are type seoul