introduction to unix/linux - hpc | usc · pdf fileintroduction to unix/linux ken-ichi nomura,...
TRANSCRIPT
![Page 1: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/1.jpg)
Introduction to Unix/Linux
Ken-ichi Nomura, Ph.D.
Center for High-Performance Computing and Communications
![Page 2: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/2.jpg)
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
![Page 3: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/3.jpg)
HPCC clusters: - HP SL230 (128GB memory), HP SL250
(64 GB memory, dual NVIDIA K20m GPUs), Dual Xeon 8-core 2.4 GHz, on 56.6-Gbit FDR Infiniband network
- 2,225-node, 4-core, 6-core, and 12-core dual-processor cluster contains Dell, Oracle Sun, HP, and IBM compute nodes on a 10-Gbit Myrinet network
- 4 large-memory nodes with 1 TB of RAM and 4x10-core Intel Xeon processors.
- 2 head nodes, hpc-login1.usc.edu and hpc-login2.usc.edu
HPCC Overview
Infiniband Myrinet
hpc-‐login1 hpc-‐login2
HPCC
Internet
![Page 4: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/4.jpg)
HPCC website: http://hpcc.usc.edu/
HPCC Overview
- About - Research - User Support - New User
Guide - Frequently
Asked Questions
- Contact Us
![Page 5: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/5.jpg)
HPCC Website: HPCC Policies
5
http://hpcc.usc.edu/support/accounts/hpcc-policies/
![Page 6: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/6.jpg)
HPCC Website: Account Application
6
http://hpcc.usc.edu/support/accounts/applying-for-a-hpcc-account/
![Page 7: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/7.jpg)
Machine Room
![Page 8: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/8.jpg)
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
![Page 9: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/9.jpg)
What is Linux?
Command Line Interface
Graphical User Interface
Operating System (OS)
![Page 10: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/10.jpg)
ls List file and/or directory names pwd Print working (current) directory mkdir/rmdir Create/remove directory cd Change directory chmod Change file or directory permission cp/mv/rm Copy/move/remove file or directory cat/more/less Display file contents chmod/chown change permission/ownership man Display online manual
Basic Commands
![Page 11: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/11.jpg)
[~]$ ls docs hpc-23 Makefile R util workshop [~]$ ls –a … [~]$ pwd /home/rcf-12/knomura [~]$ mkdir tmp [~]$ cd tmp/ [tmp]$ cp ~/.bash_profile dot_bash_profile [tmp]$ cat dot_bash_profile [tmp]$ ls -l dot_bash_profile [tmp]$ chmod u-r dot_bash_profile [tmp]$ cat dot_bash_profile cat: dot_bash_profile: Permission denied [tmp]$ chmod u+r dot_bash_profile
Basic Commands: Example
![Page 12: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/12.jpg)
[tmp]$ cp dot_bash_profile DOT_bash_profile [tmp]$ ls dot_bash_profile DOT_bash_profile [tmp]$ ls -l *_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:43 dot_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:47 DOT_bash_profile [tmp]$ mv dot_bash_profile Dot_Bash_Profile [tmp]$ ls –la … [~]$ cd .. [~]$ ls [~]$ pwd [~]$ rm -rv tmp/ removed `tmp/DOT_bash_profile' removed `tmp/Dot_Bash_Profile' removed directory: `tmp'
Basic Commands: Example
![Page 13: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/13.jpg)
Shell (bash and tcsh) • Shell is a command processor by which users tell what to do to
system. • Command-line Interface (CLI) or Character user interface (CUI) • To check which shell you are using, type echo $0 • Auto logout after 20 minutes of inactivity
![Page 14: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/14.jpg)
Bash Config Files • Configuration files to setup user environment, for
examples command prompt, path, alias so on. Sometimes called “dot file”
• .bash_profile and .bashrc are stored in each user’s home directory
• When bash is invoked as a login shell, it first reads /etc/profile, if that file exists, then looks for .bash_profile and .profile
• When bash is invoked as a non-login shell, only read .profile
![Page 15: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/15.jpg)
Csh & Tcsh Config Files
• .login & .cshrc are in each user’s home directory
• When tcsh is invoked as a login shell, it reads first .tcshrc or, if .tcshrc is not found, .cshrc, then .history, then .login, and finally .cshdirs
• When tcsh is invoked as a non-login shell, only read /etc/csh.cshrc and .cshrc
![Page 16: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/16.jpg)
- File and directory have ownership and permission - Three types of permission, readable, writeable and
executable - Each permission is given to three groups, owner, group and
others [~]$ ls -l .bash_profile -rw-r--r-- 1 knomura m-cacs 208 Sep 26 14:11 .bash_profile
r readable, w writable, x executable u user (owner), g group, o others, a all
Permission & Ownership
![Page 17: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/17.jpg)
chmod Change file/directory permission chown Change file/directory owner
Example) chmod a+w file Add W permission to all users chmod o-x file Remove E permission from others chmod 750 file Add R/W/E permission to myself, R/E
permission to group no permission to others chown usr:grp file Change ownership of a file/directory
r(4) readable, w(2) writable, x(1) executable u(I) user (owner), g(II) group, o(III) others, a all
Permission & Ownership
![Page 18: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/18.jpg)
Task 1: Change Permission and Run clock.sh
#!/bin/bash while true ; do echo -n " " date +"%r" sleep 1 done
1. Add executable permission to clock.sh
2. Type ./clock.sh 3. Terminate the script by Ctrl+c [~]$ ./clock.sh 02:21:02 PM 02:21:03 PM 02:21:04 PM
…
![Page 19: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/19.jpg)
Useful Commands env display environmental variables echo/export display/set an environmental variable grep print lines matching a pattern head/tail show first/last several lines sort sort text ps/top display currently running jobs kill/killall terminate a process (not for PBS job) tar archive/unarchive files bc calculator
![Page 20: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/20.jpg)
[~]$ env [~]$ echo $PATH [~]$ export PATH=$PATH:~/bin [~]$ env | grep PATH [~]$ env > env.txt [~]$ cat env.txt [~]$ head (or tail) env.txt [~]$ grep -n knomura env.txt [~]$ sort env.txt | grep -n knomura > env2.txt [~]$ ps -u knomura [~]$ ps -ef | grep knomura [~]$ top [~]$ kill 123456
Useful Commands: Example Archive multiple files/directories into one file, then compress it. [~]$ tar cvfz out.tar.gz file1 file2 dir1 Unzip and extract from an archived file. [~]$ tar xvfz output.tar.gz Display archived file contents [~]$ tar tvfz output.tar.gz
![Page 21: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/21.jpg)
Special Characters/Commands ~ home directory . current directory .. parent directory * wild card > redirect output >> redirect output (append) <<, <<< here document/string | pipe `command` back quotes, replace `…` with
command output
![Page 22: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/22.jpg)
tab Complete name of command, file or directory
clear/ctrl+l Clear terminal ctrl+a Go to the beginning of line ctrl+e Go to the end of line history Display command history éê Display command history ! Rerun a command in history
Special Characters/Commands
![Page 23: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/23.jpg)
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
![Page 24: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/24.jpg)
Text Editor: GNU nano
http://www.nano-editor.org/
![Page 25: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/25.jpg)
Arrow-keys Move cursor Enter Change line CTRL+a Move to the beginning of line CTRL+e Move to the end of line CTRL+v Move forward one page CTRL+y Move backward one page
Text Editor: GNU nano
![Page 26: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/26.jpg)
CTRL+o Save file CTRL+w Search text CTRL+d Delete a character CTRL+k Remove a line CTRL+u Paste buffer CTRL+x Save data and exit
Text Editor: GNU nano
![Page 27: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/27.jpg)
Task 2: bash scripting 1 #!/bin/bash 2 3 # function definition 4 function pi { 5 6 # max number of RNG. use this for normalization. 7 RANDMAX=32767 8 9 # reset the number dot within a circle 10 in=0 11 for ((n=0; n<${1}; n++)); do 12 x=${RANDOM} 13 y=${RANDOM} 14 z=`echo "(${x}^2 + ${y}^2)/(${RANDMAX}*${RANDMAX})" | bc` 15 16 [ $z -lt 1 ] && let in+=1; 17 done 18 }
pi.sh
![Page 28: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/28.jpg)
Task 2: pi.sh 20 # read total number of trial, number of procs 21 echo -n "type total number of trial: " 22 read total 23 24 # call the PI estimator 25 pi ${total} 26 27 echo "total trial=${total} PI = `echo ${in}/${total}*4 | bc -l`"
1. Fix pi.sh with text editor 2. Try 10, 50, 100, 1000 for the total number of trail
to see how the result improves
![Page 29: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/29.jpg)
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
![Page 30: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/30.jpg)
System Directories
/bin, /sbin executable files (commands) /etc system config files /home users home directories /lib, /lib64, /usr/lib64 library files /proc process information /tmp temporary space /usr/usc software repository
![Page 31: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/31.jpg)
Directory and Quota
• Home directory /home/rcf-xx/username Each user has 1 GB of disk quota and 100,000 files of file quota • Project directory /home/rcf-proj/projectid Quota on project directory varies depending on each project If you need more space, request from project page. https://www-rcf.usc.edu/rcfdocs/hpcc/allocations/
![Page 32: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/32.jpg)
Monitoring Your Quota: myquota
----------------------------------------- Disk Quota for knomura ID 55302 Used Soft Hard Files 8146 100000 101000 Bytes 839.56M 4.30G 4.30G ----------------------------------------- Disk Quota for /home/rcf-proj/hpcc ID 419 Used Soft Hard Files 400792 1000000 1100000 Bytes 236.94G 400.00G 402.00G -----------------------------------------
myquota shows the quota on your home & project directories.
• Program may crash due to the quota, being unable to write files in either home directory or project directory
• Pay attention to the file quota (max number of files) also.
![Page 33: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/33.jpg)
Account, Balance & core-hour • To be able to submit your jobs to the HPCC cluster,
your account needs some balance (core-hour) • Whenever your job finishes (successfully or
unsuccessfully), your account is charged by the number of processors used times the wallclock time the job spent
Example) If you request 2 nodes with 4 processors per nodes for 2 hours (-l nodes=2:ppn=4,walltime=2:00:00), the total charge is 2x4x2 = 16 core-hour.
![Page 34: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/34.jpg)
Monitoring Your Balance: mybalance mybalance shows the current balance of your account [~]$ mybalance -h Balance Name ----------- ------ 11106267.27 lc_an 20000.00 lc_kn1
- Job will not start if your balance is not enough. - Submit a request from HPCC website for additional
core-hour. - Make sure the ppn value matches with the number of
processors actually used. (Don’t use ppn=12 for single-core job!)
![Page 35: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/35.jpg)
Software Repository: /usr/usc Many applications are available. Compilers: gnu, intel, pgi Numerical Libraries: mpich, openmpi, cuda, fftw, petsc Molecular Simulation: NAMD, gromacs, amber Quantum Chemistry: gaussian, schrodinger Numerical Environment: matlab, R, python
hpc-login1.usc.edu for 32-bit applications hpc-login2.usc.edu for 64-bit applications
![Page 36: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/36.jpg)
Software Repository: /usr/usc How to use the installed software on HPCC: - Find application and version you want to use - Source setup script. There are two setup scripts, one for
bash the other for tcsh
Example) Matlab R2013a
[~]$ ls /usr/usc/matlab/ 2009a 2009b 2010b 2011a 2013a 2013b default Scripts [~]$ source /usr/usc/matlab/2013a/setup.sh [~]$ matlab -nodisplay
< M A T L A B (R) > Copyright 1984-2013 The MathWorks, Inc.
R2013a (8.1.0.604) 64-bit (glnxa64) February 15, 2013
>>
![Page 37: Introduction to Unix/Linux - HPC | USC · PDF fileIntroduction to Unix/Linux Ken-ichi Nomura, Ph.D. Center for High-Performance Computing and Communications](https://reader033.vdocuments.net/reader033/viewer/2022051010/5aaaa6117f8b9a7c188e57f0/html5/thumbnails/37.jpg)
System Software vs Repo Software • Some software and libraries (e.g. gcc, python, fftw) come
with OS
• Although command name is the same, the system software and repo software are often different (versions, libraries, developers). Make sure that you use what you want to use
• which command shows the absolute path of a command
[~]$ which python /usr/bin/python [~]$ source /usr/usc/python/enthought/default/setup.sh [~]$ which python /usr/usc/python/enthought/default/bin/python