class10_bash shell scripting.pdf

7/29/2019 class10_Bash Shell scripting.pdf

1/46

Bash Survival Guide for HPC

M. D. Jones, Ph.D.

Center for Computational ResearchUniversity at Buffalo

State University of New York

High Performance Computing I, 2012

M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 1 / 52


2/46

Introduction A Little History

History of bash

The concept of the shell environment has been around for a long time

(long = lifetime of UNIX operating system):

bash is a twist on one of the earliest UNIX shells (the so-called

Bourne shell, sh on most systems).bash is a superset of the (limited) Bourne shell syntax

There is quite an extended family of shells, a good link for whichcan be found here:

http://en.wikipedia.org/wiki/Comparison_of_command_shells

http://en.wikipedia.org/wiki/Comparison_of_command_shellshttp://en.wikipedia.org/wiki/Comparison_of_command_shells


3/46

Introduction Feature Subset

bash Features

Subset of important features of bash:

Basically a command line interface (CLI) to the operating system

Command line completion (via TAB key)

Command line editing, history

Output redirection

Control constructs

...

Basically all of the shells give you the ability to enter text-based

commands/queries much faster than using a GUI (if your particular OSeven supports a GUI).



4/46

Introduction Startup Scripts

bash Invocation/Startup Scripts

bash will read from various startup scripts depending on how it was

invoked:

Interactive login shell: uses /etc/profile, then

~/.bash_profile, ~/.bash_login, ~/.profile in thatorder (if they are present at all).

Interactive non-login shell: ~/.bashrc (suppressed by --norcoption, or --rcfile file alternative)

Non-interactive: uses $BASH_ENV (subject to substitution, if resultis a file, executes the file)

as sh: attempts to duplicate Bourne shell behavior, reads andexecutes /etc/profile and ~/.profile, in order

via rshd: will read and execute ~/.bashrc (if it is able todetermine that is was launched via rsh)



5/46

Introduction Navigation

CLI Navigation

bash supports a rich set of line editing features making it easier for

you to move the cursor around the command line, recall pastcommands and edit them, etc. The default syntax follows that of theemacs text editor (you can customize the syntax, of course, see manbash). Very briefly, the arrow keys can typically be used to navigatethe current command line (left and right arrow keys, up and down to

scroll though past commands), plus some common key bindings (thereare many more than this small sample):C-a go to the beginning of the line

C-e go to the end of the line

C-f,C-b go forward,back one character

M-f,M-b go forward,back one word

where C- denotes the Control key on the keyboard, and M- the METAkey - not many keyboards have META keys these days, in which case

ESC is used instead.


I d i N i i


6/46

Introduction Navigation

CLI History

Unsurprisingly, the shell records a history of what commands youexecute. The history command will list the commands in your shells

history with an identification number that you can use as shorthand forrecalling them:

[ u2 : ~ ] $ ! ! # r e p e a t v e r y l a s t commandl s l[ u 2 : ~ ] $ h i s t o r y # s p i t s o u t l a s t HISTSIZE c ommands[ u 2 : ~ ] $ ! 15 74 # p i c k one b y number , p r ef a ce w i t h !who am ijo ne sm p t s /2 3 20120525 1 2 :5 3 ( c as h . c c r . b u f f a l o . edu )[ u 2 : ~ ] $ ! 1 56 2 : s/ x/a / # p i ck one , use a r ege x t o m od if y on t he f l ypbsnodes a | l e s s


I t d ti E i t V i bl


7/46

Introduction Environment Variables

Environment Variables

Environment variables are heavily used by all shells to set the user

environment and control its behavior, and are also frequently used byapplications. For now, here are just a few of the simplest of the

defaults:

$HOME , your home directory location.

$PATH , search path for commands$LD_LIBRARY_PATH , search path for dynamic libraries

$MANPATH , search path for man pages

$PS1 , command prompt syntax

Note that the modules environment package used in CCR is designedto allow you to more easily manipulate the environment (especially$PATH, $LD_LIBRARY_PATH, and $MANPATH) for various softwarepackages.




8/46


You can set environment variables very simply by assignment or usingthe export command, the difference being whether the value is

passed on to sub-shells (i.e. if you run another script or executable) -you can also run a program with specific settings using the envcommand:

# t hes e v a r ia b le s ar e s e t o nl y i n t he c u rr e nt s h e l lNP=` cat $PBS_NODEFILE | wc l ` # counts l in es in $PBS_NODEFILENODES= ` cat $PBS_NODEFILE | un iq ` # counts unique l in es in $PBS_NODEFILE

# t h i s a pp li es t o a l l c h i ld re n o f t h i s s h e l l# ( I n t e l MKL core a f f i n i t y s e t t i n g )export KMP_AFFINITY=no war nin gs , compact# t h i s r uns a s h e l l s c r i p t us in g a l o c a l s e t t i n g f o r $HOME# ( MATLAB c o m pi le d b i n a r i e s l i k e t o abus e $HOME d i r e c t o r i e s . . . )env HOME=/tmp . / ru n _ e xt ra c t . sh

You can add common settings to your $HOME/.bashrc file if you wantthem to apply to all of your logins/shells. Using export without anyarguments will print the current settings for your shell (as will the envcommand).


Basic Syntax Format


9/46

Basic Syntax Format

The First Line

Shell scripts can be no more than an ordered list of commands to run.

By convention, the first line declares the shell:

# ! / bin / bash## pound sy mbol g e n e r a l l y s t a r t s a comment , e xc ep t f o r t h a t f i r s t l i n eecho ' H e l l o , w o r l d ! '


Basic Syntax Format


10/46

Basic Syntax Format

Variables

Variables in bash are easily declared:# some v a r i a b l e d e c l a r a t i o n s :st r1 =" Goodbye , wo rl d ! " # n o te q uo te s t o p r o t e c t t h e s pac es t r 2 = ` d at e # b a c k t i c k s t o e x e cu t e c ommands ,

# c ap t ur e th e ou tp utecho n '$USER = ' # si n g l e q u o tes do n o t e xa nd $USER

# and n d oes n ot b re ak l i n eecho "$USER" # d o u bl e q u o tes w i l l e xp an d $USER

# p r ec e di n g t w o l i n e s p r i n t o u t $USER = jo hn do e f o r u s e r i d j oh nd oe# t h e r e a re a bunc h o f p r e d e f in e d v a r i a b l e s , $USER i s oneecho " $ { s t r 1 } A ar gh ! " # b r ac es a re handy t o p r o t e c t v a r i a b l e ss t r 3 = " "i f [ n " $ s t r3 " ] ; then # n i n d i c a t es nonempty argument

echo ' $ s t r 3 i s n o t empty ! 'f i

That last bit is a good example of why variables are often enclosed inquotes (they do not have to be), in case they evaluate as the emptystring.


Basic Syntax Command Substitution


11/46

Basic Syntax Command Substitution

Command Substitution

Command substitution is an invaluable way to access the output of acommand from the environment. You can do this either with backticks,

command, or subshells, $(commands):

f i l e s = " $ ( l s ) " # l i s t i n g o f cwdh t ml _ f i l e s =` l s / va r /www/ h tml # l i s t i n g o f / v a r / www / h tm li n d e x = ` expr 4 * 2 + 1 ` # u se e x pr t o e v a l u at e math

Note that $() is very easy to nest, which gives it an edge overbackticks. Newlines are not allowed, so they are stripped out in both

methods.


Basic Syntax Arithmetic Expansion


12/46

Basic Syntax Arithmetic Expansion

Arithmetic

bash has no notion of floating-point math, although you can resort toexternal tools like bc to work around that lack. Integer arithmetic isdone using backticks, double parentheses, or let:

z = ` expr $ z + 1 ` # expr command does the workz=$ ( ( $ z+1 ))z =$ ( ( z + 1 ) ) # p a ra m et e r der e f e r e n c i ng o p t i o n a l i n s i d e d ou bl e p ar en th es es( ( z += 1 ) ) # s h o r tc u tl e t z =z +1 # l e t commandl e t " z += 1 " # q uo te s l e t you use s pa ce s


Basic Syntax I/O Redirection


13/46

y

I/O Redirection

Very handy topic - how to redirect input/output in bash. First, though,there are three default files:

stdin (keyboard), descriptor 0

stdout (screen), descriptor 1stderr (error messages to the screen), descriptor 2

Redirection just means capturing output from a file, command, or script

(or portion of a script) and sending it elsewhere as input. Note that

descriptors 3-9 are also available for use.


Basic Syntax I/O Redirection


14/46

y

Generally you can redirect input with (file), but there are quite a few more variations best illustrated byexample:

echo ' b la h ' > f i le n am e # o v e r w r i t es f i le n am eecho ' b la h ' >> f i l e n am e # a pp en ds f i l e n a m e. / somecode 1> f i l enam e # f i l e n a m e c a p t u r e s t d o u t f r om somecode

. / somecode 2> f i l enam e # f i l e na m e c a p tu r e s s t d e r r f ro m somecode. / somecode &> f i l enam e # f i le n am e c a pt u re s b ot h s t d o ut and s t d e r r

. / somecode > f i l enam e 2>&1 # same as abov e , o l d ( B our ne ) s t y l e

. / o t he r co d e < f i l e na m e # o th er co de g et s i n p u t f ro m f i le n am e

. / code < inname &> outname # c o m bi n at i on s a re okcommand1 | command2 | command3 # p ip es a re s i m i l a r t o > b ut more g en er al

# f o r c h a i n i n g commands t o g e t h e rcat * . t x t | s or t | u n i q > r e s u l t # see ?


Control Constructs Conditionals


15/46

if--fi

bash has a conditional construct, of course:

i f c o n d i t i o n A

thensta te me n t0sta te me n t1. . .

e l i f c o n d i t i o n Bthen

morestatements1morestatements2. . .

elseyetmore0yetmore1. . .

f i

We will go into the sytnax for the conditions next, but note that the

conditional branches are often written more concisely using the ";"separator:

i f c o n d i t i o n A ; thenl i s t o f s t a t e m e n t s

. . .f i




16/46

Is There in Truth No Beauty?

The test command (and its shorthand version) gets used quite often in

conditionals (and in general):

# f i r s t f o rmt e s t o pe ran d1 o p e ra to r o p eran d2# ex am pl e :

i f t es t

n "$PBS_ENVIRONMENT" ; thenecho " Hey , I am i n a PBS e n v ir o n me n t ! "f i# secon d fo rm[ o pe ra nd 1 o p e r a t o r o pe ra nd 2 ]# e xa mp le :i f [ n "$PBS_ENVIRONMENT" ] ; then

echo " Hey , I am s t i l l i n a PBS e n v ir o nm e n t ! "f i

# I c heat ed a b i t ,

n i s a u na ry o p er a to r . . .




17/46

Operators

I am not going to show the full set of test operators (see man bash),

but here is a quick subset:

Operator Operands-n 1 nonzero length-z 1 zero length-f 1 file given by operand exists

-d 1 directory given by operand exists== 2 string equality!= 2 2 string lexical-eq,-ne 2 integer (in)equality-lt,-gt 2 integer comparisons-le,-ge 2 integer comparisons

|| 2 logical OR&& 2 logical AND

Note the absence of floating-point operations (serious shortcoming).


Control Constructs Loop Constructs


18/46

for Loops

First of the main loop types in bash, for loops repeat over a simple

index of space separated items:f o r name i n " $ LIST "do

echo " working on $name"b u n ch o fsta te me n ts1

done

So the loop counter, name is set to each item in the list as the loop isexecuted. Note that the familiar form from C for integer indices is also

valid:

f o r ( ( e xp r1 ; e xp r2 ; e xp r3 ) ) ; do l i s t o f s ta t e m e nt s ; done# f o r ex am pl e :f o r ( ( i = 1 ; i < 10 ; i + + ) )

doecho " i = $ i "

done

Note that bash is pretty slow (most interpreted languages are slow),

so the C-like loop syntax is not encouraged.




19/46

Globbing

You can make handy use of substitution by globbing, or using * tomatch all filenames.

# s i m p l e g l o b b i n g e xa mp lef o r fna me i n * . c # grabs a l l . c f i l e s i n c ur re nt d i re c to r ydo

g re p ' d o ub l e ' $ fn ame # p u l l a l l o u t a l l th e d oublesdone



L


20/46

while Loops

While loops are just about what you would expect:

while l i s t ; do l i s t o f s t a t e m e n t s ; done# si mp l e e xa mp lewhile [ " $ c o u n te r " l t "$UPPER_LIMIT" ] ; do

echo " c o u n t e r = $ c o u n te r "c o u n t e r = `expr $ c o u n t e r + 1 ` # c o u nt e r =$ ( ( $ c ou n te r + 1 ) ) s h ou ld a l s o w or k

# l e t " c o u n t e r += 1 " sho uld al s o workdone

There is also an until variant of while that has the same syntax

(just negates the test).


Miscellaneous Advanced Features Conditional Execution

C diti l E ti


21/46

Conditional Execution

A very typical bash-ism is conditional execution, namely use the return

code from a command (you know that all commands have returncodes to indicate success or failure, right?):

command1 && command2 # command2 i f f command1 re t u rn s t r u e (0 )command1 | | command2 # command2 i f f command1 re tu rn s fa ls e (non

zero )

command1 && command2 | | command3 # command2 i f f command1 re tu rn s tr ue# command3 i f f command1 re t u rn s fa l s e

#rm f $ t h i s f i l e && echo " $ t h i s f i l e was removed . " | | echo " $ t h i s f i l e n ot removed . "## c an ch ec k e x i t s t a t u s o f l a s t command u s in g $?echo $?


Miscellaneous Advanced Features Functions

F ti


22/46

Functions

bash has support for functions, general syntax:

#

# g e n er a l f u n c t i o n s y nt ax# o ne way:fu n ct i on _ n a me { # body o f f u n c t i o n i n s i d e br ac es

sta te me n ts ; # s ho ul d end w i t h s em ic ol on o r n ew li ne}## a n o t he r way :fu n c ti o n f u nc t io n am e ( ) # u si n g t he f u n c ti o n b u i l t i n r eq ui re s ( ){

sta te me n t1sta te me n t2. . .statementNre tu rn # o r an e x p l i c i t r e t ur n b u i l t i n

## h andy f u n c t i o n f o r DOS / w indows s h e l l u s er sd i r ( ){

l s

F - c o l o r = a u t o

l F - c o l o r = a lw ay s "$@" | l e s s

r # $@ i s $1 , $2 , . . .}

}

Return code is the the return code from the last statement in the

function.



F ti P t


23/46

Function Parameters

You can pass arguments to functions (they behave like mini-scripts).

They become positional parameters that you can reference as $1, $2,...

## f un w it h f u n ct i on ( o r s c r i p t i t s e l f ) param eters#echo " My name i s $ 0 " # name o f t he s c r i p t

echo " I h ave $ # a rgu men ts" # n umber o f a rg um en ts i s $#i f [ n " $1 " ]then

echo " F i r s t ar gum ent i s $1 " # 1 s t a rg um en tf i## An o f t e n used f or m f o r s c r i p t s t o c he ck a rg um en t number#i f [ ! $# eq 2 ] ; th en

echo " U sage : $0 a r g1 a rg 2 "e x i t 1f i

Note that $0 remains unaffected by calling a function.



Bit More Function Trivia


24/46

Bit More Function Trivia

Functions can be recursive, and can have their own local variables

through the local builtin.


Miscellaneous Advanced Features Shell Limits

Shell Limits


25/46

Shell Limits

bash shell limits are controlled through the ulimit builtin function.

Easiest way to see that is by example:[ bono : ~ ] $ u l i m i t acore f i l e s i z e ( bl oc ks , c ) u n li m i te ddata seg s i z e ( kbytes , d ) u n l i m it e df i l e s i z e ( bl oc ks , f ) u n l i m it e dpending s i g n a l s ( i ) 1024max l oc ke d memory ( k by tes , l ) 32max memory s i z e ( kbytes , m) u n l i m it e d

open f i l e s (

n ) 8 19 2p i p e s i z e (512 bytes , p ) 8POSIX message q ue ues ( b y te s , q) 819200s t a c k s i z e ( kbytes , s ) u n li m i te dcpu t i me ( seconds , t ) u n l i m it e dmax user processes (u ) u n l i m it e dv i r t u a l memory ( kbytes , v ) u n li m i te df i l e l o c k s (x ) u n li m i te d

The values can be numerical or one of hard, soft or unlimited. Theflags -H, -S can be used to specify the hard and soft limits. Note thatthe stack limit is of particular interest to us in HPC, and very frequently

needs to be increased from its (small) default value.


Miscellaneous Advanced Features Aliases

Aliases


26/46

Aliases

Aliases are basically keyboard shortcuts that you can set up and use -

they do not do any expansion, nor can they be recursive:

## f r om my ~ / . b a sh r c#export EDITOR= v i## A l i a se s

#a l i a s rm='rm i 'a l i a s mv= 'mv i 'a l i a s c p = ' c p i 'a l i a s xe= 'xemacs 'a l i a s myps= ' ps u jonesm Lf 'a l i a s m yj ob s = ' q s t a t a | g r ep j on es m 'a l i a s p r o j = ' cd / p ro j e ct s / j o ne sm 'a l i a s w i e n = ' cd / pr oj ec ts / jonesm/ d_wien / wien2k / u2 '

a l i a s mod= ' module 'a l i a s ml = 'mo du le l o a d 'a l i a s mu= ' module unload 'a l i a s m ls = ' module l i s t '


Miscellaneous Advanced Features Loading Other Scripts

Loading Other Scripts


27/46

Loading Other Scripts

You can load other bash scripts (or just files containing bashcommands) using the source command.

#

# l o a ds I n t e l c o mp i le r e nv ir on me nt. / o p t / i n t e l / c om po se rx e / b i n / c o m p i l e r v a r s . s h i n t e l 6 4

Sometimes you will see the shorthand . used instead of source (same

meaning).


Miscellaneous Advanced Features Regular Expressions

Regular Expressions


28/46

Regular Expressions

Regular expressions (REs) occur very frequently in UNIX, especially in

sed, awk, and grep. If you have done any significant work in Perl,of course, you are likely also well versed in REs.

REs are sets of (meta)characters used for pattern matching in text

searches and string manipulation, and contain one (or more) of:characters - retain their literal meaning

anchors - designate position in the text line, ^ (beginning) and $

(end)

modifiers - expand or narrow the range of text to search, includes ,brackets, and the backslash




29/46

More on modifiers in REs:

*, matches any number of the character string or RE preceding it

., (dot) matches any non-newline character^, (carat) matches beginning of line, but also for negation (context dependent)

$, matches end of line

[,], brackets enclose a set of characters to match, including negation with ^ and rangeswith -

\, backslash escapes a special character (e.g. $)

\, escaped angle brackets match word boundaries

?, question mark matches zero or one of the preceding RE, used mostly for singlecharacters

+, matches one or more of the previous RE (like *, but does not match zero occurrences)

\{,\}, escaped curly braces indicate number of previous REs to match

(), parentheses enclose a group of REs

|, the or RE operator used in matching alternates



RE Examples


30/46

RE Examples

Used in the context of grep just as an example:

g r ep " 2 22 4* " t e x t f i l e # m at ch es 222 f o l l o w e d by a ny number o f 4 s# e . g . 222 , 2224 , 2 2244 , . . .

grep " 2 4. " t e x t f i l e # m atc hes 24 f o l l o w e d by any c h a r ac t e r , b u t n o t 24grep " ^A" t e x t f i l e # matches any l i n e b eg i nn in g w i t h Agrep " A$ " t e x t f i l e # matches any l i n e e nd in g w i t h A

grep " ^$ " t e x t f i l e # m atc hes b l an k l i n e sgr ep " [ a bc123 ] " t e x t f i l e # matches any o f t he c h a r ac t e r a , b , c , 1 , 2 , 3g re p " [ ac13] " t e x t f i l e # same t h i n g u s i n g r a ng esg re p " [ ^ ac ] " t e x t f i l e # m atches a l l c h a ra c t e rs e xc ep t acgr ep " [ Yy ] [ Ee ] [ Ss ] " t e x t f i l e # c as e i n s e n s i t i v e match f o r YES , yes , Yes , . . .grep " \ $ \ $ " t e x t f i l e # match $ $grep " \ < yes \ > " t e x t f i l e # m atches o n ly d i s t i n c t word y es

REs are a rich topic in their own right ...


Miscellaneous Advanced Features Here Documents

Here Documents


31/46

Here Documents

So-called here documents are ways to pass text on-the-fly into

programs or other code blocks. Its another form of I/O redirection, buta very handy one for running programs:

. / a . ou t


32/46

Basic Job/Process Control

You have the ability to stop and continue your processes from a fairlylow level to more sophisticated tools like screen. The basics are

controlled through the notion of a job, created by putting a processasynchronously into the background using the & operator, or C-z

(control-z), the bg and fg commands place the process intobackground or foreground, jobs will list all jobs:

[ u 2 : ~ ] $ e m a c s t e s t . c^Z # c o n tr o lz suspends j ob , j o b i d g iv en i n s qu ar e b r ac k e ts[ 1 ] + Stopped emacs t e s t . c[ u 2 : ~ ] $ bg # p u t i n t o b ac kg ro un d , now r u n ni n g s e p a r a t e l y f ro m t e r m i n a l[1 ]+ emacs t e s t . c &[ u 2 : ~ ] $ jobs # l i s t s r un ni ng j ob s[ 1 ] + Running emacs t e s t . c &[ u 2 : ~ ] $ fgemacs t e s t . c # b r in g back t o f or eg ro un d i n t e r m i na l

[ u 2 : ~ ] $ k i l l %1 # d i e j o b 1 , d i e

n ot e % f o r j o b i d

kill terminates jobs and processes (no % sign for process ids), if you

have a stuck process (identified with the ps commands) you may needto specify a higher kill level like kill -9 pid.


Miscellaneous Advanced Features Job Control

Stuff Not Covered


33/46

Stuff Not Covered

In the context of a bash survival guide, I skipped quite a bit of stuff:

Argument processing (getopts, shift) ...

List-oriented data structures

Built-in functions (only talked about a small subset)


Miscellaneous Advanced Features Additional Resources

More Resources on bash


34/46

More Resources on bash

Man page for your particular bash (definitive for your version):man bash

bash Reference Manual:

http://www.gnu.org/software/bash/manualBeginners guide to bash (LDP):http://tldp.org/LDP/Bash-Beginners-Guide

Advanced bash Scripting (LDP):

http://tldp.org/LDP/abs


Miscellaneous Utilities (not just for bash) viewers

Viewers
http://www.gnu.org/software/bash/manualhttp://tldp.org/LDP/Bash-Beginners-Guidehttp://tldp.org/LDP/abshttp://tldp.org/LDP/abshttp://tldp.org/LDP/Bash-Beginners-Guidehttp://www.gnu.org/software/bash/manual


35/46

e e s

You do not need (and often do not want) to use an editor just to look at

a file (or a long listing of results, files in a directory, etc.), soLinux/UNIX provides viewing utilities that you can use:

head - prints the first N (default 10) lines of a file, e.g., head-30 filename

tail - prints the last N (default 10) lines of a file, e.g., tail-30 filename

cat - catenates a file, i.e. just spits it out to standard output(-n shows line numbers, -t shows non- printing charactersand tabs)

more - file filter, with pagination commands like viless - alternative to more intended to be faster (does not

attempt to parse the whole file) and with more freedom ofmovement. Cf. man less


Miscellaneous Utilities (not just for bash) sed

sed


36/46

sed, the stream editor, is one of the oldest of the UNIX editors (and

therefore probably the least user-friendly). It has a lot of power,though, and you will still find it heavily utilized in shell scripts. Somevery simple examples:

[ u 2 : ~ ] $ echo ' H e l l o , w o r l d ! ' # s e l f e x pl an a to r y ( n ot e s i n g l e q uot es )#

[ u 2 : ~ ] $ echo ' H e l lo , w o rl d ! ' | sed s / Hel lo / Goodbye/Goodbye, world ![ u 2 : ~ ] $ echo ' H e l lo , w o rl d ! ' | sed " s / He l l o / Goodbye , cr u e l / "Goodbye , cru e l , wo rl d ! # n o t e q u ot e u sag e


Miscellaneous Utilities (not just for bash) grep

grep


37/46

g p

grep is a handy little utility to perform searches on files for lines

containing a given string (the origin of the name dates back to UNIXsoriginal command editor, ed, namely g/re/p.Simple examples:

[ u 2 : ~ ] $ w > t m p . w # dump o u t pu t o f w t o a t em po ra ry f i l e , c f . ' man w '[ u 2 : ~] $ g re p $USER tmp .w # my c u r r en t l o g i ns t o t h i s machine[ u 2 : ~ ] $ g re p j o eu s e r tmp . w # j oe us e r ' s l o g i n s t o t h i s machine[ u2 : ~ ] $ w | g r ep $USER # we d on ' t need no s t i n k i n ' t em po ra ry f i l e s . . .

man grep for handy options (notable, -i and -n).


Miscellaneous Utilities (not just for bash) awk

awk


38/46

awk is a very powerful utility - it harbors a complex programming

environment in its own right, but you are free to use only as much of itas you want or need to at a given time. The general syntax for awklooks like:

awk ' / p a t t e r n / { a c t i o n } ' f i l e

a given action will be performed on every record which matches thegiven /pattern/. A very handy list of the fields in the record are

given as variables, $0 (entire record), $1 (first field), $2 (second field).... Lets try an example:

[ u2 : ~ ] $ q s ta t a > t m p . q # r ec or d c u r r en t s t a t e o f t he queues

[ u 2 : ~ ] $ g re p x d ta s tmp . q # our f r i e n d grep[ u 2 : ~ ] $ awk ' / x d ta s / { p r i n t $0 } ' tmp . q # same t h i n g w i t h awk

# more i n t e r e s t i n g e xam ple[ u 2 : ~ ] $ awk ' BEGIN { c o r es = 0 } / x d t a s / { c o r es += $7 } END { p r i n t c o r es } ' t mp . q

# add nod es , j o b s ?


Miscellaneous Utilities (not just for bash) awk

Exercise


39/46

Modify the preceding awk example to also count and report the

number of jobs and nodes as well as cores.


Miscellaneous Utilities (not just for bash) find

find


40/46

find is quite a handy command for searching for files in complexdirectory layouts - see man find for various options, the very basic

syntax looks like:

f i n d [ p a t h ] [ e x p r e s s i o n ]

where the expression involves options for just about every possible file

property that you might imagine - here are a few examples:# f i n d a l l f i l e s h e r e i n w i t h s u f f i x . nc[ u 2 : / s c r a t c h / jo nes m ] $ f i n d . name \* . n c p r i n t# n ar ro w t h e s ea rc h down a b i t[ u 2 : / s c r a t c h / jo nes m ] $ f i n d . name 2010\*. nc p r i n t# l o n g l i s t ea ch one[ u 2 : / s c r a t c h / jo nes m ] $ f i n d . name 2010\*. nc exec l s l { } \ ;# show me f i l e s i n d an ge r o f b ei n g s c r ub be d f ro m my panas as s c r a t c h d i r e c t o r y[ u 2 : ~ ] $ f i n d / p anas as / s c r a t c h / jo nes m a t i me +23

Note that the xargs command is also helpful when you want to dosomething useful with copious amounts of output from a command like

find.


Miscellaneous Utilities (not just for bash) And many more

Other utilities


41/46

There is a zoo of other useful commands/utilities, here is a samplingbased on what I have found to be useful:

file determines a files type, typically text, data, or executable, but can also identifyother common file types (e.g., it will recognize windows file types).

ldd prints shared library dependencies, useful for checking to see that alldependencies are satisfied.

cut slices files in a similar fashion to awk, but does so in a less complex way. Youcan specify bytes (-b), delimiters (-d), character position (-c), or fields (-f) from

which to extract.paste merges lines from files (or standard input).

join more flexible version of paste for sources containing common fields.

sort handy sorting utility (lexical, -n numeric, etc.).

split splits files into pieces, by lines (-l) or by bytes (-b) or a maximum of some bytesper line (-C).

tr translates or deletes characters, very useful for rapidly cleaning up files(common example, tr -d \r < infile.csv > outfile.csv, toremove windows carriage returns from a file)

uniq omit (or report) repeated lines.

wc newline (-l), word (-w), character (-m), or byte (-c) counts.


Miscellaneous Utilities (not just for bash) More CLI Resources

More CLI Resources


42/46

List of CLI related cheat sheets:http://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.html

The Linux documentation project:http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.html


Examples Simple Tool Script

Simple Tool Script
http://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.htmlhttp://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.htmlhttp://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.htmlhttp://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.html


43/46

Shell scripts are quite commonly used when deploying other tools -here is an example of one that is used to wrap a Java application

called Panoply1

1 [ u 2 : ~ ] $ l s p an op ly / P an op ly J2 c o l or b ar s j a r s o v er l ay s Panoply . j a r p ano pl y . s h README. t x t

3 [ u2 : ~ ] $ ca t panoply / PanoplyJ / panoply . sh4 # ! / b i n /sh5 #67 SCRIPT = ` r e a d l i n k f $0 `8 SCRIPTDIR=` dirname $SCRIPT`9

10 j a v a Xmx1GXms128m j a r $SCRIPTDIR / j a r s / Pan op ly . j a r "$@"

1http://www.giss.nasa.gov/tools/panoply


Examples Example PBS Script

Example PBS bash Script
http://www.giss.nasa.gov/tools/panoplyhttp://www.giss.nasa.gov/tools/panoply


44/46

A more complex example script for a scientific application on thecluster, which makes use of various utilities that we have discussed.

1 #PBS

S / b i n / b ash2 #PBS q debug3 #PBS l 4 : IB2 : p pn =84 #PBS l w a l l t i m e = 0 1 :0 0 :0 05 #PBS M jonesm@ccr. buffalo .edu6 #PBS m e7 #PBS j oe8 #PBS N t i t a n _ 3 2 p9 #PBS o p b s _ s c r i p t . o u t

10 #PBS

W g r o u p _ l i s t = gmfg11 #12 # Se t e xe cu ta b l e name & mp ie xe c p a th13 # EXE = Exe cu ta b l e name14 # I NITDIR = d i r h o ld in g i n p u t & ex ec ut ab le15 # SAVEDIR = d i r t o p la ce o ut pu t f i l e s16 #17 module l oa d t i t a n / i n t e l m p i18 IN IT DI R=$PBS_O_WORKDIR

19 SAVEDIR=$PBS_O_WORKDIR20 #21 # Get l i s t o f p r oc e s so r s f ro m PBS22 p l i s t = ` cat $PBS_NODEFILE | sed " s / \ . ccr \ . b u ff a l o \ . e du / / " `23 u p l i s t = `ca t $PBS_NODEFILE | un iq | sed " s / \ . ccr \ . b u ff a l o \ . e du / / " `24 echo " u ni qu e nodes l i s t = " $ u p l i s t25 echo " f u l l p ro c es s l i s t = " $ p l i s t




45/46

26 # S ta ge o u t i nd ex ed i n p u t f i l e s ( a nd common o nes t o o ) t o eac h27 # p ro ce ss or28 cd $PBS_O_WORKDIR29 NN=130 f o r p i n $ p l i s t ; do

31 l e t "NN = $NN + 1 "32 NUM= `echo $NN | awk ' { p r i n t f "%.4d" , $1} ' `33 echo " sta g i n g o u t funky$NUM . i n p to $p . . . "34 ssh $p " cp $ INITDIR / funky$NUM . i n p $PBSTMPDIR"35 done36 # f i l e s t h a t o nl y n eed t o appear o nce on a node , no m at te r37 # how many p ro cs / n od e38 f o r p i n $ u p l i s t ; do39 ssh $p " cp r $ INITDIR / * . da ta $PBSTMPDIR/ "

40 echo " Contents of "$PBSTMPDIR" on node "$p41 ssh $p " l s l $PBSTMPDIR/ "42 done43 NP= ` cat $PBS_NODEFILE | wc l `44 NP_UNIQ=` cat $PBS_NODEFILE | un iq | wcl `45 #46 # Run code fro m $PBSTMPDIR make s ur e t h a t you c opy back f i l e s47 # t h a t y ou n eed ( $PBSTMPDIR i s e ra s ed a t c o m p l e t i on o f t h e j o b )48 # N .B . you can a ls o run t i t a n from a s hared d i r e c t o ry w / o a l l

49 # t he e x t ra f i l e s ta gi ng , as l on g as t he i / o r eq ui re me nt s a re50 # r e as o n ab l y s m a l l ( a nd y ou do n o t ex ceed , s ay , a bo ut 16p )51 #52 cd $PBSTMPDIR53 mpirun np $NPROCS $EXE




46/46

54 #55 # S tage back o ut pu t f i l e s56 #57 f o r p i n $ u p l i s t ; do58 echo " Conte nts of $PBSTMPDIR on $p: "59 ssh $p " l s l $PBSTMPDIR"60 s sh $p " c p $PBSTMPDIR / x dm f

*$SAVEDIR / ; cp $PBSTMPDIR / pe rf or m

*$SAVEDIR / ; \

61 cp $PBSTMPDIR/ output_summary * $SAVEDIR / ; cp $PBSTMPDIR/ pi l e h e i g ht * $SAVEDIR/ "62 done
