shell programming guntis barzdins girts folkmanis

90
Shell Programming Guntis Barzdins Girts Folkmanis

Upload: nickolas-shannon-houston

Post on 29-Dec-2015

231 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Shell Programming Guntis Barzdins Girts Folkmanis

Shell Programming

Guntis BarzdinsGirts Folkmanis

Page 2: Shell Programming Guntis Barzdins Girts Folkmanis

Lecture outline

Shell features Helper utilities, introduction Connecting utilities with shell scripting Helper utilities in detail Piping, advanced examples Shell scripts as files, Internal shell commands

Page 3: Shell Programming Guntis Barzdins Girts Folkmanis

Shell features

We will talk about bash, there might be differences for other shells. bash - GNU Bourne-Again Shell Authors: Brian Fox and Chet Ramey Free Software Foundation

Popular in different distributions Tip: To find your current shell, type following command

$ echo $SHELL/bin/bash

Page 4: Shell Programming Guntis Barzdins Girts Folkmanis

Shell Features

The shell itself is defined in SUS (Single UNIX Specification) as regards calling conventions and switches. The language interpreted by the shell is also part of the standard

The shell standard derives from the POSIX.2 standard, which is not freely available (the current standard, SUS, stands as IEEE Std1003.1 2001 and is identical to POSIX.2)

Page 5: Shell Programming Guntis Barzdins Girts Folkmanis

Shell features

Two types of usage: Command line - interactive Shell script, usually non-interactive

Shell script defined as: "Shell Script is series of commands written in plain

text file. Shell script is just like batch file is MS-DOS but have more power than the MS-DOS batch file."

Page 6: Shell Programming Guntis Barzdins Girts Folkmanis

Shell features

Two types of commands: Internal commands – built in the shell interpreter External commands – calling other executable files

Almost everything applies to both command line usage and shell scripts

Page 7: Shell Programming Guntis Barzdins Girts Folkmanis

External commands

Execution of external programs – most common task

girtsf@linux tmp $ ls -l /lib total 4035-rwxr-xr-x 1 root root 7488 Oct 6 12:33 cppdrwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-statedrwxr-xr-x 2 root root 1024 Jun 28 09:53 evmsdrwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables-rwxr-xr-x 1 root root 92716 Oct 14 13:10 ld-2.3.4.so-rwxr-xr-x 1 root root 22800 Oct 14 13:17 ld-linux.so.1...

External program: /bin/ls

Page 8: Shell Programming Guntis Barzdins Girts Folkmanis

External commands

Environment variable $PATH determines where to search for external programs.

girtsf@linux tmp $ echo $PATH

/bin:/usr/bin:/usr/local/bin:/opt/bin

“:” as separator Current directory “.” is usually not in PATH for

security reasons.

Page 9: Shell Programming Guntis Barzdins Girts Folkmanis

External commands

girtsf@linux tmp $ echo $PATH

/bin:/usr/bin:/usr/local/bin:/opt/bin

With /bin in path, typing “ls” suffices to run /bin/ls. Example of unsetting path: girtsf@linux tmp $ unset PATH

girtsf@linux tmp $ ls

bash: ls: No such file or directory

girtsf@linux tmp $

Page 10: Shell Programming Guntis Barzdins Girts Folkmanis

Internal commands

A large list of built in commands, that are handled internally without running an external command

Most commonly used internal command is cd, used to change the current working directory:

girtsf@linux girtsf $ cd /tmp/

girtsf@linux tmp $

Page 11: Shell Programming Guntis Barzdins Girts Folkmanis

Aliasing

Aliasing is the process of assigning a command to a shorter “alias”

This allows you to type the shorter command instead of the longer one.

Aliasing is useful for changes that you want all of the time. alias rm “rm –i”

Aliasing is similar to shell function definitions dos2unix() { cat $1 | perl -pe 's/\r\n$/\n/g'; } unix2dos() { cat $1 | perl -pe 's/\n$/\r\n/g'; }

Page 12: Shell Programming Guntis Barzdins Girts Folkmanis

Internal commands

girtsf@linux tmp $ helpGNU bash, version 2.05b.0(1)-release (i686-pc-linux-gnu)These shell commands are defined internally. Type `help' to see this

list.Type `help name' to find out more about the function `name'.Use `info bash' to find out more about the shell in general.Use `man -k' or `info' to find out more about commands not in this list.

A star (*) next to a name means that the command is disabled.

%[DIGITS | WORD] [&] (( expression )) . filename : [ arg... ] [[ expression ]] alias [-p] [name[=value] ... ] bg [job_spec] bind [-lpvsPVS] [-m keymap] [-f fi break [n] builtin [shell-builtin [arg ...]] case WORD in [PATTERN [| PATTERN]. cd [-L|-P] [dir] command [-pVv] command [arg ...]...

Page 13: Shell Programming Guntis Barzdins Girts Folkmanis

SUS: shell grammarcomplete_command : list separator | list ;list : list separator_op and_or | and_or ;and_or : pipeline | and_or AND_IF linebreak pipeline | and_or OR_IF linebreak pipeline ;pipeline : pipe_sequence | Bang pipe_sequence ;pipe_sequence : command | pipe_sequence '|' linebreak command ;command : simple_command | compound_command | compound_command redirect_list | function_definition ;compound_command : brace_group | subshell | for_clause | case_clause | if_clause | while_clause | until_clause ;subshell : '(' compound_list ')' ;compound_list : term | newline_list term | term separator | newline_list term separator ;term : term separator and_or | and_or ;for_clause : For name linebreak do_group | For name linebreak in sequential_sep do_group | For name linebreak in wordlist sequential_sep do_group ;name : NAME /* Apply rule 5 */ ;in : In /* Apply rule 6 */ ;wordlist : wordlist WORD | WORD ;case_clause : Case WORD linebreak in linebreak case_list Esac | Case WORD linebreak in linebreak case_list_ns Esac | Case WORD linebreak in linebreak Esac ;case_list_ns : case_list case_item_ns | case_item_ns ;case_list : case_list case_item | case_item ;case_item_ns : pattern ')' linebreak | pattern ')' compound_list linebreak | '(' pattern ')' linebreak | '(' pattern ')' compound_list linebreak ;case_item : pattern ')' linebreak DSEMI linebreak | pattern ')' compound_list DSEMI linebreak | '(' pattern ')' linebreak DSEMI linebreak | '(' pattern ')' compound_list DSEMI linebreak ;pattern : WORD /* Apply rule 4 */ | pattern '|' WORD /* Do not apply rule 4 */ ;if_clause : If compound_list Then compound_list else_part Fi | If compound_list Then compound_list Fi ;else_part : Elif compound_list Then else_part | Else compound_list ;while_clause : While compound_list do_group ;until_clause : Until compound_list do_group ;function_definition : fname '(' ')' linebreak function_body ;function_body : compound_command /* Apply rule 9 */ | compound_command redirect_list /* Apply rule 9 */ ;fname : NAME /* Apply rule 8 */ ;brace_group : Lbrace compound_list Rbrace ;do_group : Do compound_list Done /* Apply rule 6 */ ;simple_command : cmd_prefix cmd_word cmd_suffix | cmd_prefix cmd_word | cmd_prefix | cmd_name cmd_suffix | cmd_name ;cmd_name : WORD /* Apply rule 7a */ ;cmd_word : WORD /* Apply rule 7b */ ;cmd_prefix : io_redirect | cmd_prefix io_redirect | ASSIGNMENT_WORD | cmd_prefix ASSIGNMENT_WORD ;cmd_suffix : io_redirect | cmd_suffix io_redirect | WORD | cmd_suffix WORD ;redirect_list : io_redirect | redirect_list io_redirect ;io_redirect : io_file | IO_NUMBER io_file | io_here | IO_NUMBER io_here ;io_file : '<' filename | LESSAND filename | '>' filename | GREATAND filename | DGREAT filename | LESSGREAT filename | CLOBBER filename ;filename : WORD /* Apply rule 2 */ ;io_here : DLESS here_end | DLESSDASH here_end ;here_end : WORD /* Apply rule 3 */ ;newline_list : NEWLINE | newline_list NEWLINE ;linebreak : newline_list | /* empty */ ;separator_op : '&' | ';' ;separator : separator_op linebreak | newline_list ;sequential_sep : ';' linebreak | newline_list ;

%token WORD%token ASSIGNMENT_WORD%token NAME%token NEWLINE%token IO_NUMBER

%token AND_IF OR_IF DSEMI/* '&&' '||' ';;' */

%token DLESS DGREAT LESSAND GREATAND LESSGREAT DLESSDASH/* '<<' '>>' '<&' '>&' '<>' '<<-' */

%token CLOBBER/* '>|' */

/* The following are the reserved words. */

%token If Then Else Elif Fi Do Done/* 'if' 'then' 'else' 'elif' 'fi' 'do' 'done' */

%token Case Esac While Until For/* 'case' 'esac' 'while' 'until' 'for' */

/* These are reserved words, not operator tokens, and are recognized when reserved words are recognized. */

%token Lbrace Rbrace Bang/* '{' '}' '!' */

%token In/* 'in' */

Page 14: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

Helper utilities – various small external programs that are helpful when working with shell scripts or command line

Called from shell (scripts or command line) Somehow transforms input into output, based on

the parameters

Page 15: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

cat - concatenate files and print on the standard output

Syntax: cat [file1] [file2] … [fileN]

girtsf@linux etc $ cat gentoo-release shells Gentoo Base System version 1.4.16# /etc/shells: valid login shells# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/shells,v 1.5

2003/07/15 20:36:32 azarah Exp $/bin/sh/bin/bash/bin/tcsh/bin/csh/bin/esh/bin/ksh/bin/zsh/bin/sash

Page 16: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

echo – displays a line of text Besides a program /bin/echo, also usually

built in the shell (takes precedence) Syntax: echo [STRING] ...

girtsf@linux girtsf $ echo quick brown foxquick brown fox

Can be used to display environment variablesgirtsf@linux girtsf $ echo $HOME/home/girtsf

Page 17: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

wc - print the number of newlines, words, and bytes in files wc [options] [file1] [file2] … [fileN]

By default, newlines, words and byte counts are displayed

Options -c : print only byte count -w : print only word count -l : print only line count

Page 18: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

Example use of wc:girtsf@linux etc $ wc /etc/passwd

50 76 2257 /etc/passwd

girtsf@linux etc $ wc -l /etc/passwd

50 /etc/passwd

lines words bytes

lines only

Page 19: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

grep - print lines matching a pattern grep PATTERN [file1] [file2] … [fileN]

The lines that contain PATTERN are printed to standard output.

If no files are specified, input is taken from standard input (more later).

Advanced versions of grep allow using regular expressions in PATTERN.

Page 20: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

File “testfile” contains the following linesgirtsf@linux girtsf $ cat testfilethe quick brownfox jumped overthe lazy dog

We search for “the”:girtsf@linux girtsf $ grep the testfilethe quick brownthe lazy dog

Only lines containing the substring “the” are printed.

Page 21: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

Some useful parameters for grep: -i : ignore case (“the” finds “the”, “The”, “THE”,…) -l : output only filenames that match, not the contents -B <n> : output also n lines before the matching line -A <n>: output also n lines after the matching line

See the man page (“man grep”) for all parameters

Page 22: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

tee - read from standard input and write to standard output and files Syntax: tee [File1] [File2] .. [FileN] Example of tee taking user’s input from terminal and writing to 3 files:

girtsf@linux tmp $ tee a b csome string^Dsome stringgirtsf@linux tmp $ cat asome stringgirtsf@linux tmp $ cat bsome stringgirtsf@linux tmp $ cat csome string

In red – my input, ending with Control-D, which is the EOF (End of File) character.

This input is read as standard input by tee.

Page 23: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

Any program can be used as a helper program More examples later

Page 24: Shell Programming Guntis Barzdins Girts Folkmanis

Connecting utilities with shell scripting

Standard I/O I/O redirection to/from file I/O redirection using a pipe Backticks

Page 25: Shell Programming Guntis Barzdins Girts Folkmanis

Standard I/O

Every process, when run, has 3 already open data streams (file descriptors): Standard input Standard output Standard error

Page 26: Shell Programming Guntis Barzdins Girts Folkmanis

Standard I/O

When run interactively (from command line), these streams are attached to the terminal they are running from Standard input is attached to user’s keyboard input Standard output is attached to user’s terminal output Standard error, similarly to output, is attached to

user’s terminal output Usually referred to as stdin, stdout, stderr.

Page 27: Shell Programming Guntis Barzdins Girts Folkmanis

Standard output & error

“ls” command does not use input, but uses stdout, stderr. The second line is the stdout from “ls” command:

girtsf@linux etc $ ls -l /etc/passwd

-rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd

The second line is from stderr from “ls” command:girtsf@linux etc $ ls -l /etc/asdfasdf

ls: /etc/asdfasdf: No such file or directory

Both stdout and stderr simultaneously:girtsf@linux tmp $ ls -l /etc/passwd /etc/asdfasdf

ls: /etc/asdfasdf: No such file or directory

-rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd

Page 28: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection to/from file

By default, the 3 streams are attached to terminal This can be overridden when executing the

command and is called “redirection” “>” specifies that stdout is redirected to file “<“ specifies that stdin is taken from file “2>” specifies that stderr is redirected to file

Page 29: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection to/from file

Syntax: <cmd> [ > <file1>] [ < <file2> ] [ 2> <file3> ] For those redirections that are specified, the

respective stream will be attached to the specified file

None, one, two or all three types can be specified If output file exists: > - replace file; >> - append to file

Page 30: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection to file

Example of stdout redirection to filegirtsf@linux tmp $ ls -l /lib/ > direktorijas_saraksts

girtsf@linux tmp $ cat direktorijas_saraksts

total 4035

-rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp

drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state

drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms

drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables

...

Page 31: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection to file

Example of stdout redirection to filegirtsf@linux tmp $ ls -l /asdf > direktorijas_saraksts

ls: /asdf: No such file or directory

girtsf@linux tmp $ cat direktorijas_saraksts

The file is empty, as no output was sent to stdout, as error message was send to stderr, which still was attached to user’s terminal

Page 32: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection to file

Example of stderr redirection to filegirtsf@linux tmp $ ls -l /asdfasdf 2> errlog

girtsf@linux tmp $ cat errlog

ls: /asdfasdf: No such file or directory

Now stderr was redirected to file and file contained the error message.

Page 33: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection to file

Example of stdout, stderr redirection to filegirtsf@linux tmp $ ls -l /asdfasdf /lib 2>errlog >sargirtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directorygirtsf@linux tmp $ cat sar/lib:total 4035-rwxr-xr-x 1 root root 7488 Oct 6 12:33 cppdrwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-statedrwxr-xr-x 2 root root 1024 Jun 28 09:53 evmsdrwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables...

Page 34: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection from file

Example of stdin redirection First, we create file “a” with the following content

the quick brown foxjumped over a quick brown fox

Use wc (word count) by not supplying the file name, but redirecting standard input

girtsf@linux tmp $ wc < a 2 10 50

Page 35: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection with pipes

Task: given a file, output the total number of words in those lines, that contain substring “the”.

Example input:girtsf@linux girtsf $ cat testfile

the quick brown

fox jumped over

the lazy dog

Lines 1 and 3 match, total number of words = 6.

Page 36: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection with pipes

Solution with redirection to files: First find the lines, save them into temp file Then use wc (word count) utility to count the number

of wordsgirtsf@linux girtsf $ grep the testfile > tmpfile

girtsf@linux girtsf $ wc –w < tmpfile

6

Page 37: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection with pipes

Temporary file was used to redirect the standard output of grep to file

The standard input to wc was taken from temporary file

Easier way – connect the standard output of one program to standard input of another one directly

Page 38: Shell Programming Guntis Barzdins Girts Folkmanis

I/O Redirection with pipes

Syntax: program1 | program2 (| - pipe symbol) Called “piping output from one program to

another” This example:girtsf@linux girtsf $ grep the testfile | wc -w

6

No temporary files. Elegant!

Page 39: Shell Programming Guntis Barzdins Girts Folkmanis

Backticks

Backticks – reverse apostrophes “`” (usually the same key as tilde ~ character)

Using backticks sub-commands are executed, their result written in the place they are defined

Example:girtsf@linux tmp $ cd `echo /home`

girtsf@linux home $

Substituted with “/home”

Page 40: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

We will examine the following utilities: cut sort uniq awk sed

Page 41: Shell Programming Guntis Barzdins Girts Folkmanis

cut

cut - remove sections from each line of files Syntax: cut [OPTION]... [FILE]... Options:

-d DELIM : use DELIM instead of TAB character -f LIST : output only these fields (delimited by DELIM) -c LIST : output only these characters

See man page for more options

Page 42: Shell Programming Guntis Barzdins Girts Folkmanis

cut

Example – output second word on each line: Delimiter: space “ “ Fields: 2

girtsf@linux tmp $ cat athe quick brown foxjumped over a quick brown foxgirtsf@linux tmp $ cut -f 2 -d ' ' aquickover

Page 43: Shell Programming Guntis Barzdins Girts Folkmanis

cut

Example – output characters 1-3, 5, 7-end Use –c to choose the needed characters

girtsf@linux tmp $ cat a

the quick brown fox

jumped over a quick brown fox

girtsf@linux tmp $ cut -c 1-3,5,7- a

theqick brown fox

jume over a quick brown fox

Page 44: Shell Programming Guntis Barzdins Girts Folkmanis

sort

sort - sort lines of text files sort [OPTION]... [FILE]...

Writes sorted concatenation of all FILE(s) to standard output.

Interesting options: -r : reverse -n : compare according to string numerical value

See man page for more options

Page 45: Shell Programming Guntis Barzdins Girts Folkmanis

sort

sort - sort text file reversed

girtsf@linux tmp $ cat a fishdoganimalbirdgirtsf@linux tmp $ sort -r afishdogbirdanimal

Page 46: Shell Programming Guntis Barzdins Girts Folkmanis

sort

Sort numeric file (as text)girtsf@linux tmp $ cat a5412 this line should go last998 this line should go second50 this line should go first999 this line should go thirdgirtsf@linux tmp $ sort a50 this line should go first5412 this line should go last998 this line should go second999 this line should go third

Page 47: Shell Programming Guntis Barzdins Girts Folkmanis

sort

Sort numeric file as numbersgirtsf@linux tmp $ cat a5412 this line should go last998 this line should go second50 this line should go first999 this line should go thirdgirtsf@linux tmp $ sort -n a50 this line should go first998 this line should go second999 this line should go third5412 this line should go last

Page 48: Shell Programming Guntis Barzdins Girts Folkmanis

uniq

uniq - remove duplicate lines from a sorted file uniq [OPTION]... [INPUT [OUTPUT]]

Discards all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

Can be used together with sort, to get file without duplicate lines.

Page 49: Shell Programming Guntis Barzdins Girts Folkmanis

uniq

Just sorted:

$ cat a | sortbirdbirddogdogfishfishfly

sort | uniq:

$ cat a | sort | uniqbirddogfishfly

Page 50: Shell Programming Guntis Barzdins Girts Folkmanis

awk

gawk (GNU awk) - pattern scanning and processing language.

Gawk is the GNU Project's implementation of the AWK programming language. It conforms to the definition of the language in the POSIX 1003.2 Command Language And Utilities Standard. This version in turn is based on the description in The AWK Programming Language, by Aho, Kernighan, and Weinberger, with the additional features found in the System V Release 4 version of UNIX awk.

Page 51: Shell Programming Guntis Barzdins Girts Folkmanis

awk

Complete language interpreter Variables User defined functions …

Useful for small one-liners Just some examples will be given See man page or search for tutorials on the net

Page 52: Shell Programming Guntis Barzdins Girts Folkmanis

awk

Task: sum all the numbers in the file:$ cat a

1

2

3

4

5

$ cat a | awk '{ sum += $1 } END { print sum }'

15

executed on every line executed at the end

first field

Page 53: Shell Programming Guntis Barzdins Girts Folkmanis

awk

Sum 2nd field (separated by colons) of those lines, that contain letter “a”

girtsf@linux tmp $ cat azzz:1:azzz:2:bzzz:3:bzzz:4:bzzz:5:agirtsf@linux tmp $ cat a | awk -F ':' '{ if(/a/)

sum += $2; } END { print sum }'6

field delimiter

Page 54: Shell Programming Guntis Barzdins Girts Folkmanis

sed

sed – stream editor A stream editor is used to perform basic text

transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.

Simpler than awk, smaller feature set. Just some examples will be given. See man page or search for tutorials on the net.

Page 55: Shell Programming Guntis Barzdins Girts Folkmanis

sed

Replace some substring with another$ cat a

bird barks

mouse runs

$ sed 's/barks/flies/' < a

bird flies

mouse runs

Regular expression

Page 56: Shell Programming Guntis Barzdins Girts Folkmanis

sed

Replace some characters with others Replacing ‘b’ with ‘Q’, ‘i’ with ‘X’

girtsf@linux tmp $ cat a

bird barks

mouse runs

girtsf@linux tmp $ cat a | sed 'y/bi/QX/'

QXrd Qarks

mouse runs

Page 57: Shell Programming Guntis Barzdins Girts Folkmanis

Advanced example 1

Calculate total bytes transferred in Apache log file. First, take only successful lines (containing error code 200) Sum up byte field

Line format:

159.148.123.123 - - [28/Oct/2004:18:11:36 +0300] "GET /somefolder/file.php HTTP/1.1" 200 127602 "-" "Opera/7.54 (X11; Linux i686; U) [en]"

Page 58: Shell Programming Guntis Barzdins Girts Folkmanis

Advanced example 1 cont.

$ cat access_log | grep ' 200 ' | awk '{ bytes += $10 } END { print bytes }'

1105653994

Cat file Grep for “ 200 “ Sum up 10th column, output it at the end

Page 59: Shell Programming Guntis Barzdins Girts Folkmanis

Advanced example 2

Calculate number of hits per remote host in Apache log file, most active hosts first.

Line format:

159.148.123.123 - - [28/Oct/2004:18:11:36 +0300] "GET /somefolder/file.php HTTP/1.1" 200 127602 "-" "Opera/7.54 (X11; Linux i686; U) [en]"

Page 60: Shell Programming Guntis Barzdins Girts Folkmanis

Advanced example 2

$ cat access_log | cut -d ' ' -f 1 | sort | uniq -c | sort –n -r

First, cut out the host part (1st field), sort it get the number of repeated lines before the line (uniq –c : prefix

lines by the number of occurrences), sort it numerically, reversed so that largest number comes first Output:348698 159.148.111.222123485 159.148.48.54 12313 80.123.123.4

...

Page 61: Shell Programming Guntis Barzdins Girts Folkmanis

Helper utilities

Some more utilities that can be useful head, tail - output the first/last part of files basename - strip directory and suffix from filenames bc - an arbitrary precision calculator sleep – sleep specified number of seconds tr – translate or delete characters true, false – always return success/error code

Read the man pages

Page 62: Shell Programming Guntis Barzdins Girts Folkmanis

Tips for working in a shell

Up arrow – repeats the previous command history – internal bash command that shows command

history Use script to make a typescript of terminal session CTRL-R and a few chars recalls last command that

contains these chars: (reverse-i-search)`re': grep text file*

Page 63: Shell Programming Guntis Barzdins Girts Folkmanis

The ABCs of Unix

A is for awk, which runs like a snail B is for biff, which reads all your mail C is for cc, as hackers recall D is for dd, the command that does all E is for emacs, which rebinds your keys F is for fsck, which rebuilds your trees G is for grep, a clever detective H is for halt, which may seem defective I is for indent, which rarely amuses J is for join, which nobody uses K is for kill, which makes you the boss L is for lex, which is missing from DOS M is for more, from which less was begot

N is for nice, which really is not O is for od, which prints out things nice P is for passwd, which reads in strings twice Q is for quota, a Berkeley-type fable R is for ranlib, for sorting a table S is for spell, which attempts to belittle T is for true, which does very little U is for uniq, which is used after sort V is for vi, which is hard to abort W is for whoami, which tells you your name X is, well, X, of dubious fame Y is for yes, which makes an impression, and Z is for zcat, which handles compression

Page 64: Shell Programming Guntis Barzdins Girts Folkmanis

Another UNIX command refference

Use man pages for further details

Page 65: Shell Programming Guntis Barzdins Girts Folkmanis

Shell scripts as files

Everything that can be called from command line, can also be called from shell script

Shell Script is series of commands written in plain text file.

Shell script is just like batch file is MS-DOS but have more power than the MS-DOS batch file.

Page 66: Shell Programming Guntis Barzdins Girts Folkmanis

Running a shell script

As a parameter to shell interpreterbash some_script.sh

Specifying interpreter on first lineFirst line:

#!/bin/bashMake it executable (chmod +x some_script.sh) ./some_script.sh

Page 67: Shell Programming Guntis Barzdins Girts Folkmanis

Intercative shell coniguration files

Page 68: Shell Programming Guntis Barzdins Girts Folkmanis

Basics

Interpreted line by line The same effect when entering the lines one by

one in interactive shell

Page 69: Shell Programming Guntis Barzdins Girts Folkmanis

Sample shell script

#!/bin/bash# comment lineecho "what a fine day: "date Output, when called by “./test.sh”:what a fine day: Thu Oct 28 23:37:39 EEST 2004

Interpreter to be used

Regular commands to execute

Page 70: Shell Programming Guntis Barzdins Girts Folkmanis

Variables

Sample “hello world” with variables #!/bin/bash STR="Hello World!"echo $STR When assigning, no $ is used When getting the contents – use $ No data types – string, number, character, all the same No declaring, just assign

Page 71: Shell Programming Guntis Barzdins Girts Folkmanis

Another sample shell script

#!/bin/bashDATE=`date +%Y%m%d`WHAT='/home/girtsf'DEST="/backups/$DATE.tgz"tar cvzf $DEST $WHAT Results in calling:tar cvzf /backups/20041028.tgz /home/girtsf

Page 72: Shell Programming Guntis Barzdins Girts Folkmanis

Conditionals

Example:#!/bin/bashT1="foo"T2="bar"if [ "$T1" = "$T2" ]; thenecho expression evaluated as true

elseecho expression evaluated as false

fi

Page 73: Shell Programming Guntis Barzdins Girts Folkmanis

Command line arguments

Automatically defined variables $0 – contains shell script name $1 – contains first argument $2 – 2nd … $* - contains all arguments as a string

Page 74: Shell Programming Guntis Barzdins Girts Folkmanis

Command line arguments & conditionals example

#!/bin/sh##Script to print file#if cat $1thenecho -e "\n\nFile $1, found and successfully echoed"

fi

returns error code

usually 0 – no error

if executes branch if command returns 0

Page 75: Shell Programming Guntis Barzdins Girts Folkmanis

Comparisons

Either “test <expression>” or “ [ <expression> ] “ if test $1 -gt 0 if [ $1 -gt 0 ]

Numeric comparisons: -eq, -ne equal/not equal -lt, -le, -gt, -ge : less than, less than or equal, greater

than, greater than or equal

Page 76: Shell Programming Guntis Barzdins Girts Folkmanis

Comparisons

String comparisons string1 = string2 string1 is equal to string2 string1 != string2 string1 is NOT equal to string2 string1 string1 is NOT NULL or not defined -n string1 string1 is NOT NULL and does exist

-z string1 string1 is NULL and does exist

Page 77: Shell Programming Guntis Barzdins Girts Folkmanis

File tests

-s file Non empty file -f file File exist or normal file and not a

directory -d dir Directory exist and not a file -w file Is writeable file -r file Is read-only file -x file File is executable

Page 78: Shell Programming Guntis Barzdins Girts Folkmanis

Logical Operators

! expressionLogical NOT expression1 -a expression2 Logical AND expression1 -o expression2 Logical OR

Page 79: Shell Programming Guntis Barzdins Girts Folkmanis

Another example

#!/bin/sh# Script to test if..elif...else#if [ $1 -gt 0 ]; then echo "$1 is positive"elif [ $1 -lt 0 ]then echo "$1 is negative"elif [ $1 -eq 0 ]then echo "$1 is zero"else echo "Opps! $1 is not number, give number"fi

Page 80: Shell Programming Guntis Barzdins Girts Folkmanis

Arithmetic expansion

Arithmetic expansion allows the evaluation of an arithmetic expression and the substitution of the result. The format for arithmetic expansion is: $((expression))

A=5B=4C=$(($A*$B))echo $C

Page 81: Shell Programming Guntis Barzdins Girts Folkmanis

Loops: for

for { variable name } in { list }do

execute one for each item in the list until the list is not finished (And repeat all statement

between do and done)done

for i in 1 2 3 4 5do echo "Welcome $i times"done

Page 82: Shell Programming Guntis Barzdins Girts Folkmanis

Loops: for

for (( expr1; expr2; expr3 )) do

repeat while expr2 is truedone

for (( i = 0 ; i <= 5; i++ ))do echo "Welcome $i times"done

Page 83: Shell Programming Guntis Barzdins Girts Folkmanis

Using for

You can use for together with file name expansion to perform the same action for several files

#!/bin/shfor x in *txt; do cat $xdone;

Page 84: Shell Programming Guntis Barzdins Girts Folkmanis

Loops: while

while [ condition ]do

command1command2command3....

done

Page 85: Shell Programming Guntis Barzdins Girts Folkmanis

Many more features

See man page Linux Shell Scripting Tutorial: A Beginner's

handbook: http://www.freeos.com/guides/lsst/

Page 86: Shell Programming Guntis Barzdins Girts Folkmanis

Word guessing game

Source: http://www.ltn.lv/~guntis/unix/mini.txt

+------ |/ | | o | O | / |

MATER_ALJuusu mineejums: i

Page 87: Shell Programming Guntis Barzdins Girts Folkmanis

Main partizveleties_vardustripasgaj=0while true;do

zimet_karatavas $gaj

echo $vecho -n "Juusu mineejums: "read ievgajiens $iev

if [[ $v == $vards ]]; thenuzvara; exit

fi

if [[ $gaj -eq 10 ]]; thenzaude; exit

fi

done;

Page 88: Shell Programming Guntis Barzdins Girts Folkmanis

Getting a random word

izveleties_vardu(){local sk r

if [[ -e /usr/share/dict/words ]];then

echo Njemu vienu nejaushu vaardu no /usr/share/dict/words!sk=`cat /usr/share/dict/words|wc -l`r=$(($RANDOM % $sk + 1))vards=`tail +$r /usr/share/dict/words|head -1|tr a-z A-Z`

elseecho /usr/share/dict/words nav atrasts, njemu nokluseeto vaardu!vards="KARATAVAS"

fisleep 1}

Page 89: Shell Programming Guntis Barzdins Girts Folkmanis

Convert letters to underscores

stripas(){

local ii=${#vards}v=while [[ $i>0 ]];doi=$(($i-1))v=${v}_

done}

Page 90: Shell Programming Guntis Barzdins Girts Folkmanis

Comparison part

while [[ $i < $l ]];do

t=${vards:$i:1}if [[ $t == $b ]];then

v2=${v2}$bir=$(($ir+1))

elsev2=${v2}${v:$i:1}

fi

i=$(($i+1))done