unix_day2

8/4/2019 UNIX_DAY2

http://slidepdf.com/reader/full/unixday2 1/72

Tech Mahindra Limited confidential© Tech Mahindra Limited 2008

UNIX Fundamentals and Scripting

Day 2

8/4/2019 UNIX_DAY2


4/19/2012 CONFIDENTIAL© Copyright 2008 Tech Mahindra Limited 2

Agenda: Day 2

Regular Expressions and grep

Unix Shell

Unix Shell Environment

Unix Shell Scripting

8/4/2019 UNIX_DAY2


34/19/2012

Regular Expressions and Grep

8/4/2019 UNIX_DAY2



Regular Expression:

Often called a pattern, is an expression that describes a set of

strings

e.g.

a regular expression “amit”

may match “amit”, “amita”,”amitabh”, “namit” etc…

Many UNIX tools, primarily grep, sed & awk make use of regularexpressions in text processing

8/4/2019 UNIX_DAY2



Regular Expressions can be divided into:

Basic regular expressions (BRE) Supported by grep

support extended regular expressions (ERE) Supported by grep –E or egrep

grep –E or egrep supports both BRE as well as ERE

8/4/2019 UNIX_DAY2



Basic Regular Expression

\ Quote the next metacharacter

^ Match the beginning of the line

. Match any character

$ Match the end of the line

[] Character class

Special Operators

8/4/2019 UNIX_DAY2

http://slidepdf.com/reader/full/unixday2 7/724/19/2012 CONFIDENTIAL© Copyright 2008 Tech Mahindra Limited 7

Extended Regular Expression

| Alternation

() Grouping

Special Operators

8/4/2019 UNIX_DAY2


“a.g” matches aag,abg,a1g etc

“a[pmt]g” matches apg,amg or atg

“a[^pmt]g” matches aag,abg but not apg or amg or atg

“^ftp” matches ftp at the beginning of a line

“tle$” matches tle at the end of a line

“^$” matches a line with nothing in it

“jelly|cream” Either jelly or cream

“(eg|pe)gs” Either eggs or pegs

Examples

8/4/2019 UNIX_DAY2


BRE

ERE

* Match 0 or more times

Quantifiers: number of repeats of the previous charatcer

+ Match 1 or more times

? Match 1 or 0 times

8/4/2019 UNIX_DAY2


“adg*” ad followed by zero or more g characters

“.*” Any character, any number of times

“[qjk]” Either q or j or k

“[^qjk]” Neither q nor j nor k

“[a-z]” Anything from a to z inclusive

“[^a-z]” No lower case letters

“[a-zA-Z]” Any letter

“[a-z]+” Any non-zero sequence of lower case letters

“(da)+” Either da or dada or dadada or...

Examples

8/4/2019 UNIX_DAY2


Filters

grep: Is called as a global regular expression printer.

It searches for a given pattern in a file(s)

$ grep -[cvnl] [pattern] [files]

Option Description

-c counts the total no of lines containing the pattern.

-v displays all the lines not containing the pattern.-n displays lines with line no.

-l displays file names containing the pattern.

8/4/2019 UNIX_DAY2


Filters

grep “Agg*[ra][ra]wal” datalist

It lists all lines where the pattern matches some text The possible matches for the text are ( and many more

combinations possible)

Agrawal , Agarwal, Aggarwal, Aggrawal

grep “^A.*” datalist

This regular expression matches the lines which start with A andfollowed by any text (.*)

8/4/2019 UNIX_DAY2


Filters

egrep: Extended grep, used to search with full regularexpressions

$ egrep „ (John|Johnathon) Smith „ employee.txt

This will search for John Smith as well as for Johnathon

Smith.

grep –E “(S|Sh)arma” datalist

Matches Sarma or Sharma in the text from datalist

8/4/2019 UNIX_DAY2


Filters

fgrep: fast searching for fixed strings

$ fgrep string file(s)

It handles fixed character strings as text patterns.

Does not use regular expressions.

Faster than grep and egrep for searching text strings

Example:

fgrep “Ramesh” datalist

If found, lists the line(s) containing Ramesh

8/4/2019 UNIX_DAY2

http://slidepdf.com/reader/full/unixday2 15/7214/19/2012

Unix Shell

8/4/2019 UNIX_DAY2



Shell Features

Command Prompt Command Interpretation and Shell Meta Characters

I/O Redirection

Pipes

Shell Programming Constructs

Job Control Command History

8/4/2019 UNIX_DAY2



Shell as a Command Interpreter

Shell prints the prompt

on screen

User enters the

command

Shell interprets and

executes the particularcommand

Shell waits till thecommand finishes

execution

8/4/2019 UNIX_DAY2



Popular Unix Shells

Bourne Shell

Korn Shell

C Shell

Bourne Again Shell

8/4/2019 UNIX_DAY2



Bourne Shell

The Bourne shell is the original UNIX shell program.

It is very widely used.

Invoked using sh or /bin/sh at the command prompt.

The Bourne shell supports conditional branching in the form of

if/then/else statements.

In addition, the Bourne shell supports case statements and loops(for, while, and until).

The Bourne shell uses the $ as a prompt.

8/4/2019 UNIX_DAY2



Korn Shell

The Korn shell is a much newer variation of the Bourne shell.

It supports everything the Bourne shell does, and adds features

not available in the Bourne shell.

Invoked using ksh or /bin/ksh at the command shell prompt.

The Korn shell was originally written by David Korn and iscopyrighted by AT&T.

The programming structure of the Korn shell is very similar to thatof the Bourne shell. The Korn shell, however, is more interactive.

8/4/2019 UNIX_DAY2



C Shell

The C shell is a very commonly used shell.

Its programming structure closely resembles that of theprogramming language "C."

The C shell uses the "%" as a prompt

The C shell supports all of the features that the Bourne shellsupports, and has a more natural syntax for programming

8/4/2019 UNIX_DAY2



Bourne-Again Shell

The Bourne-Again shell is a variation of the Bourne shell

It is commonly used in Linux, but is widely available in otherstandard UNIX distributions

The Bourne Again shell is another modification of the Bourneshell, and uses the $ as a prompt

To start the Bourne Again shell, type "bash" at the shellprompt

8/4/2019 UNIX_DAY2

http://slidepdf.com/reader/full/unixday2 23/72Tech Mahindra Limited confidential© Tech Mahindra Limited 2008

Unix Shell Environment

8/4/2019 UNIX_DAY2



Environmental Variables

The Unix system is controlled by a number of shell variablesthat are separately set by the system - some during bootsequence, and some after logging in.

These variables are called system variables or environmentvariables.

The set statement displays the complete list of all thesevariables. These built-in variable names are defined inuppercase.

8/4/2019 UNIX_DAY2



Environmental Variables

PATH: is a variable that instructs the shell about the route itshould follow to locate any executable command

HOME: when you log in, UNIX normally places you in adirectory named after your login name

MAIL: determines where all incoming mail addressed to theuser is to be stored

PS1 and PS2: PS1 - your command prompt and PS2-Multi-line command string

SHELL: determines the type of shell that a user sees on

logging inNote: There are other environmental variables also but we have discussedthe important ones.

8/4/2019 UNIX_DAY2



Profile

.profile (.bash_profile in Linux): the script executed duringlogin time

The .profile must be located in your home directory, and it isexecuted after /etc/profile, the universal profile for all users

Universal environment settings are kept by the

administrator in /etc/profile so that they are available to allusers

8/4/2019 UNIX_DAY2



Setting Environment Variables

Assigning a value to a variable will set an environmentvariable temporarily

For example:

$ x=50

This example will set value 50 to the variable x

8/4/2019 UNIX_DAY2



Setting Environment Variables

The variable will not be available if you exit the shell. Addthis variable to configuration files e.g. “.profile” to set itpermanently

If you no longer want a variable to be set, you can use theunset command to erase its value

For example:$ unset x

This command will cause x to have no value set

8/4/2019 UNIX_DAY2



export

By default, the values stored in shell variables are local tothe shell, i.e., they are available only in the shell in whichthey are defined. They are not passed on to a child shell.

But the shell can also export those variables recursively toall child processes so that, once defined, they are availableglobally. This is done with the export command.

$ x=hello

$ export x

$ sh

$ echo $x

hello

8/4/2019 UNIX_DAY2



. (dot): Running commands from file

If we have a list of commands in a file “sample”, we can execute itusing command:

$ . sample

It is like executing a shell script, but with following difference:

Standard shell scripts cause a new sub shell to be created torun the script.

The dot command uses the same shell to execute the

commands.

The dot command, however, creates no child process, soany changes it produces apply to the original shell

It also does not require the script to have executablepermission

8/4/2019 UNIX_DAY2


34/19/2012

Unix Shell Scripting

Sh ll S i

8/4/2019 UNIX_DAY2



Shell Scripts

When a group of Unix commands has to be executed regularly,they are stored in a file. All such files are called as shell scripts.

There is no restrictions on extension of these files, butconventionally extension .sh is used for a shell script.

Unlike programs, shell scripts are interpreted at run time.

You can use the vi editor to create/edit the shell script.

Sh ll S i

8/4/2019 UNIX_DAY2



Shell Scripts

Shell scripting language provides following:

Scalar and array variables

Set of operators

Flow, Loop and case statements

Positional Parameters

here document

Functions

E ti h ll i t

8/4/2019 UNIX_DAY2



Executing shell script

You can execute the shell scripts using either command shor by just typing shell script name at the prompt (make surethat you have execute permission)

$ sh test.sh

$ test.sh

$ ./test.sh

Sh ll S i ti

8/4/2019 UNIX_DAY2



Shell Scripting

User-created Shell Variables:

variable=value => assigns value to variable

$variable => refers value of the variable

To display a variable:

echo $var “$var” „$var‟ the output:

hello hello $var

Sh ll S i ti

8/4/2019 UNIX_DAY2



Shell Scripting

read statement

To read a value in a variable form keyboard:

read var

To read a value in a variable from a file:

read var < file1

It reads the first line from the file into variable var

read var1 var2 < file1It reads the first word into var1 and second word into var2 from file1

Sh ll M t Ch t

8/4/2019 UNIX_DAY2



Shell Meta Characters

Wildcard substitution: * ? []

Redirection: >, >>, <, 2>, <<

Piping: |

Command substitution: ` `

Sequential commands: semicolon (;)

Inserting comment: #

Command grouping: () parenthesis

C d S b it ti R i tit d

8/4/2019 UNIX_DAY2



Command Subsitution Revistited

var1=„echo 2.5 +5.6 |bc`

var2= `grep John persons.dat`

test command

8/4/2019 UNIX_DAY2



test command

test command: used to conduct several tests on integers, strings, file

attributes It produces no output so used with if/while where its exit

status is used

Examples:

test 15 –gt 10

This command compares 15 and 10 and gives a true exit statusbut does not produce any output.

test command

8/4/2019 UNIX_DAY2



test command

Comparing numbers:

Operators:

Relational

-lt less than

-le less than equal to

-gt greater than

-ge greater than equal to

–eq equal to

–ne not equal to

Logical:

-a And

-o OR

! NOT

Test Command Strings

8/4/2019 UNIX_DAY2



Test Command - Strings

String comparison used by the command test:

String Comparison True if

string1 = string2 strings equal

string1 != string2 strings not

equal-n string string not null

-z string string is null

Test Command Files

8/4/2019 UNIX_DAY2



Test Command - Files

File tests used by the command test

Test True if

-d file file exists and is a directory

-e file file exists

-f file file exists and is a regular file-r file file exists and is a readable-s file file exists and has a size > 0

-w file file exists and is a writable

-x file file exists and is a executable

Conditional Statement: if then else

8/4/2019 UNIX_DAY2



Conditional Statement: if-then-else

The if statement takes two-way decisions depending on the condition

if condition

then

commands

fi

if condition

then

commands

else

commands

fi

if condition

then

commands

else if condition

then

commands

fi

fi

if condition

then

commands

elif condition

then

commands

else

commands

fi

Case Statement

8/4/2019 UNIX_DAY2



Case Statement

The statement matches an expression for more than one

alternative, and permits multi-way branching

case variable/expression/value in

value1) command1

command2

;;

value2) command3

;;

*) command4

esac

Example of Case Statement

8/4/2019 UNIX_DAY2



Example of Case Statement

echo “Enter the color”

read colorcase $color in

Red | red) echo “You have selected red color”

;;

Blue | blue) echo “You have selected blue

color”

;;

*) echo “Sorry! Yet to add this color”

;;

esac

Looping Statements

8/4/2019 UNIX_DAY2



Looping Statements

Looping statements are used to perform the same set of operations for more than one time till the specifiedcondition is true or false.

There are 3 types of looping statements:

1. while

2. until

3. for

Conditional Looping Statements

8/4/2019 UNIX_DAY2




1. while

while statement repeatedly performs a set of instructions till the

control command returns a true exit status

It is known as an entry-controlled loop

Syntax:

while test <condition>do

command1

command2

done

Example of While Statement

8/4/2019 UNIX_DAY2



Example of While Statement

ctrl=1while test $ctrl –le 4

do

echo $ctrl

ctrl=`expr $ctrl + 1`

done


8/4/2019 UNIX_DAY2




2. until

The set of instructions is executed repeatedly as long as

the condition remains false The until statement complements the while statement

Syntax:

until test <condition>

do

command1command2

done

Unconditional Looping Statements

8/4/2019 UNIX_DAY2



Unconditional Looping Statements

3. for

The loop is executed as many times as there are items in

the list. It doesn‟t test condition but uses a list instead.

Syntax:

for <identifier> in <val1 val2 …>

do

command1

command2

done

Example of for Loop

8/4/2019 UNIX_DAY2



Example of for Loop

for v_item in 10 20 30

do

echo $v_item

done

echo “Outside for loop”

break exit and continue Statements

8/4/2019 UNIX_DAY2



break, exit and continue Statements

break

Transfer control out of the current loop.

Mostly used in conjunction with if statement.

exit

Terminates the execution of script and transfer control tocommand prompt

continue

Skip remaining statements of the loop and continue with thenext iteration of the loop.


8/4/2019 UNIX_DAY2




clear

while test expn 1

do

stmt 1 while test expn 2

do

stmt 2

if test expn 3

then

stmt 3

break | continue | exitfi

stmt 4

done

stmt 5

done

stmt 6

break statement will pass the control to stmt 5 continue statement will pass the control to immediate done and will

then proceed to continue second while loop exit statement will exit from the script

Positional Parameters (command linearguments)

8/4/2019 UNIX_DAY2



arguments)

One can pass the command line arguments to shell scriptwhile execution

When arguments are specified with a shell script, they areassigned to variables called positional parameters

The first argument is read by the shell into the parameter

$1, the second into the parameter $2, and so on

The $# represents total number of arguments passed to thescript

The command is assigned to a variable $0

You can use these variables up to $9

Positional Parameters (command linearguments)

8/4/2019 UNIX_DAY2



arguments)

The $* indicates all arguments, in a single variable,separated by the first character in the environment variableIFS

The $@ is same as $* except when enclosed in doublequotes

The “$@” works with string input

set

8/4/2019 UNIX_DAY2



set

set command assigns values to positional parameters:

$ set 23 532

The command assigns value 23 to the positionalparameter $1, and 532 to $2

It also sets $#, $*

shift

8/4/2019 UNIX_DAY2



shift

shift command shifts command line arguments to left

The shift command copies the contents of a positional parameterto its immediate lower numbered positional parameter. Whencalled once, contents of $2 are copied to $1, $3 to $2 and so on.

$ shift 2 The command does two shifts i.e. $1=$3, $2=$4, and so on.

Using shift command we can pass more than 9 command lineparameters to shell script

Special Parameters

8/4/2019 UNIX_DAY2



Spec a a a ete s

$$ PID of current shell

$? Exit status of the last executed command

$! PID of last background process $# Total no. of Positional Parameters

$0 Name of the command being executed

$* List of shell arguments. Can‟t yield each argumentseparately

$@ Similar to $*, but yields each argument separatelywhen enclosed in double quotes.

here Document

8/4/2019 UNIX_DAY2



e e ocu e t

It allows a shell script to get its input from the same file thatholds the script

Data used this way is said to be in a here document

To create a „here document‟, we use the << operator

Any command using standard input can also have the inputfrom a „here document‟

The „here document‟ symbol (<<) followed by the data, anda delimiter (the termination string)

here Document (Contd.)

8/4/2019 UNIX_DAY2



( )

$ wc << EOF

>This is for testing the here document.

>It will give us the count of lines, words and characters

> till the terminator given after here document symbol,

> the terminator which is the

EOF

We will get output of the command as number of lines,words, characters till first occurrence of the EOF symbol inthe script

We can use any string as a terminator

Shell Debugging

8/4/2019 UNIX_DAY2



gg g

Used to trace the script and correct the errors line by line

Syntax

sh [options] <Name of the Shell Script>

Options Functionality

v

Prints the shell input lines as they are read bythe system

x Prints the commands and their arguments asthey are executed

Shell Debugging (Contd.)

8/4/2019 UNIX_DAY2



gg g ( )

Example:

(i) sh -v WhileDemo.sh

(ii) sh –x WhileDemo.sh

Arrays

8/4/2019 UNIX_DAY2



y

Unix provides one-dimensional array variables

Any variable may be used as an array

There is no maximum limit on the size of an array, nor anyrequirement that members be indexed or assignedcontiguously

Arrays (Contd.)

8/4/2019 UNIX_DAY2



y ( )

Syntax to create an array:

(i) arrayname[subscript]=valueOR

(ii) arrayname=(values)

In first method, the subscript is treated as an arithmetic

expression that must be evaluate to a number greater thanor equal to zero

In second method, the values needs to be separated by aspace

Arrays (Contd.)

8/4/2019 UNIX_DAY2



Element of an array may be referenced using

${array_name[subscript]}

Referencing an array variable without a subscript isequivalent to referencing the element zero

The unset command is used to destroy complete array or

individual array element unset array_name[subscript]

Used to destroy the array element at the specified index subscript

unset array_name

Used to remove the entire array

Arrays (Contd.)

8/4/2019 UNIX_DAY2



Example:

area[11]=23

area[13]=37area[51]=UFOs

echo -n "area[11] = ${area[11]} "

echo -n "area[13] = ${area[13]} "

fruits=(Apple Mango Banana)

echo ${fruits[1]}

echo ${fruits[2]}

Functions

8/4/2019 UNIX_DAY2



The bigger script can be divided into small modules/functions, this will give us more readability and easymaintenance.

A function consists of a group of statements which areexecuted together as a bunch.

A shell function(s) must precede the statements that call it.

Functions

8/4/2019 UNIX_DAY2



The syntax for function definition:

function_name()

{

Command1

Command2

Command3

}

When function is invoked it executes all the commandsenclosed by the curly braces

Call a function by its name only

Functions

8/4/2019 UNIX_DAY2



Shell functions can be defined at a number of places:

At the beginning of script

In a separate file, so that other scripts can also use them

In the .profile(.bash_profile for Linux), so that they areavailable in the current session

If we store the functions in a file called function_library, wemust include this file in the script using dot command asfollows:

. function_library

Functions

8/4/2019 UNIX_DAY2



The positional parameters made available to shell scriptsexternally are not available directly to a shell function

We have to store these parameters in the shell variables andthen pass them to the function

The parameters are passed on the function call statement

itself

Parameters passed to function are accessed inside functionbody in the same way we access positional parameters in ashell script

Functions

8/4/2019 UNIX_DAY2



Shell functions can change the variables of the shell, so onehas to be careful while using a variable name in a function

To return a value from a function, we have to use the echocommand example:

funct_name()

{

---------

---------echo result

}

Now comes the function call

var=`funct_name para1 para2`

The result will be stored in var variable

8/4/2019 UNIX_DAY2


Thank You

unix_day2

Documents