Agenda

• Regular Expressions (Appendix A in Text)– Definition / Purpose– Commands that Use Regular Expressions– Using Regular Expressions– Using the Replacement String– sed UNIX utility

Regular Expressions

Definition:

– A “Regular Expression” is a set of one or more strings of characters.

– Regular Expressions are used to help search and perform operations of strings that are contained inside files. Many languages including PERL use regular expressions

Regular Expressions

Purpose:

– Regular expressions can be used with Linux / UNIX commands such as grep, awk, vi and sed

– Regular expressions make it easier for user to search, change or process data in files from the issuing a command at the shell prompt or while working within vi editor

Linux (Unix) Utilities that Use Regular Expressions

grep Search for patterns.

vi Search and replace patterns withinthe vi editor in last line mode

sed Edit files matching patterns from shellprompt (without using editor)

awk Search for patterns and process matched patterns (discussed later in course)

Simple Regular Expression

• A Simple Regular Expression is a regular expression that defines a string of characters

• A regular expression such as a character is used to display all regular expressions in the file that “match”

Complex Regular Expression

• A Complex Regular Expression uses special characters with letters and numbers in order to provide an ambiguous regular expression match.

• This is similar to an ambiguous file reference, except we are matching strings contained inside a file, and rules for wildcard searches differ.

Complex Regular Expressions(Wildcard Symbols)

. matches any character (similar to ?)

* matches one or more occurrences of that character

^ matches beginning of line (must be at beginning of regular expression)

$ matches end of line (must follow ending character to match)

[ ] similar to wildcard, except if ^ is first character then matches characters not inside brackets

Complex Regular Expressions(Quoting Special Characters)

\ inhibits meaning of special characters

\< matches beginning of a word

\> matches end of a word

\<pattern\> matches only if pattern is a full word, delimited by spaces, punctuation, beginning or end of line

Longest Possible Match

– A regular expression using special characters always matches the longest possible string starting as far toward the beginning of the line as possible.

Using Regular Expressionswith grep

• Grep uses regular expressions to match patterns of strings contained in files.

Format: grep “regular expression” file

– Note that grep command places regular expression in double quotes as opposed to using a delimiters such as /

Using Regular Expressionswith vi

• Regular expressions can be used with the vi editor to make it easier to search for complex patterns that are contained in files.

• The commands to search and replace regular expressions are typed in “last-line mode”

Searching & Replacing Patterns

:s/pattern1/pattern2/ - substitutes pattern2 for the first occurrence of pattern1

:1,$ s/pattern1/pattern2/ - substitutes pattern2 for the first occurrence of pattern1 in each line of the file ($ means last line, . means current line)

:.,.+10 s/pattern1/pattern2/ - substitutes 11 lines starting with current line

:s/pattern1/ *** & ***/ - & is the value of the string matched by pattern1, and is the only character with special meaning in the second pattern

sed Utility

Used to modify standard output from a file to be stored into another file. Considered to be a streaming editor without having to “physically edit” file (like vi)

Format: sed [options] [script-file] [file-list] options:

-f read and make editing changes from script file

-n do not print lines from file-list as standard output

Streaming Editor - sed

Processing Steps using sed:– sed reads one line from the input file– sed reads first command from script file (or

command line or function) and if matched, takes specified action

– sed reads next command line from script or function and if matched, takes specified action

– Process is repeated until all commands in script file are executed

Streaming Editor - sed

Note:– sed command does not make changes to the

input file, instead it is used to modify standard output that can be redirected to other files or piped to other UNIX commands.

– For examples involving editing techniques with sed command, refer to the samples section of this week’s notes (week 12)

sed - Examples

sed ‘/abc/ p’ file– Prints standard output of file along with standard

output of pattern abc match (can use “-n” option to display pattern match only)

sed ‘5,15 d’ file– delete lines 5 to 15 of file from standard output

sed ‘/abc/ d’ file– Does not write-out lines that are matched from

standard output.

sed - Examples

sed 's/pattern1/pattern2/' file1– substitute pattern1 with pattern 2 (every

occurrence)

sed '10 q' file1– displays first 10 lines then quit

sed '/pattern/ q' file1– displays to first matching line

sed - Examples

sed '/pattern/ d' file1– same as grep -v 'pattern' file1

sed '/pattern1/s/pattern2/pattern3/g' file1– substitute only on lines matching pattern 1

sed '/pattern1/s/ /pattern3/g' file1– If pattern2 omitted, it is defaulted to same as

pattern 1

sed - Examples

sed -n '10,20 s/pattern1/pattern2/ p' file1– -n suppresses default write to output, p will

print matched lines

Additional sed Commands

a \ text• Append text, which has each embedded newline pre ceeded by a backslash.

i \ text• Insert text, which has each embedded newline pre ceeded by a backslash.

q• Immediately quit the sed script without processing any more

input, except that if auto-print is not diabled the current pattern space will be printed.

• r filename

• Append text read from filename.

Additional sed Commands

r filename• Append text read from filename.

w filename• Write text to filename.