perl for system automation - 01 advanced file processing

Perl for System Automation

1

Danairat T.

Line ID: Danairat

FB: Danairat Thanabodithammachari

+668-1559-1446

Danairat T.

Contents

Module-1: Advanced File Processing with Perl• Types of open Filehandles and File Locking

• Read/Write File line by line

– Read Line

– Write Line

– Append Line

• Lock File

• Read entire files into Array

• Read entire files into Scalar using file record separator

• Read file paragraph by paragraph using file record separator

• Read files in the Directory

• Read SubDir in the Directory

• Read files recursively in Directory

Danairat T.

Contents

Module-2: System Interaction

• Enumerating OS user identity

• Enumerating OS group identity

• Activating System Command

– Using exec()

– Using system()

• Grabbing a program's output

– Using qx()

• Display current running processes

– Using ProcessTable module

• Terminate the Process

– Built in kill() function

Danairat T.

Contents

Module-3: Managing Networks with Perl

• Host file• IP Lookup• FTP• E-mail• Web• LDAP

Danairat T.

Contents

Module-4: Perl Coding Guidelines

• Perl Coding Style Guides

– The 12 general guidelines

– Don’t do in perl codes

– To avoid in perl codes

– Performance coding guidelines

• Perl Coding Guides for Security

– Command line inputs

– The exec(), system(), qw() and Perl modules.

Danairat T.

What is Perl?(Practical Extraction and Report Language )

• Perl, release in 1987, is a high-level programming language written by Larry Wall. Perl's process, file, and text manipulation facilities make it particularly well-suited for tasks system utilities, system management tasks, database access, networking. These strengths make it especially popular with system administrators.

6

Danairat T.

Which version of Perl should I use?

• No one is actively supporting Perl 4. Five years ago it

was a dead camel carcass

• There is no Perl 6 release scheduled.

• The immediate releases are Perl 5.8 (i.e. Perl5.8.x )

7

Danairat T.

OS Platforms Support

• UNIX: More or less every UNIX or UNIX-like operating system ever created, notably

Linux , Solaris, AIX, IRIX, HP/UX, BSD, and Tru64

• MS Windows: DOS, Windows 3.1, 95, 98, NT and 2000

• Other Desktop OSs: Apple Macintosh (68k and PPC, both pre and post MacOS X),

Acorn Risc OS, Amiga, BeOS, OS/2, and many others

• Mainframes: AS/400, OS390, VMS and OpenVMS, Stratus (VOS), and Tandem

• PDAs: EPOC (Psion/Symbian), but not PalmOS or Windows CE at time of writing,

although porting efforts are being made for both platforms

8

Danairat T.

Identify the Perl version

• perl –v

• perl -V

9

Danairat T.

www.CPAN.org

• CPAN stands for

Comprehensive Perl

Archive Network,

containing libraries,

documentation, and many

Perl third-party modules

and extensions

10

Danairat T.

Install Perl

• On Unix System– pkgadd -d /cdrom/sol_10_1009_x86/Solaris_10/Product SUNWperl584core

– pkgadd –d /cdrom/sol_10_1009_x86/Solaris_10/Product SUNWperl584usr

– pkgadd –d /cdrom/sol_10_1009_x86/Solaris_10/Product SUNWperl584man

• On Windows System– run ActivePerl-5.8.9.826-MSWin32-x86-290470.msi

Available for download at www.activestate.com

11

Danairat T.

Perl file structure and location

• Perl executable: /usr/perl5/bin/perl or /usr/bin/perl

• Perl Pain Old Documentation: /usr/perl5/pod

• Perl Manual: /usr/perl5/man

12

Danairat T.

Perl Libraries (Perl Modules)

• The Perl Libraries directories

– /usr/perl5/5.8.4/lib/i86pc-solaris-64int

– /usr/perl5/5.8.4/lib

– /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int

– /usr/perl5/site_perl/5.8.4

– /usr/perl5/site_perl

– /usr/perl5/vendor_perl/5.8.4/i86pc-solaris-64int

– /usr/perl5/vendor_perl/5.8.4

– /usr/perl5/vendor_perl

13

Danairat T.

Perl Library (The modules for Unix)

http://www.cpan.org/modules/01modules.index.html

14

Danairat T.

Perl Library (PPM modules for windows)

http://ppm.activestate.com/PPMPackages/zips/8xx-builds-only/Windows/

15

Danairat T.

Install Perl Modules on Unix

gzip -dc yourmodule.tar.gz | tar -xof -

cd /your/new/lib/to/install/

perl Makefile.PL

make

make test

make install

Or, if you have internet access

perl -MCPAN -e shell

cpan> i /Time/

cpan> install Time::CTime

16

Danairat T.

Install Perl Modules on Unix

bash-3.00# cd /tmp/Class-Simple-0.19

bash-3.00# perl Makefile.PLChecking if your kit is complete...

Looks good

Writing Makefile for Class::Simple

bash-3.00# makecp lib/Class/Simple.pm blib/lib/Class/Simple.pm

Manifying blib/man3/Class::Simple.3

bash-3.00# make testPERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0,

'blib/lib', 'blib/arch')" t/*.t

t/anon.............ok

t/attributes.......ok

t/Class-Simple.....ok

t/inherit..........ok

t/isa..............ok

t/nonew............ok

t/override.........ok

All tests successful.

Files=13, Tests=95, 3 wallclock secs ( 0.99 cusr + 0.52 csys = 1.51 CPU)

bash-3.00# make installInstalling /usr/perl5/site_perl/5.8.4/Class/Simple.pm

Installing /usr/perl5/5.8.4/man/man3/Class::Simple.3

Writing /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int/auto/Class/Simple/.packli

st

Appending installation info to /usr/perl5/5.8.4/lib/i86pc-solaris-64int/perlloca

l.pod

bash-3.00#

Example:-

17

Danairat T.

Installing Perl modules on Sun Solaris 10

1. Login as root

2. Add path /usr/sfw/bin to /.profile

3. Download and unzip the Perl module then cd

into its directory

4. Follow the below command steps:-

– /usr/perl5/bin/perlgcc Makefile.PL

– gmake

– gmake test

– gmake install

18

Danairat T.

Install Perl Module on Windows

1. Unzip the lib file

2. Cd to the downloaded lib

3. ppm install yourdownloadedlib.ppd

19

Module-1: Advanced File

Processing with Perl

20

Danairat T.

Topics

• Types of open Filehandles and File Locking

• Read/Write File line by line

– Read Line

– Write Line

– Append Line

• Lock File

• Read entire files into Array

• Read entire files into Scalar using file record separator

• Read file paragraph by paragraph using file record separator

• Read many files in a Directory

• Read SubDir in the Directory

• Read files recursively from Directories

21

Danairat T.

File Handle Options

22

mode operand create

delete and recreate

file if file exists

read <

write > ✓ ✓

append >> ✓

read/write +<

read/write +> ✓ ✓

read/append +>> ✓

Danairat T.

Open File Options

23

• Using < for file reading.

#!/usr/bin/perluse strict;use warnings;

my $myFile = "fileread.txt"; # the file “filetest.txt” must be existmy $myLine;if (open (MYFILEHANDLE, '<' , $myFile)) { # using ‘<‘ and . for file read

while ($myLine = <MYFILEHANDLE>) { # read linechomp($myLine); # trim whitespace at end of lineprint "$myLine \n";

}close (MYFILEHANDLE);

} else {print "File could not be opened. \n";

}exit(0);

OpenFileReadEx01.pl

Results:-

<print the file content>

Danairat T.

Open File Options

24

• Using > for file writing to new file.


my $myFile = "filewrite.txt"; my @myData = ("line1", "line2", "line3");

if (open (MYFILEHANDLE, '>' , $myFile)) {foreach my $myLine (@myData) {

print MYFILEHANDLE "$myLine \n"; # print to filehandle}close (MYFILEHANDLE);


}exit(0);

OpenFileWriteEx01.pl

Results:-

<see from the output file>

Danairat T.

Open File Options

25

• Using >> to append data to file. If the file does not exist then it is create a

new file.


my $myFile = "filewrite.txt"; my @myData = ("line4", "line5", "line6");

if (open (MYFILEHANDLE, ‘>>' , $myFile)) {foreach my $myLine (@myData) {

print MYFILEHANDLE "$myLine \n"; # print to filehandle}close (MYFILEHANDLE);


}exit(0);

OpenFileAppendEx01.pl

Results:-


Danairat T.

File Locking

• Lock File for Reading (shared lock): Allow other to

open the file but no one can modify the file

• Lock File for Writing (exclusive lock): NOT allow

anyone to open the file either for reading or for

writing

• Unlock file is activated when close the file

26

Shared lock: 1Exclusive lock: 2Unlock: 8

Danairat T.

File Locking – Exclusive Locking

27

#!/usr/bin/perluse strict;use warnings;use Fcntl;

my $file = 'testfile.txt';

# open the fileopen (FILE, ">>", "$file") || die "problem opening $file\n";

# immediately lock the fileflock (FILE, 2);

# test keeping the lock on the file for ~20 secondsmy $count = 0;while ($count++ < 30){print "count = $count\n";print FILE "count = $count\n";sleep 1;

}

# close the file, which also removes the lockclose (FILE);

exit(0);

FileExLockEx01.pl

Please run this concurrence

with FileExLockEx02.pl, see

next page.

Results:-


Danairat T.

File Locking – Exclusive Locking

28



# open the fileopen (FILE, ">>", "$file") || die "problem opening $file\n";


# test keeping the lock on the file for ~20 secondsmy $count = 0;while ($count++ < 30){print "count : $count\n";print FILE "count : $count\n";sleep 1;

}


exit(0);

FileExLockEx02.pl

Please run this concurrency

with FileExLockEx01.pl

Results:-


Danairat T.

File Locking – Shared Locking

29



# open the fileopen (FILE, "<", "$file") || die "problem opening $file\n";


# test keeping the lock on the file for ~20 secondsmy $count = 0;while ($count++ < 30){print "Shared Locking\n";sleep 1;

}


exit(0);

FileShLockEx01.pl

Please run this concurrency

with FileExLockEx02.pl

Results:-


Danairat T.

Read Files from Command Line

• The command line argument will be stored in

@ARGV as default or you may straight away using <>

for file read.

• The below codes is to read file from command line

argument and print their content to the screen.

30

#!/usr/bin/perl -wwhile (<>) { print $_; }

Danairat T.

Read Files from Command Line

31


my $lineNo = 1;

# read values from @ARGVforeach my $myFile (@ARGV) {

# read each fileif (open (FILEREAD, "<", $myFile)) {

flock (FILEREAD,1);print "====Start $myFile====\n\n";while (my $myLine = <FILEREAD>) {

chomp($myLine);print "$lineNo: $myLine";print "\n";$lineNo++;

}close (FILEREAD);

} else {print "Failed to open $myFile\n\n";

}}

exit(0);

01_ReadFilesFromARGVEx01.pl

perl 01_ReadFilesFromARGVEx01.pl data01.dat data02.dat

Please see the result screen.

Danairat T.

Read File as a whole into Array

• Perl allows you to read file into array in a single

statement using = (assign) operator.

32

open (FILEREAD, $myFile);@myLines = <FILEREAD>;close (FILEREAD);

Danairat T.

Read File as a whole into Array

33



# read each filedie "Failed to open $myFile\n\n" unless (open (FILEREAD, "<", $myFile));flock (FILEREAD,1);print "====Start $myFile====\n\n";my @myLines = <FILEREAD>; # read file into single arrayclose(FILEREAD);

my $lineNo = 1;foreach my $myLine (@myLines) {

chomp($myLine);print $lineNo++ . ": ". $myLine . "\n";

}}

exit(0);

02_ReadFilesIntoArrayEx01.pl

perl 02_ReadFilesIntoArrayEx01.pl data01.dat data02.dat


Danairat T.

Read File as a whole into Scalar

• Perl provides you the quick way to read file into a

single scalar variable and process in your program

structure. Eg. You may need to read many input files

and append to one output file.

• $/ is a file record separator. Default is \n. You need to

turn it off otherwise you can only read just one line.

34

undef $/; # undefine the file record separatormy $myLines = <FILEREAD>;

Danairat T.

Read File as a whole into Scalar

35



# read each filedie "Failed to open $myFile\n\n" unless (open (FILEREAD, "<", $myFile));flock (FILEREAD,1);print "====Start $myFile====\n\n";undef $/; # undefine the file record separatormy $myLines = <FILEREAD>;close(FILEREAD);

print "$myLines\n";}

exit(0);

03_ReadFilesIntoScalarEx01.pl

perl 03_ReadFilesIntoScalarEx01.pl data01.dat data02.dat


Danairat T.

Read File as Paragraph

• To read file paragraph by paragraph, you need to

assign “” the empty string to the file record separator

$/ for perl to detect the blank line as a break of

elements until match with the next line content

36

$/ = ""; # assign empty string to the file record separatormy @myParagraphs = <FILEREAD>;

Danairat T.

Read File as Paragraph

37



# read each filedie "Failed to open $myFile\n\n" unless (open (FILEREAD, "<", $myFile));flock (FILEREAD,1);print "====Start $myFile====\n\n";$/ = ""; # assign empty string to the file record separatormy @myParagraphs = <FILEREAD>;close(FILEREAD);

my $paragraphNo = 1;foreach my $myParagraph (@myParagraphs ) {

chomp($myParagraph);print "====Paragraph " . $paragraphNo++ . "====\n" . $myParagraph . "\n";

}

}

exit(0);

04_ReadFilesParagraphEx01.pl

perl 04_ReadFilesParagraphEx01.pl data01.dat data02.dat


Danairat T.

Read many files in the Directory

• Perl allow you to open the directory using:-

– opendir (MYDIR, “/mydirectory”);

• To read the item from directory, using:-

– readdir (MYDIR);

• To close the directory, using:-

– closedir (MYDIR, “/mydirectory”);

38

#!/usr/bin/perl –wopendir(DIR, '.') or die "Couldn't open directory, $!"; foreach (sort grep(/^.*\.txt$/,readdir(DIR))) {

print "$_\n"; } closedir (DIR);

Danairat T.

Read Files in the Directory

39

#!/usr/bin/perl -wuse strict;

my $dir = '.';

opendir(DIR, $dir) or die $!;while (my $file = readdir(DIR)) {

# We only want filesnext unless (-f "$dir/$file");

# Use a regular expression to find files ending with .txtnext unless ($file =~ /\.txt$/);

print "$file\n"; # you can open the file here}

closedir(DIR);

exit(0);

05_ReadFilesFromDirectory.pl


Danairat T.

Read SubDir in the Directory

40

#!/usr/bin/perl -wuse strict;

my $dir = '../';

opendir(DIR, $dir) or die $!;while (my $subDirName = readdir(DIR)) {

next unless (-d "$dir/$subDirName");

print "$subDirName";

}

closedir(DIR);exit 0;

06_ReadSubDirFromDirectory.pl


Danairat T.

Read File Recursively in Directory

1. The Cwd module for you to find the current

working directory.

2. The chdir() function is to use when you want

to change “perl program working directory”

to the another directory.

3. The recursive programming provides the

repeat of the same action with keeping your

history variables in the program stack and

automatically pop it up to complete the

program

41

Danairat T.

Read File Recursively in Directory

42

#!/usr/bin/perl -wuse strict;use Cwd; # module for finding the current working directory

my $myDirectory = '../../';ScanDirectory($myDirectory);

exit 0;sub ScanDirectory {

my $workdir = shift; # same as shift(@_)my $startdir = cwd; # keep track of where we began chdir $workdir or die "Unable to enter dir $workdir: $!\n";opendir (DIR, '.') or die "Unable to open $workdir: $!\n";my @names = readdir (DIR) or die "Unable to read $workdir: $!\n";closedir (DIR);foreach my $name (@names) {

next if ( $name eq '.' );next if ( $name eq '..' );if ( -d $name ) { # is this a directory?

ScanDirectory($name);next;

}if ( $name =~ /\.txt$/i ) { # is this a file named "core"?

print "$startdir/$workdir/$name \n";}

}chdir $startdir or die "Unable to change to dir $startdir: $!\n";

}

07_ReadFilesFromDirectoryRecursive_Ex01.pl


Danairat T.

Line ID: Danairat

FB: Danairat Thanabodithammachari

+668-1559-1446

Thank you

perl for system automation - 01 advanced file processing

Software