rolando v. raqueñowednesday, june 10, 2015 statistics function implementation traditional...
Post on 19-Dec-2015
215 views
TRANSCRIPT
Rolando V. Raqueño Tuesday, April 18, 2023
Statistics Function Implementation
Traditional Programming Approach
Rolando V. Raqueño Tuesday, April 18, 2023
minimum.pro
function minimum, xn = n_elements(x)answer = x(0)for i=1L, n-1 do begin
if( answer gt x(i) ) then $answer = x(i)
endforreturn, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
sum.pro
function sum, x
n = n_elements( x )
answer = 0
for i = 0, n-1 do begin
answer = answer + x(i)
endfor
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
Overflow Problems
• What if the array is an integer array?
• Sums will have a maximum limit based on the number of bits specified by the integer type.
Rolando V. Raqueño Tuesday, April 18, 2023
Corrected sum.pro
function sum, x
n = n_elements( x )
answer = 0.0D
for i = 0L, n-1 do begin
answer = answer + x(i)
endfor
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
mean.pro
function mean, x
n = n_elements(x)
answer = sum(x) / n
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
Integer Problems
IDL> b=9/5
IDL> print,b
1
Rolando V. Raqueño Tuesday, April 18, 2023
Integer Problems
IDL> b=9/double(5)
IDL> print,b
1.8000000
Rolando V. Raqueño Tuesday, April 18, 2023
Corrected mean.pro
function mean, x
n = n_elements(x)
answer = sum(x) / double(n)
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
sum_squares.pro
function sum_squares, x
squares = x^2.0
answer= sum( squares )
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
variance.pro
function variance, x
n = double(n_elements(x))
answer=(sum_squares(x)-(sum(x)^2.0/n))/(n-1)
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
sd.pro
function sd, x
answer = sqrt(variance(x))
return, answer
end
Rolando V. Raqueño Tuesday, April 18, 2023
Shell Programming
Rolando V. Raqueño Tuesday, April 18, 2023
UNIX Tip of the Day
• Directory maneuvering commands pushd, popd, and, dirs
% cd /usr/tmp% pwd/usr/tmp% pushd ~rvrpci% pwd/cis/staff/rvrpci
Rolando V. Raqueño Tuesday, April 18, 2023
UNIX Tip of the Day
% dirs
/cis/staff/rvrpci /usr/tmp
% pushd
% pwd
/usr/tmp
% dirs
/usr/tmp /cis/staff/rvrpci
% pushd /usr/local/bin
Rolando V. Raqueño Tuesday, April 18, 2023
UNIX Tip of the Day
% dirs
/usr/local/bin /usr/tmp
/cis/staff/rvrpci
% pwd
/usr/local/bin
% pushd
% dirs
/usr/tmp /usr/local/bin
/cis/staff/rvrpci
Rolando V. Raqueño Tuesday, April 18, 2023
UNIX Tip of the Day% dirs
/usr/tmp /usr/local/bin
/cis/staff/rvrpci
% pwd
/usr/tmp
% pushd +2
% pwd
/cis/staff/rvrpci
Rolando V. Raqueño Tuesday, April 18, 2023
UNIX Tip of the Day% dirs
/cis/staff/rvrpci /usr/tmp
/usr/local/bin
% popd
% dirs
/usr/tmp /usr/local/bin
Rolando V. Raqueño Tuesday, April 18, 2023
IMPORTANT UNIX Concepts• Environment and Shell Variables
– These allow you to customize your UNIX environment
– They are different in terms of their SCOPE• SCOPE determines the visibility of a variable
Rolando V. Raqueño Tuesday, April 18, 2023
Other IMPORTANT UNIX Concepts• Environment Variable
– Examples are TERM and DISPLAY– Set a particular variable to a value by using the setenv command
– You can print the value of a particular variable or all the environment variable using the printenv command
Rolando V. Raqueño Tuesday, April 18, 2023
% Environment Variables %• Examples
– To set environment variables
% setenv TERM vt100% setenv DOG Goofy– print out the terminal type
% printenv TERM vt100– print out all environment variables
% printenv
Rolando V. Raqueño Tuesday, April 18, 2023
Shell Variables
• Shell variables are similar to Environment variables except they have a limited scope, i.e., they exist only in the shell which they are defined.
• Environment variables on the other hand, exist in all its children shells
• To illustrate this concept, let us look at the following example
Rolando V. Raqueño Tuesday, April 18, 2023
Environment vs. Shell Variables% set prompt = "Parent Shell > "Parent Shell > setenv DOG GoofyParent Shell > set mouse=MickeyParent Shell > printenv DOGGoofyParent Shell > echo $mouseMickeyParent Shell > xterm &
(YOU SHOULD NOW HAVE A NEW xterm WINDOW)THIS IS KNOWN AS
“SPAWNING A NEW (OR CHILD) PROCESS”
Rolando V. Raqueño Tuesday, April 18, 2023
Environment vs. Shell Variables(IN THE NEW xterm WINDOW, DO THE FOLLOWING)
% set prompt = "Child Shell > "Child Shell > printenv DOGGoofyChild Shell > echo $mousemouse: Undefined variable.
Rolando V. Raqueño Tuesday, April 18, 2023
Environment vs. Shell VariablesChild Shell > setenv DOG PlutoChild Shell > set mouse=MinnieChild Shell > printenv DOGPlutoChild Shell > echo $mouseMinnieChild Shell > exit
(THE xterm WINDOW SHOULD NOW GO AWAY - THIS PROCESS HAS NOW BEEN KILLED)
Rolando V. Raqueño Tuesday, April 18, 2023
Environment vs. Shell VariablesParent Shell >Parent Shell > printenv DOGGoofyParent Shell > echo $mouseMickeyParent Shell >
Rolando V. Raqueño Tuesday, April 18, 2023
Environment & Shell Variables
• Why is this important?– UNIX uses Environment and Shell Variables
control a number of processes– Customizes your working environment– Variables used for UNIX Scripts
• They are typically defined and initialized in your .login and .cshrc files
Rolando V. Raqueño Tuesday, April 18, 2023
Useful Shell Variables
filec#Allows file completionpath #List of command
directoriescdpath #List of candidate
directories to cd intohistory #Number of commands to
remember
Rolando V. Raqueño Tuesday, April 18, 2023
What is shell programming?
• Shell programming – automate a set of UNIX commands.– Just like any programming language– “wrappers”
• black box a customized collection of UNIX commands.
– Example of shell programs.login.cshrc
Rolando V. Raqueño Tuesday, April 18, 2023
.login file
set path=($HOME/bin /usr/local/bin \/usr/ucb /usr/sbin /bin /usr/bin \/usr/bin/X11 .)stty dec newtset -I -Qset mail=/usr/spool/mail/$USERset editmode = emacsumask 077biff ndate
Rolando V. Raqueño Tuesday, April 18, 2023
.cshrc file
if ($?prompt) then
set notify
set history = 100
set savehist = 100
alias pd pushd
alias pop popd
alias vt100 "set term = vt100"
endif
Rolando V. Raqueño Tuesday, April 18, 2023
When these files are executed?
.cshrc – is automatically executed when you start a new
shell
.login – only gets executed once when you first login
Can be re-executed by giving the source command% source .cshrc
Rolando V. Raqueño Tuesday, April 18, 2023
Other useful .login and .cshrc entries
set filec
set cdpath=(~ ~rvrpci/pub ~/mythesis)
Other common entries
set path=( $path /usr/local/bin)
set path=(/usr/local/bin $path)
Rolando V. Raqueño Tuesday, April 18, 2023
User defined shell program
• Determine name of command
• Determine input, output, and option arguments
• Determine UNIX commands to execute
• Establish error trapping
• Make shell program executable
Rolando V. Raqueño Tuesday, April 18, 2023
A simple shell program
• dd command to swap bytes
% dd if=input.dat of=output.dat bs=2 conv=swab
• Very difficult to remember• Very little utility to non-UNIX geeks
(normal people)
Rolando V. Raqueño Tuesday, April 18, 2023
We would rather see...
% swap_bytes input.dat output.dat
Rolando V. Raqueño Tuesday, April 18, 2023
Special Shell Variables Set
% swap_bytes input.dat output.dat
$0 $1 $2$argv[1] $argv[2]command
Rolando V. Raqueño Tuesday, April 18, 2023
Another Special Shell Variables
% swap_bytes input.dat output.dat
$#argv
Indicates how many arguments are present
In this case, 2
Rolando V. Raqueño Tuesday, April 18, 2023
shell program swap_bytes
#!/bin/csh -f
dd if=$1 of=$2 bs=2 conv=swab
Rolando V. Raqueño Tuesday, April 18, 2023
Making swap_bytes shell script executable
% ls -l swap_bytes
-rw------- ... swap_bytes
% chmod u+x swap_bytes
% ls -l swap_bytes
-rwx------ ... swap_bytes
Rolando V. Raqueño Tuesday, April 18, 2023
To run swap_bytes
• swap_bytes becomes just another unix command!
% swap_bytes input.dat output.dat
Rolando V. Raqueño Tuesday, April 18, 2023
Limitation of swap_bytes
• No error trapping
• Should give usage when typing command
% swap_bytes
usage: swap_bytes input_file output_file
Rolando V. Raqueño Tuesday, April 18, 2023
Improvement to swap_bytes
#!/bin/csh -f if ( $#argv != 2 ) then
echo "usage: $0 input_file output_file"
exit 1
endif
dd if=$1 of=$2 bs=2 conv=swab
Rolando V. Raqueño Tuesday, April 18, 2023
Commad exit status
• By convention
exit 0
Indicates successful command completion
exit 1 (or non-zero)
Indicates some error condition
Rolando V. Raqueño Tuesday, April 18, 2023
Informational message from swap_bytes
• UNIX style informational message
% swap_bytes
usage: swap_bytes input_file output_file
Rolando V. Raqueño Tuesday, April 18, 2023
Interactive swap_bytes
• If you want a “friendlier” shell program– Have it query the user for the inputs
• Another special shell variable can be used
$<
Rolando V. Raqueño Tuesday, April 18, 2023
Interactive swap_bytes
#!/bin/csh -f
if ( $#argv != 2 ) then
echo -n "Please enter the input file> "
set input=$<
echo -n "Please enter the output file> "
set output=$<
endif
dd if=$input of=$output bs=2 conv=swab
Rolando V. Raqueño Tuesday, April 18, 2023
Interactive swap_bytes example
• User simply types the command
% swap_bytes
Please enter the input file> input.dat
Please enter the output file> output.dat
Rolando V. Raqueño Tuesday, April 18, 2023
UNIX Quotes
Rolando V. Raqueño Tuesday, April 18, 2023
A note about quotes in UNIX
% set a=ls
% echo a
% echo $a
% set b=“$a”
% echo $b
% set b=‘$a’
% echo $b
% set b=`$a`
% echo $b
Rolando V. Raqueño Tuesday, April 18, 2023
A note about shell variables
• Shell variables can also double up as arrays
• Using the previous example,
% echo $b
% echo $b[1]
% echo $#b
% echo $b[$#b]
Rolando V. Raqueño Tuesday, April 18, 2023
A more complex shell program
• In pbmplus utilities, rawtopgm conversion existspgmtoraw conversion does not
• A version of pgmtoraw in a programming language like C– Time consuming– Program will likely be used infrequently
• Solution: shell program
Rolando V. Raqueño Tuesday, April 18, 2023
pgmtoraw shell script design
• Define input and output files
• Figure out dimensions of input image
• Determine number of bytes for input image
• Determine number of bytes for header
• Need to strip out the header bytes
• Write out headerless image data
Rolando V. Raqueño Tuesday, April 18, 2023
Define input and output files pgmtoraw
#!/bin/csh -fset command_name=$0set number_args=$#argvif( $number_args != 1 ) then echo “usage:$command_name input_file_name” exit 1endif...
Rolando V. Raqueño Tuesday, April 18, 2023
Dimensions of input image ( pnmfile)
% more test_data.pgmP23 32551 2 34 5 67 8 9
% pnmfile test_data.pgmtest_data.pgm: PGM plain, 3 by 3 maxval 255
Rolando V. Raqueño Tuesday, April 18, 2023
pgmtoraw (continued)
set input_file=$1
set pnm_info = `pnmfile $input_file`
set image_type = $pnm_info[2]
set data_type = $pnm_info[3]
set width = $pnm_info[4]
set height = $pnm_info[6]
set maxval = $pnm_info[8]
set pixel_bytes = 1
@ image_bytes = $width * $height
Rolando V. Raqueño Tuesday, April 18, 2023
pgmtoraw (continued)
set file_info=`wc -c $input_file`
set bytes_in_file = $file_info[1]
@ header = $bytes_in_file - $image_bytes
dd if=$input_file bs=$pixel_bytes skip=$header
Rolando V. Raqueño Tuesday, April 18, 2023
Resulting pgmtoraw utility
• Uses existing routines to obtain informationpnmfile
wc
dd
• Functional tool written in 20 command lines
Rolando V. Raqueño Tuesday, April 18, 2023
Current Limitations of Resulting pgmtoraw utility
• No check between “ASCII” vs. “RAW” pgmif( data_type == ‘plain,’) ...
• No provisions for multibyte per pixel case– Use pnmfile results to check for above cases– endian case needs to be addressed for multibyte
case
• Above conditions can be addressed by suite of UNIX commands
Rolando V. Raqueño Tuesday, April 18, 2023
Shell Scripts Wrappers and IDL
• Another utility formerly missing in pbmplusjpegtopnm or pnmtojpeg
• IDL has jpeg read/write capability– Create a “wrapper” that makes an idl
program pbmplus-like
Rolando V. Raqueño Tuesday, April 18, 2023
pnmtojpeg.pro
pro pnmtojpeg, input_file, output_file
read_ppm, input_file, image
write_jpeg, output_file, image
end
Rolando V. Raqueño Tuesday, April 18, 2023
Usage of pnmtojpeg.pro in IDL
IDL> pnmtojpeg,‘image.pnm’,’image.jpg’
Rolando V. Raqueño Tuesday, April 18, 2023
Usage of pnmtojpeg.pro in IDL
IDL> pnmtojpeg,‘image.pnm’,’image.jpg’
• For IDL to automatically find pnmtojpeg.pro
– It must be in the current working directory
– Directory containing pnmtojpeg.pro must be defined in the ENVIRONMENT VARIABLE•IDL_PATH
Rolando V. Raqueño Tuesday, April 18, 2023
IDL_PATH environment variable
setenv IDL_DIR /cis/common/rsi/idl_5
setenv IDL_PATH \+$IDL_DIR/lib:\+$IDL_DIR/examples:
\+/dirs/common/rsi/idl_5:\+/dirs/common/lib/idl:\+~/lib/idl
Rolando V. Raqueño Tuesday, April 18, 2023
pnmtojpeg.csh
#!/bin/csh -f
echo pnmtojpeg “,” “’”$1”’” “,” “’”$2”’” | idl
Rolando V. Raqueño Tuesday, April 18, 2023
Usage of pnmtojpeg.csh
% pnmtojpeg.csh image.pnm image.jpg
Rolando V. Raqueño Tuesday, April 18, 2023
Limitation of pnmtojpeg.csh
• Does not conform to pbmplus piping, i.e.,% tifftopnm file.tif | pnmscale 2.0 > new_file.pnm
• No error trapping
Rolando V. Raqueño Tuesday, April 18, 2023
Usage cases of pnmtojpeg(no command line arguments)
% tifftopnm file.tif | pnmscale 2.0 | pnmtojpeg > new_file.jpg
Rolando V. Raqueño Tuesday, April 18, 2023
Usage cases of pnmtojpeg(1 command line argument)
% pnmtojpeg image.pnm > image.jpg
Rolando V. Raqueño Tuesday, April 18, 2023
Usage cases of pnmtojpeg(2 command line arguments)
% pnmtojpeg image.pnm image.jpg
Rolando V. Raqueño Tuesday, April 18, 2023
Yet another wrapper pnmtojpeg
#!/bin/csh -f
# If user interrupts process, jump to stop
onintr stop
# $0 is the command name
# $#argv is the number of arguments
# $$ is the process id
Rolando V. Raqueño Tuesday, April 18, 2023
Code for no argument case
if($#argv == 0) then
set input_file = /usr/tmp/$0_input_$$
set output_file = /usr/tmp/$0_output_$$
cat > $input_file
pnmtojpeg.csh $input_file $output_file
cat $output_file
.
.
.
Rolando V. Raqueño Tuesday, April 18, 2023
Code for 1 argument case
else if($#argv ==1) then
set input_file = $1
set output_file = /usr/tmp/$0_output_$$
pnmtojpeg.csh $input_file $output_file
cat $output_file
.
.
.
Rolando V. Raqueño Tuesday, April 18, 2023
Code for 2 argument caseelse set input_file = $1 set output_file = $2 pnmtojpeg.csh $input_file $output_fileendif
#clean up when finishedstop: rm -f /usr/tmp/$0_input_$$ rm -f /usr/tmp/$0_output_$$
Rolando V. Raqueño Tuesday, April 18, 2023
pnmtojpeg summary
• Produced a “new” pbmplus utility• Used UNIX shell scripting
– Argument handling– Scratch space /usr/tmp– Process id handling– Clean up
• Integrated IDL program and commands• 21 lines of UNIX commands
Rolando V. Raqueño Tuesday, April 18, 2023
Summary
• The “dot” files
• Basics of Shell Scripting
• Special Shell Variables
• Seamless integration of UNIX to other utilities (IDL)
Rolando V. Raqueño Tuesday, April 18, 2023
Other Shell Constructs to keep in mind
• foreach• while• case