unix tutorial - advanced

8/4/2019 Unix Tutorial - Advanced

1/26

1

Advanced UNIX Tutorial

Shelling,Otheus

For University of Innsbruck, March 2007

NOTE:NOTTHEFINALVERSION!!

Topics CoveredWhy Unix?

Brief History

Architecture

Inter-processCommunication

Filesystem

Regular Expressions

Program Execution PATH to enlightenment

Dynamic Linking

The Shells BASH Programming

Shell Startup

Short-cuts

Cool Commands

root:/ $

Why UNIX? And Why Not? High Performance

Fine-grain

control overhardware

Fine-graincontrol over OS

WYSIWYG .. NotWYSIAYG

Windows has caught up inperformance

Windows is easier toadministrate

Windows systems dont requirefine-grain control


2/26

2

Flavors, Versions, DistributionsFlavors

UNIX comes in variousFlavors

System V, BSD, Linux

Distributions

A particular Flavor will be packaged and marketed

through variousDistributions

FreeBSD, RedHat, Gentoo

Versions

A particularDistribution will be released with a

specific Version number

RedHat EL4 US, Solaris 10 (2.10)

Other version designations

The kernel and various software tools and

applications each have their own version numbering

X11R6, Linux 2.6, Perl 5.6

UNIX Architecture System LevelDevices -

managed by drivers

Drivers -

interact through anAPI with the Kernel

Kernel -

manages core resources

Processes

Memory (physical andvirtual)

Interprocesscommunication (IPC)

presents to users:

system calls

device files

UNIX Architecture User LevelUser:

interacts with an application through a

device

Application:commands

ls, qsub,

run-time environment

bash, perl, python,

Java, PHP, mozilla

subsystem

X Windows, SGE

Shared libraries:glibc, gtk

Each application is in its ownprocess and has its

own private data. The code can be shared acrossinstances.

Each process runs under the permissions of the

user that started it effective user id

The application and its libraries make system

calls and operates onfiles.


3/26

3

Inter-Process Communication

Inter-Process communication (IPC) Semaphores

Shared Memory

Filesystem

Pipes

Signals

*otheus* ~/eg$ ls -1

example

example2

exists

myexample

ozzie

*otheus* ~/eg$ ls -1 >files*otheus* ~/eg$ sort -k 1.2,1.4 filesfiles

example

example2

exists

myexample

ozzie

*otheus* ~/eg$


Shared Memory

Filesystem

Pipes

Unnamed

Named / FIFOs

Network Sockets

RPC

Signals

*otheus* ~/eg$ ls -1

example

example2

existsfiles

myexample

ozzie

*otheus* ~/eg$ ls -1 | sort -k 1.2,1.4

files

example

example2

exists

myexample

ozzie

*otheus* ~/eg$


4/26

4


Shared Memory

Filesystem

Pipes

Unnamed

Named / FIFOs

Network Sockets

RPC

Signals

*otheus* ~/eg$ mkfifo newfifo*otheus* ~/eg$*otheus* ~/eg$ ls -l newfifoprw-r--r-- 1 otheus dps 0 Mar 28 19:52 newfifo

*otheus* ~/eg$ { while true;> do echo $RANDOM >>/home/otheus/eg/newfifo ;> sleep 5; done } &[1] 15533

*otheus* ~/eg$ ls -l newfifoprw-r--r-- 1 otheus dps 0 Mar 28 19:52 newfifo

*otheus* ~/eg$ date ; cat newfifo ; dateWed Mar 28 19:52:49 CEST 200732102Wed Mar 28 19:52:49 CEST 2007

*otheus* ~/eg$ date ; cat newfifo ; dateWed Mar 28 19:52:51 CEST 200726331Wed Mar 28 19:52:54 CEST 2007

*otheus* ~/eg$


Shared Memory

Filesystem

Pipes

Unnamed

Named / FIFOs

Network Sockets2-way pipes over a network

RPC

Signals

*otheus* ~/eg$ sudo netstat -anp |> grep -E "(181):(3306|80|22) " |> head -3

tcp 0 0 138.232.66.181:220.0.0.0:* LISTEN 4176/sshd

tcp 0 48 138.232.66.181:22192.168.64.52:2131 ESTABLISHED 9343/5

tcp 0 0 138.232.66.181:3306138.232.66.164:49325 TIME_WAIT -


Shared Memory

Filesystem

Pipes

RPC

RPC process

Portmapper

process

YP/NIS & NFS

Signals

*otheus* ~/eg$ ps -e -o uid,pid,cmd |> grep rpc

29 3338 rpc.statd

0 3599 rpc.idmapd0 4093 rpc.yppasswdd -e chsh -e chfn0 4274 rpc.rquotad0 4312 [rpciod]0 4323 rpc.mountd

201 17112 grep rpc

*otheus* ~/eg$ rpcinfo -p | uniq -f 4

program vers proto port100000 2 tcp 111 portmapper100024 1 udp 32768 status100007 2 udp 856 ypbind100004 2 udp 845 ypserv100009 1 udp 878 yppasswdd100011 1 udp 640 rquotad100003 2 udp 2049 nfs100021 1 udp 32774 nlockmgr100005 1 udp 686 mountd


5/26

5


Shared

Memory

Filesystem

Pipes

RPC

Signals Only within a

process family

Processes 0, -1

SIGHUP

SIGTERM

SIGKILL

SIGSEGV

*otheus* ~/eg$ kill -l | head -81) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR213) SIGPIPE 14) SIGALRM 15) SIGTERM 17) SIGCHLD18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMI

*otheus* ~/eg$ echo $$2432*otheus* ~/eg$ /bin/ash

*\u* \w$ echo $$18201*\u* \w$ kill -HUP 18201Hangup

*otheus* ~/eg$ echo $$2432

*otheus* ~/eg$ trapped &[1] 18664*otheus* ~/eg$ kill -1 18664 ; sleep 1Caught signal!*otheus* ~/eg$ kill 18664 ; sleep 1[1]+ Terminated trapped*otheus* ~/eg$

UNIX Filesystem

UNIX Filesystem Tree Structure Resembles a tree

File-types

Mounting drives Breaking

structure

Promiscuous

Permissions


6/26

6


File-types

Mounting drives

Breaking

structure

Promiscuous

Permissions


File-types

Mounting drives

Breaking

structure

Soft-links

Loopback mounts

Bind mounts

Promiscuous

Permissions


File-types

Mounting drives Breaking

structure

Soft-links

Loopback mounts

Bind mounts

Promiscuous

Permissions


7/26

7


File-types

Mounting drives

Breaking

structure

Soft-links

Loopback mounts

Bind mounts

Promiscuous

Permissions

UNIX Filesystem Very Promiscuous Resembles a tree

Promiscuous:

Nearly everything in UNIX can be accessed through the fi lesystem

Devices

Processes (Linux & Solaris) via /proc filesystem

Kernel stats and configuration (Linux: /proc)

Exception: Network interfaces

Permissions


Promiscuous

Devices

Processes

Kernel stats and

configuration

Permissions

MAKEDEV loop5 ra m14 skip t ty2XOR loop6 ram15 stderr tty3arpd loop7 ram2 stdin tty4cdrom lp0 ram3 stdout tty5cdrom1 lp1 ram4 systty tty6

console lp2 ram5 tap0 tty7core lp3 ram6 tap1 tty8cpu mapper ram7 tap10 tty9cpu0 mcelog ram8 tap11 urandomcpu1 md0 ram9 tap12 usersockcpu2 mem ramdisk tap13 vcscpu3 mice random tap14 vcs1dnrtmsg mosal rawctl tap15 vcs2dvd net root tap2 vcs3fd nflog route tap3 vcs4floppy null route6 tap4 vcs5full parport0 rtc tap5 vcs6fwmonitor parport1 scd0 tap6 vcsahda parport2 sda tap7 vcsa1hw_random parport3 sdb tap8 vcsa2initctl port sdc tap9 vcsa3input ppp sdd tcpdiag vcsa4ip6_fw ptmx sde ts_ua0 vcsa5js0 pts sde1 ts_ua1 vcsa6kmsg ram sde2 ts_ua2 vipkllog ram0 sde3 ts_ua3 xfrmloop0 ram1 sdf ts_ua4 zeroloop1 ram10 sdf1 ts_ua5loop2 ram11 sdf2 ts_ua6loop3 ram12 sdf3 tty0loop4 ram 13 shm t ty1


8/26

8


Promiscuous

Devices

Processes

Kernel stats and

configuration

Permissions

*otheus* /proc$ ls -Cd [1-2]*1 14642 17888 21007 21942 258

10 14644 17889 21008 21943 2619

10377 14657 17890 21009 2251 2620

10395 15 17892 21010 2269 2647

11 155 17894 21017 2270 2648

11639 17473 19570 21019 2312 2649

12 17476 2 21020 2313 2650

13 17478 20998 21021 240 2658

13956 17511 20999 21022 241 27265

13964 17871 21000 2121 2431 286

14 17884 21002 21646 2432

1433 17887 21003 21941 257

*otheus* /proc$ cat 1/cmdline; echoinit [4]

*otheus* /proc$ ls -F 1attr/ environ maps stat wchan

auxv exe@ mem statm

cmdline fd/ mounts status

cwd@ loginuid root@ task/

*otheus* /proc$


Promiscuous

Devices

Processes

Kernel stats and

configuration

Permissions

*otheus* /proc$ ls -CdF [a-z]*acpi/ iomem net/buddyinfo ioports partitionsbus/ irq/ pcicmdline kallsyms scsi/cpuinfo kcore self@crypto key-users slabinfodevices keys statdiskstats kmsg swapsdma loadavg sys/driver/ locks sysrq-triggerexecdomains mdstat sysvipc/fb meminfo topspin@filesystems misc tty/fs/ modules uptimeide/ mounts@ versioninfiniband/ mpt/ vmstatinterrupts mtrr

*otheus* /proc$ cat uptime704398.43 593929.58*otheus* /proc$ uptime20:56:58 up 8 days, 3:39, 7 users,

load average: 1.72, 1.75, 1.67*otheus* /proc$

UNIX Filesystem Security Resembles a tree

Promiscuous

Permissions

3 Classes:

User

Group

Other (World)

4 bits:

Read

Write

Execute

Special


9/26

9


Promiscuous

Permissions

3 Classes:

User

Group

Other (World)

4 bits:

Read

Write

Execute

Special


Promiscuous

Permissions

4 bits:

Read

Write

Execute

Special

Directories

Set-uid

Set-gid


Promiscuous

Permissions

4 bits:

Read

Write

Execute

Special

Directories

Set-uid

Set-gid


10/26

10

Regular Expressions

Regular Expressions Quick Overview

Regular Expressions Examples


11/26

11



Program Execution


12/26

12

Program Execution - Overview

Program Execution Example

Program Execution Example


13/26

13

The Shells

The Shells Run applications

and commands

Interactively

Batch mode

In parallel

Interact with

Files

Manage

environment used

by applications


and commands

Interact withFiles

Send files to

commands.

Send commands

output to files.

Manage

environment used

by applications


14/26

14


and commands

Interact with

Files

Manage

environment used

by applications

Environment

variables

File creation

mask

The Shells Flavors

Bourne (sh)

Korn (ksh)

Bourne-Again(bash)

Almquist (ash)

Z shell (zsh)

C Shell (csh)

tcsh

Exotic

fish

zoidberg

GUI

Konqueror

Text-GUI

MidnightCommander (mc)

The ShellsMans Best Friend

help (BASH only)

man

more or less

man k

man (Solaris)

man (GNU/BSD)

info


15/26

15

The ShellsMans Best Friend

help (BASH only)

man

info

Hyperlink viewer

CTRL-H

--vi-keys

File: info.info, Node: Top, Next: Getting St\arted, Up: (dir)

Info: An Introduction*********************

The GNU Project distributes most of its on-lin\e manuals in the "Infoformat", which you read using an "Info reader"\. You are probably usingan Info reader to read this now.

If you are new to the Info reader and want \to learn how to use it,type the command `h' now. It brings you to a \programmed instruction--zz-Info: (info.info.gz)Top, 35 lines --Top---Basic Commands in Info Windows******************************

C-x 0 Quit this help.C-x C-c Quit Info altogether.h Invoke the Info tutorial.

Selecting other nodes:----------------------n Move to the "next" node of this no\de.p Move to the "previous" node of thi\s node.u Move "up" from this node.m Pick menu item specified by name.-----Info: *Info Help*, 763 lines --Top--------No cross references in this node.

The Shells Using AliasesWhat is it?

Creating an Alias

Automatic

aliasing (bash)

hash

type / which

unalias

alias

The Shells Using AliasesWhat is it?

Creating an Alias

Automatic

aliasing (bash)

hash

type / which

unalias

alias


16/26

16

Advanced Shell Programming

Advanced Shell Programming with BASHWorking Directory

Variable Creation

& Expansion

Job Control

File Manipulation

File

Specification

Negative Logic

Execution Control Functions

exec

fg / bg

eval

export

set / echo

cd

pwd

Advanced Shell Programming with BASH Working Directory

Variable Creation

& Expansion Job Control

File Manipulation

File

Specification

Negative Logic

Execution Control

Functions

exec

fg / bg

eval

export

set / echo

cd

pwd


17/26

17


Variable Creation

& Expansion

Job Control

File Manipulation

File

Specification

Negative Logic

Execution Control

Functions

ls

kill

jobs

exec

fg / bg

eval

export


Variable Creation

& Expansion

Job Control

File Redirection

and Manipulation

Listing Files

Viewing Files

File

Specification

Negative Logic

Execution Control

Functions

rm /cp / mv

grep

head / tail

more / less

cat

ls

exec


Variable Creation

& Expansion

Job Control

File Manipulation

standard streams

Copying, moving,

renaming, new dir

File

Specification

Negative Logic

Execution Control

Functions

newgrp

id / groups

chmod / chown

mkdir / rmdir

rm /cp /mv

grep

head / tail


18/26

18


Variable Creation

& Expansion

Job Control

File Manipulation

Changing owner

Changing

permissions

File

Specification

Negative Logic

Execution Control

Functions

umask

newgrp

id / groups

chmod / chown

mkdir / rmdir


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

File globs

Negative Logic Execution Control

Functions

exec

fg / bg

xargs

find

umask

newgrp

id / groups


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

find and xargs

Negative Logic

Execution Control

Functions

exec

fg / bg

xargs

find

umask

newgrp

id / groups


19/26

19


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

Functions

for VAR in .. do ..

done

case W in .. esac

if .. fi

true / false

test

eval


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control if/elif/else/fi

Functions

while.. do..

done

for VAR in ..

do .. done

case W in ..

esac

if .. fi

true / false


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

case WORD in ...

esac

Functions

while.. do..

done

for VAR in ..

do .. done

case W in ..

esac

if .. fi

true / false


20/26

20


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

case WORD in ...

esac

Functions

while.. do..

done

for VAR in ..

do .. done

case W in ..

esac

if .. fi

true / false


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control for VAR in ...

done

Functions

while.. do..

done

for VAR in ..

do .. done

case W in ..

esac

if .. fi

true / false


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

while ... do ...

done

Functions

while.. do..

done

for VAR in ..

do .. done

case W in ..

esac

if .. fi

true / false


21/26

21


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

break, next, exit

||, &&

type, which,whence

Functions

|| and &&

exit

next

break

while.. do..

done


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

break, next, exit ||, &&

type, which,whence

Functions

whence

which

type


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

Functions

Declaring,

returning

declare

command

return


22/26

22


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

Functions Using

Overridingbuiltins &commands

declare

builtin

command


Variable Creation

& Expansion

Job Control

File Redirection

File

Specification

Negative Logic

Execution Control

Functions Using

Overridingbuiltins &commands

type

builtin

command

Shell Startup


23/26

23

Shell Startup (bash)

Shell Startup (tcsh)

Cool Commands


24/26

24

Cool Commands

Cool Commands

Cool Commands


25/26

25

Complex Examples

Complex Examples

Complex Examples


26/26

26

Complex Examples

Complex Examples

Complex Examples

unix tutorial - advanced

Documents