Abstract

Linux continues to gain momentum as a cost-effective operating system [OS]. Many companies are obtaining significant ROI & savings by migrating to this platform from other proprietary flavors of Unix.

In today's business environment, Informix DBA's must be proficient with the Linux toolset. If you are a DBA in a Linux shop, then it is "vital" that you know how to use the OS effectively.

Items discussed

o How to manage server software

o View & configure system resources

o Analyze server performance

o Handle server log files

o Miscellaneous tips, tricks & treats

Flavors of Linuxo Debian – debian.org

o Fedora – fedoraproject.org [Redhat community project]

o Mandrake or Mandriva - www2.mandriva.com/en

o Redhat – redhat.com

o SuSE – novell.com/linux

o Ubuntu - ubuntu.com [derivative of Debian]

Managing server software

o rpm – Redhat Package Manager & now stands for RPM Package Manager [used by many Linux distributions]. o up2date – pulls rpm's & associated dependencieso yum - Yellowdog Updater, Modified [uses a xml repository]o apt – Advanced Packaging Tool [Debian & variants and modified to work with rpm via apt-rpm]

Package extensiono .src.rpm – download source & compile the rpmo .noarch – no dependency on architecture of the system

Managing server softwareo RPM system comprises of a local database in /var/lib/rpm,

rpm executable & rpm package files. Local database houses metadata information about installed

packages including package prerequisites & file attributes. Therefore, removing a package becomes relatively simple.

o Install package: rpm -ihv <package-name> i – install; h – hash marks; v – verbose outputo Identify rpm package to install: rpm -qf <filename> q – query; f – file; filename - /usr/bin/gedit [executable] or

libc.iso [shared object], for example

Managing server softwareo View contents of package & peek at destination where files

will be extracted: rpm -qlp <package-name> q – query; l – list; p – view information directly from packageo Upgrade an existing package: rpm -Uhv <package-name> or --

upgrade option [original package removal & new one installed]

-F or -- freshen will not install package if the package doesn't already exist

o Remove a package: rpm -e <package-name> or – erase avoid dependency check -- nodeps option [not recommended]

View & configure system resourceso View system information uname -a cat /proc/versionThe Linux /proc virtual filesystem acts as an interface for viewing and configuring kernel parameters.

The /proc directory is a hierarchy of files and subdirectories that contain the current settings of kernel values [appropriately named because this sends information to other system processes & virtual because its files don’t actually reside on disk.

View & configure system resourcesDescriptions of Virtual Files in the /proc Directory

File Name Contains Information Regarding

____________________________________________________

/proc/cpuinfo CPU and system architecture

/proc/meminfo Free and used memory for both physical RAM and swap

/proc/net Directory containing network information

/proc/mounts All mounted filesystems

/proc/diskstats Disk I/O statistics for each disk

View & configure system resourcesDescriptions of Virtual Files in the /proc Directory

File Name Contains Information Regarding

____________________________________________________

/proc/devices PCI devices

/proc/filesystems Filesystems compiled into the kernel

/proc/sys Contains subdirectories and files pertaining to kernel variables [some can

be configured with sysctl command]

/proc/cmdline Parameters passed to the kernel at boot time

/proc/version Version of the operating systems

View & configure system resourceso Some of /proc/<filename> may need root access to reviewo man proc will show documentation on /proc virtual filesystemo top & free extract some information from /proc virtual files Mem Total – Memory Physical RAM Swap Total – Memory Virtualo View per-process memory consumption cat /proc/<PID>/maps ; cat /proc/<PID>/statuso Viewing kernel parameters grep . /proc/sys/kernel/* [some may need root access to

review]

View & configure system resourceso Viewing kernel parameters - sysctl -a -w to dynamically modify kernel parameterso For making changes persistent across reboots, add parameters to /etc/sysctl.conf or do a "sysctl -w" & then issue a "sysctl -p" [entries in this file are instantiated as current values used by the linux kernel] It helps to keep a backup copy of kernel parameters before change – in case one needs to revert back + maintain an audit trail of all kernel changes [use diff command to display differences in file versions] cp /etc/sysctl.conf /etc/sysctl.conf.20110306

View & configure system resourceso Network settings sysctl -a | grep -i net

o The /etc/sysctl.conf file is read from top to bottom – last entry is set if multiple lines configure the same kernel parameter echo 65536 >> /proc/sys/fs/file-max or sysctl -w fs.file-max=65536 [remember to add an entry to /etc/sysctl.conf to make the change persistent across reboots]

View & configure system resourceso Not all virtual files in the /proc/sys directory can be modified. One quick way to determine whether a /proc/sys file can be altered is to check the permissions – any file that shows the writeable permission can be changed.

o Shared memory segments – messages, queues, semaphores A semaphore is a construct which signifies whether a resource is busy [or not] Inter Process Communication Status: ipcs -a - - displays list of currently allocated shared components

View & configure system resourceso Linux imposes a limit on the overall # of files simultaneously open on a server. Those that host database applications tend to have many simultaneously open files. If the default value for the maximum # of open files is too low, one will most likely have to increase it for preventing errors related to "running out of file handles".

cat /proc/sys/fs/file-nr - - this output will show the current allocated,

free & maximum file handle #'s respectively.

View & configure system resourceso A logon shell often imposes default maximum limits on various resources that a process can use such as the number of open files, processes per user, amount of memory allocated, and so on. These shell limits are defined by the ulimit command. Each resource has soft & hard limit settings. The soft limit setting establishes the default resource limit when a user logs on to system. If the user process exceeds resource soft limit setting, an error will be obtained.

List of system resource soft & hard limits - ulimit -a ; ulimit -aH

View & configure system resourceso A user can modify the hard limit down but cannot modify the hard limit up. This means if one sets a hard limit to a lower value, one cannot reset it to its original value. Only the root user can modify a hard limit to a higher value.

o The default values for soft and hard limits on the server are established by adding entries into /etc/security/limits.conf file.

o Database processes often tend to consume more resources than what default shell limits allow.

Analyze server performanceOS monitoring can assist in providing answers to questions like -

o Details of resources an application is using

o Applications doing work or waiting for other resources

o Server bottlenecks possibly affecting database performance

o Potential resources used by database engine which likely can't be monitored with bundled-in database commands & tools

Analyze server performanceTool Performance Monitoring Purpose

_______________________________________________

df Reports on free disk space

du Displays disk usage

free Displays free and used memory

iostat Displays disk I/O statistics

mpstat Reports CPU statistics

netstat Reports on network statistics

Analyze server performanceTool Performance Monitoring Purpose

_______________________________________________

ps Identifies highest CPU- and memory-consuming sessions

sar Displays CPU, memory, disk I/O, and network usage, both current and historical

top Identifies sessions consuming the most resources

vmstat Monitors processes, CPU, memory, or disk I/O bottlenecks

watch Periodically runs another command

{io|mp|net|vm}stat <interval-sec> <interval-#>

Analyze server performanceIdentifying System BottlenecksProblem: The application users are reporting that the database seems slow. One wants to determine whether there are any system resource bottlenecks on the database server.Solution: The vmstat (virtual memory statistics) tool is intended to help in quickly identifying bottlenecks on a server. It displays real-time performance information about processes, memory, paging, disk I/O, and CPU usage.Some general techniques one can use for interpreting the output of vmstat are in the following slide.

Analyze server performanceo If b (processes sleeping) is consistently > 0, then one may not have enough CPU processing power. Recipe 1 aids in identifying items consuming the most CPU.o If so/si (memory swapped out to/in from disk) are consistently > 0, one may have a memory bottleneck. Recipe 3 aids in identifying items consuming the most memory.o If wa (time waiting for I/O) column is high, this is usually an indication that the storage subsystem is overloaded. Recipes 4 & 5 aid in identifying the sources of I/O contention.

Note: The Linux vmstat command does not count itself as a currently running process.

Analyze server performanceBy default, only 1 line of server statistics is displayed when running vmstat (without supplying any options) - average statistics calculated from the last time the system was rebooted.One can also redirect vmstat output to a file - this is useful for analyzing historical performance over a period of time.Another useful way to use vmstat is with watch which is used to execute another program on a periodic basis: For example, watch -n <interval-sec> -d vmstatWhen running vmstat in watch -d (differences) mode, one will visually see changes on screen as they alter from snapshot to snapshot. To exit from watch, press Ctrl+C.

Analyze server performanceRecipe 1 - Identifying CPU-Intensive ProcessesProblem: Identify database sessions consuming the most CPU on the server [if running a SQL query, then display related SQL also]Solution A: The ps command identifies session process IDs consuming most CPU. The command below displays the top 10 CPU-consuming statements and the associated process ids -ps -e -o pcpu,pid,user,tty,args | grep -i informix | sort -n -k 1 -r | head -10 Solution B: Use top command – with batch mode [-b option] to run until ending (with a Ctrl+C) or a specified # of iterations.

Note: These may be tuned/terminated as needed

Analyze server performanceRecipe 2 – Analyzing Historical CPU LoadProblem: View CPU load over past several daysSolution: Use sar with -u option to report on CPU statistics – by default, it will report on current day activities. The -f option reports on the previous day and files that sar uses to report on statistics for different days of the month are located in the /var/log/sa<##> directory where ## is two-digit day of month.For example, sar -u -f /var/log/sar10 [here ## is 10th day of month]Note: Since reports are on events in past, determine what system activity was taking place that caused any unusual spikes.

Analyze server performanceRecipe 3 - Identifying Memory-Intensive ProcessesProblem: Identify database sessions consuming the most memory [if running a SQL query, then display related SQL also]Solution A: The ps command identifies session process IDs consuming most memory. The command below displays the top 10 memory-consuming statements & the associated process ids -ps -e -o pmem,pid,user,tty,args | grep -i informix | sort -n -k 1 -r | head -10 Solution B: Use top command – with batch mode [-b option] to run until ending (with a Ctrl+C) or a specified # of iterations.

Note: These may be tuned/terminated as needed

Analyze server performanceRecipe 4 - Identifying I/O-Intensive ProcessesProblem: Identify database sessions consuming the most I/O [if running a SQL query, then display related SQL also]Solution: The iostat & top commands identify devices with disk bottlenecks. For example, iostat -xd <interval-sec> When trying to determine whether device I/O is a bottleneck, the general guidelines below will aid in examining iostat output:o Devices with abnormally high blocks read/written per secondo If any device is near 100% utilization, that’s a strong indicator of an I/O bottleneck Note: These may be tuned/terminated as needed

Analyze server performanceRecipe 5 – Analyzing Historical I/O LoadProblem: View I/O load over past several daysSolution: Use sar with -d option to report on I/O statistics – by default, it will report on current day activities. The -f option reports on the previous day and files that sar uses to report on statistics for different days of the month are located in the /var/log/sa<##> directory where ## is two-digit day of month.For example, sar -d -f /var/log/sar10 [here ## is 10th day of month]Note: Since reports are on events in past, determine what system activity was taking place that caused any unusual spikes.

Analyze server performanceRecipe 6 - Identifying Network-Intensive Processes

Problem: Identify database sessions with the most network traffic [if running a SQL query, then display related SQL also]

Solution: The netstat & top commands provide reports on network traffic. For example, netstat -ptc will display the process id & tcp connections. Note: These may be tuned/terminated as needed

Handle server log fileso Use the logwatch utility to monitor server log fileslogwatch --printParameter file - /etc/logd/logwatch.conf

o Use the logrotate utility to perform tasks such as renaming, compressing & removing old logfiles

logrotate -f -s <status-file> <config-file>-f – force logrotate to do a rotation

Miscellaneous tips, tricks & treatsDBA's often tar & untar binaries from one server to anotherThe command below assists one in not incurring double storage with storing a local & remote copy of tar archive -tar cvf - {<dir-name>} | ssh { <target-server>} "cd <path-name> ; tar xvf - "The tar command piped to ssh ensures that symbolic links get copied over as symbolic links

Scheduling: o use crontab command for jobs which need to be run ofteno use at command to run a job once at a specified time in future

Miscellaneous tips, tricks & treatsTemporarily adding swap space to a server [done by root user]:If short on swap space, one can temporarily add a swap file on the server [commands will add around 1GB of swap space] - # dd if=/dev/zero of=tempswap bs=1k count=1000000# chmod 600 tempswap; mkswap tempswap; swapon tempswapVerify that the swap space was added - swapon -sTo remove this temporary swap file - swapoff tempswap; rm tempswapAfter disabling the swap file, swap space in /proc/meminfo should return to its original value.

Miscellaneous tips, tricks & treatso Menu-driven setup utility on SuSE – yast [Yet Another Simple Tool] & yast2 o Nmon – Swiss army-knife tool for monitoring performance http://nmon.sourceforge.net/pmwiki.phpo Data Studio – http://www.ibm.com/software/data/optim/data-studio/o Determining ip-address & domain for a servername: hostname -i <servername> – ip-address hostname -d <servername> – domain [on other OS like AIX, a separate command like "nslookup <servername>" determines this information]

Miscellaneous tips, tricks & treatso Removing in-memory structures When a database crashes unexpectedly, it may happen that for some reason semaphores and shared memory don't get released. If other databases are running on the server, one cannot reboot the system to release the shared memory objects. One can manually remove these orphaned memory structures. ipcs -sm - - view a list of structures to be removed; choose with care ipcrm -m <id> ipcrm -q <id> ipcrm -s <id>

Miscellaneous tips, tricks & treatso Linux user has default shell of bash – if korn shell is preferred, then chsh /bin/ksho man command provides online help - "<command> --help"o du -sk * . | sort -nr | head -10 provides a report of the top 5 sub- directories consuming space in a particular directory with a comparison to the total $ alias tn [acronym for top #] = du -sk * . | sort -nr | head -10 or function tnf [short for top # function] { du -sk * . |sort -nr | head -10 } $ tnf 10

Linux meets

INFORMation on unIX's needs!

Questions ?!?