1

1

Linux Performance TuningLinux Performance Tuning

Jaqui Lynch

USCMGJaqui.lynch@mainline.com

2

AgendaAgenda

IntroductionVarious Commands– CPU– Memory– Disk and I/O– Network

2

3

IntroductionIntroductionLinux is Linux– Suse, Redhat, Debian, IBM– Security and performance

Capacity & Scalability Viewpoint– XPZ – Test on X, Develop on P, Production on Z

Much the same as tuning any other system– Processor, Memory, I/O, network

Focus of this presentation is Redhat Fedora

4

Kernel ChangesKernel Changes

Redhat– /etc/sysctl.confEcho “value” >/proc/kernel/parameter– Is reset at bootRemove all unneeded services – /etc/xinetd.d/– X, pop, ….

3

5

Commands for CPUCommands for CPU

vmstattopulimitProvided by sysstat rpm– mpstat– iostat– sar

/proc/cpuinfops

6

ulimitulimit --aa# ulimit -a

core file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedfile size (blocks, -f) unlimitedmax locked memory (kbytes, -l) unlimitedmax memory size (kbytes, -m) unlimitedopen files (-n) 1024pipe size (512 bytes, -p) 8stack size (kbytes, -s) 10240cpu time (seconds, -t) unlimitedmax user processes (-u) 3072virtual memory (kbytes, -v) unlimited

ulimit –u unlimitedSet max procs per user to unlimited

4

7

sarsar ––v Kernel Valuesv Kernel ValuesLinux 2.4.20-30.9 (biteme.xxx.com) 03/30/2004

12:13:28 PM dentunusd file-sz %file-sz inode-sz12:13:30 PM 115509 107 0.27 11857312:13:32 PM 115509 107 0.27 11857312:13:34 PM 115509 107 0.27 11857312:13:36 PM 115509 107 0.27 11857312:13:38 PM 115509 107 0.27 118573……..Average: 115509 107 0.27 118573

12:13:28 PM super-sz %super-sz dquot-sz %dquot-sz rtsig-sz %rtsig-sz12:13:30 PM 0 0.00 0 0.00 0 0.0012:13:32 PM 0 0.00 0 0.00 0 0.0012:13:34 PM 0 0.00 0 0.00 0 0.0012:13:36 PM 0 0.00 0 0.00 0 0.0012:13:38 PM 0 0.00 0 0.00 0 0.00……..Average: 0 0.00 0 0.00 0 0.00

8

more /proc/more /proc/cpuinfocpuinfo

processor : 0vendor_id : GenuineIntelcpu family : 6model : 7model name : Pentium III (Katmai)cpu MHz : 448.972cache size : 512 KB

Above are just a few of the items shown

5

9

CPU TimeCPU Time

Real– Wallclock time

User state CPU– Time spent running a user’s program– Includes library calls– Can be affected by compile optimization and writing efficient code

System code CPU– Time spent in the system state on the user’s behalf– Kernel calls & all I/O routines– Can be affected by using blocking for I/O transfers

I/O Time & Network Time– Time spent moving data and servicing I/O requests

10

uptimeuptime12:23pm up 86 days, 4:42, 4 users, load average: 0.44, 0.61,0.6811:53am up 56 days 4:59 1 user load average: 5.45, 5.45, 5.4609:20am up 23 days 15:23 1 user load average: 0.61, 0.65, 0.56

Provides system load averageRough IndicatorCounts as runnable all jobs waiting on disk or nfs I/O Last minute, last 5 minutes, last 15 minutes– Single user workstation <2– <3 Light 4-7 heavy >7 ??????

6

top on top on linuxlinux12:24pm up 86 days, 4:43, 4 users, load average: 0.66, 0.63, 0.68119 processes: 115 sleeping, 3 running, 0 zombie, 1 stoppedCPU states: 25.3% user, 7.6% system, 0.0% nice, 67.0% idleMem: 517356K av, 514272K used, 3084K free, 550780K shrd, 33728K buffSwap: 525288K av, 6916K used, 518372K free 152520K cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND22339 sybase 19 0 16384 15M 16236 R 0 12.8 3.1 3095m datasv14590 jaqui 4 0 1268 1268 1016 S 0 7.9 0.2 0:00 ssh14511 webid 12 0 12448 12M 8828 S 0 3.8 2.4 0:00 httpd25935 webid 7 0 14828 14M 6888 S 0 2.7 2.8 2:43 httpd14586 root 6 0 1060 1060 836 R 0 2.3 0.2 0:00 top10589 webid 2 0 13256 12M 7348 S 0 0.5 2.5 0:26 httpd14514 webid 1 0 12588 12M 8616 S 0 0.3 2.4 0:00 httpd14585 webid 0 0 11336 11M 11000 S 0 0.3 2.1 0:00 httpd25930 root 1 0 11160 10M 11008 S 0 0.1 2.1 0:11 httpd25938 webid 0 0 14740 14M 7072 S 0 0.1 2.8 2:31 httpd25970 webid 0 0 15548 15M 6868 S 0 0.1 3.0 2:16 httpd13675 root 0 0 1272 1240 1028 R 0 0.1 0.2 0:01 sshd

7

nmonnmon

14

mpstatmpstatmpstat 2 10 needs sysstat

mpstat 2 10 (phntest)Linux 2.4.20-30.9 (biteme.xxx.com) 03/30/200401:33:28 PM CPU %user %nice %system %idle intr/s01:33:30 PM all 100.00 0.00 0.00 0.00 104.0001:33:32 PM all 100.00 0.00 0.00 0.00 100.0001:33:34 PM all 100.00 0.00 0.00 0.00 102.5001:33:36 PM all 100.00 0.00 0.00 0.00 100.0001:33:38 PM all 100.00 0.00 0.00 0.00 100.0001:33:40 PM all 100.00 0.00 0.00 0.00 101.5001:33:42 PM all 100.00 0.00 0.00 0.00 100.0001:33:44 PM all 99.50 0.00 0.50 0.00 101.5001:33:46 PM all 100.00 0.00 0.00 0.00 100.0001:33:48 PM all 100.00 0.00 0.00 0.00 100.00Average: all 99.95 0.00 0.05 0.00 100.95

8

15

mpstatmpstat ––P ALLP ALLmpstat -P ALL 2 10 needs sysstat

Linux 2.2.17 (jaqui.abc.com) 05/15/01

11:29:18 AM CPU %user %nice %system %idle intr/s11:29:20 AM all 1.50 0.00 3.25 95.25 102.0011:29:20 AM 0 0.00 0.00 0.00 100.00 102.0011:29:20 AM 1 3.00 0.00 6.50 90.50 102.00

11:29:20 AM CPU %user %nice %system %idle intr/s11:29:22 AM all 1.50 0.00 1.00 97.50 101.5011:29:22 AM 0 0.00 0.00 0.00 100.00 101.5011:29:22 AM 1 3.00 0.00 2.00 95.00 101.50

………Average: CPU %user %nice %system %idle intr/sAverage: all 0.30 0.00 0.42 99.28 103.00Average: 0 0.00 0.00 0.00 100.00 103.00Average: 1 0.60 0.00 0.85 98.55 103.00

16

sarsar --uuSar does not come with most Linux systems – you need todownload it as part of the sysstat rpm

Create file using sar –a –o sardata.out

sar -u -f sardata.out

Linux 2.4.20-30.9 (biteme.xxx.com) 03/30/2004

12:34:44 CPU %user %nice %system %idle12:34:46 all 53.50 0.00 9.00 37.5012:34:48 all 79.00 0.00 11.50 9.5012:34:50 all 80.00 0.00 6.50 13.5012:34:52 all 59.00 0.00 8.00 33.0012:34:54 all 75.50 0.00 15.50 9.00Average: all 69.40 0.00 10.10 20.50

9

17

sarsar --qq# sar -q -f sardata.out

12:13:28 PM runq-sz plist-sz ldavg-1 ldavg-512:13:30 PM 1 63 1.03 0.9612:13:32 PM 1 63 1.03 0.9612:13:34 PM 1 63 1.03 0.9612:13:36 PM 1 63 1.03 0.9612:13:38 PM 1 63 1.02 0.96……..12:14:02 PM 1 63 1.02 0.9612:14:04 PM 1 63 1.02 0.9612:14:06 PM 1 63 1.02 0.9612:14:08 PM 1 63 1.01 0.96Average: 1 63 1.02 0.96

18

psps auxauxps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 1096 52 ? S Feb18 0:03 init [3]root 2 0.0 0.0 0 0 ? SW Feb18 4:31 [kflushd]root 3 0.0 0.0 0 0 ? SW Feb18 0:00 [kpiod]root 4 0.0 0.0 0 0 ? SW Feb18 7:54 [kswapd]root 5 0.0 0.0 0 0 ? SW< Feb18 0:00 [mdrecoveryd]bin 254 0.0 0.0 1092 216 ? S Feb18 0:01 portmaproot 301 0.0 0.0 1280 372 ? S Feb18 0:24 syslogd -m 0jaqui 29044 0.0 0.1 1264 548 pts/3 S 10:08 0:00 -kshwebid 29173 0.5 2.4 14432 12640 ? S 10:14 0:02 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29174 0.3 2.4 14840 12664 ? S 10:14 0:01 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29175 0.2 2.4 14728 12580 ? S 10:14 0:01 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29176 0.3 2.3 13908 12308 ? S 10:14 0:01 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29177 0.2 2.4 14724 12612 ? S 10:14 0:00 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29178 0.1 2.3 13784 12148 ? S 10:14 0:00 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29195 0.1 2.3 13888 12208 ? S 10:15 0:00 /usr/local/apache/bin/httpd -d /usr/local/apachewebid 29206 0.4 2.4 14728 12788 ? S 10:15 0:01 /usr/local/apache/bin/httpd -d /usr/local/apache

10

19

Commands for MemoryCommands for Memory

/proc/meminfosar/proc/sys/vm/*vmstatlsmod or look at /proc/modulesvmstatswapon & swapofffreetop

20

UNIX and MemoryUNIX and MemoryUses ALWAYS FULL modelKernel Space– Memory reserved for the kernel– Used for device drivers, scheduling and to keep

the kernel in memoryUser Space– Memory for application code – includes servers

Buffer Space– Used to buffer I/O for disks and network– Used for DMA transfers between devices– Applied to both read and write operations

11

21

vmstatvmstat (Linux)(Linux)

vmstat 2 10Can use vmstat –m or vmstat –g to see in MB or GB

procs memory swap io system cpur b w swpd free buff cache si so bi bo in cs us sy id1 0 0 6916 13700 27876 156100 0 0 4 4 1 4 3 4 12 0 0 6916 9520 28196 160324 0 0 2121 348 202 138 79 4 174 0 0 6916 4020 28560 163352 0 0 1520 0 219 272 80 15 53 0 0 6916 1732 28532 165408 0 0 1226 250 234 208 73 8 201 2 0 6916 3016 25628 164360 0 0 1250 569 247 395 78 9 132 1 0 6916 2296 21976 164476 0 0 1430 485 245 250 79 11 102 0 0 6916 2444 20208 164480 0 0 660 140 372 396 74 8 182 0 0 6916 2360 19084 165736 0 0 1835 134 211 248 82 9 91 0 0 6916 1192 17908 168180 0 0 2443 161 185 138 88 5 72 0 0 6916 2824 16000 168544 0 0 2122 251 202 160 81 4 14

22

Swap DevicesSwap Devices

Swap device is a disk partition Can be a logical volume with some effortSwap device in /etc/fstab defaults to activeswapon –s list active swap devices

Filename Type Size Used Priority

/dev/hdb3 partition 1020116 1872 -1

12

23

swaponswapon and freeand freeFreeTest:

total used free shared buffers cachedMem: 517356 515076 2280 465748 10708 179340-/+ buffers/cache: 325028 192328Swap: 525288 6916 518372

Biteme:Mem: 384568 372860 11708 0 93332 130044-/+ buffers/cache: 149484 235084Swap: 1020116 1872 1018244

free -mtotal used free shared buffers cached

Mem: 505 477 28 443 36 120-/+ buffers/cache: 320 184Swap: 512 6 506

24

/proc//proc/meminfomeminfo

more /proc/meminfo

Mem: 529772544 436813824 92958720 552947712 32686080 62316544Swap: 537894912 7081984 530812928MemTotal: 517356 kBMemFree: 90780 kBMemShared: 539988 kBBuffers: 31920 kBCached: 60856 kBSwapTotal: 525288 kBSwapFree: 518372 kB

13

25

/proc/sys//proc/sys/vmvm

/proc/sys/vm

cat bdflush50 500 0 0 500 3000 80 50 0

cat max-readahead31

cat min-readahead3

26

sarsar --BBsar -B -f sardata.out

12:13:28 PM pgpgin/s pgpgout/s activepg inadtypg inaclnpg inatarpg12:13:30 PM 0.00 40.00 47258 0 1596 1161412:13:32 PM 0.00 8.00 47260 0 1596 1161412:13:34 PM 0.00 8.00 47262 0 1596 1161512:13:36 PM 0.00 16.00 47264 0 1596 1161512:13:38 PM 0.00 8.00 47266 0 1596 11616………….12:14:06 PM 0.00 8.00 47294 0 1596 1162212:14:08 PM 0.00 8.00 47296 0 1596 11622Average: 0.00 12.50 47277 0 1596 11618

sar -B -f sardata.out

12:34:44 pgpgin/s pgpgout/s activepg inadtypg inaclnpg inatarpg12:34:46 376.00 1.50 0 0 0 012:34:48 441.50 1.50 0 0 0 012:34:50 568.00 71.50 0 0 0 012:34:52 437.00 1.50 0 0 0 012:34:54 450.50 22.50 0 0 0 0Average: 454.60 19.70 0 0 0 0

14

27

sarsar --rrsar -r -f sardata.out

12:13:28 PM kbmem kbmem %mem kbmem kb kb kbswp kbswp %swpfree used used shrd buffers cached free used used

12:13:30 PM 10268 374300 97.33 0 90408 129280 1018268 1848 0.1812:13:32 PM 10268 374300 97.33 0 90416 129280 1018268 1848 0.1812:13:34 PM 10268 374300 97.33 0 90424 129280 1018268 1848 0.1812:13:36 PM 10268 374300 97.33 0 90432 129284 1018268 1848 0.1812:13:38 PM 10268 374300 97.33 0 90440 129284 1018268 1848 0.18………..12:14:06 PM 10268 374300 97.33 0 90552 129296 1018268 1848 0.1812:14:08 PM 10268 374300 97.33 0 90560 129296 1018268 1848 0.18Average: 10268 374300 97.33 0 90484 129288 1018268 1848 0.18

28

Commands for I/OCommands for I/O

iostatsar/proc/sys/fs/*/proc/partitionshdparm/proc/statdf

15

29

dfdfDf or df -k

Filesystem 1k-blocks Used Available Use% Mounted on/dev/sda1 127880 52265 69011 43% //dev/sda8 12762679 9957815 2141620 82% /home/dev/sda5 2971111 2097239 720222 74% /usr/dev/sda6 762122 277769 444981 38% /var

df -hFilesystem Size Used Available Use% Mounted on/dev/sda5 539M 155M 383M 29% //dev/sdb6 21G 14G 7.5G 65% /home/dev/sda6 2.9G 1007M 1.9G 34% /usr/dev/sda9 1.9G 358M 1.6G 18% /usr/local/dev/sda8 992M 96M 896M 10% /var/dev/sdb7 19G 231M 19G 1% /web

30

dfdf ––i i InodesInodes[root@biteme root]# df -iFilesystem Inodes IUsed IFree IUse% Mounted on/dev/hdb2 640000 28460 611540 5% //dev/hdb1 115712 64 115648 1% /boot/dev/hdb7 640000 769 639231 1% /home/dev/hdb11 127744 11 127733 1% /optnone 48140 1 48139 1% /dev/shm/dev/hdb10 256512 150 256362 1% /tmp/dev/hdb8 640000 176782 463218 28% /usr/dev/hdb5 1281696 43068 1238628 4% /usr/local/dev/hdb6 1281696 5634 1276062 1% /usr/local/software/dev/hdb9 256512 5262 251250 3% /var

16

31

iostatiostatiostat 2 10

Linux 2.4.20-30.9 (biteme.wideopenwest.com) 03/30/2004

avg-cpu: %user %nice %sys %idle3.14 0.31 0.16 96.39

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtndev3-0 0.00 0.00 0.00 256 0dev3-1 0.82 4.90 9.69 447714 886136

avg-cpu: %user %nice %sys %idle100.00 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtndev3-0 0.00 0.00 0.00 0 0dev3-1 0.00 0.00 0.00 0 0

32

sarsar I/O ratesI/O rates

sar -b -f sardata.outLinux 2.4.20-30.9 (biteme.xxx.com) 03/30/2004

12:13:28 PM tps rtps wtps bread/s bwrtn/s12:13:30 PM 3.50 0.00 3.50 0.00 80.0012:13:32 PM 1.50 0.00 1.50 0.00 16.0012:13:34 PM 1.50 0.00 1.50 0.00 16.0012:13:36 PM 1.50 0.00 1.50 0.00 32.0012:13:38 PM 1.50 0.00 1.50 0.00 16.0012:13:40 PM 1.50 0.00 1.50 0.00 16.00………12:14:08 PM 1.50 0.00 1.50 0.00 16.00Average: 1.88 0.00 1.88 0.00 25.00

17

33

/proc/sys//proc/sys/fsfs

Max values can be set in /proc/sys/fs/*-maxcat file-max– 39321

cat file-nr– 425 114 39321

cat inode-nr– 172688 72170

hdparm– Use man hdparm– Sets filesystem read ahead count, drive write cache, drive lookahead

prefetch count, drive lookahead ….

34

/proc/partitions/proc/partitions

18

35

lsoflsof

lsof is a 3rd party tool that you can use to look at open files and network connectionsftp://vic.cc.purdue.edu/pub/tools/unix/lsof/READMEcd /usr/localtar -xvf soft/tarfiles/lsof_4. 71.tarcd lsof_4. 71./Configure linux

ensure you set HASSECURITY to yesmakecp lsof /usr/local/bincp lsof.8 /usr/local/man/man8Try lsof | grep TCP or lsof | grep UDP or lsof | more

36

Intra and Inter PoliciesIntra and Inter Policies

Intra Policy– How data is laid out on the disk– Outer edge, outer middle, center, inner

middle, inner edge

Inter Policy– How data is laid out between/across disks

19

37

RAID LevelsRAID Levels

Raid-0– Disks combined into single volume stripeset– Data striped across the disks

Raid-1– Every disk mirrored to another– Full redundancy of data but needs extra disks– At least 2 I/Os per random write

Raid-0+1– Striped mirroring– Combines redundancy and performance

38

RAID LevelsRAID Levels

RAID-5– Data striped across a set of disks– 1 more disk used for parity bits– Parity may be striped across the disks also– At least 4 I/Os per random write

(read/write to data and read/write to parity)

– Uses hot spare technology

20

39

Other I/O performance notesOther I/O performance notes

Mirroring of disksWrite verifySCSI (1 disk at a time) vs scsi-2 fw diffRAID and SSAI/O Pacing and Async I/OStriping

40

NetworkingNetworking

netstatnfsstatlsof | grep TCPlsof | grep UDP

21

41

/proc/sys/net/core/proc/sys/net/core

Critical on high speed networks such as z/390 hypersockets (24gb/sec) or gigabit ethernetImproves TCP throughputTCP window sizes (2.1.90 and >)– rmem_max & wmem_max– Default is 64kb – can increase it

Default Buffer Settings– Max socket buffer size 64kb– Default TCP socket buffer size 32kb– Default UDP socket buffer size 32kb?

42

/proc/sys/net/core on Fedora/proc/sys/net/core on Fedora

/proc/sys/net/core

[jaqui@biteme core]$ cat optmem_max10240[jaqui@biteme core]$ cat rmem_default65535[jaqui@biteme core]$ cat rmem_max65535[jaqui@biteme core]$ cat wmem_default65535[jaqui@biteme core]$ cat wmem_max65535

22

43

/proc/sys/net/ipv4 on Fedora/proc/sys/net/ipv4 on Fedora

/proc/sys/net/ipv4

[jaqui@biteme ipv4]$ cat ip_forward0[jaqui@biteme ipv4]$ cat tcp_rmem4096 87380 174760[jaqui@biteme ipv4]$ cat tcp_mem97280 97792 98304[jaqui@biteme ipv4]$ cat tcp_wmem4096 16384 131072[jaqui@biteme ipv4]$ cat tcp_window_scaling1

44

netstatnetstat --ii

Kernel Interface tableIface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1500 0 94408545 0 0 0 99872636 0 0 0 BRUlo 3924 0 287611603 0 0 0 287611603 0 0 0 LRU

netstat –i (Fedora)

Kernel Interface tableIface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1500 0 12023 0 0 0 10340 0 0 0 BMRUlo 16436 0 70 0 0 0 70 0 0 0 LRU

23

45

nfsstatnfsstat

Client and Server NFS Infonfsstat -cn, -r, -s– retransmissions due to errors– badcalls– timeouts– waits– reads

46

Useful ReferencesUseful References

http://www.tunelinux.comhttp://linuxperf.nl.linux.orghttp://www.psc.edu/networking/perf_tune.html– Enabling high performance data transfers on hosts

http://linuxperf.nl.linux.org/links.htmlhttp://www-124.ibm.com/developerworks/opensource/linuxperf/http://www.ibm.com/developerworks/eserver/articles/analyze_aix/

24

47

SummarySummary

Linux has useful free performance toolsDownload iptraf and sysstat toolsets

jaqui@circle4.comhttp://www.circle4.com/jaqui/http://www.mainline.com