Top 10 Performance Hints

Adam BackmanAdam Backman

White Star SoftwareWhite Star Software

adam@wss.comadam@wss.com

What is performance tuning?

Performance tuning is a process where Performance tuning is a process where application response times are recorded and application response times are recorded and modifications to the environment are made modifications to the environment are made is a systematic manner in an effort to reduce is a systematic manner in an effort to reduce the amount of time it takes to complete the amount of time it takes to complete system tasks.system tasks.

What is performance tuning?What is performance tuning?

A process of moving the bottleneck to the A process of moving the bottleneck to the fastest resource.fastest resource.

Hardware Resources(Slowest to Fastest)Hardware Resources(Slowest to Fastest)

NetworkNetwork DiskDisk Memory Memory CPUCPU

When to Tune Performance?

Installation or upgrade of hardwareInstallation or upgrade of hardware Installation or upgrade of softwareInstallation or upgrade of software Change in workloadChange in workload

Number of usersNumber of users Volume of dataVolume of data

How Fast is Fast?

Remember: Performance is relative so it is Remember: Performance is relative so it is important to have a baseline. important to have a baseline.

Application Performance Application Performance

The single largest component of performance The single largest component of performance tuning is application performance. tuning is application performance.

A well written application will run well given A well written application will run well given the proper resources while a poorly written the proper resources while a poorly written application will rarely run well despite application will rarely run well despite resource allocation.resource allocation.

A few application hintsA few application hints

Use indexes - not USE-INDEXUse indexes - not USE-INDEX Use NO-UNDOUse NO-UNDO Use TEMP-TABLES not WORKFILESUse TEMP-TABLES not WORKFILES Use NO-LOCKUse NO-LOCK Use FIELD-LISTUse FIELD-LIST Go n-tier - Progress application serverGo n-tier - Progress application server

But enough about applications let’s talk about database and system tuning

Network contentionNetwork contention

The network is the slowest resource for The network is the slowest resource for client/server applications so you want to client/server applications so you want to eliminate contention for this resource before eliminate contention for this resource before moving on to the other resourcesmoving on to the other resources

Networking tipsNetworking tips

Keep things localKeep things local No temp files on network drivesNo temp files on network drives Move the application “close” to the userMove the application “close” to the user

Use -cache to speed initial connectionUse -cache to speed initial connection Use -pls if you are using program libraries Use -pls if you are using program libraries

over the networkover the network Application issues are magnified over a Application issues are magnified over a

network (field-lists, no-lock, indexes, …) network (field-lists, no-lock, indexes, …)

Odd Network IssuesOdd Network Issues

BandwidthBandwidth Speed (10/100/Auto)Speed (10/100/Auto) Duplex (Always full)Duplex (Always full) Sharing TCP/IP and AppleTalkSharing TCP/IP and AppleTalk Latency (WAN Issue)Latency (WAN Issue) Mixing hardware (Cisco and Fore)Mixing hardware (Cisco and Fore)

Disk contentionDisk contention

In most environments disks are the largest In most environments disks are the largest area for improvement. All of the data flows area for improvement. All of the data flows from the disks to the other resources so this from the disks to the other resources so this effects both local and networked users.effects both local and networked users.

Disk tipsDisk tips

No RAID-5No RAID-5 Use storage areasUse storage areas Use 8k block sizeUse 8k block size Use the correct BI cluster sizeUse the correct BI cluster size Use page writersUse page writers Use private buffers (-Bp)Use private buffers (-Bp) Use -T to eliminate varianceUse -T to eliminate variance

Disk Tips (continued)Disk Tips (continued)

Buy many small disksBuy many small disksTwo heads are better than oneTwo heads are better than one

Buy fast disksBuy fast disksBuy at least 10,000 RMPBuy at least 10,000 RMP

Buy fast controllersBuy fast controllersFibre channel is better than SCSIFibre channel is better than SCSI

No RAID 5No RAID 5

Poor man’s mirroring - This is the kiss of Poor man’s mirroring - This is the kiss of death for OLTP performancedeath for OLTP performance

User information is stripedUser information is striped Parity information is striped WITH user Parity information is striped WITH user

informationinformation OK for 100% read only applicationsOK for 100% read only applications Poor performance for writesPoor performance for writes

If no RAID 5 what do I do?If no RAID 5 what do I do?

RAID 0+1 for the databaseRAID 0+1 for the database RAID 1 for the AI and BI logsRAID 1 for the AI and BI logs Hardware RAID is almost always better Hardware RAID is almost always better

than software RAIDthan software RAID

Storage Areas

• Progress-specific way to distribute I/OProgress-specific way to distribute I/O• Only way to eliminate I/O indirection in a Only way to eliminate I/O indirection in a

Progress environmentProgress environment• Only way to pre-allocate database blocksOnly way to pre-allocate database blocks• Every database has at least 1 storage area in Every database has at least 1 storage area in

Progress version 9Progress version 9

Storage AreasStorage Areas

Pros:Pros: Reduce utility timeReduce utility time Increase sequential access performanceIncrease sequential access performance

Cons:Cons: Adds complexity Adds complexity More potential room for failureMore potential room for failure

8k block size8k block size

Most systems will benefit from using 8k Most systems will benefit from using 8k block size (NT should use 4k)block size (NT should use 4k)

You will retrieve more information per You will retrieve more information per physical I/O especially on index readsphysical I/O especially on index reads

I/O is done how the operating likes it to be I/O is done how the operating likes it to be done.done.

Remember to modify records per block if Remember to modify records per block if necessarynecessary

BI Cluster sizeBI Cluster size

Somewhere between 1MB and 4MB works Somewhere between 1MB and 4MB works for most peoplefor most people

If you are checkpointing every 2 minutes or If you are checkpointing every 2 minutes or more often during peak periods increase the more often during peak periods increase the cluster sizecluster size

If you a “workgroup” version of Progress If you a “workgroup” version of Progress leave your cluster size alone (512kb)leave your cluster size alone (512kb)

Progress page writers

Every database that does updates should Every database that does updates should have a before image writer (BIW)have a before image writer (BIW)

Every database that does updates should Every database that does updates should have at least 1 asynchronous page writer have at least 1 asynchronous page writer (APW)(APW)

Every database that is using after imaging Every database that is using after imaging should have a after image writer (AIW) should have a after image writer (AIW)

Tuning APWsTuning APWs

Start with 1 APWStart with 1 APW Monitor buffers flushed at checkpoint on Monitor buffers flushed at checkpoint on

the activity screen (option 5) in promonthe activity screen (option 5) in promon If buffers flushed increases during the If buffers flushed increases during the

“important” hours of the day add 1 APW“important” hours of the day add 1 APW

Private Buffers (-Bp)Private Buffers (-Bp)

Not really privateNot really private Use different LRU chain Use different LRU chain Good for read only clients (reports)Good for read only clients (reports) _MyConnection._MyConn-_MyConnection._MyConn-

NumSeqBuffers to allocate or NumSeqBuffers to allocate or deallocate from the application deallocate from the application levellevel

Use -T to level disk I/OUse -T to level disk I/O

Local (host based) users and batch jobs Local (host based) users and batch jobs should use the -T parameter to place their should use the -T parameter to place their temporary file (.srt, .pge, .lbi, …) I/O on a temporary file (.srt, .pge, .lbi, …) I/O on a drive that is not working as hard as the drive that is not working as hard as the other drives on the systemother drives on the system

Note: -T should never point to a network Note: -T should never point to a network drivedrive

Memory contentionMemory contention

Memory should be used to reduce disk I/O. Memory should be used to reduce disk I/O. Broker (server) side parameters should be Broker (server) side parameters should be tuned first and then user parameters can be tuned first and then user parameters can be modified. In a memory lean situation, modified. In a memory lean situation, memory should be taken away from memory should be taken away from individual users before reducing broker individual users before reducing broker parameters.parameters.

Memory hintsMemory hints

Swapping and/or excessive paging is bad, Swapping and/or excessive paging is bad, buy more memory or reduce parameters to buy more memory or reduce parameters to avoid itavoid it

Increase -B in 10% increments until the Increase -B in 10% increments until the point of diminishing returns or swapping, point of diminishing returns or swapping, whichever comes firstwhichever comes first

Use V9 private buffers (-Bp) for reportingUse V9 private buffers (-Bp) for reporting Do not use private buffers (-I) prior to V9Do not use private buffers (-I) prior to V9

Memory hints (continued)Memory hints (continued)

Use memory for the users closest to the customer Use memory for the users closest to the customer first (developers increase last)first (developers increase last)

Use -Bt for large temp tablesUse -Bt for large temp tables Set -bibufs between 50 and 120. Look at the Set -bibufs between 50 and 120. Look at the

activity screen in promon (BI buffer waits) to see activity screen in promon (BI buffer waits) to see if additional tuning is necessary.if additional tuning is necessary.

Use –pinshm to keep broker memory from being Use –pinshm to keep broker memory from being paged out.paged out.

Use -directio to bypass the OS buffer poolUse -directio to bypass the OS buffer pool

CPU contentionCPU contention

High CPU activity is not bad in and of itself High CPU activity is not bad in and of itself but high system CPU activity is bad and but high system CPU activity is bad and should be corrected.should be corrected.

Components of CPU activityComponents of CPU activity

USER - This is what you paid forUSER - This is what you paid for SYSTEM - This is overheadSYSTEM - This is overhead WAIT - This is wasteWAIT - This is waste IDLE - This is nothing ;-)IDLE - This is nothing ;-)

CPU activity goalsCPU activity goals

The goal is to have as much USER time as The goal is to have as much USER time as possible with as little SYSTEM and WAIT.possible with as little SYSTEM and WAIT.

A practical split is USER: A practical split is USER: 70%70%

SYSTEM:SYSTEM: 20%20%

WAIT:WAIT: 0% 0%

IDLE: IDLE: 10%10%

Eliminating high SYSTEM CPU activityEliminating high SYSTEM CPU activity

Always use -spinAlways use -spin Use a setting of 1 for single CPU systemsUse a setting of 1 for single CPU systems Use a higher setting for multiple CPU systemsUse a higher setting for multiple CPU systems

Testing has shown that the optimal setting for -Testing has shown that the optimal setting for -spin is somewhere between 2000 and 10000. spin is somewhere between 2000 and 10000. First try 2000First try 2000

-napmax should default to 5000 but in some late -napmax should default to 5000 but in some late 7 and early 8 versions of Progress it is set to 100 7 and early 8 versions of Progress it is set to 100 which is way too lowwhich is way too low

Eliminating high WAIT CPU activityEliminating high WAIT CPU activity

WAIT = Waiting on I/OWAIT = Waiting on I/O If you still have IDLE time it generally is If you still have IDLE time it generally is

not a big problemnot a big problem Look at paging/swapping firstLook at paging/swapping first Next look at your disk I/ONext look at your disk I/O

Points to rememberPoints to remember

Document where you are now Document where you are now before before changing anythingchanging anything

Change 1 thing at a time so you know what Change 1 thing at a time so you know what made a differencemade a difference

Move from the slowest to the fastest resourceMove from the slowest to the fastest resource Buying faster disks generally does more Buying faster disks generally does more

good than buying faster CPUsgood than buying faster CPUs

QuestionsQuestions