copyright © 2003, sas institute inc. all rights reserved. where's waldo uncovering...

Copyright © 2003, SAS Institute Inc. All rights reserved.

Where's Waldo Uncovering Hard-to-Find Application Killers Claire CatesSAS Institute, [email protected]


Always something different

Just like the Waldo pictures, performance is always different and not easy to find.• Performance based on the test that was run

• Performance based on the data that was used

• Performance based on the hardware used

• Performance based on the number of users


What is a Performance Problem When jobs cannot execute within a required time period.

Something that makes the system run slower than it should

When a system cannot sustain expected or unexpected growth (scalability) • More users

• More data

• More processors


Causes of Performance Problems

Hardware

Software

Integration of software systems

Network

Web


How Do You Find Performance Problems

• Vendor profilers

• Looking at system variables (cache hits, IO timings,…)

• Profilers that analyze the code

• Code reviews

• Network sniffers

• Profiles that profile the entire machine

• Code instrumentation


Common Software Performance Problems Antipatterns

Excessive Memory Allocation

The Ramp

Unnecessary Processing

One Lane Bridge

How Many Times Do I have to tell you.

Unbalanced Processing

More is Less


Excessive Memory Allocation Memory Allocations/Frees are expensive

• OS calls

• If your system is multi-threaded, locking may need to occur

Often times memory is allocated temporarily within a function

Often times structures are allocated and freed and not reused

Memory allocations in loops.

Possible Fixes• Use a stack variable instead of temporary memory allocations

• Maintain a free chain for structures that are used often

• Make one large allocation outside loop – split it within the loop


The Ramp When processing slows down over time

Can be caused by long lists that grow and need to be searched

Can be caused by too many resources being used

Another kind of Ramp happens when the system doesn’t scale well.

Possible fixes• Use a different algorithm when you know large amounts of data

will be required

• Make sure all resources (especially memory) are returned.


Unnecessary Processing Loops that contain statements that do not need to

execute on each iteration of the loop

Macros that are inefficient because they are too general

Zeroing of allocated memory that does not need to be zeroed.

Inefficient algorithms

Possible fixes• Rework algorithm

• Don’t use macros


One Lane Bridge

Resource contention – every thread waiting on the same resource

Eventually causes a traffic jam

Possible fixes• Change the algorithm to use less resources that are shared


How Many Times Do I Have To Tell You Calling a routine too often with the same parameters

Sometimes a routine (foo) is the hot spot yet, (foo) is fine, it is just that foo is called to often.• Most common occurrence of this that I see occurs when

(afoo)

(bfoo)

(cfoo)

(foo)

So cfoo is called twice and foo is called 4 times all because afoo does not know what bfoo and cfoo actually call


Unbalanced Processing Occurs in a multi-threaded environment running on a

multi-processor machine.

When one or two threads do all the processing while other threads are idle

Possible fixes• Rebalance the work in the threads so that all threads process a

similar amount of work


More is Less or Thundering Herd Adding threads does not make a system run faster

The system spends its time thrashing

Thrashing can occur because of • Excessive thread switching

• Destroying the system memory or I/O caches when each thread runs

Possible fixes• Don’t create more threads than there is work to be done


Common Profilers in Use at SAS InTune – z/OS

VTune – Intel – PCs

Caliper – HP

Rational Quantify – Windows, Solaris, HP, Linux

Jprobe

OS tools – not really profilers but gives information on− System calls

− IO bottlenecks

− Caching problems


SAS’s use of Quantify Other Profilers we use

• Event / timer based

• Based more on the function level, not the source level

• Network traffic & other processes on the machine distorted the results

Great results from all profilers

Tips I’m using with Quantify do apply to these other tools


Quantify differences… Quantify

• Counts instruction cycles for the user code

• Times the system calls (can be turned off)

• Does not have to run on a clean machine (a machine with no other activity) – developer machine

• Provides performance information down to the source line level

• Performance can be broken down by threads


A Little about Quantify

Quantify integrates itself with your system. As your system executes, Quantify will accumulate a mass of data.• Function counts for every function used

• The amount of time spent in every function

• The amount of time spent in every function plus the amount of time spent in all its descendent routines.

• Full Call tree information


Quantify Quick Overview


Excessive Memory Allocation


How Many Times Do I Have to Tell You?


More Is Less


Unnecessary Processing


One Lane Bridge


Unbalanced Processing


Ramp


Another Useful Hint


Summary

Profilers are useful in helping to uncover performance problems in a software application

By being patient and looking at the mass of data provided by the tools you can find out a tremendous amount about what your system is doing and how to optimize the performance.


References

AntiPatterns – Refactoring Software, Architectures, and Projects in Crisis - Brown, Malveau, McCormick & Mowbray

• ISBN 0-471-19713-0

New Software Performance AntiPatterns: More Ways to Shoot Yourself in the Foot. • Smith & Williams CMG 2002 Paper Presentation

Computer Measurement Group • www.cmg.org

copyright © 2003, sas institute inc. all rights reserved. where's waldo uncovering...

Documents

loop slide

shared slide

times memory

temporary memory allocations

different algorithm

ramp unnecessary processing

unnecessary processing

large allocation