bbworld 2009 performance forensics workshop

Source: http://www.youtube.com/watch?v=-LkusicUL2s

2

Did you all get a chance to read that? As a public company we need to have Vista is the only solution built from the ground up on true enterprise

3

our disclosure statement before all presentations. If you have any questions

on what it means please speak with our General Counsel.

Vista is the only solution built from the ground up on true enterprise

technology --- allowing you to ensure that you continue to provide your

faculty and students an outstanding experience

Source: http://www.flickr.com/photos/turkguy19/1018419391/

9

Our session isn’t about crime scene investigation. I’m not going to attempt to

10

talk about digital forensics from a security and fraud perspective. I’m here to

put context around performance and scalability incidents that happen in your

application environments every day, week, month and year. There’s no

reason to assume that performance problems can only be solved by “black

magic”. A rational, evidence-supported process can be used to solve a

performance problem.

The first definition about forensics assumes that “something” occurs in the

form of a crime. For all intensive purposes a crime is considered pre-

meditated. Often crimes such as denial of service, script injection and spam

attacks can cause performance problems. Performance incidents can

happen when they are not necessarily premeditated. They happen because

of failure. Things break…it’s a fact of life.

When breakage or breakdowns occur they are not visibly apparent or

obvious to every set of eyes looking on. This can be problematic when

diagnosis bias is introduced or value attribution is incorrectly established.

http://www.flickr.com/photos/wwarby/3297205226/

11

Search Google and you will be amazed how little meaningful information you get by the words “performance forensics” in the

13

information you get by the words “performance forensics” in the context of computers and software. One paper by Bob Sneed from Sun Microsystems (http://www.sun.com/blueprints/1203/817-4444.pdf) is out there, but very little else.

So you will have to trust me in my primitive definition of performance forensics. You might even offer to help make it better.

Performance forensics is like any other forensics process. It begins with collective evidence. If you are lucky and have a lot of tools in place you will have a starting point of data to sift through. More often then not, the data is not there. You are not always lucky to have the data when you need it and/or it might not be in the best format for getting to the root cause of a problem.

Evidence as we will discuss later can be collected after the fact. Techniques such as discrete simulation can be used to re-enact an incident. When that does happen, you have the ability to capture all of the data you want. You simply need to know what data to collect. It’s a circuitous loop of sorts…mainly because you might not know what data to collect to begin with.

It’s like when I look under the hood of my car. I have no idea what I’m looking at…Maybe it’s that smoking gun I’m in search of. Yeah, I guess if I see some kind of corrosive, smoke or leak it might be painfully obvious…but it never is. Not with today’s cars…Computers and software specifically are the same. Rarely is there that smoking gun sitting in front of your face waiting to be found. Thus evidence is

Source: http://www.flickr.com/photos/t_squared/152270386/

14

http://www.flickr.com/photos/7563125@N08/2830710184/

16

Source: http://farm4.static.flickr.com/3396/3507282396_3756634f01_m.jpg

23

Source: http://www.flickr.com/photos/psilver/412264230/

24

http://crackerjackonlinemarketing.com/blog/wp-

content/uploads/2008/07/working-woman-with-octopus-hands.jpg

25

Source: http://farm2.static.flickr.com/1330/3174009125_ec49351a6d_m.jpg

26

Problems are not always easily identifiable. When I say that I feel off or sick, I leave the listener desiring more information. They might infer that I have a stomach pain, a cold or a

28

listener desiring more information. They might infer that I have a stomach pain, a cold or a headache. It could be that I am tired or I have a broken arm. A more related example that I often hear is that my system is slow. What defines slow? Can you show me? Can I experience the slowness?

Is it always slow every single day and every minute? Are all of the components that make up the physical architecture necessarily slow? Are particular use cases experiencing latency? Do they always experience latency or is it at specific times? Is it specific users who experience latency? Are the users different is some kind of fashion? Does the problem happen after a particular interaction pattern? Does it happen with a particular piece of data?

When a problem is easily identifiable, define a clear, intelligible problem statement. The problem statement is used to aid the investigation so the forensics process can focus on collecting meaningful data to get to root cause analysis.

Narrowing down to a problem statement from the unknown can be an exhaustive effort. Start with questioning (not formal interviewing) in which your goal is to exclusively narrow down the chasm of possibilities. Start with the “Lassie Question: Can you show me?” Experiencing the problem first hand provides basic context. If the problem can’t be reproduced, try to provide supporting clues so that the unpredictable can become more predictable. You can’t necessarily replicate the performance problem at will. Do you have supporting data about your experience? Can you explain what happened to you? Do you know when it happened (smallest time window)? Has it happened before? If so when? Try to get down to the exact minute if possible. Has it happened to anyone else? What were they doing? Did it happen to them at the same time as you?

It comes off like you are asking dozens and dozens of questions, but in reality you are not. You are gathering basic context: Who, What, Where and When.

Be unwilling to announce a problem statement until you have confidence in the development of the problem statement (not the cause of the issue). Remember we are not diagnosing, we are simply collecting and announcing symptoms.

I’m not the creator of this methodology. I’m quite sure that others who are far more knowledgeable on the subject would tell you I’m possibly missing a step or that I am drawing out the process too far. A picture is truly worth a thousand words.

30

would tell you I’m possibly missing a step or that I am drawing out the process too far. A picture is truly worth a thousand words.

I will breakdown each element of the methodology in subsequent slides. I’ve designed a circular visualization for the obvious conclusion that I’ve come to over the years in which the process must revolve in order to come to root cause analysis.

Performance forensics doesn’t necessarily begin with evidence collection. Rather, it potentially begins long before an incident occurs. Let’s take an abstract example such as a person complains about chest pain. The person tells their spouse that at times they have unbearable pains, but eventually it goes away. It doesn’t happen enough and the pain isn’t so severe that it’s worth the time or the effort to go to the doctor. The process of convincing yourself that the symptoms you are experiencing is not what you really have is called diagnosis bias. I will talk about this in greater detail later.

This pain might go on and on for quite some time until it progresses. Analysis could be initiated at any point. More often then not, the complaints go unrealized and forensics is placed on hold. It comes back later on. The question is when. Typically when a terrible even occurs. It could be a heart attack or sadly a loss of life. The forensic engineer is tasked with tracing back why it happened, was foul play suspected and could it have been avoided.

I propose that at any time the methodology can be initiated. No major issue has to occur for performance forensics to begin. Symptoms do not necessarily have to show-up for the process to begin. You can call this what you want, but basically the collection of evidence, interviewing, modeling/visualizing and planning for the future is most commonly referred to as capacity planning. It’s not the much different from what we are trying to accomplish with performance forensics. The key difference is proactive behavior versus reactive behavior.

The methodology begins with the collection of data. We can call this data evidence. Evidence is collected in two ways: intended data collection and simulated data collection. When data is not available, we often go through the process of putting data collectors in place. The thought behind this is that if something happened once, it’s bound to happen again.

Interviewing is incorporated into the methodology. I will discuss techniques for interviewing. Understand that when humans are involved and asked to participate, you run the greatest chance for diagnosis bias and value attribution (two topics I will present in greater detail).

Next I will discuss why modeling and visualizing a problem can be critical at getting to the root cause of a performance issue.

The best analogy to depict the execution model for wait events is the grocery

32

store checkout line. Assume the cashier is the CPU. The customer who is

currently being checked out by the cashier is the running session. The

customers who are waiting in line represent the runnable queue.

If customer1 who is being checked out requires a price check on a product,

customer1 must wait until the price check is completed. Meanwhile, the next

in line, customer2, is immediately checked out by the cashier until the price

check is completed for customer1.

When the price check is completed, the cashier can resume the check out of

customer1. This is the simplest illustration of the wait event execution

model.

The best analogy to depict the execution model for wait events is the grocery

33

store checkout line. Assume the cashier is the CPU. The customer who is

currently being checked out by the cashier is the running session. The

customers who are waiting in line represent the runnable queue.

If customer1 who is being checked out requires a price check on a product,

customer1 must wait until the price check is completed. Meanwhile, the next

in line, customer2, is immediately checked out by the cashier until the price

check is completed for customer1.

When the price check is completed, the cashier can resume the check out of

customer1. This is the simplest illustration of the wait event execution

model.

JConsole on Steroids

39

Great presentation: http://www.javapassion.com/javase/VisualVM.pdf

Another Great Presentation: http://weblogs.java.net/blog/mandychung/archive/VisualVM-BOF-2007.pdf

http://www.stevesouders.com/blog/2009/06/30/firefox-35-at-the-top/

44

http://assets.en.oreilly.com/1/event/29/The%20User%20and%20Business%20Impact%20of%20Server%20Delays,%20Additional%20Bytes,%20and%20HTTP%20Chunking%20in%20Web%20Search%20Presentation.pptx

Source: http://www.flickr.com/photos/kaptainkobold/83359336/

45

http://www.phpied.com/image-optimization-7-mistakes/

46

Using pngrewrite to optimize this image

(http://entropymine.com/jason/pngrewrite/)

Cost Savings: 3KB or roughly 15%

47

Using optipng to optimize this image

(http://sourceforge.net/projects/optipng/)

Cost Savings: 4KB or roughly 9%

48

Using optipng to optimize this image

(http://sourceforge.net/projects/optipng/)

Cost Savings: 4MB or roughly 47.50%

49

http://code.google.com/speed/page-speed/

50

http://developer.yahoo.com/yslow/

http://www.fiddler2.com/Fiddler2/version.asp

Great add-on to Fiddler2 is neXpert

http://videos.visitmix.com/MIX09/T53F

Great presentation highlighting differences between tools: http://assets.en.oreilly.com/1/event/29/Website%20Performance%20Analysi

51

http://assets.en.oreilly.com/1/event/29/Website%20Performance%20Analysis%20Presentation.ppt

Source: http://www.flickr.com/photos/pollyann/2877940383/

53

http://www.fiddler2.com/Fiddler2/version.asp

55

Demo videos: http://www.fiddler2.com/Fiddler/help/video/default.asp

Example of use as a reverse proxy: http://blogs.msdn.com/nexpert/archive/2009/06/04/capturing-http-with-fiddler-as-a-reverse-proxy.aspx

WebCast of neXpert:http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?E

56

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032398774&EventCategory=5&culture=en-US&CountryCode=US

Can also consider using Microsoft VRTA: http://www.microsoft.com/downloads/details.aspx?FamilyID=119f3477-dced-

58

http://www.microsoft.com/downloads/details.aspx?FamilyID=119f3477-dced-41e3-a0e7-d8b5cae893a3&displaylang=en

Source: http://www.flickr.com/photos/turkguy19/1018419787/

60

Easily configured Virtual Machine by cloning VMs using XenCenter

61

bbworld 2009 performance forensics workshop

Education