intro to linux performance analysis
DESCRIPTION
LOPSA SD 2014.03.27 Presentation on Linux Performance Analysis An introduction using the USE method and showing how several tools fit into those resource evaluations.TRANSCRIPT
![Page 1: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/1.jpg)
Intro to Linux Performance
AnalysisChris McEniry
LOPSA-SD March 27, 2014
![Page 2: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/2.jpg)
Me
• Systems Architect
• Sony Network Entertainment
• 18 years running stuff
• Majority of the last 14 years: medium-large Internet services
![Page 3: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/3.jpg)
Read this book…
And look here:
http://www.brendangregg.com/
http://www.brendangregg.com/methodology.html
http://www.brendangregg.com/Slides/LISA2012_methodologies.pdf
http://www.amazon.com/Systems-Performance-Enterprise-Brendan-Gregg/dp/0133390098
![Page 4: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/4.jpg)
The website is down!!! It’s just too slow! The DB is too slow! The disk is too slow!
SLOW!!!
http://farm4.staticflickr.com/3190/2976755407_6a6a574596_o.jpg
![Page 5: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/5.jpg)
SLOW!!!!
• What does slow mean anyways?
• Is it not transferring fast enough?
• Is it handling (not) too many requests?
http://commons.wikimedia.org/wiki/File:United_States_sign_-_Slow_Traffic_Ahead.svg
![Page 6: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/6.jpg)
Slow can mean…
• Latency: How long it takes
• ms, s, request time, etc
• Throughput: How much can happen at the same time
• bandwidth, IOPS, rps, tps, etc
http://upload.wikimedia.org/wikipedia/commons/2/2e/Miniature_DNF_Dictionary_055_ubt.JPG
![Page 7: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/7.jpg)
Slowness comes from…
• Full utilization of a resource
• Waiting in a saturated queue
• Generated errors!
!
• The USE Method
http://farm6.staticflickr.com/5181/5614813544_a30d693a50_o.jpg
![Page 8: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/8.jpg)
Utilization
• You have fully used up what’s been allocated
• aka 5 lb bag
http://farm3.staticflickr.com/2524/4000641774_3331fe06fb_o.jpg
![Page 9: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/9.jpg)
Saturation
• Waiting for someone else to get done so you can do yours
• Typically because a resource is fully utilized, but not necessarily directly
http://www.fotocommunity.com/pc/pc/display/30396619
![Page 10: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/10.jpg)
Errors
• Dropped packets
• Incorrect responses
• Deadlocks
• Timeouts
!
• Not all failures fail fast
http://farm8.staticflickr.com/7001/6509400855_aaaf915871_b.jpg
![Page 11: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/11.jpg)
How do we determine?
• Different types of tools for different examinations
• Depends on what you’re looking for (which can be a problem in and of itself)
http://farm5.staticflickr.com/4083/5086955738_61f6455ace_b.jpg
![Page 12: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/12.jpg)
Resource vs Transaction• Do you care if…
• a CPU is maxed out?
• processes are blocked?
• packets are lost?
• or if…
• a user’s request fails?
• a user gives up on waiting for a response?
![Page 13: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/13.jpg)
Maturity
• Tracing tools, especially using in production, requires a level of maturity
• I’m not that mature… ;)
• No, really just focusing on the basics first
http://upload.wikimedia.org/wikipedia/commons/b/bd/OFLC_large_R18%2B.svg
![Page 14: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/14.jpg)
http://image.slidesharecdn.com/scalelinuxperformance-130224171331-phpapp01/95/slide-15-638.jpg?cb=1362166290
![Page 15: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/15.jpg)
http://image.slidesharecdn.com/scalelinuxperformance-130224171331-phpapp01/95/slide-16-638.jpg?cb=1362166290
![Page 16: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/16.jpg)
General
![Page 17: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/17.jpg)
?
/var/log/messages
![Page 18: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/18.jpg)
Errors !(mostly - sometimes stats go here)
/var/log/messages
![Page 19: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/19.jpg)
CPU
![Page 20: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/20.jpg)
?
uptime
![Page 21: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/21.jpg)
Saturation of the scheduler
uptime
![Page 22: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/22.jpg)
?
top
![Page 23: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/23.jpg)
topSaturation
Utilization
![Page 24: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/24.jpg)
Memory
![Page 25: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/25.jpg)
?
free
![Page 26: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/26.jpg)
Utilization
free
![Page 27: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/27.jpg)
?
vmstat
![Page 28: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/28.jpg)
vmstat
SaturationUtilization
Counts
![Page 29: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/29.jpg)
?
slabtop
![Page 30: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/30.jpg)
Utilization
slabtop
![Page 31: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/31.jpg)
Disk
![Page 32: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/32.jpg)
?
df
![Page 33: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/33.jpg)
Utilization
df
![Page 34: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/34.jpg)
?
iostat -x
![Page 35: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/35.jpg)
Maybe you can get additional utilization if you know the max r/s or w/s - but not as clear based on different properties.
iostat -x
![Page 36: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/36.jpg)
IO (Network)
![Page 37: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/37.jpg)
?
ping
![Page 38: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/38.jpg)
Errors
ping
![Page 39: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/39.jpg)
?
netstat
![Page 40: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/40.jpg)
Saturation
netstat
![Page 41: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/41.jpg)
?
netstat -s
![Page 42: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/42.jpg)
Errors
netstat -s
![Page 43: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/43.jpg)
?
ifconfig
![Page 44: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/44.jpg)
ifconfigSaturation
UtilizationErrors
![Page 45: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/45.jpg)
What are your examples?
http://upload.wikimedia.org/wikipedia/commons/f/f3/Uncle_Sam_(pointing_finger).jpg
![Page 46: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/46.jpg)
Applications
![Page 47: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/47.jpg)
Running out of Apache Threads
• Lots of incoming requests
• Apache hits ServerLimit of threads (Utilization!)
• Requests start to get stuck in TCP backlog (Saturation!)
• Apache endpoints are removed from load balancers (Error!)
• Fail!
http://upload.wikimedia.org/wikipedia/commons/9/96/Colorful_Threads_(3965274345).jpg
![Page 48: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/48.jpg)
Cold DB Start• DB’s like to be in memory, but
can’t start that way
• All data requests go to disk (which is SAN backed)
• SAN controller CPU gets maxed out (Utilization!)
• HBA queues get deep (Saturation!)
• Requests timeout (Error!)
• Fail!
![Page 49: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/49.jpg)
Summary
![Page 50: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/50.jpg)
Methods > Tools
• Don’t let tools get in the way of solutions
• It’s easy to think that all your missing a tool.
• But are you actually following a method to your performance madness?
http://upload.wikimedia.org/wikipedia/commons/6/6d/Three_Card_Monte.jpg
![Page 51: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/51.jpg)
Anti-Methods• Blame Someone Else
• Streetlight
• Drunk Man
• Random Change
• Passive Benchmark
!
• Don’t do these…
http://www.brendangregg.com/methodology.html http://upload.wikimedia.org/wikipedia/commons/a/af/Villainc.svg
![Page 52: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/52.jpg)
Methods• Ad Hoc Checklist
• Problem Statement
• Scientific
• Workload Characterization
• Drill-down Analysis
• By-layer
• Latency Analysis
• Tools
• Stack Profile
• Off-CPU Analysis
• Thread State Analysis
• Active Benchmarkhttp://www.brendangregg.com/methodology.html http://memegenerator.net/instance/9192015
![Page 53: Intro to linux performance analysis](https://reader033.vdocuments.net/reader033/viewer/2022052618/554c5240b4c905452e8b4870/html5/thumbnails/53.jpg)
Linux Performance Tools
Chris McEniry LOPSA-SD
March 27, 2014