the art and methodology of troubleshooting alexey diomin, [email protected]
TRANSCRIPT
![Page 2: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/2.jpg)
Production
![Page 3: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/3.jpg)
Agenda
Logging Offline analysis Online analysis
![Page 4: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/4.jpg)
Monitoring
System monitoring Zabbix Nagios
Application Monitoring New Relic Logstash Kibana
![Page 5: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/5.jpg)
Zabbix. CPU load example
![Page 6: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/6.jpg)
Zabbix. Memory example
![Page 7: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/7.jpg)
Zabbix. CMS Old Gen
![Page 8: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/8.jpg)
Logstash. Kibana
![Page 9: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/9.jpg)
“We need to go deeper”
![Page 10: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/10.jpg)
“We need to go deeper”
jstack (thread dump) jstack <pid> kill -QUIT <pid>
![Page 11: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/11.jpg)
jstack
Very simple profiler Lock detection
![Page 12: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/12.jpg)
jstack example
![Page 13: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/13.jpg)
“We need to go deeper”
jstack (thread dump) jstack <pid> kill -QUIT <pid>
jmap jmap -dump <pid> jmap -histo <pid|file>
![Page 14: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/14.jpg)
jmap
Eclipse Memory Analyzer (MAT) http://www.eclipse.org/mat/
jvisualvm http://visualvm.java.net/
![Page 15: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/15.jpg)
MAT
![Page 16: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/16.jpg)
MAT
![Page 17: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/17.jpg)
Leak Detection
![Page 18: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/18.jpg)
RingBuffer
![Page 19: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/19.jpg)
MAT
![Page 20: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/20.jpg)
“We need to go deeper”
jstack (thread dump) jstack <pid> kill -QUIT <pid>
jmap jmap -dump <pid> jmap -histo <pid|file>
JFR / JMC
![Page 21: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/21.jpg)
JFR
![Page 22: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/22.jpg)
JFR
![Page 23: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/23.jpg)
Event Type
![Page 24: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/24.jpg)
JFR
Basic features: Code Profile Allocation profile
![Page 25: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/25.jpg)
JFR
Basic features: Code Profile Allocation profile
Killer features: Contentions Exceptions rate
![Page 26: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/26.jpg)
Code profile
![Page 27: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/27.jpg)
Allocation profile
![Page 28: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/28.jpg)
Netty’s bug example
![Page 29: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/29.jpg)
Contention
![Page 30: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/30.jpg)
Hbase latency
![Page 31: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/31.jpg)
Contentions example
![Page 32: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/32.jpg)
Exceptions rate
![Page 33: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/33.jpg)
Context Switch
![Page 34: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/34.jpg)
Exception example
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException
1. “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ….”
SQL syntax for prepared statements does not support multi-statements (that is, multiple statements within a single string separated by “;” characters).
2. “Can't create more than max_prepared_stmt_count statements”
useServerPrepStmts
Use server-side prepared statements if the server supports them
![Page 35: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/35.jpg)
useServerPrepStmts
![Page 36: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/36.jpg)
Summary
Logging Offline analysis Online analysis
![Page 37: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/37.jpg)
Links
Java Performance: The Definitive Guide http://shop.oreilly.com/product/0636920028499.do
Troubleshooting Guide for Java SE 6 with HotSpot VM
http://www.oracle.com/technetwork/java/javase/tsg-vm-149989.pdf
![Page 38: THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com](https://reader030.vdocuments.net/reader030/viewer/2022032703/56649cfa5503460f949cc43d/html5/thumbnails/38.jpg)
Black boxes? Not in
troubleshooting!!!