java black box profiling
DESCRIPTION
Slide deck from tech talk in Nignyi NovgorodTRANSCRIPT
The Problem
• There are application• It doesn’t meat its SLA• Users/client/stakeholders are unhappy
You are one to fix it!
What to do?
• Do not panic!• Write out all moving parts in system• Elaborate acceptance criteria• Make sure you understand KPI and SLA• Is problem in Java application?
Yes - You are ready to start profiling
Moving parts
ServerBrowser
Application
Database
Caching
JavaScript
HTTP
Network Network
SQL performance
CPU Memory / Swapping
Disk IO Virtualization
Understanding targets
• Business transaction ≠ Page• Business transaction ≠ SQL transaction• Page ≠ HTTP request• HTTP request ≠ SQL transaction
Make sure you know how your KPI are translating into milliseconds
Just be before you start profiling
Take a deep breath Make sure problem is in Java part Make sure you understand end goal
These points are trivial so it is very easy to forget them under stress.Do not make that mistake!
Few more rule before start
Every system is a black box Do you think that know what is inside? You don’t!
There 3 kinds of data produced by profiling Lies Darn lies Statistics
Data incorrectly interpreted
Data incorrectly collected
Profiling
Reproducing performance issue
Stationary process• Uniform load reasonable simulating real one• Goal is optimizing average performance indicators
Transition process• Goal is optimize prolonged process• Load profile is changing over time systematically
Profiling single operation• Goal is to optimize particular operation
Profiling in JVM
• Stack trace sampling• Byte code instrumentation• Thread MBeans• GC logs• Heap dumps / Heap histogramMore diagnostic sources• JMX• Performance counters• Flight recorder / mission control
Stack trace sampling
• Work best with stationary process• Uniformly impacts system under test• Do not require “calibrating”• Result from multiple runs could be combined• You deal with probabilities, not hard numbers• Measure wall clock, not CPU time
Byte code instrumentation
• Useful for all types of experiments• Produce some absolute numbers
Number of calls is accurate Time is predictably skewed
• Significantly skew performance of system under test• Require careful targeting• Could measure CPU time
Threading MBean
Standard JVM Threading MBean CPU usage per thread (user/sys) Memory allocation per thread Blocking / waiting statistics
Tools
Swiss Java Knife (SJK) - https://github.com/aragozin/jvm-tools
• Thread monitor• Stack Trace Recorder/Analyzer• Dead object histogram• and more
BTrace - https://kenai.com/projects/btrace • Incrementing profiler• Used via CLI or API• Scriptable in Java
Profiling strategy
Classify your bottleneck CPU• Application CPU usage• Excessive memory allocation / GC
Disk IO Network latency Network bandwidth Inter thread contention
Recipes: CPU hogs
1. You sampling to identify suspects• Frame frequency histogram + meditation• Call tree + deep meditation• Iterative classification
2. Instrument methods to get hard prove• relative number of calls between methods
3. Iterate to pin point root cause
Recipes: CPU hogs
Iterative classification (JBoss + Seam)
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Base
Other
DefaultServlet.doGet
LifeCycleImpl (render)
LifeCycleImpl (execute)
Business logic
Seam interceptor (lock contention)
Seam interceptor (inject/outject)
Resource bundle (getObject)
Resource bundle (missing)
Facelet compile
Hibernate (rest)
Hibernate (autoFlush)
JDBC
Recipes: CPU hogs
Iterative classification (JBoss + Seam)
[ROOT] org.apache.catalina.connector.CoyoteAdapter.service
[JDBC] org.jboss.jca.adapters.jdbc.
[Hibernate (autoFlush)] org.hibernate.internal.SessionImpl.autoFlushIfRequired
[Hibernate (rest)] org.hibernate.
[Facelets compile] com.sun.faces.facelets.compiler.Compiler.compile
[XML] javax.xml.
[ResourceBundle - missing resource] java.util.ResourceBundle.getObject(ResourceBundle.java:395)
[ResourceBundle - getObject] java.util.ResourceBundle.getObject
[Seam bean interceptor - inject/disinject/outject] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed !ANY org.jboss.seam.Component.inject
org.jboss.seam.Component.disinject org.jboss.seam.Component.outject
[Seam bean interceptor - lock contention] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed java.util.concurrent.locks.ReentrantLock [Seam bean interceptor - other] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed
[WorkItemController.doselect] org.cp.shark.WorkItemController.doselect
[LifecycleImpl.execute] com.sun.faces.lifecycle.LifecycleImpl.execute
[LifecycleImpl.render] com.sun.faces.lifecycle.LifecycleImpl.render
[DefaultServlet.doGet] org.apache.catalina.servlets.DefaultServlet.doGet
[other]
Recipes: Memory issues
1. Indentify thread producing garbage• SJK: ttop command
2. Classify garbage• Class histogram of dead object in heap• SJK: hh --dead
3. Investigate suspect classes• Heap dump• Debugging / Instrumentation
Recipes: contention
• Stack trace sampling Good for “bad” contention cases
• Thread MBean Contention statistics could be enabled
• BTrace Analyze lock access pattern using your
codebase knowledge
Sub millisecond profiling
Low latency profiling requires indirect measuring.You should not measure sub ms directly!
• Make a hypothesis• Arrange experiment to prove / disprove• Collect aggregated metrics• Compare, analyze and iterate
Flight record / Mission Control
Tight integration with JVM• Low overhead• Access to intimate JVM areas
Well balanced set of metrics• Unobtrusive• Indicative
Good user interfaceMay be one day I will abandon my SJK
Thank you
Alexey Ragozin [email protected]
http://blog.ragozin.info- my articleshttp://aragozin.timepad.ru- community events in Moscow