Bill Au

CBS Interactive

http://www.apache.org/~billa/oscon2012/

Troubleshooting Slow or Hung Java Applications

2

Agenda

»Tools of the Trade»Case Studies

2

3

Tools – Out Of The Box

» jinfo, jmap, jstack, jstat»JConsole, VisualVM

– Monitor CPU usage by using JTop plug-in ($JAVA_HOME/demo/management/JTop)

» java.lang.management– Platform MXBeans: runtime, operating system, class

loading, compilation, garbage collection, memory, thread– Sample code in $JAVA_HOME/demo/management– JDK demo and sample was part of JDK download for

jdk6u31 or older, separate download for u32+

3

4

Tools – java.lang.management

»Sample code:import java.lang.management.*;

ThreadMXBean tb=ManagementFactory.getThreadMXBean();

if (tb.isThreadCpuTimeSupported()) {

long[] tids=tb.getAllThreadIds();

for (long tid : tids) {

System.out.println(“cputime:”+tb.getThreadCpuTime(tid));

System.out.println(“usertime:”+tb.getThreadUserTime(tid));

}

}

4

5

Tools

»Garbage collection log– -Xloggc:<filename>– Timestamp and duration of GC events– Sizes before and after GC events– More options to explore:

− -XX:+PrintGCDateStamps− -XX:+PrintGCDetails − -XX:+PrintTenuringDistribution− -XX:+PrintGCApplicationsStoppedTime

– Some GC flags can be set dynamically by jinfo/JConsole/VisualVM: PrintGC, PrintGCDetails, PrintGCTimeStamps

»visualgc – GUI for garbage collection monitoring

5

6

Tools

»Thread dump– kill -3 (or kill –SIGHUP)– JConsole, VisualVM– ThreadMXBean– jstack (-F)

»Consecutive thread dumps– Stop motion animation

6

7

Hung or Slow App

»Long GC pauses and/or high GC overhead– Basic GC tuning

−max heap size: -Xmx− type of collector used

− -XX:+UserSerialGC− -XX:+UseParallelGC/-XX:+UseParallelOldGC− -XX:+UseConcMarkSweepGC− -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC (jdk6u14+)

– Advanced GC tuning− young generation size: -Xmn

– Baseline, tune, measure one change at a time

»Demo: HPjmeter – garbage collection analysis (and more…)

7

8

Hung or Slow App

»Deadlock– Take thread dump to run the JVM’s deadlock detector

»Looping Threads– Monitor CPU times of running threads

−ThreadMXBean− JTop plug-in for JConsole/VisualVM

– Inspect stack trace of suspicious threads

8

9

Slow or Hung App

»Blocked threads– Look for blocked threads waiting on the same condition– Identify bottleneck for resource contention– Inspect code for incautious and/or improper synchronization

»Demo: samurai – thread dump analysis tool»Stuck threads

– Look for runnable threads with the same stack trace across consecutive thread dumps

– Some common causes:−Blocking for a network call without specifying connect and/or

read timeout (default may be no timeout)−Long execution time for calls to backend servers

9

10

Links

»Presentation http://www.apache.org/~billa/oscon2012

»HPjmeter https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER

»samurai http://yusuke.homeip.net/samurai/en/index.html https://github.com/yusuke/samurai

10