Efficient Tracing and Versatile AnalysisEfficient Tracing and Versatile Analysisof Lock Contention in Java Applicationsof Lock Contention in Java Applications

on the Virtual Machine Levelon the Virtual Machine Level

Peter HoferPeter HoferDavid GnedtDavid Gnedt

Andreas SchörgenhumerAndreas SchörgenhumerHanspeter MössenböckHanspeter Mössenböck

16 March 201616 March 2016

The Need for Analyzing Lock Contention

T1

T2

T3

T4

t

final Segment<K,V>[] segments = this.segments;int size;boolean overflow; // true if size overflows 32 bitslong sum; // sum of modCountslong last = 0L; // previous sumint retries = -1; // first iteration isn't retrytry { for (;;) { if (retries++ == RETRIES_BEFORE_LOCK) { for (int j = 0; j < segments.length; ++j) ensureSegment(j).lock(); // force creation } sum = 0L; size = 0; overflow = false; for (int j = 0; j < segments.length; ++j) { Segment<K,V> seg = segmentAt(segments, j); if (seg != null) { sum += seg.modCount; int c = seg.count; if (c < 0 || (size += c) < 0) overflow = true; } } if (sum == last) break; last = sum; }} finally { if (retries > RETRIES_BEFORE_LOCK) { for (int j = 0; j < segments.length; ++j) segmentAt(segments, j).unlock(); }}return overflow ? Integer.MAX_VALUE : size;

protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException{ synchronized (getClassLoadingLock(name)) { // First, check if the class has already been loaded Class<?> c = findLoadedClass(name); if (c == null) { long t0 = System.nanoTime(); try { if (parent != null) { c = parent.loadClass(name, false); } else { c = findBootstrapClassOrNull(name); } } catch (ClassNotFoundException e) { // ClassNotFoundException thrown if class not found // from the non-null parent class loader }

if (c == null) { // If still not found, then invoke findClass in order // to find the class. long t1 = System.nanoTime(); c = findClass(name);

// this is the defining class loader; record the stats sun.misc.PerfCounter.getParentDelegationTime().addTime(t1 - t0); sun.misc.PerfCounter.getFindClassTime().addElapsedTimeFrom(t1); sun.misc.PerfCounter.getFindClasses().increment(); } } if (resolve) { resolveClass(c); } return c; }}

/** * Removes all of the mappings from this map. */public void clear() { long delta = 0L; // negative number of deletions int i = 0; Node<K,V>[] tab = table; while (tab != null && i < tab.length) { int fh; Node<K,V> f = tabAt(tab, i); if (f == null) ++i; else if ((fh = f.hash) == MOVED) { tab = helpTransfer(tab, f); i = 0; // restart } else { synchronized (f) { if (tabAt(tab, i) == f) { Node<K,V> p = (fh >= 0 ? f : (f instanceof TreeBin) ? ((TreeBin<K,V>)f).first : null); while (p != null) { --delta; p = p.next; } setTabAt(tab, i++, null); } } } } if (delta != 0L) addCount(delta, -1);}

/** * Removes all of the mappings from this map. */public void clear() { long delta = 0L; // negative number of deletions int i = 0; Node<K,V>[] tab = table; while (tab != null && i < tab.length) { int fh; Node<K,V> f = tabAt(tab, i); if (f == null) ++i; else if ((fh = f.hash) == MOVED) { tab = helpTransfer(tab, f); i = 0; // restart } else { synchronized (f) { if (tabAt(tab, i) == f) { Node<K,V> p = (fh >= 0 ? f : (f instanceof TreeBin) ? ((TreeBin<K,V>)f).first : null); while (p != null) { --delta; p = p.next; } setTabAt(tab, i++, null); } } } } if (delta != 0L) addCount(delta, -1);}

Images: Intel Haswell Chip, Intel Free Press, licensed under Creative Commons BY-SA 2.0 / System Lock, Yuri Samoilov, CC BY 2.0

3

Java Intrinsic Object Locksclass SynchronizedArrayList { ArrayList list = new ArrayList();

void get(int index) { synchronized(list) { return list.get(index); } }

void add(int index, Object obj) { synchronized(list) { return list.add(index, obj); } }}

4

java.util.concurrent Locksclass SynchronizedArrayList { ArrayList list = new ArrayList(); final ReentrantLock lock = new ReentrantLock();

void get(int index) { lock.lock(); try { return list.get(index); } finally { lock.unlock(); } }

void add(int index, Object obj) { lock.lock(); try { return list.add(index, obj); } finally { lock.unlock(); } }}

Java Synchronization

Locking is cheap: fast compare-and-set operations

Contention is expensive

no progress in blocked threads

threads must coordinate

T2

T1

t

Images: KDE Oxygen Icon Project, licensed under the LGPL

6

Blame

get() add()

get() add()

Time spent waiting at...

Time spent waiting for...

7

Tracing Monitors in the HotSpot Java VM

t

T1

Contended Exit

T3

Contended Exit

T2

Contended Enter Entered

8

Metadata

Metadata recorded in special events

Call chains

execution state – no identity

many identical call chains

89?

89

Call Chain Metadata Eventid: 89frames: ...

Contended Exit Eventcall chain: 89

...lock-free!

9

Trace Analysis

Trace Parser

EEE

Metadata Resolver

MD

Event Rearranger

EEE

EEE

IL1IL2

41 594

IL Dispatcher

E

E

E

IL1 Analyzer

ILn Analyzer

EE

EE

CC

Aggregation

CC

java.util.concurrent Analysis

10

0%

100%

110%

120%

130%

140%

150%

160%

170%

180%

190%

No Tracing Compressed Output Online Analysis

Overhead

multi-threaded single-threaded

DaCapo and scalabench benchmarks (except batik and eclipse)2x Xeon E5-2670v2 = 20 cores + HT, Oracle Linux 7, OpenJDK 8u45-b14

11

0 B/s

5 MB/s

10 MB/s

15 MB/s

20 MB/s

25 MB/s

30 MB/s

actorsapparat

avrorah2 jython

luindex

lusearch

pmdscalac

scaladoc

scalatest

sunflow

tmttomcat

tradebeans

tradesoap

xalan

Uncompressed Compressed

Bytes per Second DaCapo and scalabench benchmarks (except batik and eclipse)2x Xeon E5-2670v2 = 20 cores + HT, Oracle Linux 7, OpenJDK 8u45-b14

multi-threaded

12

Demonstration

13

14

15

16

17

18

19

20

21

22

23

24

25

26

Read our paper

for more details on tracing and analysis,especially java.util.concurrent

Download our JDK and visualization tool:

http://mevss.jku.at/ → Tools → Lock Contention Tracing

Ask questions!