java jvm troubleshooting guide

8/9/2019 Java JVM Troubleshooting Guide

1/127

Troubleshoot the JVM like never before

JVM TroubleshootingGuide

Pierre-Hugues CharbonneauIlias Tsagklis


2/127

JVM Troubleshooting Handbook www.javacodegeeks.com

Table of ContentsOracle HotSpot JVM Memory ...................................................................................................................

Java HotSpot VM Heap space ...............................................................................................................

Java HotSpot VM !erm"en space ........................................................................................................ #

$%M JVM Memory .................................................................................................................................... &Oracle J'ockit JVM Memory .................................................................................................................... (

Tips )or proper Java Heap si*e ................................................................................................................... +

Java Threading, JVM 'etained memory analysis .................................................................................... -#Java +, rom !erm"en to Metaspace ......................................................................................................./-

H!'O 0 Memory leak analysis with 1clipse Memory 2naly*er Tool 3M2T4 ....................................... /&

JVM verbose "5 output tutorial .............................................................................................................. 2naly*ing thread dumps .......................................................................................................................... #6

$ntroduction to thread dump analysis .................................................................................................. #6

Thread 7ump, Thread Stack Trace analysis ........................................................................................#(Java Thread 5!8 analysis on 9indows ...................................................................................................#:

5ase Study 0 Too many open )iles ............................................................................................................ ;#"5 overhead limit e ............................................................................. +&

OutO)Memory1rror, Out o) swap space 0 !roblem !atterns .............................................................. +(

OutO)Memory1rror, unable to create new native thread .................................................................... +:5lass?otound1


3/127


Oracle HotSpot JVM Memory

Java HotSpot VM Heap space

This section will provide you with a high level overview of the different Java Heap memory spaces ofthe Oracle Java HotSpot VM. This understanding is quite important for any individual involved inproduction support given how frequent memory problems are observed. roper !nowledge of the JavaVM Heap Space is critical.

"our Java VM is basically the foundation of your Java program which provides you with dynamicmemory management services# garbage collection# Threads# $O and native operations and more.

The Java Heap Space is the memory %container% of you runtime Java program which provides to yourJava program the proper memory spaces it needs &Java Heap# 'ative Heap( and managed by the

JVM itself.

The JVM HotSpot memory is split between ) memory spaces*

The Java Heap

The erm+en &permanent generation( space

The 'ative Heap &,-Heap(

Here is the brea!down for each one of them*

Memory Space Start-up argumentsand tuning

Monitoring strategies Description

Java Heap -m/ &ma/imum Heapspace(

-ms &minimum Heapsi0e(

1*-m/2345m-ms2345m

- verbose +,- JM 6$- J,onsole- Other monitoring tools

The Java Heap isstoring your primaryJava program ,lassinstances.

erm+en -*Ma/ermSi0e&ma/imum si0e(

-*ermSi0e&minimum si0e(

1*-*Ma/ermSi0e7824

- verbose +,- JM 6$- J,onsole

- Other monitoring tools

The Java HotSpot VMpermanent generationspace is the JVM

storage used mainly tostore your Java ,lassob9ects such as namesand method of the,lasses# internal JVMob9ects and other J$Toptimi0ation related

o) -/(
http://www.javacodegeeks.com/http://www.javacodegeeks.com/


4/127


m-*ermSi0e748:m

data.

'ative Heap&,-Heap(

'ot configurabledirectly.

;or a )4-bit VM# the ,-Heap capacity 7 5 +ig virtual memory

< Java Heap - erm+en

- Total process si0echec! in ?indows and

@inu/- pmap command onSolaris > @inu/- svmon command on

6$

The ,-Heap is storingob9ects such as MM6

file# other JVM and thirdparty native codeob9ects.

Java Heap Space - Overview > life cycle

"our Java program life cycle typically loo!s li!e this*

Java program coding &via 1clipse $A1 etc.( e.g. Hello?orld.9ava

Java program compilation &Java compiler or third party build tools such as 6pache 6nt# 6pacheMaven..( e.g. Hello?ord.class

Java program start-up and runtime e/ecution e.g. via your Hello?orld.main&( method

'ow letBs dissect your Hello?orld.class program so you can better understand.

6t start-up# your JVM will load and cache some of your static program and JAC libraries to the'ative Heap# including native libraries# Mapped ;iles such as your program Jar file&s(# Threadssuch as the main start-up Thread of your program etc.

"our JVM will then store the %static% data of your Hello?orld.class Java program to theerm+en space &,lass metadata# descriptors# etc.(.

Once your program is started# the JVM will then manage and dynamically allocate the memoryof your Java program to the Java Heap &"oung+en > Old+en(. This is why it is so importantthat you understand how much memory your Java program needs to you can properly fine-tuned the capacity of your Java Heap controlled via -ms > -m/ JVM parameters. rofiling#Heap Aump analysis allow you to determine your Java program memory footprint.

;inally# the JVM has to also dynamically release the memory from the Java Heap Space thatyour program no longer needD this is called the garbage collection process. This process canbe easily monitored via the JVM verbose +, or a monitoring tool of your choice such as

Jconsole.

Java HotSpot VM PermGen space

The Java HotSpot VM permanent generation space is the JVM storage used mainly to store your Java,lass ob9ects. The Java Heap is the primary storage that is storing the actual short and long term

# o) -/(


5/127


instances of your erm+en ,lass ob9ects.

The erm+en space is fairly static by nature unless using third party tool andEor Java =eflection 6$which relies heavily on dynamic class loading.

$t is important to note that this memory storage is applicable only for a Java HotSpot VMD other JVMvendors such as $FM and Oracle J=oc!it do not have such fi/ed and configurable erm+en storageand are using other techniques to manage the non Java Heap memory &native memory(.

;ind below a graphical view of a JVM HotSpot Java Heap vs. erm+en space brea!down along withits associated attributes and capacity tuning arguments.

6part from the Oracle HotSpot JVM# there are other virtual machines provided by differented vendors.The following sections e/amine the memory configurations used by other JVMs. Gnderstanding thoseis quite important given the implementation and naming convention differences between HotSpot and

; o) -/(


6/127


the other JVMs.

IBM JVM Memory

The $FM VM memory is split between 4 memory spaces*

The Java Heap &nursery and tenured spaces(

The 'ative Heap &,-Heap(

Here is the brea!down for each one of them*



Java Heap -m/ &ma/imum Heap

space(

-ms &minimum Heapsi0e(- verbose +,- JM 6$- $FM monitoring tools

1*-m/2345m-ms2345m

+, policy 1/*-gcpolicy*gencon&enable gencon +,policy(

- verbose +,

- JM 6$- $FM monitoring tools

The $FM Java Heap is

typically split betweenthe nursery and tenuredspace &"oung+en#Old+en(.

The gencon +, policy&combo of concurrentand generational +,( istypically used for Java11 platforms in order tominimi0e the +, pausetime.

'ative Heap&,-Heap(

'ot configurabledirectly.

;or a )4-bit VM# the ,-Heap capacity 7 5 +ig virtual memory

< Java Heap

- svmon command The ,-Heap is storingclass metadata ob9ectsincluding library files#other JVM and thirdparty native codeob9ects.

6s you might noticed# there is no erm+en space for the $FM VM. The erm+en space is onlyapplicable to the HotSpot VM. The $FM VM is using the 'ative Heap for ,lass metadata related data.

& o) -/(


7/127


6lso note that Oracle is also starting to remove the erm+en space for the HotSpot VM# as we willdiscuss in a ne/t section.

Oracle Joc!it JVM Memory

The J=oc!it VM memory is split between 4 memory spaces*

The Java Heap &"oung+en and Old+en(

The 'ative memory space &,lasses pool# ,-Heap# Threads...(



Java Heap -m/ &ma/imum Heapspace(

-ms &minimum Heapsi0e(

1*-m/2345m-ms2345m

- verbose +,- JM 6$

- J=oc!it Mission,ontrol tools suite

The J=oc!it Java Heapis typically split between

the "oung+en &short-lived ob9ects(# Old+en&long-lived ob9ects(.

'ative memory space 'ot configurabledirectly.

;or a )4-bit VM# thenative memory spacecapacity 7 4-5 +ig virtual memory < JavaHeap

- Total process si0echec! in ?indows and@inu/- pmap command onSolaris > @inu/- J=oc!it J=,MA tool

The J=oc!it 'ativememory space isstoring the Java,lasses metadata#Threads and ob9ectssuch as library files#other JVM and thirdparty native codeob9ects.

Similar to the $FM VM# there is no erm+en space for the J=oc!it VM. The erm+en space is onlyapplicable to the HotSpot VM. The J=oc!it VM is using the 'ative Heap for ,lass metadata relateddata.

( o) -/(


8/127


The J=oc!it VM tend to uses more native memory in e/change for better performance. J=oc!it doesnot have an interpretation mode# compilation only# so due to its additional native memory needs theprocess si0e tends to use a couple of hundred MF larger than the equivalent Sun JVM si0e. Thisshould not be a big problem unless you are using a )4-bit J=oc!it with a large Java Heap requirementDin this scenario# the ris! of OutOfMemory1rror due to 'ative Heap depletion is higher for a J=oc!it VM&e.g. for a )4-bit VM# bigger is the Java Heap# smaller is memory left for the 'ative Heap(.

OracleBs strategy# being the vendor for both HotSpot and J=oc!it product lines# is to merge the twoVms to a single JVM pro9ect that will include the best features of each one. This will also simplify JVMtuning since right now failure to understand the differences between these 4 VMBs can lead to badtuning recommendations and performance problems.

Tips for proper Ja"a Heap si#e

Aetermination of proper Java Heap si0e for a production system is not a straightforward e/ercise.Multiple performance problem can occur due to inadequate Java Heap capacity and tuning. Thissection will provide some tips that can help you determine the optimal Java heap si0e# as a startingpoint# for your current or new production environment. Some of these tips are also very usefulregarding the prevention and resolution of OutOfMemory1rror problems# including memory lea!s.

lease note that these tips are intended to help youI determine proper Java Heap si0e. Since each $Tenvironment is unique# you are actually in the best position to determine precisely the required JavaHeap specifications of your clients environment.

K2 - JVM* you always fear what you donBt understand

How can you e/pect to configure# tune and troubleshoot something that you donBt understandL "oumay never have the chance to write and improve Java VM specifications but you are still free to learnits foundation in order to improve your !nowledge and troubleshooting s!ills. Some may disagree# butfrom my perspective# the thin!ing that Java programmers are not required to !now the internal JVMmemory management is an illusion.

Java Heap tuning and troubleshooting can especially be a challenge for Java > Java 11 beginners.;ind below a typical scenario*

"our client production environment is facing OutOfMemory1rror on a regular basis and causinglot of business impact. "our support team is under pressure to resolve this problem.

6 quic! +oogle search allows you to find e/amples of similar problems and you now believe

&and assume( that you are facing the same problem. "ou then grab JVM -ms and -m/ values from another person OutOfMemory1rror problem

case# hoping to quic!ly resolve your clientBs problem.

"ou then proceed and implement the same tuning to your environment. 4 days later you reali0eproblem is still happening &even worse or little better(...the struggle continues...

?hat went wrongL

+ o) -/(


9/127


"ou failed to first acquire proper understanding of the root cause of your problem.

"ou may also have failed to properly understand your production environment at a deeper level&specifications# load situation etc.(. ?eb searches is a great way to learn and share !nowledge

but you have to perform your own due diligence and root cause analysis. "ou may also be lac!ing some basic !nowledge of the JVM and its internal memory

management# preventing you to connect all the dots together.

My K2 tip and recommendation to you is to learn and understand the basic JVM principles along withits different memory spaces. Such !nowledge is critical as it will allow you to ma!e validrecommendations to your clients and properly understand the possible impact and ris! associated withfuture tuning considerations.

6s a reminder# the Java VM memory is split up to ) memory spaces*

The Java Heap* 6pplicable for all JVM vendors# usually split between "oung+en &nursery( >

Old+en &tenured( spaces. The erm+en &permanent generation(* 6pplicable to the Sun HotSpot VM only &erm+en

space will be removed in future Java updates(

The 'ative Heap &,-Heap(* 6pplicable for all JVM vendors.

6s you can see# the Java VM memory management is more comple/ than 9ust setting up the biggestvalue possible via


10/127


K4 - Aata and application is !ing* review your static footprint requirement

"our application&s( along with its associated data will dictate the Java Heap footprint requirement. Fystatic memory# $ mean %predictable% memory requirements as per below.

Aetermine how many different applications you are planning to deploy to a single JVM processe.g. number of 16= files# ?6= files# 9ar files etc. The more applications you deploy to a singleJVM# higher demand on native Heap.

Aetermine how many Java classes will be potentially loaded at runtimeD including third part6$Bs. The more class loaders and classes that you load at runtime# higher demand on theHotSpot VM erm+en space and internal J$T related optimi0ation ob9ects.

Aetermine data cache footprint e.g. internal cache data structures loaded by your application

&and third party 6$Bs( such as cached data from a database# data read from a file etc. Themore data caching that you use# higher demand on the Java Heap Old+en space.

Aetermine the number of Threads that your middleware is allowed to create. This is veryimportant since Java threads require enough native memory or OutOfMemory1rror will bethrown.

;or e/ample# you will need much more native memory and erm+en space if you are planning todeploy 23 separate 16= applications on a single JVM process vs. only 4 or ). Aata caching not

-6 o) -/(


11/127


seriali0ed to a dis! or database will require e/tra memory from the Old+en space.

Try to come up with reasonable estimates of the static memory footprint requirement. This will be veryuseful to setup some starting point JVM capacity figures before your true measurement e/ercise &e.g.tip K5(. ;or )4-bit JVM# $ usually do not recommend a Java Heap si0e high than 4 +F &-ms435Nm#-m/435Nm( since you need enough memory for erm+en and native Heap for your Java 11applications and threads.

This assessment is especially important since too many applications deployed in a single )4-bit JVMprocess can easily lead to native Heap depletionD especially in a multi threads environment.

;or a :5-bit JVM# a Java Heap si0e of ) +F or 5 +F per JVM process is usually my recommendedstarting point.

K) - Fusiness traffic set the rules* review your dynamic footprint requirement

"our business traffic will typically dictate your dynamic memory footprint. ,oncurrent users > requestsgenerate the JVM +, %heartbeat% that you can observe from various monitoring tools due to veryfrequent creation and garbage collections of short > long lived ob9ects. 6s you saw from the aboveJVM diagram# a typical ratio of "oung+en vs. Old+en is 2*) or )).

;or a typical )4-bit JVM# a Java Heap si0e setup at 4 +F &using generational > concurrent collector(will typically allocate 833 MF for "oung+en space and 2.8 +F for the Old+en space.

Minimi0ing the frequency of ma9or +, collections is a !ey aspect for optimal performance so it is veryimportant that you understand and estimate how much memory you need during your pea! volume.

6gain# your type of application and data will dictate how much memory you need. Shopping cart typeof applications &long lived ob9ects( involving large and non-seriali0ed session data typically need large

Java Heap and lot of Old+en space. Stateless and M@ processing heavy applications &lot of shortlived ob9ects( require proper "oung+en space in order to minimi0e frequency of ma9or collections.

1/ample*

"ou have 8 16= applications &P4 thousands of Java classes( to deploy &which includemiddleware code as well...(.

"our native heap requirement is estimated at 2 +F &has to be large enough to handle Threadscreation etc.(.

"our erm+en space is estimated at 824 MF.

"our internal static data caching is estimated at 833 MF.

"our total forecast traffic is 8333 concurrent users at pea! hours. 1ach user session data footprint is estimated at 833 C.

Total footprint requirement for session data alone is 4.8 +F under pea! volume.

6s you can see# with such requirement# there is no way you can have all this traffic sent to a singleJVM )4-bit process. 6 typical solution involves splitting &tip K8( traffic across a few JVM processes andE or physical host &assuming you have enough hardware and ,G cores available(.

-- o) -/(


12/127


However# for this e/ample# given the high demand on static memory and to ensure a scalableenvironment in the long run# $ would also recommend :5-bit VM but with a smaller Java Heap as astarting point such as ) +F to minimi0e the +, cost. "ou definitely want to have e/tra buffer for theOld+en space so $ typically recommend up to 83 memory footprint post ma9or collection in order to!eep the frequency of ;ull +, low and enough buffer for fail-over scenarios.

Most of the time# your business traffic will drive most of your memory footprint# unless you needsignificant amount of data caching to achieve proper performance which is typical for portal &media(heavy applications. Too much data caching should raise a yellow flag that you may need to revisitsome design elements sooner than later.

K5 - AonBt guess it# measure itQ

6t this point you should*

Gnderstand the basic JVM principles and memory spaces

Have a deep view and understanding of all applications along with their characteristics &si0e#type# dynamic traffic# stateless vs. stateful ob9ects# internal memory caches etc.(

Have a very good view or forecast on the business traffic &K of concurrent users etc.( and foreach application

Some ideas if you need a :5-bit VM or not and which JVM settings to start with

Some ideas if you need more than one JVM &middleware( processes

Fut wait# your wor! is not done yet. ?hile this above information is crucial and great for you to comeup with %best guess% Java Heap settings# it is always best and recommended to simulate yourapplication&s( behaviour and validate the Java Heap memory requirement via proper profiling# load >performance testing.

"ou can learn and ta!e advantage of tools such as Jrofiler. ;rom my perspective# learning how touse a profiler is the best way to properly understand your application memory footprint. 6notherapproach $ use for e/isting production environments is heap dump analysis using the 1clipse M6Ttool. Heap Aump analysis is very powerful and allow you to view and understand the entire memoryfootprint of the Java Heap# including class loader related data and is a must do e/ercise in anymemory footprint analysisD especially memory lea!s.

-/ o) -/(


13/127


Java profilers and heap dump analysis tools allow you to understand and validate your applicationmemory footprint# including detection and resolution of memory lea!s. @oad and performance testingis also a must since this will allow you to validate your earlier estimates by simulating your forecastconcurrent users. $t will also e/pose your application bottlenec!s and allow you to further fine tuneyour JVM settings. "ou can use tools such as 6pache JMeter which is very easy to learn and use ore/plore other commercial products.

;inally# $ have seen quite often Java 11 environments running perfectly fine until the day where onepiece of the infrastructure start to fail e.g. hardware failure. Suddenly the environment is running atreduced capacity &reduced K of JVM processes( and the whole environment goes down. ?hat

happenedL

There are many scenarios that can lead to domino effects but lac! of JVM tuning and capacity tohandle fail-over &short term e/tra load( is very common. $f your JVM processes are running at N3Old+en space capacity with frequent garbage collections# how can you e/pect to handle any fail-overscenarioL

"our load and performance testing e/ercise performed earlier should simulate such scenario and you

- o) -/(


14/127


should ad9ust your tuning settings properly so your Java Heap has enough buffer to handle e/tra load&e/tra ob9ects( at short term. This is mainly applicable for the dynamic memory footprint since fail-overmeans redirecting a certain of your concurrent users to the available JVM processes &middlewareinstances(.

K8 - Aivide and conquer

6t this point you have performed do0ens of load testing iterations. "ou !now that your JVM is notlea!ing memory. "our application memory footprint cannot be reduced any further. "ou tried severaltuning strategies such as using a large :5-bit Java Heap space of 23 +F# multiple +, policies butstill not finding your performance level acceptableL

$n my e/perience $ found that# with current JVM specifications# proper vertical and hori0ontal scalingwhich involved creating a few JVM processes per physical host and across several hosts will give youthe throughput and capacity that you are loo!ing for. "our $T environment will also more fault tolerant ifyou brea! your application list in a few logical silos# with their own JVM process# Threads and tuningvalues.

This %divide and conquer% strategy involves splitting your application&s( traffic to multiple JVMprocesses and will provide you with*

=educed Java Heap si0e per JVM process &both static > dynamic footprint(

=educed comple/ity of JVM tuning

=educed +, elapsed and pause time per JVM process

$ncreased redundancy and fail-over capabilities

6ligned with latest ,loud and $T virtuali0ation strategies

The bottom line is that when you find yourself spending too much time in tuning that single elephant

:5-bit JVM process# it is time to revisit your middleware and JVM deployment strategy and ta!eadvantage of vertical > hori0ontal scaling. This implementation strategy is more ta/ing for thehardware but will really pay off in the long run.

Ja"a T$reading% JVM etained memory analysis

Having discussed the various heap spaces of the JVM# this section will provide you with a tutorialallowing you to determine how much and where Java heap space is retained from your activeapplication Java threads. 6 true case study from an Oracle ?eblogic 23.3 production environment willbe presented in order for you to better understand the analysis process.

?e will also attempt to demonstrate that e/cessive garbage collection or Java heap space memoryfootprint problems are often not caused by true memory lea!s but instead due to thread e/ecutionpatterns and high amount of short lived ob9ects.

Fac!ground

Java threads are part of the JVM fundamentals. "our Java heap space memory footprint is driven not

-# o) -/(


15/127


only by static and long lived ob9ects but also by short lived ob9ects.

OutOfMemory1rror problems are often wrongly assumed to be due to memory lea!s. ?e oftenoverloo! faulty thread e/ecution patterns and short lived ob9ects they %retain% on the Java heap untiltheir e/ecutions are completed. $n this problematic scenario*

"our %e/pected% application short lived E stateless ob9ects &M@# JSO' data payload etc.(become retained by the threads for too long &thread loc! contention# huge data payload# slowresponse time from remote system etc.(.

1ventually such short lived ob9ects get promoted to the long lived ob9ect space e.g.Old+enEtenured space by the garbage collector.

6s a side effect# this is causing the Old+en space to fill up rapidly# increasing the ;ull +,&ma9or collections( frequency.

Aepending of the severity of the situation this can lead to e/cessive +, garbage collection#increased JVM paused time and ultimately OutOfMemory1rror* Java heap spaceI.

"our application is now down# you are now pu00led on what is going on.

;inally# you are thin!ing to either increase the Java heap or loo! for memory lea!s...are youreally on the right trac!L

$n the above scenario# you need to loo! at the thread e/ecution patterns and determine how muchmemory each of them retain at a given time.

OC $ get the picture but what about the thread stac! si0eL

$t is very important to avoid any confusion between thread stac! si0e and Java memory retention. Thethread stac! si0e is a special memory space used by the JVM to store each method call. ?hen athread calls method 6# it %pushes% the call onto the stac!. $f method 6 calls method F# it gets alsopushed onto the stac!. Once the method e/ecution completes# the call is %popped% off the stac!.

The Java ob9ects created as a result of such thread method calls are allocated on the Java heapspace. $ncreasing the thread stac! si0e will definitely not have any effect. Tuning of the thread stac!si0e is normally required when dealing with 9ava.lang.stac!overflowerror or OutOfMemory1rror*unable to create new native threadI problems.

-; o) -/(


16/127


,ase study and problem conte/t

The following analysis is based on a true production problem we investigated recently.

2. Severe performance degradation was observed from a ?eblogic 23.3 production environmentfollowing some changes to the user web interface &using +oogle ?eb Tool!it and JSO' asdata payload(.

4. $nitial analysis did reveal several occurrences of OutOfMemory1rror* Java heap spaceI errorsalong with e/cessive garbage collection. Java heap dump files were generated automatically &-*HeapAumpOnOutOfMemory1rror( following OOM events.

). 6nalysis of the verbose*gc logs did confirm full depletion of the )4-bit HotSpot JVM Old+enspace &2 +F capacity(.

5. Thread dump snapshots were also generated before and during the problem.8. The only problem mitigation available at that time was to restart the affected ?eblogic server

when problem was observed.:. 6 rollbac! of the changes was eventually performed which did resolve the situation.

The team first suspected a memory lea! problem from the new code introduced.

Thread dump analysis* loo!ing for suspects

The first step we too! was to perform an analysis of the generated thread dump data. Thread dumpwill often show you the culprit threads allocating memory on the Java heap. $t will also reveal anyhogging or stuc! thread attempting to send and receive data payload from a remote system.

The first pattern we noticed was a good correlation between OOM events and STG,C threadsobserved from the ?eblogic managed servers &JVM processes(. ;ind below the primary thread

-& o) -/(


17/127


pattern found*

-( o) -/(


18/127


6s you can see# the above thread appears to be STG,C or ta!ing very long time to read and receivethe JSO' response from the remote server. Once we found that pattern# the ne/t step was to correlatethis finding with the JVM heap dump analysis and determine how much memory these stuc! threadswere ta!ing from the Java heap.

Heap dump analysis* retained ob9ects e/posedQ

The Java heap dump analysis was performed using M6T. ?e will now list the different analysis stepswhich did allow us to pinpoint the retained memory si0e and source.

2. @oad the HotSpot JVM heap dump

4. Select the H$STO+=6M view and filter by %1/ecuteThread%

* ExecuteTre!d is te "!#! cl!ss used $y te %e$logic &ernel for tre!d cre!tion ' execution *

-+ o) -/(


19/127


6s you can see# this view was quite revealing. ?e can see a total of 423 ?eblogic threads created.The total retained memory footprint from these threads is N3: MF. This is pretty significant for a )4-bitJVM process with 2 +F Old+en space. This view alone is telling us that the core of the problem andmemory retention originates from the threads themselves.

). Aeep dive into the thread memory footprint analysis

The ne/t step was to deep dive into the thread memory retention. To do this# simply right clic! over the1/ecuteThread class and select* @ist ob9ects R with outgoing references.

-: o) -/(


20/127


6s you can see# we were able to correlate STG,C threads from the thread dump analysis with highmemory retention from the heap dump analysis. The finding was quite surprising.

5. Thread Java @ocal variables identification

The final analysis step did require us to e/pand a few thread samples and understand the primarysource of memory retention.

/6 o) -/(


21/127


6s you can see# this last analysis step did reveal huge JSO' response data payload at the root cause.That pattern was also e/posed earlier via the thread dump analysis where we found a few threadsta!ing very long time to read > receive the JSO' responseD a clear symptom of huge data payloadfootprint.

$t is crucial to note that short lived ob9ects created via local method variables will show up in the heapdump analysis. However# some of those will only be visible from their parent threads since they are notreferenced by other ob9ects# li!e in this case. "ou will also need to analy0e the thread stac! trace inorder to identify the true caller# followed by a code review to confirm the root cause.

;ollowing this finding# our delivery team was able to determine that the recent JSO' faulty codechanges were generating# under some scenarios# huge JSO' data payload up to 58 MF. +iven the

fact that this environment is using a )4-bit JVM with only 2 +F of Old+en space# you can understandthat only a few threads were enough to trigger severe performance degradation.

This case study is clearly showing the importance of proper capacity planning and Java heap analysis#including the memory retained from your active application > Java 11 container threads.

Ja"a &% 'rom (erm)en to Metaspace

Java N will introduce some new language and runtime features. One of these features is the completeremoval of the ermanent +eneration &erm+en( space which has been announced by Oracle since

the release of JAC . $nterned strings# for e/ample# have already been removed from the erm+enspace since JAC . The JAC N release finali0es its decommissioning. $n this section# we shall discussthe erm+en successor* Metaspace.

?e will also compare the runtime behavior of the HotSpot 2. vs. HotSpot 2.N &b8( when e/ecuting aJava program lea!ingI class metadata ob9ects.

The final specifications# tuning flags and documentation around Metaspace should be available once

/- o) -/(


22/127


Java N is officially released.

Metaspace* 6 new memory space is born

The JAC N HotSpot JVM is now using native memory for the representation of class metadata and iscalled MetaspaceD similar to the Oracle J=oc!it and $FM JVMBs.

The good news is that it means no more 9ava.lang.OutOfMemory1rror* erm+en space problems andno need for you to tune and monitor this memory space anymorenot so fast. ?hile this change isinvisible by default# we will show you ne/t that you will still need to worry about the class metadatamemory footprint. lease also !eep in mind that this new feature does not magically eliminate classand classloader memory lea!s. "ou will need to trac! down these problems using a different approachand by learning the new naming convention.

$n summary*

erm+en space situation

This memory space is completely removed.

The ermSi0e and Ma/ermSi0e JVM arguments are ignored and a warning is issued ifpresent at start-up.

Metaspace memory allocation model

Most allocations for the class metadata are now allocated out of native memory.

The classes that were used to describe class metadata have been removed.

Metaspace capacity

Fy default class metadata allocation is limited by the amount of available native memory&capacity will of course depend if you use a )4-bit JVM vs. :5-bit along with OS virtualmemory availability(.

6 new flag is available &Ma/MetaspaceSi0e(# allowing you to limit the amount of nativememory used for class metadata. $f you dont specify this flag# the Metaspace willdynamically re-si0e depending of the application demand at runtime.

Metaspace garbage collection

+arbage collection of the dead classes and classloaders is triggered once the classmetadata usage reaches the Ma/MetaspaceSi0eI.

roper monitoring > tuning of the Metaspace will obviously be required in order to limit thefrequency or delay of such garbage collections. 1/cessive Metaspace garbage collectionsmay be a symptom of classes# classloaders memory lea! or inadequate si0ing for yourapplication.

Java heap space impact

Some miscellaneous data has been moved to the Java heap space. This means you may observe

// o) -/(


23/127


an increase of the Java heap space following a future JAC N upgrade.

Metaspace monitoring

Metaspace usage is available from the HotSpot 2.N verbose +, log output.

Jstat > JVisualVM have not been updated at this point based on our testing with b8 and theold erm+en space references are still present.

1nough theory now# lets see this new memory space in action via our lea!ing Java program

erm+en vs. Metaspace runtime comparison

$n order to better understand the runtime behavior of the new Metaspace memory space# we created aclass metadata lea!ing Java program. "ou can download the source here.

The following scenarios will be tested*

=un the Java program using JAC 2. in order to monitor > deplete the erm+en memoryspace set at 24N MF.

=un the Java program using JAC 2.N &b8( in order to monitor the dynamic increase andgarbage collection of the new Metaspace memory space.

=un the Java program using JAC 2.N &b8( in order to simulate the depletion of the Metaspaceby setting the Ma/MetaspaceSi0e value at 24N MF.

JAC 2. U:5-bit < erm+en depletion

Java program with 83C configured iterations

Java heap space of 2345 MF

Java erm+en space of 24N MF &-*Ma/ermSi0e724Nm(

6s you can see form JVisualVM# the erm+en depletion was reached after loading about )3Cclasses. ?e can also see this depletion from the program and +, output.

/ o) -/(
http://www.javacodegeeks.com/https://docs.google.com/file/d/0B6UjfNcYT7yGbjVLNlpJRWlKUWc/edit?usp=sharinghttps://docs.google.com/file/d/0B6UjfNcYT7yGbjVLNlpJRWlKUWc/edit?usp=sharinghttp://www.javacodegeeks.com/


24/127


'ow lets e/ecute the program using the HotSpot JAC 2.N J=1.

JAC 2.N U:5-bit < Metaspace dynamic re-si0e



Java Metaspace space* unbounded &default(

/# o) -/(


25/127


6s you can see from the verbose +, output# the JVM Metaspace did e/pand dynamically from 43 MFup to )4N MF of reserved native memory in order to honor the increased class metadata memoryfootprint from our Java program. ?e could also observe garbage collection events in the attempt by

the JVM to destroy any dead class or classloader ob9ect. Since our Java program is lea!ing# the JVMhad no choice but to dynamically e/pand the Metaspace memory space.

The program was able to run its 83C of iterations with no OOM event and loaded 83C ,lasses.

@etBs move to our last testing scenario.

JAC 2.N U:5-bit < Metaspace depletion



Java Metaspace space* 24N MF &-*Ma/MetaspaceSi0e724Nm(

/; o) -/(


26/127


6s you can see form JVisualVM# the Metaspace depletion was reached after loading about )3CclassesD very similar to the run with the JAC 2.. ?e can also see this from the program and +,output. 6nother interesting observation is that the native memory footprint reserved was twice as much

as the ma/imum si0e specified. This may indicate some opportunities to fine tune the Metaspaceresi0e policy# if possible# in order to avoid native memory waste.

,apping the Metaspace at 24N MF li!e we did for the baseline run with JAC 2. did not allow us tocomplete the 83C iterations of our program. 6 new OOM error was thrown by the JVM. The aboveOOM event was thrown by the JVM from the Metaspace following a memory allocation failure.

;inal ?ords on Metaspace

The current observations definitely indicate that proper monitoring > tuning will be required in order tostay away from problems such as e/cessive Metaspace +, or OOM conditions triggered from our lasttesting scenario.

H(O' - Memory lea! analysis *it$ +clipse Memory ,naly#erTool M,T.

$n this section# we will show you how you can analy0e a JVM memory lea! problem by generating andanaly0ing a HotSpot JVM H=O; Heap Aump file.

/& o) -/(


27/127


6 real life case study will be used for that purpose* ?eblogic .4 memory lea! affecting the ?eblogic6dmin server.

1nvironment specifications

Java 11 server* Oracle ?eblogic Server .4 M2

Middleware OS* Solaris 23

Java VM* Sun HotSpot 2.8.3W44

latform type* Middle tier

Monitoring and troubleshooting tools

Xuest ;oglight &JVM and garbage collection monitoring(

9map &hprof E Heap Aump generation tool(

Memory 6naly0er 2.2 via $FM support assistant &hprof Heap Aump analysis(

latform type* Middle tier

Step K2 - ?@S .4 6dmin server JVM monitoring and lea! confirmation

The Xuest ;oglight Java 11 monitoring tool was quite useful to identify a Java Heap lea! from our?eblogic 6dmin server. 6s you can see below# the Java Heap memory is growing over time.

$f you are not using any monitoring tool for your ?eblogic environment# my recommendation to you isto at least enable verbose*gc of your HotSpot VM. lease visit the Java verbose*gc tutorial below onthis sub9ect for more detailed instructions.

Step K4 - +enerate a Heap Aump from your lea!ing JVM

/( o) -/(


28/127


;ollowing the discovery of a JVM memory lea!# the goal is to generate a Heap Aump file &binaryformat( by using the Sun JAC 9map utility.

** Ple!se note t!t (m!p )e!p ump gener!tion +ill c!use your "- to $ecome unresponsi#e sople!se ensure t!t no more tr!ffic is sent to your !ffected . le!&ing "- $efore running te (m!p utility**

This command will generate a Heap Aump binary file &heap.bin( of your lea!ing JVM. The si0e of thefile and elapsed time of the generation process will depend of your JVM si0e and machinespecifications E speed.

;or our case study# a binary Heap Aump file of P4+F was generated in about 2 hour elapsed time.

Sun HotSpot 2.8E2.:E2. Heap Aump file will also be generated automatically as a result of aOutOfMemory1rror and by adding -*HeapAumpOnOutOfMemory1rror in your JVM start-uparguments.

Step K) - @oad your Heap Aump file in Memory 6naly0er tool

$t is now time to load your Heap Aump file in the Memory 6naly0er tool. The loading process will ta!eseveral minutes depending of the si0e of your Heap Aump and speed of your machine.

/+ o) -/(

/bin/jmap -heap:format=b


29/127


/: o) -/(


30/127


Step K5 - 6naly0e your Heap Aump

The Memory 6naly0er provides you with many features# including a @ea! Suspect report. ;or this casestudy# the Java Heap histogram was used as a starting point to analy0e the lea!ing ob9ects and thesource.

6 o) -/(


31/127


;or our case study# 9ava.lang.String and charYZ data were found as the lea!ing Ob9ects. 'ow questionis what is the source of the lea! e.g. references of those lea!ing Ob9ects. Simply right clic! over yourlea!ing ob9ects and select RR @ist Ob9ects R with incoming references.

- o) -/(


32/127


6s you can see# 9ava/.management.Ob9ect'ame ob9ects were found as the source of the lea!ingString > charYZ data. The ?eblogic 6dmin server is communicating and pulling stats from its managedservers via MFeans E JM which create 9ava/.management.Ob9ect'ame for any MFean ob9ect type.'ow question is why ?eblogic .4 is not releasing properly such Ob9ects

=oot cause* ?eblogic 9ava/.management.Ob9ect'ame lea!Q

;ollowing our Heap Aump analysis# a review of the ?eblogic !nown issues was performed which didreveal the following ?eblogic .4 bug below*

?eblogic Fug $A*,=)4):N Aescription* Memory lea! of 9ava/.management.Ob9ect'ame ob9ects on the 6dministration

Server used to cause OutOfMemory error on the 6dministration Server.

6ffected ?eblogic version&s(* ?@S .4

;i/ed in* ?@S 23 M2

This finding was quite conclusive given the perfect match of our Heap Aump analysis# ?@S version

/ o) -/(
http://www.javacodegeeks.com/http://download.oracle.com/docs/cd/E11035_01/wls100/issues/known_resolved.htmlhttp://download.oracle.com/docs/cd/E11035_01/wls100/issues/known_resolved.htmlhttp://download.oracle.com/docs/cd/E11035_01/wls100/issues/known_resolved.htmlhttp://www.javacodegeeks.com/


33/127


and this !nown problem description.

Hopefully# this tutorial along with the case study has helped you understand how you can pinpoint thesource of a Java Heap lea! using 9map and the Memory 6naly0er tool.

JVM "erbose )C output tutorial

This section will provide you with a detailed tutorial on how to enable and read Java HotSpot VMverbose gc output data.

$ recommend that you compile and run the sample program on your end as well.

?e also recently created a Java verbose +, tutorial videoe/plaining this analysis process.

Tutorial specifications and tools

OS* ?indows - :5-bit

Java VM* Sun Java HotSpot &build 42.3-b2(

$A1* 1clipse Java 11 $A1 for ?eb Aeveloper v5.2

Step K2 - ,ompile our sample Java program

?e created a sample Java program in order to load the Java Heap and trigger an e/plicit +, in orderto generate some interesting verbose +, output. This program is simply loading about ) millioninstances of 9ava.lang.String in a static Map data structure and triggers an e/plicit +, &viaSystem.gc&(( followed by the removal of 4 million instances along with a second e/plicit +, beforee/iting.

o) -/(
http://www.javacodegeeks.com/http://www.youtube.com/watch?v=7CJCMKNoICEhttp://www.youtube.com/watch?v=7CJCMKNoICEhttp://www.youtube.com/watch?v=7CJCMKNoICEhttp://www.javacodegeeks.com/


34/127


# o) -/(

packageorg.ph.javaee.tools.jdk7;

importjava.util.Map;

importjava.util.HashMap;

/*** JavaHeapVerboseGCTest

* @authorPierreHuguesCharbo!!eau*

*/

publicclassJavaHeapVerboseGCTest "

privatestaticMap#$tri!g% $tri!g& mapContainer' newHashMap#$tri!g% $tri!g&();

/**

* @paramargs

*/

publicstaticvoidai!($tri!g+, args) "

$-ste.out.pri!tl!(Java 7 Hot$pot Verbose GC Test Progra v.0);

$-ste.out.pri!tl!(1uthor2 PierreHugues Charbo!!eau);

$-ste.out.pri!tl!(http2//javaeesupportpatter!s.blogspot.3o/ );

$tri!g stri!g4ataPre5i6 ' stri!g4ataPre5i6;

// oad Java Heap 8ith 9 M java.la!g.$tri!g i!sta!3es

for(inti'0; i#9000000; i::) "

$tri!g !e8$tri!g4ata ' stri!g4ataPre5i6 : i;

mapContainer.put(!e8$tri!g4ata% !e8$tri!g4ata);

$-ste.out.pri!tl!(M1P si


35/127


Step K4 - 1nable verbose +, via the JVM start-up arguments

The ne/t step is to enable the verbose +, via the JVM start-up arguments and specify a name andlocation for our +, log file.

Step K) - 1/ecute our sample Java program

6t this point# it is now time to e/ecute our sample program and generate the JVM verbose +, output.

; o) -/(


36/127


Verbose +, output high level analysis

$t is now time to review the generated +, output.

;irst# letBs start with the raw data. 6s you can see below# the +, output is divided into ) main sections*

8 Minor collections &"oung+en space collections( identified as S"oung+en

4 Ma9or collections &triggered by System.gc&(( identified as ;ull +, &System(

6 detailed Java Heap brea!down of each memory space

& o) -/(


37/127


Verbose +, data interpretation and sequence

6s you can see from the verbose +, output# the Old+en space was at )53 MF after the initial loadingof )M String instances in our HashMap. $t did go down to 24: MF following the removal of 4M String

( o) -/(

0.437: [GC [PSYoungGen: 262208K->43632K(305856K)]

262208K->137900K(1004928K), 0.1396040 e!]

["#$e: ue%&0.45 '&0.01, %e&0.14 e!]


400108K->291080K(1004928K), 0.2197630 e!]

["#$e: ue%&0.56 '&0.03, %e&0.22 e!]


412192K->340488K(1004928K), 0.0878209 e!]

["#$e: ue%&0.37 '&0.00, %e&0.09 e!]

1.188: [*u GC (S'+e$) [PSYoungGen: 43632K->0K(305856K)]

[PSGen: 296856K->340433K(699072K)]

340488K->340433K(1004928K)

[PSPe%$Gen: 1554K->1554K(16384K)], 0.4053311 e!]

["#$e: ue%&0.41 '&0.00, %e&0.40 e!]


602641K->340449K(1004928K), 0.0326756 e!]

["#$e: ue%&0.09 '&0.00, %e&0.03 e!]


432556K->340433K(1004928K), 0.0161477 e!]

["#$e: ue%&0.06 '&0.00, %e&0.02 e!]

2.020: [*u GC (S'+e$) [PSYoungGen: 0K->0K(305856K)]

[PSGen: 340433K->125968K(699072K)]340433K->125968K(1004928K)

[PSPe%$Gen: 1555K->1555K(16384K)], 0.2302415 e!]

["#$e: ue%&0.23 '&0.00, %e&0.23 e!]

e/PSYoungGen +o+ 305856K, ue 5244K [03!0000, 053010000, 053010000)

een /!e 262208K, 2 ue [03!0000,03168,040000)

%o$ /!e 43648K, 0 ue [040000,040000,050570000)+o /!e 43648K, 0 ue [050570000,050570000,053010000)PSGen +o+ 699072K, ue 125968K [013010000, 03!0000, 03!0000)

oe!+ /!e 699072K, 18 ue [013010000,011408,03!0000)

PSPe%$Gen +o+ 16384K, ue 1560K [00010000, 010010000, 013010000)oe!+ /!e 16384K, 9 ue [00010000,0019600,010010000)


38/127


instances.

'ow find below e/planation and snapshots on how you can read the +, output data in more detail foreach Java Heap space.

KK "oung+en space analysis

KK Old+en space analysis

KK erm+en space analysis

+ o) -/(


39/127


KK Java Heap brea!down analysis

Hopefully this sample Java program and verbose +, output analysis has helped you understand howto read and interpret this critical data.

: o) -/(


40/127


,naly#ing t$read dumps

Introduction to thread dump analysis

This section will teach you how to analy0e a JVM Thread Aump and pinpoint the root cause of yourproblem&s(. ;rom my perspective# Thread Aump analysis is the most important s!illset to master forany individual involved in Java 11 production support. The amount of information that you can derivefrom Thread Aump snapshots is often much beyond than what you can thin! of.

"ou may find complementary training videos hereand here.

Fefore going deeper into Thread Aump analysis and problem patterns# it is very important that youunderstand the fundamentals.

Java VM overview

The Java virtual machine is really the foundation of any Java 11 platform. This is where yourmiddleware and applications are deployed and active.

The JVM provides the middleware software and your Java E Java 11 program with*

6 runtime environment for your Java E Java 11 program &bytecode format(.

Several program features and utilities &$O facilities# data structure# Threads management#security# monitoring etc.(.

Aynamic memory allocation and management via the garbage collector.

"our JVM can reside on many OS &Solaris# 6$# ?indows etc.( and depending of your physical serverspecifications# you can install 2...n JVM processes per physical E virtual server.

JVM and Middleware software interactions

;ind below a diagram showing you a high level interaction view between the JVM# middleware andapplication&s(.

#6 o) -/(
http://www.javacodegeeks.com/http://www.youtube.com/watch?v=YQgmF8I-zhkhttp://www.youtube.com/watch?v=YQgmF8I-zhkhttp://youtu.be/3dKufRRT_3Ehttp://www.youtube.com/watch?v=YQgmF8I-zhkhttp://youtu.be/3dKufRRT_3Ehttp://www.javacodegeeks.com/


41/127


This is showing you a typical and simple interaction diagram between the JVM# middleware andapplication. 6s you can see# the Threads allocation for a standard Java 11 application are donemainly between the middleware !ernel itself and JVM &there are some e/ceptions when applicationitself or some 6$s create Threads directly but this is not common and must be done very carefully(.

6lso# please note that certain Threads are managed internally within the JVM itself such as +,&garbage collection( Threads in order to handle concurrent garbage collections.

Since most of the Thread allocations are done by the Java 11 container# it is important that youunderstand and recogni0e the Thread Stac! Trace and identify it properly from the Thread Aump data.This will allow you to understand quic!ly the type of request that the Java 11 container is attemptingto e/ecute.

;rom a Thread Aump analysis perspective# you will learn how to differentiate between the differentThread ools found from the JVM and identify the request type.

#- o) -/(


42/127


JVM Thread Aump - what is itL

6 JVM Thread Aump is a snapshot ta!en at a given time which provides you with a complete listing ofall created Java Threads.

1ach individual Java Thread found gives you information such as*

- Thread nameD often used by middleware vendors to identify the Thread $d along with its associatedThread ool name and state &running# stuc! etc.(

- Thread type > prioritye/*e$on /%#o&3 ** middle+!re soft+!res typic!lly cre!te teir Tre!ds!s d!emon me!ning teir Tre!ds !re running in $!c&ground/ pro#iding ser#ices to its user e0g0 your

"!#! EE !pplic!tion **

- Java Thread $Ae/*+#&0000000011e52800 ** Tis is te "!#! Tre!d 1d o$t!ined #i!(!#!0l!ng0Tre!d0get1d !nd usu!lly implemented !s !n !utoincrementing long 500n**

- 'ative Thread $Ae/*n#&0251!** 6ruci!l inform!tion !s tis n!ti#e Tre!d 1d !llo+s you tocorrel!te for ex!mple +ic Tre!ds from !n OS perspecti#e !re using te most 6P7 +itin your "-etc0 **

- Java Thread State and detaile/*#+#ng o% $on#+o% en+%' [0e5000].ng."%e.S++e: CK; (on oe!+ $on#+o%)

**8llo+s to 9uic&ly le!rn !$out Tre!d st!te !nd its potenti!l current $loc&ing condition **

- Java Thread Stac! TraceD this is by far the most important data that you will find from the ThreadAump. This is also where you will spent most of your analysis time since the Java Stac! Traceprovides you with 3 of the information that you need in order to pinpoint root cause of many

problem pattern types as you will learn later in the training sessions

- Java Heap brea!downD starting with HotSpot VM 2.:# you will also find at the bottom of the ThreadAump snapshot a brea!down of the HotSpot memory spaces utili0ation such as your Java Heap&"oung+en# Old+en( > erm+en space. This is quite useful when e/cessive +, is suspected as apossible root cause so you can do out-of-the-bo/ correlation with Thread data E patterns found

#/ o) -/(


43/127


Thread Aump brea!down overview

$n order for you to better understand# find below a diagram showing you a visual brea!down of aHotSpot VM Thread Aump and its common Thread ools found*

# o) -/(

e/PSYoungGen+o+ 466944K, ue 178734K [045!00000, 070800000,070800000)

een /!e 233472K, 76 ue

[045!00000,0507!50,054000000)%o$ /!e 233472K, 0 ue[062400000,062400000,070800000)

+o /!e 233472K, 0 ue[054000000,054000000,062400000)PSGen +o+ 1400832K, ue 1400831K [0e0400000,045!00000, 045!00000)

oe!+ /!e 1400832K, 99 ue[0e0400000,0458,045!00000)PSPe%$Gen +o+ 262144K, ue 248475K [0e0400000,0ee0400000, 0e0400000)

oe!+ /!e 262144K, 94 ue[0e0400000,0e6608,0ee0400000)


44/127


6s you can there are several pieces of information that you can find from a HotSpot VM Thread Aump.Some of these pieces will be more important than others depending of your problem pattern.

;or now# find below a detailed e/planation for each Thread Aump section as per our sampleHotSpotThread Aump*

K ;ull thread dump identifierThis is basically the unique !eyword that you will find in your middleware E standalong Java standardoutput log once you generate a Thread Aump &e/* via !ill -) [$AR for G'$(. This is the beginning ofthe Thread Aump snapshot data.

## o) -/(

*u +%e u$/


45/127


K Java 11 middleware# third party > custom application ThreadsThis portion is the core of the Thread Aump and where you will typically spend most of your analysistime. The number of Threads found will depend on your middleware software that you use# third partylibraries &that might have its own Threads( and your application &if creating any custom Thread# whichis generally not a best practice(.

$n our sample Thread Aump# ?eblogic is the middleware used. Starting with ?eblogic .4# a self-tuning Thread ool is used with unique identifier %Bweblogic.!ernel.Aefault &self-tuning(%

K HotSpot VM Thread

This is an internal Thread managed by the HotSpot VM in order to perform internal native operations.Typically you should not worry about this one unless you see high ,G &via Thread Aump > prstat Enative Thread id correlation(.

K HotSpot +, Thread

?hen using HotSpot parallel +, &quite common these days when using multi physical coreshardware(# the HotSpot VM create by default or as per your JVM tuning a certain K of +, Threads.These +, Threads allow the VM to perform its periodic +, cleanups in a parallel manner# leading toan overall reduction of the +, timeD at the e/pense of increased ,G utili0ation.

This is crucial data as well since when facing +, related problems such as e/cessive +,# memorylea!s etc# you will be able to correlate any high ,G observed from the OS E Java process&es( withthese Threads using their native id value &nid73/)(.

#; o) -/(

?[S"@A;Y] e!u+e"%e: B414B o% ueue: Beog#!.De%ne.;eu+ (e-+un#ng)B? e$on /%#o&3 +#&0000000010916800 n#&02613 #n e!+.#+()[0e9e000]

.ng."%e.S++e: E@F"FAG (on oe!+ $on#+o%)

+ .ng.e!+.#+(A+#e =e+o)

- #+#ng on 02744e0> ( eog#!.o%D.e!u+e"%e)

+ .ng.e!+.#+(e!+.:485) + eog#!.o%D.e!u+e"%e.#+*o%Heue+(e!u+e"%e.:160)

- o!De 02744e0> ( eog#!.o%D.e!u+e"%e)

+ eog#!.o%D.e!u+e"%e.%un(e!u+e"%e.:181)

?= Pe%#o#! "D "%e? /%#o&3 +#&00000000101238800 n#&019 #+#ng on!on#+#on

"GC task thread#0 (ParallelGC)" prio=3 tid=0x0000000100120000 nid=0x3 runnable

"GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100131000 nid=0x4 runnable


46/127


K J'$ global references count

J'$ &Java 'ative $nterface( global references are basically Ob9ect references from the native code to aJava ob9ect managed by the Java garbage collector. $ts role is to prevent collection of an ob9ect that isstill in use by native code but technically with no %live% references in the Java code.

$t is also important to !eep an eye on J'$ references in order to detect J'$ related lea!s. This canhappen if you program use J'$ directly or using third party tools li!e monitoring tools which are proneto native memory lea!s.

K Java Heap utili0ation view

This data was added bac! to JAC 2 .: and provides you with a short and fast view of your HotSpotHeap. $ find it quite useful when troubleshooting +, related problems along with H$+H ,G since youget both Thread Aump > Java Heap in a single snapshot allowing you to determine &or to rule out( anypressure point in a particular Java Heap memory space along with current Thread computing currentlybeing done at that time. 6s you can see in our sample Thread Aump# the Java Heap Old+en is ma/edoutQ

$n order for you to quic!ly identify a problem pattern from a Thread Aump# you first need to understandhow to read a Thread Stac! Trace and how to get the storyI right. This means that if $ as! you to tell

me what the Thread K)N is doingD you should be able to precisely answerD including if Thread Stac!Trace is showing a healthy &normal( vs. hang condition.

Java Stac! Trace revisited

Most of you are familiar with Java stac! traces. This is typical data that we find from server andapplication log files when a Java 1/ception is thrown. $n this conte/t# a Java stac! trace is giving usthe code e/ecution path of the Thread that triggered the Java 1/ception such as a

#& o) -/(


47/127


9ava.lang.'o,lassAef;ound1rror# 9ava.lang.'ullpointer1/ception etc. Such code e/ecution pathallows us to see the different layers of code that ultimately lead to the Java 1/ception.

Java stac! traces must always be read from bottom-up*

The line at the bottom will e/pose the originator of the request such as a Java E Java 11container Thread.

The first line at the top of the stac! trace will show you the Java class where that last 1/ceptiongot triggered.

@etBs go through this process via a simple e/ample. ?e created a sample Java program simplye/ecuting some ,lass methods calls and throwing an 1/ception. The program output generated is asper below*

Java program JavaSTSimulator is invo!ed &via the %main% Thread(

The simulator then invo!es method call&( from ,lass2

,lass2 method call&( then invo!es ,lass4 method call&(

,lass4 method call&(throws a Java 1/ception* 9ava.lang.$llegal6rgument1/ception

The Java 1/ception is then displayed in the log E standard output

6s you can see# the code e/ecution path that lead to this 1/ception is always displayed from bottom-up.

The above analysis process should be well !nown for any Java programmer. ?hat you will see ne/t isthat the Thread Aump Thread stac! trace analysis process is very similar to above Java stac! traceanalysis.

Thread Dump: Thread Stack Trace analysis

Thread Aump generated from the JVM provides you with a code level e/ecution snapshot of all the%created% Threads of the entire JVM process. ,reated Threads does not mean that all these Threadsare actually doing something. $n a typical Thread Aump snapshot generated from a Java 11 containerJVM*

Some Threads could be performing raw computing tas!s such as M@ parsing# $O E dis!access etc.

Some Threads could be waiting for some bloc!ing $O calls such as a remote ?eb Service call#

#( o) -/(


48/127


a AF E JAF, query etc.

Some Threads could be involved in garbage collection at that time e.g. +, Threads

Some Threads will be waiting for some wor! to do &Threads not doing any wor! typically go inwait&( state(

Some Threads could be waiting for some other Threads wor! to complete e.g. Threads waitingto acquire a monitor loc! &synchroni0ed bloc!\]( on some ob9ects

6 Thread stac! trace provides you with a snapshot of its current e/ecution. The first line typicallyincludes native information of the Thread such as its name# state# address etc. The current e/ecutionstac! trace has to be read from bottom-up. lease follow the analysis process below. The moree/perience you get with Thread Aump analysis# the faster you will able to read and identify veryquic!ly the wor! performed by each Thread*

Start to read the Thread stac! trace from the bottom.

;irst# identify the originator &Java 11 container Thread# custom Thread# +, Thread# JVMinternal Thread# standalone Java program %main% Thread etc.(.

The ne/t step is to identify the type of request the Thread is e/ecuting &?eb6pp# ?eb Service#JMS# =emote 1JF &=M$(# internal Java 11 container etc.(.

The ne/t step is to identify from the e/ecution stac! trace your application module&s( involvedthe actual core wor! the Thread is trying to perform. The comple/ity of analysis will depend ofthe layers of abstraction of your middleware environment and application.

The ne/t step is to loo! at the last P23-43 lines prior to the first line. $dentify the protocol orwor! the Thread is involved with e.g. HTT call# Soc!et communication# JAF, or rawcomputing tas!s such as dis! access# class loading etc.

The ne/t step is to loo! at the first line. The first line usually tells a @OT on the Thread statesince it is the current piece of code e/ecuted at the time you too! the snapshot.

The combination of the last 4 steps is what will give you the core of information to conclude ofwhat wor! and E or hanging condition the Thread is involved with.

'ow find below a visual brea!down of the above steps using a real e/ample of a Thread Aump Threadstac! trace captured from a JFoss 8 production environment. $n this e/ample# many Threads wereshowing a similar problem pattern of e/cessive $O when creating new instances of J6-?S Serviceinstances.

#+ o) -/(


49/127


6s you can see# the last 23 lines along with the first line will tell us what hanging or slow condition the

Thread is involved with# if any. The lines from the bottom will give us detail of the originator and type ofrequest.

Ja"a T$read C(/ analysis on 0indo*s

This section will provide you with a tutorial on how you can quic!ly pinpoint the Java thread

#: o) -/(


50/127


contributors to a high ,G problem on the ?indows OS. ?indows# li!e other OS such as @inu/#Solaris > 6$ allow you to monitor the ,G utili0ation at the process level but also for individualThread e/ecuting a tas! within a process.

;or this tutorial# we created a simple Java program that will allow you to learn this technique in a stepby step manner.

Troubleshooting tools

The following tools will be used below for this tutorial*

?indows rocess 1/plorer &to pinpoint high ,G Thread contributors(

JVM Thread Aump &for Thread correlation and root cause analysis at code level(

High ,G simulator Java program

The simple program below is simply looping and creating new String ob9ects. $t will allow us to performthis ,G per Thread analysis. $ recommend that you import it in an $A1 of your choice e.g. 1clipseand run it from there. "ou should observe an increase of ,G on your ?indows machine as soon asyou e/ecute it.

;6 o) -/(


51/127


;- o) -/(

packageorg.ph.javaee.tool.3pu;

/**

* HighCPA$iulator

* @authorPierreHuguesCharbo!!eau * http2//javaeesupportpatter!s.blogspot.3o

*

*/

publicclassHighCPA$iulator "

privatefinalstaticintNB_ITERATIONS' B00000000;

// DE data 5ootpri!t

privatefinalstatic$tri!g DATA_PREFIX'datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatada

tadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata ;

publicstaticvoidai!($tri!g+, args) "

$-ste.out.pri!tl!(HFGH CPA $iulator .0);

$-ste.out.pri!tl!(1uthor2 PierreHugues Charbo!!eau);

$-ste.out.pri!tl!(http2//javaeesupportpatter!s.blogspot.3o/ );

try"

for(inti ' 0; i # NB_ITERATIONS; i::) "

// Per5or soe $tri!g a!ipulatio!sto slo8do8! a!d e6pose loopi!g pro3ess...

$tri!g data ' DATA_PREFIX: i;

catch(Thro8able a!-) "

$-ste.out.pri!tl!(A!e6pe3ted =63eptio!> :a!-.getMessage()

: +: a!- :,);

$-ste.out.pri!tl!(HighCPA$iulator do!e>);


52/127


Step K2 - @aunch rocess 1/plorer

The rocess 1/plorer tool visually shows the ,G usage dynamically. $t is good for live analysis. $fyou need historical data on ,G per Thread then you can also use ?indows perfmon with rocessor Time > Thread $d data counters. "ou can download rocess 1/plorer from the lin! here.

$n our e/ample# you can see that the 1clipse 9avaw.e/e process is now using P48 of total ,Gutili0ation following the e/ecution of our sample program.

Step K4 - @aunch rocess 1/plorer Threads view

The ne/t step is to display the Threads view of the 9avaw.e/e process. Simply right clic! on the9avaw.e/e process and select roperties. The Threads view will be opened as per below snapshot*

;/ o) -/(
http://www.javacodegeeks.com/http://technet.microsoft.com/en-us/sysinternals/bb896653http://technet.microsoft.com/en-us/sysinternals/bb896653http://www.javacodegeeks.com/


53/127


The first column is the Thread $d &decimal format(

The second column is the ,G utili0ation used by each Thread

The third column is also another counter indicating if Thread is running on the ,G

$n our e/ample# we can see our primary culprit is Thread $d K8: using P 48 of ,G.

Step K) - +enerate a JVM Thread Aump

6t this point# rocess 1/plorer will no longer be useful. The goal was to pinpoint one or multiple JavaThreads consuming most of the Java process ,G utili0ation which is what we achieved. $n order togo the ne/t level in your analysis you will need to capture a JVM Thread Aump. This will allow you tocorrelate the Thread $d with the Thread Stac! Trace so you can pinpoint that type of processing isconsuming such high ,G.

JVM Thread Aump generation can be done in a few manners. $f you are using J=oc!it VM you cansimply use the 9rcmd tool as per below e/ample*

; o) -/(


54/127


Once you have the Thread Aump data# simply search for the Thread $d and locate the Thread Stac!Trace that you are interested in.

;or our e/ample# the Thread %Main Thread% which was fired from 1clipse got e/posed as the primaryculprit which is e/actly what we wanted to demonstrate.

Step K5 - 6naly0e the culprit Thread&s( Stac! Trace and determine root cause

6t this point you should have everything that you need to move forward with the root cause analysis."ou will need to review each Thread Stac! Trace and determine what type of problem you are dealingwith. That final step is typically where you will spend most of your time and problem can be simplesuch as infinite looping or comple/ such as garbage collection related problems.

$n our e/ample# the Thread Aump did reveal the high ,G originates from our sample Java program

around line )2. 6s e/pected# it did reveal the looping condition that we engineered on purpose for thistutorial.

Case Study - Too many open files

This case study describes the complete root cause analysis and resolution of a ;ile Aescriptor &Too

;# o) -/(

?=#n "%e? #&1 #&04 +#&5996/%#o&5 #e, n+#eJo!De

+ o%gI/IeeI+ooI!/uI#gCPS#$u+o%.$#n

(#gCPS#$u+o%.:31)

+ %o!D#+I$IHAF.!2(FFFFF)(A+#e =e+o)

-- en o +%!e


55/127


many open files( related problem that we faced following a migration from Oracle 6@SF 4.: running onSolaris OS to Oracle OSF 22g running on 6$.

This section will also provide you with proper 6$ OS commands you can use to troubleshoot andvalidate the ;ile Aescriptor configuration of your Java VM process.

1nvironment specifications

Java 11 server* Oracle Service Fus 22g

Middleware OS* $FM 6$ :.2

Java VM* $FM J=1 2.:.3 S= - :5 bit

latform type* Service Fus - Middle Tier

roblem overview

roblem type* 9ava.net.Soc!et1/ception* Too many open files error was observed under heavy load

causing our Oracle OSF managed servers to suddenly hang.

Such problem was observed only during high load and did require our support team to ta!e correctiveaction e.g. shutdown and restart the affected ?eblogic OSF managed servers.

+athering and validation of facts

6s usual# a Java 11 problem investigation requires gathering of technical and non technical facts sowe can either derive other facts andEor conclude on the root cause. Fefore applying a correctivemeasure# the facts below were verified in order to conclude on the root cause*

?hat is the client impactL )1G)/ :ull "- !ng

=ecent change of the affected platformL ;es, recent migr!tion from 8 OS

6ny recent traffic increase to the affected platformL ?o

?hat is the health of the ?eblogic serverL8ffected m!n!ged ser#ers +ere no longerresponsi#e !long +it closure of te %e$logic )TTP Ser#er Soc&et port

Aid a restart of the ?eblogic $ntegration server resolve the problemL ;es $ut tempor!rily only

,onclusion K2* The problem appears to be load related

?eblogic server log files review

6 quic! review of the affected managed servers log did reveal the error below*

This error indicates that our Java VM process was running out of ;ile Aescriptor. This is a severecondition that will affect the whole Java VM process and cause ?eblogic to close its internal ServerSoc!et port &HTTEHTTS port( preventing any further inbound > outbound communication to theaffected managed server&s(.

;; o) -/(

.ne+.So!De+!e/+#on: "oo $n' o/en #e


56/127


;ile Aescriptor - ?hy so important for an Oracle OSF environmentL

The ;ile Aescriptor capacity is quite important for your Java VM process. The !ey concept you mustunderstand is that ;ile Aescriptors are not only required for pure ;ile Handles but also for inbound andoutbound Soc!et communication. 1ach new Java Soc!et created to &inbound( or from &outound( yourJava VM by ?eblogic !ernel Soc!et Mu/er requires a ;ile Aescriptor allocation at the OS level.

6n Oracle OSF environment can require a significant number of Soc!ets depending how muchinbound load it receives and how much outbound connections &Java Soc!ets( it has to create in orderto send and receive data from e/ternal E downstream systems &System 1nd oints(.

;or that reason# you must ensure that you allocate enough ;ile Aescriptors E Soc!ets to your Java VMprocess in order to support your daily loadD including problematic scenarios such as sudden slowdownof e/ternal systems which typically increase the demand on the ;ile Aescriptor allocation.

=untime ;ile Aescriptor capacity chec! for Java VM and 6$ OS

;ollowing the discovery of this error# our technical team did perform a quic! review of the currentobserved runtime ;ile Aescriptor capacity > utili0ation of our OSF Java VM processes. This can bedone easily via the following 6$ command*

KK Java VM process ;ile Aescriptor total capacity

KK Java VM process ;ile Aescriptor current utili0ation

6s you can see# the current capacity was found at 4333D which is quite low for a medium si0e OracleOSF environment. The average utili0ation under heavy load was also found to be quite close to theupper limit of 4333.

The ne/t step was to verify the default 6$ OS ;ile Aescriptor limit via the command*

,onclusion K4* The current ;ile Aescriptor limit for both OS and OSF Java VM appears to be quite lowand setup at 4333. The ;ile Aescriptor utili0ation was also found to be quite close to the upper limit

;& o) -/(

procfiles | grep rlimit & lsof -p | wc -l

>> procfiles 5425732 | grep rlimit

Current rlimit: 2000file descriptors

>> lsof -p | wc -l

1920

>> ulimit -S -n

2000


57/127


which e/plains why so many JVM failures were observed at pea! load.

?eblogic ;ile Aescriptor configuration review

The ;ile Aescriptor limit can typically be overwritten when you start your ?eblogic Java VM. Suchconfiguration is managed by the ?@S core layer and script can be found at the following location*

=oot cause* ;ile Aescriptor override only wor!ing for Solaris OSQ

6s you can see with the script screenshot below# the override of the ;ile Aescriptor limit via ulimit isonly applicable for Solaris OS &SunOS( which e/plains why our current OSF Java VM running on 6$OS did end up with the default value of 4333 vs. our older 6@SF 4.: environment running on SolarisOS which had a ;ile Aescriptor limit of :88):.

Solution* script twea!ing for 6$ OS

The resolution of this problem was done by modifying the ?eblogic comm1nv script as per below.This change did ensure a configuration of :88): ;ile Aescriptor &from 4333(D including for the 6$ OS*

;( o) -/(

EJ=>Ie%e%J10.3I!o$$onI#nI!o$$n.


58/127


** Ple!se note t!t te !cti#!tion of !ny c!nge to te %e$logic :ile escriptor configur!tion re9uires! rest!rt of $ot te ?ode -!n!ger if used !long +it te m!n!ged ser#ers0 **

6 runtime validation was also performed following the activation of the new configuration which did

confirm the new active ;ile Aescriptor limit*

'o failure has been observed since then.

,onclusion and recommendations

?hen upgrading your ?eblogic Java 11 container to a new version# please ensure that you verifyyour current ;ile Aescriptor limit as per the above case study. ;rom a capacity planning perspective#please ensure that you monitor your ;ile Aescriptor utili0aiton on a regular basis in order to identifyany potential capacity problem# Soc!et lea! etc.

)C o"er$ead limit e1ceeded 2 ,nalysis and (atterns

This section will provide you with a description of this new JVM 2.: HotSpot OutOfMemory1rror errormessage and how you should attac! this problem until its resolution.

;+ o) -/(

>> procfiles 6416839 | grep rlimit

Current rlimit: 65536 file descriptors


59/127


"ou can also refer to this post for a real case studyon a Java Heap problem &OutOf-emoryError@ G6o#ere!d limit exceeded( affecting a JFoss production system.

9ava.lang.OutOfMemory1rror* +, overhead limit e/ceeded - what is itL

1veryone involved in Java 11 production support is familiar with OutOfMemory1rror problems sincethey are one of the most common problem type you can face. However# if your environment recentlyupgraded to Java HotSpot 2.: VM# you may have observed this error message in the logs*

(!#!0l!ng0OutOf-emoryError@ G6 o#ere!d limit exceeded.

+, overhead limit e/ceeded is a new policy that was added by default for the Java HotSpot VM 2.:only. $t basically allows the VM to detect potential OutOfMemory1rror conditions earlier and before itruns out of Java Heap spaceD allowing the JVM to abort the current Thread&s( processing with thisOOM error.

The official Sun statement is provided below*

Te p!r!llel . concurrent collector +ill tro+ !n OutOf-emoryError if too muc time is $eing spent ing!r$!ge collection@ if more t!n AC of te tot!l time is spent in g!r$!ge collection !nd less t!n 2Cof te e!p is reco#ered, !n OutOf-emoryError +ill $e tro+n0 Tis fe!ture is designed to pre#ent!pplic!tions from running for !n extended period of time +ile m!&ing little or no progress $ec!usete e!p is too sm!ll0 1f necess!ry, tis fe!ture c!n $e dis!$led $y !dding te option >>@7seG6O#ere!d


60/127


lease !eep in mind that this error is very li!ely to be a symptom of a JVM Heap E tuning problem somy recommendation to you is always to focus on the root cause as opposed to the symptom.

9ava.lang.OutOfMemory1rror* +, overhead limit e/ceeded - how can $ fi/ itL

"ou should not worry too much about the +, overhead limit error itself since itBs very li!ely 9ust asymptom E hint. ?hat you must focus on is on your potential Java Heap problem&s( such as JavaHeap lea!# improper Java Heap tuning etc. ;ind below a list of high level steps to troubleshoot further*

$f not done already# enabled verbose +, RR -verbose*gc

6naly0e the verbose +, output and determine the memory footprint of the Java HeapDincluding the ratio of "oung +en vs. Old +en. Having an old gen footprint too high will lead totoo many frequent ;ull +, and ultimately to the OOM* +, overhead limit e/ceeded.

6naly0e the verbose +, output or use a tool li!e J,onsole to determine if your Java Heap islea!ing over time. This can be observed via monitoring of the HotSpot old gen space.

@oo! at your young +en requirement as well# if you application generates a lot of short liveob9ects then your Java Heap space must be big enough in order for the VM to allocate a bigger"oung +en space.

$f facing a Java Heap lea! andEor if you have concern on your Old +en footprint then add thefollowing parameter to your start-up JVM arguments* -*HeapAumpOnOutOfMemory1rror.This will generate a Heap Aump &hprof format( on OOM event that you can analy0e using atool li!e Memory 6naly0er or Jhat.

Heap 6nalysis

'ow we will provide you with a sample program and a tutorial on how to analy0e your Java HotSpotHeap footprint using Memory 6naly0er following an OutOfMemory1rror. $ highly recommend that youe/ecute and analy

java jvm troubleshooting guide

Documents