performance & jvm & gc › ... › 09 › performance-jvm-gc.pdf · performance & jvm...

Performance & JVM & GC 1 of 36

Performance & JVM & GC Tips & Tricks & Troubleshoot

Table of Content --------------------------------------------------

Performance 3

Performance Optimization Architecture 3

Memory Leak Types 4

Top 10 Most Common Java Performance Problems 5

DataBase 5

Persistence Configuration 5

Caching 5

Pool Connections 5

Memory Optimization 5

Garbage Collector 5

Concurrency 5

Address Memory Leaks 5

J2EE Application Performance Optimization 6

Overview 6

Identify Bottlenecks 6

Use Performance Monitors Error! Bookmark not defined.

Application Server Tuning 8

Database Tuning 8

Code optimization 8

10 Application Performance Tuning Tips (Architect perspective) 8

Define the requirements (benchmarking) 8

Measure don’t guess 9

Automate 9

Only Optimize if Needed 9

Learn to Parallelize 9

Learn to Scale 9

Cache It 9

Right Abstraction 9

Top 10 Causes of Java EE Enterprise Performance Problems 10

Lack of proper capacity planning 10


Inadequate Java EE middleware environment specifications 12

Too many or poor integration with external systems 13

Lack of proper database SQL tuning & capacity planning 13

Application specific performance problems 13

Java EE middleware tuning problems 13

Insufficient proactive monitoring 13

Saturated hardware on common infrastructure 14

Network latency problems 15

Solving common Java EE performance problems 17

Out-of-memory errors 17

Java Memory problems 17

Problem Groups 17

Memory Leaks 17

Unnecessarily high memory usage 17

Inefficient object creation 17

Inefficient garbage collector behaviour 17

HTTP Session as Cache 18

ThreadLocal Memory Leak 19

Large Temporary Objects 20

Bad Garbage Collector Configuration 20

ClassLoader Leaks 20

Conclusion 21

Common Performance problems 21

Commonly seen reasons behind performance issues 22

Memory management algorithms/management in java 22

Explain types of references in Java? 23

Talk about garbage collector for various references! 27

Explain garbage collection on Remote Objects or Distributed Garbage collection. 28

Does OutOfMemoryError and StackOverFlowError cause JVM crash? 29

Different OutOfMemory errors!! 30

Why does the JVM crash with a core dump or a Dr.Watson error? 31

Memory Leaks Examples 31

Auto Boxing 31

JVM & Garbage Collection 34

What flags can I use to tune the JVM and GC? 34


Performance & JVM & GC Tips & Tricks & Troubleshoot

--------------------------------------------------

Performance

Performance Optimization Architecture

____________________________________________________________________________


Memory Leak Types

____________________________________________________________________________


Top 10 Most Common Java Performance Problems

● DataBase ○ Connection Pools

○ Indexes & Defragmentation

○ Views

○ Concurrency Controls (Optimistic & Pessimistic)

○ DB optimization techniques i.e. Tablespaces in Oracle

● Persistence Configuration ○ Lazy Loading (Right Fetch Types)

○ Enable Hibernate statistics & Improve Slow queries

○ Second Level Cache

○ Avoid N + 1 problems

○ Prefer Batch operations

● Caching ○ ORM Caches

○ Application Cache like Oracle Coherence

● Pool Connections ○ Database

○ Thread pool

○ EJB pools

● Memory Optimization

● Garbage Collector ○ Type of GC

● Concurrency

● Address Memory Leaks

____________________________________________________________________________


J2EE Application Performance Optimization

● Overview

○ Set Goal

○ Identify Problem Area

○ Follow Methodical & Focussed Path

● Identify Bottlenecks

○ Use load runners & stress test tools

● Use Performance Monitors


○ Perfmon tools in windows OS


● Application Server Tuning

○ Memory

○ GC tuning

○ Connection & Thread & EJB pools

○ Pre compiled JSP objects

○ NO hot deployment

● Database Tuning

○ SQL query optimization

○ Short transactions

○ Avoid select *

○ Avoid distinct

○ Avoid String operations like concat etc

○ Use Indices whenever possible

● Code optimization

○ Avoid sync blocks

○ Logging less or async log in production

○ Instead of create & Destroy, try to use object pool etc

○ Less overhead on HTTP session

○ Use HTTP forward instead of sendredirect (involves browser roundtrip)

____________________________________________________________________________

10 Application Performance Tuning Tips (Architect perspective)

● Define the requirements (benchmarking)

Performance requirements should be defined SMART

(https://en.wikipedia.org/wiki/SMART_criteria) – like “95% of the Login requests should respond

in less than 2 seconds measured on the web server“.


● Measure don’t guess

This tip is easy: You need a tool to measure and optimize your code. For Java this can be

included tools like jstat or jvisualvm. I prefer a good profiler like JProfiler and an APM solution

like AppDynamics for production analysis.

● Automate

Continuous Delivery pipeline which includes profiling and performance analysis of acceptance

and load tests and a performance report for each build/release – including comparison of

performance metrics for two builds.

● Only Optimize if Needed

● Learn to Parallelize

○ Scale Up

● Learn to Scale

○ Scale Out

○ Clusters

○ CDN (Functional Scalability)

● Cache It

○ Distributed Cache like MemCached or Oracle’s Coherence

○ CDN (Functional Scalability)

● Right Abstraction

This is also a simple rule: Don’t be too abstract in your application design.

E.g. don’t create a layer above your database that abstracts away all special features of

Oracle/DB2/MSSQL/……this will make your application slow, as you are not using the features

you have paid for. Yes, I know you did this to have less effort to replace the database if

needed…but believe me, this will not happen in the next 10 years and if it happens you will

have other problems…

____________________________________________________________________________


Top 10 Causes of Java EE Enterprise Performance Problems

● Lack of proper capacity planning

Capacity planning can be defined as a comprehensive and evolutive process measuring and

predicting current and future required IT environment capacity. A proper implemented capacity

planning process will not only ensure and keep track of current IT production capacity and

stability but also ensure that new projects can be deployed with minimal risk in the existing

production environment. Such exercise can also conclude that extra capacity (hardware,

middleware, JVM, tuning, etc.) is required prior to project deployment.

In my experience, this is often the most common "process" problem that can lead to short- and

long- term performance problems. The following are some examples.


● Inadequate Java EE middleware environment specifications

The second most common cause of performance problems I have observed for Java EE

enterprise systems is an inadequate Java EE middleware environment and / or infrastructure.

Not making proper decisions at the beginning of new platform can result in major stability

problems and increased costs for your client in the long term. For that reason, it is important to

spend enough time brainstorming on required Java EE middleware specifications. This exercise

should be combined with an initial capacity planning iteration since the business processes,

expected traffic, and application(s) footprint will ultimately dictate the initial IT environment

capacity requirements.

Now, find below typical examples of problems I have observed in my past experience:

● Deployment of too many Java EE applications in a single 32-bit JVM

● Deployment of too many Java EE applications in a single middleware domain

● Lack of proper vertical scaling and under-utilized hardware (e.g., traffic driven by one or

just a few JVM processes)

● Excessive vertical scaling and over-utilized hardware (e.g., too many JVM processes vs

available CPU cores and RAM)

● Lack of environment redundancy and fail-over capabilities

● Trying to leverage a single middleware and / or JVM for many large Java EE applications

can be quite attractive from a cost perspective. However, this can result in an operation

nightmare and severe performance problems such as excessive JVM garbage collection

and many domino effect scenarios (e.g., Stuck Threads) causing high business impact


(e.g., App A causing App B, App C, and App D to go down because a full JVM restart is

often required to resolve problems).

● Too many or poor integration with external systems

○ Like Rest, JDBC, EJB calls - use appropriate connection pool mechanism, adequate

number of calls etc

○ Timeout configuration

● Lack of proper database SQL tuning & capacity planning

● Application specific performance problems

○ Correct Collections & Algorithms

○ Object Scopes

○ Mutables

○ Singletons

○ Avoid Sync blocks

○ Lack Of Data Caching

○ Excessive Logging

● Java EE middleware tuning problems

○ Configure Thread pools

○ Connection Pools

○ EJB Pools

○ Thin Sessions

○ JDBC tuning

○ JMS tuning

● Insufficient proactive monitoring

Lack of monitoring is not actually "causing" performance problems, but it can prevent you from

understanding the Java EE platform capacity and health situation. Eventually, the environment

can reach a breakpoint, which may expose several gaps and problems (JVM memory leak, etc.).


From my experience, it is much harder to stabilize an environment after months or years of

operation as opposed to having proper monitoring, tools, and processes implemented from day

one.

That being said, it is never too late to improve an existing environment. Monitoring can be

implemented fairly easily. My recommendations follow.

● Review your current Java EE environment monitoring capabilities and identify

improvement opportunities.

● Your monitoring solution should cover the end-to-end environment as much as possible;

including proactive alerts.

● The monitoring solution should be aligned with your capacity planning process

discussed in our first section.

● Saturated hardware on common infrastructure

Another common source of performance problems is hardware saturation. This problem is often

observed when too many Java EE middleware environments along with its JVM processes are

deployed on existing hardware. Too many JVM processes vs. availability of physical CPU cores

can be a real problem killing your application performance. Again, your capacity planning

process should also take care of hardware capacity as your client business is growing.

My primary recommendation is to look at hardware virtualization. Such an approach is quite

common these days and has quite a few benefits such as reduced physical servers, data center

size, dedicated physical resources per virtual host, fast implementation, and reduced costs for

your client. Dedicated physical resources per virtual host is quite important since the last thing

you want is one Java EE container bringing down all others due to excessive CPU utilization.


● Network latency problems

Our last source of performance problems is the network. Major network problems can happen

from time to time such as router, switch, and DNS server failures. However, the more common

problems observed are typically due to regular or intermittent latency when working on a highly

distributed IT environment. The diagram below highlights an example of network latency gaps

between two geographic regions of a Weblogic cluster communicating with an Oracle database

server located in one geographic region only.


Intermittent or regular latency problems can definitely trigger some major performance

problems and affect your Java EE application in different ways.

● Applications using database queries with large datasets are fully exposed to network latency

due to high number of fetch iterations (back and forward across network).

● Applications dealing with large data payloads (such as large XML data) from external

systems are also exposed to network latency that can trigger intermittent high-response

time when sending and receiving responses.

● Java EE container replication process (clustering) can be affected and put at risk its failover

capabilities (e.g., multicast or unicast packet losses).

Tuning strategies such as JDBC row data "prefetch", XML data compression, and data

caching can help mitigate network latency. But such latency problems should be reviewed

closely when first designing the network topology of a new IT environment.

____________________________________________________________________________


Solving common Java EE performance problems

● Out-of-memory errors

One of the most common problems that plagues enterprise applications is the dreaded

OutOfMemoryError. The error is typically followed by one of the following:

● An application server crash

● Degraded performance

● A seemingly endless loop of repeated garbage collections that nearly halts processing

and usually leads to an application server crash

Regardless of the symptoms, you will most likely need to reboot the application server before

performance returns to normal.

____________________________________________________________________________

Java Memory problems

Problem Groups

● Memory Leaks

in Java are created by referencing objects that are no longer used. This easily happens when multiple

references to objects exist and developer forget to clear them, when the object is no longer needed.

● Unnecessarily high memory usage

aused by implementations consuming too much memory. This is very often a problem in web

applications where a large amount of state information is managed for “user comfort”. When the

number of active users increases, memory limits are reached very fast. Unbound or inefficiently

configured caches are another source of constant high memory usage.

● Inefficient object creation

easily results in a performance problem when user load increases, as the garbage collector must

constantly clean up the heap. This leads to unnecessarily high CPU consumption by the garbage

collector. As the CPU is blocked by garbage collection, application response times increases often

already under moderate load. This behaviour is also referred to as GC trashing.

● Inefficient garbage collector behavior

is caused by missing or wrong configuration of the garbage collector. The garbage collector will

take care that object are cleaned up. How and when this should happen must however be

configured by the programmer or system architect. Very often people simply “forget” to


properly configure and tune the garbage collector. I was involved in a number of performance

workshops where a “simple” parameter change resulted in a performance improvement of up

to 25 percent.

In most cases memory problems affect not only performance but also scalability. The higher the

amount of consumed memory per request, user or session the less parallel transactions can be

executed. In some cases memory problems also affect availability. When the JVM runs out of memory or

it is close to memory limits it will quit with an OutOfMemory error. This is when management enters

your office and you know you are in serious trouble.

Memory problems are often difficult to resolve for two reasons: In some case analysis will get complex

and difficult – especially if you are missing the right methodology to resolve them. Secondly their

foundation is often in the architecture of the application. Simple code changes will not help to resolve

them.

In order to make life easier I present a couple of memory antipatterns which are often found in real

world world applications. Those patterns should help to be able to already avoid memory problems

during development.

HTTP Session as Cache

This antipattern refers to the misuse of the HTTPSession object as a data cache. The session object

serves as means to store information which should “survive” a single HTTP request. This is also referred

to a as conversational state. Meaning data is stored over a couple of requests until it is finally processed.

This approach can be found in any non-trivial web application. Web applications have no other means

than storing this information on the server. Well, some information can be put into the cookie, but this

has a number of other implications.

It is important to keep as few data as possible and as short as possible. It can easily happen that the

session contains megabytes of object data. This immediately results in high heap usage and memory

shortages. At the same time the number of parallel users is very limited. The JVM will respond to an

increasing number of users with an OutOfMemoryError. Large user sessions have other performance

penalties as well. In case of session replication in clusters increased serialization and communication

effort will result in additional performance and scalability problems.

In some projects the answer to this kind of problems is increasing the amount of memory and switching

to 64bit JVMs. They cannot resisit the temptation of just increasing heap size up to several gigabytes.

However this is often only hiding symptoms than providing a cure to the real problem. This “solution” is

only temporal and also introduces a new problem. Bigger and bigger heaps make it more difficult to find

“real” memory problems. Memory dumps for very large heaps (greater 6 gigabytes) cannot be

processed by most available analysis tools. We at dynaTrace invested a lot of R&D effort to be able to


efficiently analyze large memory dumps. As this problem is gaining more and more importance a new

JSR specification is also addressing it.

Session caching problems often arise because the application architecture has not been clearly defined.

During development data is simply put into the session as it is comfortable. This very often happens in

an “add and forget” manner, as nobody ensures that this data is removed when no longer needed.

Normally unneeded session data should be handled by the session timeout. In enterprise applications

which are constantly under heavy use the session timeout, this will not work. Additionally very often

very high session timeouts are used – up to 24 hours – to provide additional “comfort” to users so that

they do not have to login again.

A practical example is putting selection choices from list, which have to be fetched from the database, in

the session. The intention is to avoid unnecessary database queries. (Smells like premature optimization

– doesn’t it). This results in several kilobytes being put into the session object for every single user.

While it is reasonable to cache this information the user session is definitely the wrong place for it.

Another example is abusing the Hibernate session for managing conversational state. The Hibernate

session object is simply put into the HTTP session to be have fast access to data. This however results in

much more data to be stored as necessary and the memory consumption per users rises significantly.

In modern AJAX applications conversational state can also be managed at the client side. Ideally this

leads to a stateless or nearly stateless server application which also scales significantly better.

ThreadLocal Memory Leak

ThreadLocal variables are used in Java to bind variables to a specific thread. This means every thread

gets it’s own single instance. This approach is used to handle status information within a thread. An

example would be user credentials. The lifecycle of a ThreadLocal variable is however related to the

lifecycle of the thread. ThreadLocal variables are cleaned up when the thread is terminated and

removed by the garbage collector – if not explicitly removed by the programmer.

Forgotten ThreadLocal variables can especially in application servers easily result in memory problems.

Application servers uses ThreadPools in avoid constant creation and destruction of threads. An

HTTPServletRequest for example gets a free thread assigned at runtime, which is passed back to the

ThreadPool after execution. If the application logic uses ThreadLocal variables and forget to explicitly

remove them, the memory will not be freed up.

Depending on the pool size – in production systems this can be several hundred threads – and the size

of the objects reference by the ThreadLocal variable this can lead to problems. A pool of 200 threads

and a ThreadLocal size of 5MB will in the worst case lead to 1 GB of unnecessarily occupied memory.


This will immediately result in high GC activity leading to bad response times and potentially to an

OutOfMemoryError.

A practical example was a bug in jbossws version 1.2.0 which was fixed in version 1.2.1 – “DOMUtils

doesn’t clear thread locals”. The problem was a ThreadLocal variable which referenced a parsed

document having a size of 14 MB.

Large Temporary Objects

Large temporary objects can in the worst case also lead to OutOfMemoryErrors or at least to high GC

activity. This will for example happen if very big documents (XML, PDF, images, …) have to be read and

processed. In a specific case the application was not responsive for a couple of minutes or performance

was so limited that it was not practically usable. The root cause was the garbage collection going crazy.

Bad Garbage Collector Configuration

In the scenarios presented so far the problem was caused by the application code. In a lot of cause the

root cause however is wrong – or missing – configuration of the garbage collector. I frequently see

people trusting the default settings of their application servers and believing these application server

guys know best what is ideal for their application. The configuration of the heap however strongly

depends on the application and the actual usage scenario. Depending on the scenario parameters have

to adopted to get a well performing application. An application processing a high number of short

lasting requests has to be configured completely different than a batch application, which is execution

long lasting tasks. The actual configuration additionally also depends from the used JVM. What works

fine for Sun JVM might be a nightmare for IBM (or at least not ideal).

Misconfigured garbage collectors are often not immediately identified as the root cause of a

performance problem (unless you monitor Garbage Collector acitvity anyway). Often the visual

manifestation of problems are bad response times. Understand the relation of garbage collector activity

to response times is not obivous. If garbage collector times cannot be correlated to response times,

people find themselves hunting a very complex performance problem. Response times and execution

time metric problems will manifest across the applications – at different places without an obvious

pattern behind this phenomenon.

ClassLoader Leaks

When talking about memory leaks most people primarily think about objects on the heap. Besides

objects, classes and constants are also managed on the heap. Depending on the JVM they are put into

specific areas of the heap. The Sun JVM for example uses the so called permanent generation or

PermGen. Classes very often are put on the heap several times. Simply because they have been loaded

by different classloaders. The memory occupation of loaded classes can be up to several hundred MB in

modern enterprise applications.


Key is to avoid unecessarily increasing the size of classes. A good sample is the definition of large

amount of String constants – for example in GUI applications. Here all texts are often stored in

constants. While the approach of using constants for Strings is in principle a good design approach, the

memory consumption should not be neglected. In a real world case all constants where define in one

class per language in an internationalized application. A not obviously visibile coding error resulted in all

of this classed being loaded. The result was a JVM crash with an OutOfMemoryError in the PermGen of

the application.

Application servers suffer additional problems with classloader leaks. These leaks are causes as

classloaders cannot be garbage collected because an object of one of the classes of this classloader is

still alive. As a result memory occupied by these classes will not be freed up. While this problem today is

handled well by J EE application server, it seems to appear more often in OSGI-based application

environments.

Conclusion

Memory problems in Java applications are manifold und easily lead to performance and scalability

problems. Especially in J EE applications with a high number of parallel users memory management must

be a central part of the application architecture.

While the garbage collector takes care that unreferenced objects are clean up, the developer still is

responsible for proper memory management. In addition to application design memory management is

a central part of application configuration.

____________________________________________________________________________

Common Performance problems

● Java engine hangs during startup.

● Java engine takes a long time to return a response. For example, search operations take long

time to respond in portal.

● Time-out error for portal and other java applications.

● Long running or frequent garbage collection activity (Full GCs).

● Heavy paging activity.

● High CPU usage.

● Connectivity between servernode and message server fails (missed broadcast, delay while

getting response from message server/servernode. All -33X exit codes)

● System or applications threads maxed out.

● System slows down and crashes with out of memory (exitcode 666). <std_serverN.out> contains

java.lang.OutOfMemoryError or Out of memory. System will write heap dumps if the

parameters as per Sap note 1004255 are maintained.


● Unable to login to java applications or login takes a lot of time.

● Intermittent servernode restarts.

● Portal or other java application pages remain blank after login.

____________________________________________________________________________

Commonly seen reasons behind performance issues

● Insufficient heap, permanent or physical memory.

● Insufficient number of application and/or system threads

● Too many logs are written due to log severity settings.

● Message server connectivity issues - problem with message server timeout settings.

● Java threads being blocked by certain applications.

● Slow response from LDAP server.

● Incorrect JVM parameters settings.

● Outdated JVM used.

● High CPU consumption by Java processes.

● Outdated database statistics lead to slow portal performance.

● Network issues.

● Certain applications consuming majority of heap.

● Inconsistent support pack levels leading system to crash.

● Timed out java services.

● Dispatcher node hangs (valid for releases lower that Netweaver 7.1).

● Communication error between java and SCS instances..

● All system threads are in use.

● All application threads are in use.

____________________________________________________________________________

Memory management algorithms/management in java

In java, memory is managed via garbage collector. Few techniques for memory management are:

1. Reference Counting: A count of references to each object is maintained. When garbage collector runs,

it deletes objects with zero reference count.

Drawback: Circular references are maintained in memory.

2. Tracing collectors/Copy Collector/Stop and copy collector: Start from a root object and keep a track of

all references which have direct/indirect reference to the root object. Then all the live objects are

moved to another heap, taking care of references properly.


Drawback: At each point of time, you will have 2 heaps thus consuming twice the memory.

3. Mark sweep collectors/Stop and work collector: Similar to tracing collector except that instead of

copying the references to the new heap, they are swept out of memory, after a list of live and dead

objects is known.

Mark and sweep is a stop-the-world garbage collection technique; that is all application threads stop

until garbage collection completes or until a higher-priority thread interrupts the garbage collector. If

the garbage collector is interrupted it must restart which can lead to application thrashing with little

apparent result.

____________________________________________________________________________

Explain types of references in Java?

There are actually four different degrees of reference strength: strong, soft, weak, and phantom, in

order from strongest to weakest:

Strong Reference: By default.

Weak references: A weak reference is a reference that isn't strong enough to force an object to remain

in memory. Weak references allow you to leverage the garbage collector's ability to determine

reachability for you, so you don't have to do it yourself. You create a weak reference like this:

WeakReference<Widget> weakWidget = new WeakReference<Widget>(widget);

weakWidget.get() // get the actual Widget

Of course the weak reference isn't strong enough to prevent garbage collection, so you may find (if

there are no strong references to the widget) that weakWidget.get() suddenly starts returning null.

Soft references: A soft reference is exactly like a weak reference, except that it is less eager to throw

away the object to which it refers. An object which is only weakly reachable (the strongest references to

it are WeakReferences) will be discarded at the next garbage collection cycle, but an object which is

softly reachable will generally stick around for a while. Soft References aren't required to behave any

differently than WeakReferences, but in practice softly reachable objects are generally retained as long

as memory is in plentiful supply. This makes them an excellent foundation for a cache, since you can let

the garbage collector worry about both how reachable the objects are and how badly it needs the

memory they are consuming.

Phantom references


A phantom reference is quite different than either SoftReference or WeakReference. Its grip on its

object is so tenuous that you can't even retrieve the object -- its get() method always returns null. The

only use for such a reference is keeping track of when it gets enqueued into a ReferenceQueue, as at

that point you know the object to which it pointed is dead. How is that different from WeakReference,

though?

The difference is in exactly when the enqueuing happens. WeakReferences are enqueued as soon as the

object to which they point becomes weakly reachable. This is before finalization or garbage collection

has actually happened. In case of Weak Reference, object could even be "resurrected" by an finalize()

method, but the WeakReference would remain dead. PhantomReferences are enqueued only when the

object is physically removed from memory, and the get() method always returns null specifically to

prevent you from being able to "resurrect" an almost-dead object.

Use of Phantom Reference:

1. They allow you to determine exactly when an object was removed from memory. They are in fact the

only way to determine that. This isn't generally that useful, but might come in handy in certain very

specific circumstances like manipulating large images: if you know for sure that an image should be

garbage collected, you can wait until it actually is before attempting to load the next image, and

therefore make the dreaded OutOfMemoryError less likely.

2. PhantomReferences avoid a fundamental problem with finalization – resurrection. With

PhantomReference, resurrection is impossible. When a PhantomReference is enqueued, there is

absolutely no way to get a pointer to the now-dead object.Arguably, the finalize() method should never

have been provided in the first place. PhantomReferences are definitely safer and more efficient to use,

and eliminating finalize() would have made parts of the VM considerably simpler. But, they're also more

work to implement, so I confess to still using finalize() most of the time. The good news is that at least

you have a choice.

____________________________________________________________________________

Phantom Reference Notes

The object is strongly reachable as long as there's a reference to it; either in a static field, an instance

field of another strongly reachable object, or a local variable that is currently within scope.

When there are no more strong references to the object, it becomes softly reachable. Softly reachable

objects stay softly reachable until the garbage collector determines that memory is running out, and

they need to make room for other objects. The garbage collector then "clears" all the soft references

(meaning, the soft reference will no longer refer to the object it used to refer to), and enqueues the

references.


After that, the object becomes weakly reachable. The garbage collector will immediately clear all weak

references to the object, and enqueue them. The object is then eligible for finalization.

Now, the garbage collection will run the finalize() method on the object, if it hasn't already done so.

After the finalize() method is done, the object is finalized, and should** become phantom reachable. So

an object that's phantom reachable only has phantom references to it. Phantom reachable objects are

removed from memory; the space they take up is reclaimed by the garbage collector.

Phantom references are a special case. They *always* return null when asked about their referent. This

is to prevent people from getting strong references to the object after it's been finalized. Secondly,

phantom references aren't automatically cleared by the garbage collector before they are enqueued.

You need to clear them manually after you've retrieved them from the reference queue, and performed

cleanup operations.

Once all the phantom references to the object have been cleared or finalized, the object is no longer

reachable. The object is unreachable.

I'll give some examples of use cases for these references. Soft references are only cleared when the JVM

runs out of memory (this actually isn't a guarantee, the garbage collector may clear the references even

if there's plenty of space). Therefore, they are great for building caches. They keep an object in memory

as long as there's space, but they don't prevent the object from being collected when space runs out.

Weak references are great when you want to store extra information about an object, but that

information is only useful for as long as the object exists. You can then have a weak reference to the

object, which also strongly refers to the extra information. When the object has no strong references to

it anymore, it becomes weakly reachable, is automatically cleared, and the weak reference is enqueued.

The thread that deals with the reference queue can then clean up the extra information when it

retrieves the weak reference from the queue.

Finally, Phantom references are good for two things only. One is determining that the object is *really*

dead and removed from memory. The second thing is performing cleanup. If you need to clean up a

bunch of other objects when the first object is dead, you can do so, and then clear the phantom

reference. This is exactly the same as just performing the finalize() method, except it's safer, because the

finalize() method can perform cleanup before the object is physically removed from memory.

**An object that has been finalized *should* become phantom reachable. However, this is not always

true. The finalize() method can resurrect a dead object by storing a reference to it in a variable, thereby

making it strongly reachable once again. This is why cleanup in the finalize() method is dangerous. It's

possible to clean up other objects, but then resurrecting the original object. When you perform

operations on the object, the program may fail catastrophically.


Example

package org.karanki.play.gc; import java.lang.ref.PhantomReference; import java.lang.ref.ReferenceQueue; public class TestPhantomRefQueue { public static void main(String[] args) throws InterruptedException { Object obj = new Object(); final ReferenceQueue queue = new ReferenceQueue(); PhantomReference pRef = new PhantomReference(obj, queue); obj = null; new Thread(new Runnable() { public void run() { try { System.out.println("Awaiting for GC"); // This will block till it is GCd PhantomReference pRef = (PhantomReference) queue.remove(); System.out.println("Referenced GC'd"); } catch (InterruptedException e) { e.printStackTrace(); } } }).start(); // Wait for 2nd thread to start


Thread.sleep(2000); System.out.println("Invoking GC"); System.gc(); } }

____________________________________________________________________________

Talk about garbage collector for various references!

Talk about garbage collector for various references.

Ans. If an element is determined to be eligible for processing, GC must determine if it is eligible for

collection. The first criterion here is simple. Is the referent marked? If it is marked, the reference object

is not eligible for collection and GC moves onto the next element of the list. However, if the referent is

not marked and is eligible for collection, the process differs for each reference type.

Soft references are collected if their referent has not been marked for the previous 32 garbage

collection cycles. You adjust the frequency of collection with the -Xsoftrefthreshold option. If there is a

shortage of available storage, all soft references are cleared. All soft references are guaranteed to have

been cleared before the OutOfMemoryError is thrown.

Weak and phantom references are always collected if their referent is not marked.


____________________________________________________________________________

Explain garbage collection on Remote Objects or Distributed Garbage

collection.

In a distributed system, just as in the local system, it is desirable to automatically delete those remote

objects that are no longer referenced by any client. This frees the programmer from needing to keep

track of the remote objects' clients so that it can terminate appropriately. RMI uses a reference-counting

garbage collection algorithm for the same.

To accomplish reference-counting garbage collection, the RMI runtime keeps track of all live references

within each Java virtual machine. When a live reference enters a Java virtual machine for first time, it

sends a "referenced" message to the server for the object. Going forward, whenever a live reference

enters JVM, its reference count is incremented and is decremented as soon as it leaves the JVM. When

the last reference has been discarded, an unreferenced message is sent to the server. Many subtleties

exist in the protocol; most of these are related to maintaining the ordering of referenced and

unreferenced messages in order to ensure that the object is not prematurely collected.

When a remote object is not referenced by any client, the RMI runtime refers to it using a weak

reference. The weak reference allows the Java virtual machine's garbage collector to discard the object

if no other local references to the object exist. As long as a local reference to a remote object exists, it

cannot be garbage-collected and it can be passed in remote calls or returned to clients. Remote objects


are only collected when no more references, either local or remote, still exist. The distributed garbage

collection algorithm interacts with the local Java virtual machine's garbage collector in the usual ways by

holding normal or weak references to objects.

In addition to the reference counting mechanism, a live client reference has a lease with a specified

time. When the client is done with the reference and allows the remote stub to go out of scope, or when

the lease on the object expires, the reference layer on the host automatically deletes the record of the

remote reference and notifies the client's reference layer that this remote reference has expired. The

lease time is controlled by the system property java.rmi.dgc.leaseValue. The value is in milliseconds and

defaults to 10 minutes. The concept of expirable leases, as opposed to strict on/off references, is used

to deal with situations where a client-side failure or a network failure keeps the client from notifying the

server that it is done with its reference to an object.

A remote object needing unreferenced notification must implement the java.rmi.server.Unreferenced

interface. When those references no longer exist, the unreferenced method will be invoked.

____________________________________________________________________________

Does OutOfMemoryError and StackOverFlowError cause JVM crash?

Any problem in PURE Java code throws a Java exception or error. Java exceptions or errors will NOT

cause a core dump (on UNIX systems) or a Dr.Watson error (on WIN32systems). Any serious Java

problem will result in an OutOfMemoryError thrown by the JVM with the stack trace and consequently

JVM will exit. An OutOfMemoryError (not jvm crash) can be thrown due to one of the following 4

reasons:

1. JVM may have a memory leak due to a bug in its internal heap management implementation. But this

is highly unlikely because JVMs are well tested for this.

2. The application may not have enough heap memory allocated for its running. You can allocate more

JVM heap size (with –Xmx parameter to the JVM) or decrease the amount of memory your application

takes to overcome this. You can increase heap size as below:

java -Xms1024M -Xmx1024M

Care should be taken not to make the –Xmx value too large because it can slow down your application.

3. Another not so prevalent cause is the running out of a memory area called the “perm” which sits next

to the heap. All the binary code of currently running classes is archived in the “perm” area. The ‘perm’

area is important if your application or any of the third party jar files you use dynamically generate

classes.

For example: “perm” space is consumed when XSLT templates are dynamically compiled into classes,

J2EE application servers, JasperReports, JAXB etc use Java reflection to dynamically generate classes

and/or large amount of classes in your application. To increase perm space:


java -XX:PermSize=256M -XX:MaxPermSize=256M

4. The fourth and the most common reason is that you may have a memory leak in your application.

____________________________________________________________________________

Different OutOfMemory errors!!

Let’s have a look at the Sun HotSpot JVM and its concrete implementation of OutOfMemoryError errors.

1. In the heap we get an OutOfMemoryError, if the garbage collector cannot reclaim enough memory

for a new object. In such situation the Sun HotSpot JVM shows this error message:

java.lang.OutOfMemoryError: Java heap space

2. An alternative for this is as below, it occurs when application tries to create an array on the heap

that is bigger than the total heap size.

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

3. If there is not enough memory in the method area for creating a new class, the Sun HotSpot

implementation gets an error in the permanent generation:

java.lang.OutOfMemoryError: PermGen space

4. OutOfMemory errors in thread exclusive memory areas occur less frequently and are identified by

the following error messages in the Sun HotSpot JVM:

java.lang.OutOfMemoryError: unable to create new native thread

This occurs if there are too many threads in the JVM and there is not enough memory left to create a

new thread. I’ve seen this because the memory limits of a process have been reached (especially in

32bit operating systems, e.g. on Windows 32bit it is 2GB) or the maximum number of file handles for

the user that executes the java process has been reached.

5. It indicates that a memory allocation error on a native stack (JNI method call) has occured.

java.lang.OutOfMemoryError: <reason> <stacktrace> (Native method)

6. It is also interesting that a memory allocation error on the JVM stack (too many frames on the stack)

does not throw an Java OutOfMemory error but as the JVM specification mandates.

java.lang.StackOverflowError

7. The last variant of the OutOfMemoryError is out of swap space. This error is thrown if there is not

enough memory left on the operating system level – which is normally true if other processes are using

all of the available memory or the swap space is configured too small.

java.lang.OutOfMemoryError: request <size> bytes for <reason>.


____________________________________________________________________________

Why does the JVM crash with a core dump or a Dr.Watson error?

Both the core dump on UNIX operating system and Dr.Watson error on WIN32 systems mean the same

thing. If you define a crash as an unhandled problem (i.e. no Java Exception or Error); then this cannot

be done from within Java. The JVM is a process like any other and when a process crashes a core dump

is created. A core dump is a memory map of a running process. This can happen due to one of the

following reasons:

1. Using JNI (Java Native Interface) code containing a fatal bug in it. Typical crashes in native code

happen by dereferencing pointers to wrong memory areas (like Nullpointer) or illegal opcodes.

For ex: using Oracle OCI drivers, which are written partially in native code or JDBC-ODBC bridge drivers,

which are written in non Java code. Using 100% pure Java drivers (communicates directly with the

database instead of through client software utilizing the JNI) instead of native drivers can solve this

problem.

2. The OS on which your JVM is running might require a patch or service pack.

3. The JVM implementation may have a bug in translating system resources like threads, file handles,

sockets etc from the platform neutral Java byte code into platform specific operations. If this JVM’s

translated native code performs an illegal operation then the operating system will instantly kill the

process and mostly will generate a core dump file.

The core dump files are generated by the operating system in response to certain signals. The JVM can

also intercept certain signals like SIGQUIT which is kill -3 <pid> from the operating system and it

responds to this signal by printing out a Java stack trace and then continue to run. On the other hand

signals like SIGSTOP (kill -23 <pid>) and SIGKILL (kill -9 <pid>) will cause the JVM process to stop or die.

The JVM argument "java –Xsqnopause" will indicate JVM not to pause on SIGQUIT signal from OS.

4. On Linux/Unix, it is easy to crash JVM crash by sending it a Signal to the running process.

____________________________________________________________________________

Memory Leaks Examples

Auto Boxing

package org.karanki.play.performance;


public class Adder { public long addIncremental(long l) { long sum=0L; sum =sum+l; return sum; } public static void main(String[] args) { Adder adder = new Adder(); for(long i = 0; i < 10000000; i++) { adder.addIncremental(i); System.out.println(i + " being added..."); try { Thread.sleep(50); } catch (InterruptedException e) { e.printStackTrace(); } } } }

The Long wrapper creates Long object every time when sum = sum + 1 is performed. Hence using

primitives is much better.

____________________________________________________________________________


JVM & Garbage Collection

What flags can I use to tune the JVM and GC?

-XX:-UseConcMarkSweepGC: Use the CMS collector for the old gen.

-XX:-UseParallelGC: Use Parallel GC for New Gen

-XX:-UseParallelOldGC: Use Parallel GC for Old and New Gen.

-XX:-HeapDumpOnOutOfMemoryError: Create a thread dump when the application runs out of

memory. Very useful for diagnostics.

-XX:-PrintGCDetails: Log out details of Garbage Collection.

-Xms512m: Sets the initial heap size to 512m

-Xmx1024m: Sets the maximum heap size to 1024m

-XX:NewSize and -XX:MaxNewSize: Specifically set the default and max size of the New Generation

- XX:NewRatio=3: Set the size of the Young Generation as a ratio of the size of the Old Generation.

-XX:SurvivorRatio=10: Set the size of Eden space relative to the size of a survivor space.

____________________________________________________________________________

Head Dump vs Thread Dump vs Core Dump

jmap command is used to generate heap dump if your using Sun JDK.

jrcmd command is used to generate heap dump if your using Jrocket.

Don’t generate heap dump multiple times as it causes your application performance degradation as

generating heap dump will take at least 15 to 30 minutes and generated file size will be minimum 2 GB.

And use Eclipse mat only to analyze it. IBM heap analyzer will not help you in this case due to IBM heap

analyzer supports hprof formated files only to my knowledge.

Heap dump – Collection of objects that are in memory (JVM)


Thread dump – Shows what each thread in a process is doing at a given point in time along with the

stack trace.

Core dump – O/S level dump file which has O/S level info in addition to the heap dump.

Heap dump – is useful to analyse OOM situations.

Thread dump – To troubleshoot slow running of your application.

Core dump – When your JVM has crashed abruptly. To find details about native calls and so on.

jmap -heap:live,format=b,file=filename pid

jrcmd pid hprofdump filename=name_of_dump_file

A thread dump is a dump of the stacks of all live threads. Thus useful for analyzing what an app is up to

at some point in time, and if done at intervals handy in diagnosing some kinds of ‘execution’ problems

(e.g. thread deadlock).

A heap dump is a dump of the state of the Java heap memory. Thus useful for analyzing what use of

memory an app is making at some point in time so handy in diagnosing some memory issues, and if

done at intervals handy in diagnosing memory leaks.

Heapdump can be taken in number of ways:

Via Java VM parameters:

-XX:+HeapDumpOnOutOfMemoryError writes heap dump on OutOfMemoryError (recommended)

-XX:+HeapDumpOnCtrlBreak writes heap dump together with thread dump on CTRL+BREAK

Using JRockit:

-jrcmd pid hprofdump filename=name_of_dump_file

Using Jmap:

-jmap -heap:format=b pid

Note: use -J-d64 jvm option if your JVM is 64 Bit Jvm “jmap -J-d64 -heap pid”

You can also manually generate a heap dump with tool VisualVM.

Using HPROF:


You can use HPROF: Heap and CPU Profiling Agent.

A complete dump of the current live objects in the heap can be obtained with:

-java -agentlib:hprof=heap=dump,format=b -jar application

This will automatically dump heap when java application is terminated. You can also force heap dump by

sending QUIT signal to java process with kill -QUIT pid command.

Analysing Heapdump file using Jhat

You can use jhat (Java Heap Analysis Tool) to read the generated file:

– jhat [ options ]

The jhat command parses a java heap dump file and launches a webserver. jhat enables you to browse

heap dumps using your favorite webbrowser.

Note that you should have a hprof binary format output to be able to parse it with jhat. You can

useformat=b option to generate the dump in this format.

performance & jvm & gc › ... › 09 › performance-jvm-gc.pdf · performance & jvm...

Documents