oracle weblogic troubleshooting performance

Upload: luis-yanez

Post on 01-Jun-2018

252 views

Category:

Documents


0 download

TRANSCRIPT

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    1/37

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    2/37

    Oracle Weblogic ServerBest Practices for Troubleshooting PerformanceIssues - CON8307

    Laurent GoldsztejnFusion Middleware Proactive Support

    September 2014@weblogicsupport

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    3/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Safe Harbor StatementThe following is intended to outline our general product direction. It is inteinformation purposes only, and may not be incorporated into any contract.commitment to deliver any material, code, or functionality, and should notin making purchasing decisions. The development, release, and timing of afunctionality described for Oracle’s products remains at the sole discretion

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    4/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Program Agenda

    Performance issues

    Best practices with server logs

    Best practices with thread dumps

    Deconstructing Java threads

    Examples & Demo

    1

    2

    3

    4

    5

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    5/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Performance Issues

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    6/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Performance IssuesServer hanging

    The server does not respond to new requests

    Requests time out

    Requests take longer and longer to process

    The server is no longer reported as running

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    7/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Performance IssuesIssues that could lead to server hang or other severe performance issues

    Application code or WLS internal issue

    Too high number of file descriptors or open s

    JVM spending too much time in GC

    Long running JDBC requests

    Lock contention between NM and Managed Server

    Excessive or unneeded Remote JNDI look

    WLS Session Replication problem

    Memory leak

    Deadlock

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    8/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Performance IssuesData needed to analyze a performance issue

    Server logs, -verbose:gc to print information at every collection

    Stdout redirected to server log:

    Servers->Logging->General->Advanced-> Redirect Stdout Logging Enabled

    -Dweblogic.log.RedirectStdoutToServerLogEnabled=true

    Java Flight Recordings (JFR)

    Collections of general data about the application (GCs, Heap, Optimizations, Objects stat

    RDA collections or configuration files

    Thread dumps / Heap dumps

    WLDF to collect metrics, setup watch and notifications, and to define instrumentatio

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    9/37Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with server logs

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    10/37Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with server logsErrors in Server Log file

    Check for StuckThreads

    Check for Too ManyOpen Files

    Check foOutOfMem

    message

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    11/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with server logsExample of Stuck Threads

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    12/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with server logs

    • Socket connections use file descriptors

    • A socket can remain in TIME_WAIT after they are closed to guarantee that delayed packets in the the corresponding socket

    tcp_time_wait_interval  on Linux (default 4 minutes)

    HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters on Windows (default 1 minute) 

    • If many connections are being opened and closed quickly then socket's in TIME_WAIT may begia system

     – View sockets in TIME_WAIT using netstat 

    Too Many Open Files

    Support Pattern: How To Troubleshoot Too Many Open Files Problems [ID 8674

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    13/37

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    14/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with thread dumps

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    15/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with thread dumps

    A java thread dump is a snapshot that shows what every threaprocess is doing at a particular point in time

    Definitions

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    16/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with thread dumpsWhat does a Thread Dump contain?

    • State

    • Name,Queue

    • tid, nid, prio

    • Classes

    • Packages

    • Method Calls

    • Locks

    • Vendor• Version

    • Build

    • Architecture

    • Mode

    • Date andtime thedump wascreated

    Time Stamp JVM Info

    ThreadsStack

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    17/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with thread dumpsDifferent ways to take thread dumps

    Remote DiagnosticAgent (RDA)

    Unix Command-linekill -3

    WLST threadDump()

     jcmprin

    Hot

    WLS Admin Console Jstack Java VisualVM

    T

    (Javac

    Different ways to take thread dumps in WebLogic Server [ID 1098691.1]

    How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?

    https://blogs.oracle.com/fmwinstallproactive/entry/how_to_use_rda_to

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    18/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with thread dumps

    • Different JVM Vendors display the data in different formats• Markers for start/end of thread dumps

    • Reporting of locks

    • Thread states and method signatures

    •The underlying data exposed remains the same across vendors

    • Thread dumps can contain lots of data

    • Trying to read them in a text editor can be very tricky

    Viewing a Thread Dump

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    19/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Best practices with thread dumpsFree parsing tools

    Tool Description

    Samuraihttp://yusuke.homeip.net/samurai/en/index.ht

    ml 

    Light weight open source tool that

    thread dumps and GC stats from lo

    and presents them in different tab

    with color code for easy visual ana

    Thread Dump Analyzer (TDA)http://java.net/projects/tda/pages/Home 

    Swing GUI for analyzing Thread DuHeap Information

    Thread Logichttp://java.net/projects/threadlogic/ 

    Built upon TDA by adding logic for

    patterns found in application serve

    http://yusuke.homeip.net/samurai/en/index.htmlhttp://yusuke.homeip.net/samurai/en/index.htmlhttp://java.net/projects/tda/pages/Homehttp://java.net/projects/threadlogic/http://java.net/projects/threadlogic/http://java.net/projects/tda/pages/Homehttp://yusuke.homeip.net/samurai/en/index.htmlhttp://yusuke.homeip.net/samurai/en/index.html

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    20/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    TroubleshootingThread Dump Analysis – Five High level steps

    Look at default and custom execute threads

    Identity the cause of stuck threads by looking at the last method and java class

    Ignore the idle threads

    Check thread dumps from different managed servers to find potential relationshithe stuck threads

    In most cases, WLS internal threads, such as muxer and listen threads, can be

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    21/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threads

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    22/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threads

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    23/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threadsThread.State

    State Description

    NEW Available thread ready to process work request for any work manager

    BLOCKED

    Thread blocked by another and waiting for the other thread to complete its o

    release its locks. If more than one threads attempt to acquire the monitor of

    object then only one thread (selected by the JVM scheduler ) is granted the m

    other threads are put into BLOCKED state

    RUNNING Thread actively working processing a task and therefore using some CPU

    TIMED_WAITING

    Thread in sleep, wait, join or park method and in a state of monitoring (waiti

    condition) - waiting for another thread to perform an action for up to a specif

    time is in this state

    WAITINGThe thread is waiting by using a wait, join or park method without any maxim

    indefinitely for another thread to perform a particular action is in this state

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    24/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threads

    Mechanism used by Java to support thread synchronization – Mutual exclusion

    • Supported via object locks

    • Enables multiple threads to independently work on shared data without interfering with eac

     – Cooperation

    • Supported via the wait and notify methods of class Object

    • Enables threads to work together towards a common goal

    • Package java.util.concurrent

     – Utility classes commonly useful in concurrent programming

    Java monitors

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    25/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threadsJava Monitors - Example

    "Thread-0" … in Object.wait()[…] at java.lang.Object.wait(Native Method)

    - waiting on (a a.b.c.thread.MyClass)

    at java.lang.Object.wait(Unknown Source)

    at a.b.c.thread.Drop.take(ClassB.java:44)

    - locked (a a.b.c.thread.ClassB)

    at a.b.c.thread.exec(Test.java:15)

    at java.lang.Thread.exec(Unknown Source)

    "Thread-1" … waiting on condition[…] 

    at java.lang.Thread.sleep(Native Method)

    at x.y.z.thread.run(ClassA.java:33)

    at java.lang.Thread.run(Unknown Source)

    "Thread-0" is waiting for anotification after it called

    Object.wait() 

    "Thread-1" is sleeping on a

    condition after it called

    Thread.sleep

    Conditions provide a means for one thread to suspend execution and wait until

    notified by another thread that some state condition may now be true

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    26/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threads

    • Locks are used to prevent that more than one thread accesses and workobject

    • Only one thread can own a Lock object at a time

    • Lock objects support a wait/notify mechanism

    • A lock is called Thin with little or no contention

    • A lock is promoted to Fat when contention increases

    Java locks

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    27/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threads

    If the lock is not held by another thread then the thread becomes the nelock until it relinquishes it

    • If the lock is currently held then the thread requesting it joins the blockedcontenders for this lock

    • Threads holding locks should relinquish them as soon as they stop needinavoid impairing on other threads

    Thread Requesting Locks

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    28/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Deconstructing java threads

    Deadlocks describe a situation where two or more threads are blocked wother (circular dependency ). Neither block will ever end because each thfor the other

    • Deadlocks can occur when multiple threads need the same locks, at the sobtain them in different order

    • Restart of the server is the only option

    Deadlocks

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    29/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Examples & Demo

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    30/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Examples and demo

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    31/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Examples and demo

    [STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x128 tid=15140 prio=1 alive, in native

    at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)

    at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

    at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]

    at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]

    … 

    at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]

    ^-- Holding lock: java/io/BufferedInputStream@0x291bf2b8[thin lock]

    at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:75)

    at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1440)

    … 

    at javax/mail/Service.connect(Service.java:105)

    at javax/mail/Transport.send0(Transport.java:168)

    … 

    Thread hanging on SocketRead from SMTP call

    The thread i

    receive data

    mail server.

    used as instr

    communicatio

    needs to

    A lock i

    BufferInpu

    safegua

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    32/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Examples and demo

    "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x000000004c8a2000 nid=0

    [0x000000004296c000]

     java.lang.Thread.State: WAITING (on object monitor)

    at java.lang.Object.wait(Native Method)

    - waiting on  (a netscape.ldap.LDAPSearchListener)

    at java.lang.Object.wait(Object.java:485)

    at netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200)

    - locked  (a netscape.ldap.LDAPSearchListener)

    at netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101)

    … 

    at weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3625)

    at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:220)

    at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)

    at java.security.AccessController.doPrivileged(Native Method)

    Thread hanging on LDAP response

    The thread is waiting t

    synchronized block and

    Java monitor

    Its wait will be over

    receives the message

    the LDAP serve

    The health of the LDA

    needs to be checkunderstand why it’s pe

    slowly

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    33/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Examples and demo

    "[ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)'" id=313028 idx=0x1d0 tid=4686

    waiting, daemon

    -- Waiting for notification on: weblogic/work/ExecuteThread@0x739b42e0[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

    at java/lang/Object.wait(Object.java:474)[inlined]

    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x739b42e0[fat lock]

    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

    -- end of trace

    Idle Thread waiting for new task

    The thread

    notified of a ne

    Several availa

    are waitin

    ExecuteThre

    results

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    34/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Examples and demo

    "[ACTIVE] ExecuteThread: '44' for queue: 'weblogic.kernel.Default (self-tuning)'" id=372 idx=0xfa t id=4257

    prio=5 alive, in native, blocked, daemon

    -- Blocked trying to get lock: weblogic/utils/KeyTable@0x16939098[thin lock]

    at jrockit/vm/Threads.sleep(I)V(Native Method)

    ...

    - Holding lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]

    at weblogic/rjvm/ResponseImpl.getTxContext()Ljava/lang/Object;(ResponseImpl.java:100)[inlined]

    ..."[ACTIVE] ExecuteThread: '134' for queue: 'weblogic.kernel.Default (self-tuning)'" id=473785 idx=0x398 tid=32638 prio=5 alivedaemon

    -- Blocked trying to get lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]

    at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)

    ...

    ^-- Holding lock: weblogic/utils/KeyTable@0x16939098[thin lock]

    ...

    Deadlock

    Thread #134

    that thread

    the same tim

    lock that thre

    Both thread

    blocked wait

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    35/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

    Drinks. Food. Fun.My Oracle Support M

    Tonight! 

    Monday, September 296:00 to 8:00 p.m.ThirstyBear Brewing Company(only ½ block from Moscone Center)

    Oracle Confidential – Inte

    Join us for a relaxing Happy Hour after a busy day at Oracle Open

    •Take a break and unwind with your peers

    •Get to know the Oracle support engineers you depend on

    •Meet My Oracle Support executives and developers

    •Enjoy drinks and hors d’oevres 

    •Admission is free with your Oracle OpenWorld badgeEvent details atwww.oracle.com

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    36/37

    Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

  • 8/9/2019 Oracle WebLogic Troubleshooting Performance

    37/37