Advanced Enterprise Debugging

Techniques

Neal FordApplication ArchitectThoughtWorks

www.nealford.comwww.thoughtworks.com

nford@thoughtworks.commemeagora.blogspot.com

2

Questions, Slides, and Samples

• Please feel free to ask questions anytime• The slides and samples will be available at

www.nealford.com (I’ll show this again at the end)

3

What This Session Covers:

• Remote debugging • Remote debugging in IDE’s• (Don’t scoff) jdb• Forensic debugging• Debugging web applications• Debugging class loaders• Groovy enterprise debugging• Automating common tasks

4

Remote Debugging in Java

• Built into the platform• Found in JavaDocs at

jdk1.5.0/docs/guide/jpda/architecture.html

5

Enable Debugging on the VM

• Add some parameters to your JVM startup

Parameter Purpose

-Xdebug Enables debugging support in the VM

-Xrunjdwp:<options> Loads the in-process debugging libraries and selects the transport mechanism for communciations

6

runjdwp Options

Name Description

help Prints a brief help message and exits the VM.

transport Contains the name of the transport you want to use when connecting to the debugger. 

server If y, listens for a debugger application to attach; otherwise, attaches to the debugger application at the specified address. If y and no address is specified, chooses a transport address at which to listen for a debugger application, and prints the address to the standard output stream.

7

runjdwp Options

Name Description

address Contains the transport address for the connection. If server=n, attempts to attach to debugger application at this address. If server=y, listens for a connection at this address.

launch At completion of Java Debugging Wire Protocol (JDWP) initialization, launches the process given in this string. This option is used in combination with onthrow and/or onuncaught to provide just-in-time debugging, in which a debugger process is launched when a particular event occurs in this VM.

onthrow Delays initialization of the JDWP library until an exception of the given class is thrown in this VM. The class name is package-qualified. Connection establishment is included in JDWP initialization, so it will not begin until the exception is thrown.

8

runjdwp Options

Name Description

onuncaught If y, delays initialization of the JDWP library until an uncaught exception is thrown in this VM. Connection establishment is included in JDWP initialization, so it will not begin until the exception is thrown.

stdalloc By default, the JDWP reference implementation uses an alternate allocator for its memory allocation. If y, the standard C runtime library allocator will be used. This option is mainly for testing; use it with care. Deadlocks can occur in this VM if the alternative allocator is disabled.

9

runjdwp Options

Name Description

strict If y, assumes strict Java Virtual Machine Debugging Interface (JVMDI) conformance. This will disable all workarounds to known bugs in JVMDI implementations. This option is mainly for testing and should be used with care.

suspend If y, VMStartEvent has a suspendPolicy of SUSPEND_ALL. If n" VMStartEvent has a suspendPolicy of SUSPEND_NONE.

10

The Ones You Care About

• Transport• dt_shmem- Uses Windows shared memory primitives to

communicate information between debugger applications and the target VM.

• Shortcomings

• Only works on Windows

• Only works when both VMs are running on the same machine

• dt_socket- Allows the debugger and debuggee to communicate over a socket

• Advantages

• Works across platform

• Works across machines

11

The Ones You Care About

• Address• dt-shmem – The “address” of the shared memory

• An arbitrary string that the debugger and debuggee agree upon

• dt_socket – The socket used for communication

• Server• Set to “Y”

• Suspend• Set to “N” unless you are debugging startup behavior

12

Remote Debugging in Eclipse

13

Remote Debugging in IntelliJ

14

Remote Debugging in JBuilder

15

When It's All You've Got: jdb

• What’s the only debugger that you absolutely know will be available everywhere?• On you development machine• On the production server• At a remote client’s site

• jdb – the command-line debugger that comes with the SDK

• It is surprisingly capable once you learn how to use it• To see a list of commands, start it and type ?

16

The jdb Designer Surface

17

Some choice jdb commands

Command Description

run Starts execution of the application’s main class

Threads Lists the threads in the application, both running and suspended

thread Set the default thread

where <thread id> Dumps a thread’s stack

print <expr> Prints the value of an expression

stop at <class id>:

<line>

Sets a breakpoint at a line

stop at <class id>.

<method>

Creates a method breakpoint

18

The jdb Recipe

1. Start the application server in debug mode.2. Attach to the source path for your code.3. Set one or more breakpoints, either on line numbers or contingent on

method invocations.4. Run the application.5. Find the thread ID for your application.6. Set it as the default thread.7. Invoke your application in a browser and interact with it until you hit a

breakpoint.8. Use step and/or next to walk through your code.9. Use dump, locals, and eval to analyze your variables.10. If you need to make a small change to a file and test it, use redefine to

load the new class file.11. Use cont to resume execution of the application until the next breakpoint.12. Iterate until done.

19

Effective Debugging

• To often, developers only set line breakpoints• Every debugger (including jdb) allows you to set a

variety of breakpoint types (all conditional)• Line• Method • Class• Exception

• Use the debugger wisely• If you know that a certain method causes a problem when a

condition becomes true, set that up as a breakpoint

20

Effective Debugging

• One of the keys to effective debugging is to not use brute force• System.out.println()

• Line breakpoints

• There is always a sense of urgency when debugging – stop and think!• I know that you are behind schedule, but you’re just making it

worse!• Step away from the keyboard• Don’t ever say: “But that’s impossible!”• Or, “But it works on my machine!”

21

Forensic debugging using loggers

• Loggers are the best way to do forensic debugging• Logging setup is well documented just about

everywhere• Use the levels wisely• If you do need to get down to the “debug” level, use

tools• grep will help you find just what you are looking for• Don’t go scroll blind• Use ChainSaw

22

Using Aspects to Instrument Loggers

• One of the best use of Aspects is to inject logging code• Even though great effort has been made to make the

logging tests cheap, there is still overhead• There’s no overhead if the code isn’t there!• Aspects allow you to specifically target code only when

you need to

23

Debugging Web Applications

• Firefox developer’s toolbar• http://www.chrispederick.com/work/firefox/webdeveloper/

• Bookmarklets• Bookmarks (generally JavaScript) that expose information

about the page• It is sometimes amazing how much you can find out• Search for Bookmarklets

• www.bookmarklets.com/

• www.squarefree.com/bookmarklets/

24

Tapestry

• Debugging JSP’s is tough• Mixed HTML, JavaScript, JSP custom tags, and JSP• Tapestry has the best debugging aid of any web

framework• Inspector

25

Tapestry’s Inspector

26

Debugging Hibernate

• With some JDBC drivers (i.e., PostgreSQL) and batched updates with Hibernate:• An exception is thrown (e.g. when a foreign key constraint is

violated or not null column is left out, etc.)• This SQLException is wrapped in a PBatchUpdateException • This PBatchUpdateException is wrapped in a

HibernateException • The HibernateException is written to the log

• Hibernate uses standard stacktrace routines from JDK 1.4

• The call to getNextException() from SQLException is never made

27

Debugging Hibernate: 2 Solutions

• Disable batch updating in the hibernate.properties file:hibernate.jdbc.batch_size = 0

• “Walk” the exception hierarchy yourself, using a combination of getNextException() and getCause()

• Note that Hibernate isn’t walking the exception chain in every case• May be losing information even without batch updates• Exceptions aren’t linked via the 1.4 mechanism

• This is deliberate – each exception in the chain may have an independent cause)

• Hibernate needs fixing!

28

Debugging Class Loaders

• One of the bane’s of application servers are class loader issues

• Class loaders are responsible for loading classes within the VM

• When you execute a Java application, the native Java class loader loads (the bootstrap class loader)

• The JVM loads 2 other class loaders by default• Extension class loader• Application class loader

29

Class loaders

Core Java Classes

Extensions (javax.* and classes from the ext directory)

Your application

30

Class loaders

• Class loaders use the delegation model to load classes• Each class loader defers to its parent

• In application servers, each deployed application generally gets its own class loader• EAR files get a single class loader for both web and EJB

• These class loaders are responsible for loading and unloading classes

31

Application Server Class Loaders

32

Why Should You Care?

• Shared libraries• Logger packages (like Log4J)• XML parsers

• You have several options:• Deploy the shared library on the application server’s class

loader• Move it to the app server’s lib directory

• Can cause problems

33

Class Loader Recipe

• If the utility class is required only by your web application, move it to WEB-INF/lib

• If you have a web application and EJB’s that don’t share a class loader• Make an entry in your manifest file for the EJB • Specify all the utility classes using the Classpath entry• Keep this utility JAR file either in a shared location where both

web and EJB’s can see it (but off the app server’s boot classpath)

34

Debugging Class Loader Problems

• You usually don’t see a problem when the class loads (or fails to load)

• The toString() methods of the JDK class loaders don’t provide much information

• What to do when you get the dreaded ClassNotFound exception

1. Look for the line where your class loader's loadClass() call is involved

2. Figure out what class loader is used there. 3. Figure out the parent class loaders recursively until you find

the bootstrap class loader. 4. Figure out why the class cannot be found by any of these

class loaders.

35

Debugging Class Loader Problems

• Some causes:• An archive, directory, or other source for the classes was not

added to the class loader asked to load the class, or to its parent.

• A class loader's parent is not set correctly. • The wrong class loader is used to load the class in question.

36

Debugging Class Loader Problems

• The 3rd option happens very easily• In J2EE, an EJB is required to use the thread context class

loader (Thread.currentThread.getContextClassLoader()) to load classes

• Some developers use Class.forName() to load a class without specifying the thread context class loader as the current class loader

37

Symbolic-link Class Not Found

• If a symbolic link cannot be found, a NoClassDefFoundError occurs

• This means that the code has been successfully compiled but can’t find the referenced class at runtime

• How to investigate this problem:1. Find the NoClassDefFoundError and print out its stack trace.

2. Find the topmost line in the stack trace that is not class-loader-related. This most likely indicates the class of the symbolic link that was not found.

38

Symbolic-link Class Not Found

• How to investigate this problem:3. Figure out the class loader of the class containing the

offending symbolic link.

4. List the parent class loaders recursively.

5. Figure out why the class is not available to any of these class loaders.

• Potential causes:• An archive, directory, or other source for the classes was not

added to the class loader asked to load the class, or to its parent.

• A class loader's parent is not set correctly. • Symbolic links in a class are unaccessible by the containing

class's class loader

39

Last Resort to Solve CL Issues

• Patch the JDK’s Class Loader with an improved toString() method

1. Take the source of java.lang.ClassLoader, java.security.SecureClassLoader and java.net.URLClassLoader and copy them.

2. Add the desired toString() method and make sure that every class loader prints out its own address by using super.toString() so that you can test if class loaders from the same class are identical or not.

3. Jar them up

4. Add -Xbootclasspath/p:<the archive you just have created> to your script that starts Java.

40

Last Resort

• We need 3 pieces of information from the class loader• An indicator of the instance• What archives are available to this class loader• Information about the parent class loader

41

Sample toString() from the Updated System CL

public String toString() { if( getParent() != null ) { return "java.net.URLClassLoader:\n" + "hashcode: " + hashCode() + "\n" + "URLs: " + java.util.Arrays.asList( getURLs() ) + "\n" + "parent { " + getParent() + " }\n"; } else { return "java.net.URLClassLoader:\n" + "hashcode: " + hashCode() + "\n" + "URLs: " + java.util.Arrays.asList( getURLs() ) + "\n"; }}

42

Dynamic Server-side Debugging

• Sometimes you need a little snippet of information from the application server that is very difficult to get from a debugger without Herculean effort

• Chris Judd’s Script Debugger• A servlet that includes Groovy and a way to dynamically

execute Groovy scripts• Groovy allows you to ferret out all sorts of server side

information

43

Judd Script Debugger

• Need to know which XML parser is being loaded?

• Interested in a particular JNDI resource?

import javax.xml.parsers.DocumentBuilderFactory as dbf

clazz = dbf.newInstance().newDocumentBuilder().class

clazz.protectionDomain.codeSource.location

import javax.naming.*

ctx = new InitialContext()

ctx.lookup("java:/DefaultDS")

44

Judd Script Debugger

• Want to see all session variables?

output = ""

sessionVars = [:]

for (n in session.attributeNames) {

sessionVars[n] = session.getAttribute(n)

}

sessionVars.each { s |

output = output + "${s.key} = ${s.value} <br>"

}

output

45

Automating Debugging Tasks

• How many times have you walked the same path through a web application to get to the point that you want to debug?

• Stop working so hard for your computer!• Set up jWebUnit to “walk through” the application for you

to a certain point• jWebUnit is designed for unit testing but it has great

automation for driving web applications• While you are there, go ahead and write a unit test!

46

Automating Debugging Tasks

• What if you need to interact with the application once you’ve gotten to the critical point?

• Selenium• A testing tool for web applications• Selenium uses JavaScript and IFrames to embed a test

automation engine in your browser• This technique works with any JavaScript-enabled browser• Developed by ThoughtWorks for internal use• It was so useful that we’ve open-sourced it• Still in early stages (0.4 is latest version)

47

How Does Selenium Work?

48

Using Selenium as a Debugging Aid

• Test cases are trivial to write• Use Selenium to “walk” your web application to the point

where you want to debug it• Either

• Take over by hand• Use Selenium’s step behavior to check individual behavior

49

Words of Debugging Wisdom

• From The Pragmatic Programmer:• “Embrace the fact that debugging is just problem solving, and

attack it as such.”• Tip #24: Fix the problem, not the blame.• Tip #25: Don’t Panic

• The Pragmatic Programmer has lots of information about the debugging mindset

• Don’t use brute force!• Work the problem• Apply as much ingenuity to debugging as to the design

that got you here

Neal Ford

www.nealford.com

nford@thoughtworks.com

memeagora.blogspot.com

Questions?

Samples & slides at www.nealford.com

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license. http://creativecommons.org/licenses/by-nc-sa/2.5/