python + gdb = javaデバッガ

2016.5.22

数村憲治

Python + GDB = Javaデバッガ

Copyright 2016 FUJITSU LIMITED

JJUG CCC 2016 Spring

M-5

アジェンダ


やりたいこと

GDB Pythonインタフェース

VMStruct

backtraceの実装

TODO

1

やりたいこと


coreファイルをデバッグ

coreからJava VM(C++)とJavaコードを同時に操作

なぜcoreなのか？

透過的なスタックトレース

Javaオブジェクトのダンプ

Javaならjdb、JVMならGDB。Java+JVMは？

2

jstack - man


jstack(1) Troubleshooting Tools jstack(1)

NAME

jstack - Prints Java thread stack traces for a Java process, core file, or remote debug

server. This command is experimental and unsupported.

SYNOPSIS

jstack [ options ] pid

jstack [ options ] executable core

jstack [ options ] [ server-id@ ] remote-hostname-or-IP

options

The command-line options. See Options.

pid The process ID for which the stack trace is printed. The process must be a Java

process. To get a list of Java processes running on a machine, use the jps(1) command.

executable

The Java executable from which the core dump was produced.

core The core file for which the stack trace is to be printed.

3

jstack - man


DESCRIPTION

The jstack command prints Java stack traces of Java threads for a specified Java process, core

file, or remote debug server. For each Java frame, the full class name, method name, byte code

index (BCI), and line number, when available, are printed. With the -m option,

the jstack command prints both Java and native frames

of all threads with the program counter (PC).

OPTIONS

-F

Force a stack dump when jstack [-l] pid does not respond.

-l

Long listing. Prints additional information about locks such as a list of owned

java.util.concurrent ownable synchronizers. See the AbstractOwnableSynchronizer class

description at

http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/AbstractOwnableSynchronizer.html

-m

Prints a mixed mode stack trace that has both Java and

native C/C++ frames.4

jstack -m


% /v3/java/jdk1.8.0_66/bin/jstack -m /v3/java/jdk1.8.0_66/bin/java core.2017

Attaching to core core.2017 from executable /v3/java/jdk1.8.0_66/bin/java, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 25.66-b17

Deadlock Detection:

No deadlocks found.

----------------- 2018 -----------------

0x00007f3492d03149 __pthread_cond_timedwait + 0x129

0x00007f3491e54ff3 _ZN2os5sleepEP6Threadlb + 0x283

0x00007f3491c56f02 JVM_Sleep + 0x3b2

0x00007f348bc15994 * java.lang.Thread.sleep(long) bci:0 (Interpreted frame)

0x00007f348bc07c4d * a.xxx3() bci:11 line:15 (Interpreted frame)



0x00007f348bc07c4d * a.main(java.lang.String[]) bci:0 line:4 (Interpreted frame)

0x00007f348bc007a7 <StubRoutines>

0x00007f3491bc3c46 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056

0x00007f3491c05262 _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPushe

0x00007f3491c21c6a jni_CallStaticVoidMethod + 0x17a

0x00007f3492ae7bcc JavaMain + 0x80c

----------------- 2020 -----------------

0x00007f3492d02da0 __pthread_cond_wait + 0xc0

0x00007f3491e0ef07 _ZN7Monitor5IWaitEP6Threadl + 0xf7

0x00007f3491e0f826 _ZN7Monitor4waitEblb + 0x256

0x00007f3491b0ad93 _ZN13GCTaskManager8get_taskEj + 0x43

0x00007f3491b0bc28 _ZN12GCTaskThread3runEv + 0x188

5

jstackの課題


スタックトレースしか出ない

コアを他のマシンに移すと動作しない

拡張性に乏しい

修正が大変

6

GDB - backtrace


(gdb) bt

#0 0x00007f3492d03149 in pthread_cond_timedwait@@GLIBC_2.3.2 ()

at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238

#1 0x00007f3491e5361f in os::PlatformEvent::park(long) ()

from /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so

#2 0x00007f3491e54ff3 in os::sleep(Thread*, long, bool) ()


#3 0x00007f3491c56f02 in JVM_Sleep ()


#4 0x00007f348bc15994 in ?? ()

#5 0x00007f348c009800 in ?? ()

#6 0x00007f348bc156e2 in ?? ()

…..

#10 0x00007f348b3b3480 in ?? ()

#11 0x0000000000000000 in ?? ()7

stack unwinding


呼び出し元fp

pc (戻りアドレス)

スタック

呼び出し元fp

pc (戻りアドレス)

呼び出し先フレーム

呼び出し元フレーム

fp(フレームポインタ)

sp(スタックポインタ)

デバッガはこのアドレスから該当シンボルを探す

8

コーリングコンベンション


GDBは各言語のコーリングコンベンションを意識してstack unwindingしている

Javaは難しい？

インタプリタ・JIT・ネイティブのミックスフレーム

このフレームがインタプリタフレームかどうか、どうやって判断する？

コードが動的に生成されるテンプレートインタプリタ・JIT

9

Javaコーリングコンベンション


A frame represents a physical stack frame (an activation).

Frames can be C or Java frames,

and the Java frames can be interpreted or compiled.

In contrast, vframes represent source-level activations,

so that one physical frame can correspond to

multiple source level frames because of inlining.

A frame is comprised of {pc, fp, sp}

frame_x86.hpp

frameinterpreted

compiled vframe

vframe

vframe

10

Javaコーリングコンベンション


// Layout of asm interpreter frame:

// [expression stack ] * <- sp

// [monitors ] ¥

// ... | monitor block size

// [monitors ] /

// [monitor block size ]

// [byte code index/pointr] = bcx() bcx_offset

// [pointer to locals ] = locals() locals_offset

// [constant pool cache ] = cache() cache_offset

// [methodData ] = mdp() mdx_offset

// [Method* ] = method() method_offset

// [last sp ] = last_sp() last_sp_offset

// [old stack pointer ] (sender_sp) sender_sp_offset

// [old frame pointer ] <- fp = link() return_addr_offset

// [return pc ]

// [oop temp ] (only for native calls)

// [locals and parameters ]

// <- sender sp11

これまでのアプローチ


GDB自身の改造

OS毎にGDBをビルド

最新のGDBがすぐに使えない

修正変更の管理

HotSpotのコーリングコンベンションの追加

12

他のアプローチ


GDB JIT Compilation Interfacehttps://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html#JIT-Interface

JITの修正が必要。インタプリタはダメ。

GDB script

ifとかwhileはあるが。。。

13

アジェンダ


やりたいこと


VMStruct

backtraceの実装

TODO

14

Pythonインタフェース


https://sourceware.org/gdb/onlinedocs/gdb/Extending-GDB.html#Extending-GDB

GDBでPythonスクリプトが使える

PythonプログラムをGDBでデバッグすることではない

configure時に--with-pythonを指定する

% gdb -configuration

This GDB was configured as follows:

configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu

--with-auto-load-dir=$debugdir:$datadir/auto-load

……..

--with-jit-reader-dir=/usr/lib/gdb (relocatable)

--without-libunwind-ia64

--with-lzma

--with-python=/usr (relocatable)

15

Inferiorオブジェクト


メモリアクセス

— Function: Inferior.read_memory (address, length)

Read length addressable memory units from the

inferior, starting at address. Returns a buffer object,

which behaves much like an array or a string. It can be

modified and given to the Inferior.write_memory

function. In Python 3, the return value is a

memoryview object

16

メモリアクセス関数


def readNbyteAsLittle(addr, size):

try:

buf = gdb.selected_inferior().read_memory(addr, size)

except:

return None

i = size - 1

n = 0

while i >= 0:

n = n << 8

m = ord(buf[i])

n += m

i -= 1

return n

17

関数選択


if (target.isSparc):

readNbyte = readNbyteAsBig

sizeof_pointer = 4

elif (target.isX86):

readNbyte = readNbyteAsLittle

sizeof_pointer = 4

elif (target.isX64):

readNbyte = readNbyteAsLittle

sizeof_pointer = 8

動作環境によって、関数を自動的に切り替え

18

アジェンダ


やりたいこと


VMStruct

backtraceの実装

TODO

19

VMStruct


JVM自身のデバッグ情報

JVM内部にテーブルとして保持

VMStructのアドバンテージ (vmStruct.hpp)

クラス、フィールド、サイズ等

従来、DWARF等で定義されていたもの

プラットフォーム・コンパイラ非依存

デバッグ用ビルドを作らなくてよい

Javaのサイズを保持 (「jlong」 v.s. 「long long」)

20

class VMStruct (vmStruct.hpp)


class VMStructs {

public:

// The last entry is identified over in the serviceability agent by

// the fact that it has a NULL fieldName

static VMStructEntry localHotSpotVMStructs[];


// the fact that it has a NULL typeName

static VMTypeEntry localHotSpotVMTypes[];

// Table of integer constants required by the serviceability agent.


// the fact that it has a NULL typeName

static VMIntConstantEntry localHotSpotVMIntConstants[];

…….21

VMStructEntry


typedef struct {

const char* typeName;

// The type name containing the given field (example: "Klass")

const char* fieldName;

// The field name within the type (example: "_name")

const char* typeString;

// Quoted name of the type of this field (example: "Symbol*";

int32_t isStatic;

// Indicates whether following field is an offset or an address

uint64_t offset;

// Offset of field within structure; only used for nonstatic fields

void* address;

// Address of field; only used for static fields

} VMStructEntry;

Hotspot VMを構成する(C++の)クラス・フィールドを表現

22

VMStructEntry –対応例


typedef struct {

const char* typeName;

const char* fieldName;

const char* typeString;

int32_t isStatic;

uint64_t offset;

void* address;

}

class Klass : public Metadata {

friend class VMStructs;

…

Symbol* _name;

vmStruct.hpp klass.hpp

_nameの(Klassクラスの先頭からの)オフセット使わない

(staticのみ)

0

23

vmstructコマンド


class VMStruct (gdb.Command):

def __init__ (self):

super (VMStruct, self).__init__ ("info vmstruct", gdb.COMMAND

self.gHotSpot = memory.getSymbolAddr("gHotSpotVMStructs")

self.typeNameOffset = memory.readUint64FromSymbol( ¥

"gHotSpotVMStructEntryTypeNameOffset")

self.fieldNameOffset = memory.readUint64FromSymbol( ¥

"gHotSpotVMStructEntryFieldNameOffset")

self.typeStringOffset = memory.readUint64FromSymbol( ¥

"gHotSpotVMStructEntryTypeStringOffset")

self.isStaticOffset = memory.readUint64FromSymbol( ¥

"gHotSpotVMStructEntryIsStaticOffset")

24



src = memory.readPointer(self.gHotSpot)

self.vmstruct = []

while (True) :

entry = self.readVMStructEntry(src)

if (entry == None):

break

self.vmstruct.append(entry)

src += VMStructEntry.entrySize

__init__の続き

25



def readVMStructEntry(self, addr):

#TypeName

target = memory.readPointer(addr + self.typeNameOffset)

if (target == 0):

return None

else:

typeName = memory.readString(target)

if (typeName == ""):

return None

#FieldName

target = memory.readPointer(addr + self.fieldNameOffset)

if (target == 0):

fieldName= ""

else:

fieldName = memory.readString(target)

26



def invoke(self, arg, from_tty):

args = arg.split()

argNum = len(args)

if (argNum == 0):

for entry in self.vmstruct:

entry.printEntry()

elif (argNum == 1):

typeName = args[0]

for entry in self.vmstruct:

if entry.typeName.startswith(arg):

entry.printEntry()

elif (argNum == 2):

…

else:

print ("usage : info vmstruct")

print (" : info vmstruct <typeName>")

27

アジェンダ


やりたいこと


VMStruct

backtraceの実装

TODO

29

backtraceの実現ステップ


現在のフレームが、Javaのフレームかどうか判断する

現在のフレームを実行しているメソッド名を求める

前のフレーム(fp等)をセットする

unwinderの登録

frame filterの登録

30

unwinder


フレームをたどる処理

GDBにunwinderを登録しておくと、フレーム処理のたびに呼び出される

unwinderでは、自分が処理すべきフレームであればgdb.UnwindInfoを、そうでなければNoneを返却。

31

unwinder登録


if (target.isX64):

jvmunwinder = JVMUnwinderX64()

gdb.unwinder.register_unwinder(None, jvmunwinder)

class JVMUnwinderX64(AbstractJVMUnwinder):

name = "JVMUnwinderX64"

enabled = True

def __init__(self):

super(JVMUnwinderX64, self).__init__()

32

unwinder呼び出し


def __call__(self, pending_frame):

pc = pending_frame.read_register('rip')

sp = pending_frame.read_register('rsp')

bp = pending_frame.read_register('rbp')

33

unwinder呼び出し(続き)


if self.isInterpretedFrame(pc) :

sp = bp + sender_sp_offset * memory.sizeof_pointer

pc = gdb.Value(memory.readPointer(bp + ¥

return_addr_offset * memory.sizeof_pointer))

bp = gdb.Value(memory.readPointer(bp))

uinfo = pending_frame.create_unwind_info(FrameId(sp))

uinfo.add_saved_register(6, bp)

uinfo.add_saved_register(7, sp)

uinfo.add_saved_register(16, pc)

return uinfo

elif self.iscCompiledFrame(pc) :

…

else:

return None

34

Javaコーリングコンベンション(再掲)


// Layout of asm interpreter frame:

// [expression stack ] * <- sp

// [monitors ] ¥

// ... | monitor block size

// [monitors ] /

// [monitor block size ]

// [byte code index/pointr] = bcx() bcx_offset

// [pointer to locals ] = locals() locals_offset

// [constant pool cache ] = cache() cache_offset

// [methodData ] = mdp() mdx_offset

// [Method* ] = method() method_offset

// [last sp ] = last_sp() last_sp_offset

// [old stack pointer ] (sender_sp) sender_sp_offset

// [old frame pointer ] <- fp = link() return_addr_offset

// [return pc ]

// [oop temp ] (only for native calls)

// [locals and parameters ]

// <- sender sp35

isInterpretedFrame


class AbstractJVMUnwinder:

__ai = memory.readPointer( ¥

vmstruct.getValue("AbstractInterpreter", "_code"))

__sq_sb = memory.readPointer(

__ai + vmstruct.getValue("StubQueue", "_stub_buffer"))

__sq_bl= memory.readNbyte(

__ai + vmstruct.getValue("StubQueue", "_buffer_limit"), ¥

vmtype.getValue("int"))

def isInterpretedFrame(self, addr):

return ((addr >= self.__sq_sb) and ¥

(addr < self.__sq_sb + self.__sq_bl))

36

AbstractInterpreter


class AbstractInterpreter: AllStatic {


protected:

static StubQueue* _code; // the interpreter code (codelets)

abstractInterpreter.hpp

stubs.hpp

class StubQueue: public CHeapObj<mtCode> {


private:

StubInterface* _stub_interface; // the interface prototype

address _stub_buffer; // where all stubs are stored

int _buffer_size; // the buffer size in bytes

37

frame filter登録


class JavaFilter():

def __init__(self):

self.name = "JavaFilter"

self.priority = 100

self.enabled = True

gdb.frame_filters[self.name] = self

def filter(self, frame_iter):

frame_iter = map(JavaDecorator, frame_iter)

return frame_iter

38

decorator登録


class JavaDecorator(gdb.FrameDecorator.FrameDecorator):

def __init__(self, fobj):

super(JavaDecorator, self).__init__(fobj)

def function(self):

frame = self.inferior_frame()

pc = frame.pc()

if jvmunwinder.isInterpretedFrame(pc) :

methodOop = jvmunwinder.getMethodOop(frame)

name = oop.getMethodName(methodOop)

return ("j " + name)

elif jvmunwinder.isEntryFrame(pc) :

return " <<entryframe>>"

return str(frame.name())

39

backtrace


(gdb) bt

#0 0x00007f3492d03149 in pthread_cond_timedwait@@GLIBC_2.3.2 ()

at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238

#1 0x00007f3491e5361f in os::PlatformEvent::park(long) ()

at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so

#2 0x00007f3491e54ff3 in os::sleep(Thread*, long, bool) ()


#3 0x00007f3491c56f02 in JVM_Sleep () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so

#4 0x00007f348bc15994 in j java/lang/Thread#sleep(J)V ()

#5 0x00007f348bc07c4d in j a#xxx3()V ()

#6 0x00007f348bc07c4d in j a#xxx2()V ()

#7 0x00007f348bc07c4d in j a#xxx1()V ()

#8 0x00007f348bc07c4d in j a#main([Ljava/lang/String;)V ()

#9 0x00007f348bc007a7 in <<entryframe>> ()

#10 0x00007f3491bc3c46 in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) ()


#11 0x00007f3491c05262 in jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*,

#12 0x00007f3491c21c6a in jni_CallStaticVoidMethod () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so

#13 0x00007f3492ae7bcc in JavaMain () at /v3/java/jdk1.8.0_66/bin/../lib/amd64/jli/libjli.so

#14 0x00007f3492cfd6aa in start_thread (arg=0x7f349311b700) at pthread_create.c:333

#15 0x00007f3492618eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

40

アジェンダ


やりたいこと


VMStruct

backtraceの実装

TODO

42

TODO


各種プラットフォーム展開

コマンド充実

oopコマンド

ライブデバッグ

JDK9

43

oopコマンド


(instanceOop*) 0xefe983d0 : oop is java.net.SocksSocketImpl instance.

+00 markOop _mark = 0x3ce7e9f8

+04 klassOop _klass = 0xcd649f38

=== instance value ===

*** java.net.SocketImpl ***

+08 Ljava.net.Socket; socket = 0xefe97db8

+0c Ljava.net.ServerSocket; serverSocket = 0x00000000

+10 protected Ljava.io.FileDescriptor; fd = 0xefe98478

+14 protected Ljava.net.InetAddress; address = 0xdf0cb5b0

+18 protected I port = 0x00009726

+1c protected I localport = 0x0000a5b1

*** java.net.PlainSocketImpl ***

+20 private Ljava.net.SocketInputStream; socketInputStream = 0x00000000

+24 private Ljava.lang.Object; fdLock = 0xefe98468

+28 private Ljava.lang.Object; resetLock = 0xefe98470

+2c private Ljava.io.FileDescriptor; fd1 = 0x00000000

+30 private Ljava.net.InetAddress; anyLocalBoundAddr = 0xd8e4b968

44

python + gdb = javaデバッガ

Software