Software & Services Group
1
PinADX: Customizable Debugging with Dynamic Instrumentation
Gregory Lueck, Harish Patil, Cristiano Pereira
Intel Corporation
CGO 2012, San Jose, USA
Software & Services Group
2
Hypothetical Problem 1
(gdb) runProgram received signal SIGSEGV, Segmentation fault.0x0000000000000000 in ?? ()(gdb) bt#0 0x0000000000000000 in ?? ()#1 0x0000000000000000 in ?? ()
Crash with bad PC and no stack traceCorrupted return address someplace ...Want to stop BEFORE bad “ret” instruction
Software & Services Group
3
Hypothetical Problem 2
thread stack
?
thread stack
?
thread stack
?
thread stack
?...
Massively threaded applicationHow much stack space needed?At what point does each thread use its max stack?
Software & Services Group
Traditional Debugger Breakpoints?
4
OriginalApplication
… 1. Debugger catches trap2. Check if “ret” is to good PC3. If yes, resume
How can debugger find all “ret” instructions?Horribly slow to trap on each return
Foo:ret
Bar: …ret
ApplicationIn Debugger
…Foo:trap
Bar: …trap
Overwrite each “ret” with trap ret
ret
Software & Services Group
5
Dynamic Binary Instrumenation (DBI)Application
if (return to bad PC) Breakpoint()
Applicationif (stack too big) Breakpoint()
Much faster – avoids trap overheadDBI can find all “ret” instructions reliablyGeneral approach – solves stack problem (and others)BUT difficult to integrate with debugger
sub 0x60, %sp…Foo:
retBar: …
ret
Foo:
Bar:
sub 0x10, %sp
…
…Instrumentation
Software & Services Group
Pin Overview
Tool
JIT compiler
Appl
icat
ion
Code Cache
Trac
es
Tool controls instrumentation(e.g. “if return to bad PC”)
JIT compiler fetches application instructions, calls tool to instrument
Instrumented instructions stored in code cache for efficiency
FetchIn
stru
men
t Optim
izeStore &
execute
6
Software & Services Group
Original Code Code Cache
JIT Compiler Overview
1’
2
1
6
3
5
2’
6’
Pin
4
3’
5’
Tool inserts instrumentation (e.g. check if return to bad PC)Dynamic
recompilation makes debugging hard
7
Software & Services Group
8
Process running under Pin
PinADX Architecture
Tool
Pin
Appl
icat
ion Debugger
PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.
Tool extends debugger via instrumentation.
PinADXcore
GDBor
Microsoft Visual Studio 11
Supports Linux & Windows
Software & Services Group
9
Rest of the Talk• Introduction / Motivation• Example: Using “Stack Debugger”
extension• Example: Authoring “Stack Debugger”
extension• Implementing PinADX
Software & Services Group
10
$ gdb ./my-application(gdb) target remote :1234(gdb) break PrintHelloBreakpoint 1 at 0x4004dd: file hw.c, line 13(gdb) contBreakpoint 1, PrintHello () at hw.c:13(gdb) backtrace#0 PrintHello () at hw.c:13#1 main () at hw.c:7(gdb) x/2i $pc=> 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi 0x4004e2 <PrintHello+9>: callq 0x4003b8
Example – Stack Debugger$ pin –appdebug –t stack-debugger.so -- ./my-applicationApplication stopped until continued from debugger.Start GDB, then issue this command at the (gdb) prompt: target remote :1234
Run application under Pin
Debugger connected to Pin
(gdb) contBreakpoint 1, PrintHello () at hw.c:13(gdb) backtrace#0 PrintHello () at hw.c:13#1 main () at hw.c:7(gdb) x/2i $pc=> 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi 0x4004e2 <PrintHello+9>: callq 0x4003b8
(gdb) break PrintHelloBreakpoint 1 at 0x4004dd: file hw.c, line 13
Software & Services Group
11
(gdb) monitor stackbreak 4000Break when thread uses 4000 stack bytes(gdb) contStopped: Thread uses 4004 bytes of stack(gdb) backtrace#0 0x3f07214445 in _dl_runtime_resolve ()#1 0x00004004e7 in PrintHello () at hw.c:13#2 0x00004004d2 in main () at hw.c:7(gdb) monitor stackbreak 10000Break when thread uses 10000 stack bytes(gdb) break exitBreakpoint 2 at 0x7fffe60f9650(gdb) contBreakpoint 2, 0x7fffe60f9650 in exit ()(gdb) monitor statsMaximum stack usage: 8560 bytes.
Example – Stack DebuggerStop when application uses too much
stack(gdb) monitor stackbreak 4000Break when thread uses 4000 stack bytes(gdb) contStopped: Thread uses 4004 bytes of stack
(gdb) monitor stackbreak 10000Break when thread uses 10000 stack bytes(gdb) break exitBreakpoint 2 at 0x7fffe60f9650(gdb) contBreakpoint 2, 0x7fffe60f9650 in exit ()
(gdb) backtrace#0 0x3f07214445 in _dl_runtime_resolve ()#1 0x00004004e7 in PrintHello () at hw.c:13#2 0x00004004d2 in main () at hw.c:7
(gdb) monitor statsMaximum stack usage: 8560 bytes.
Software & Services Group
12
Rest of the Talk• Introduction / Motivation• Example: Using “Stack Debugger”
extension• Example: Authoring “Stack Debugger”
extension• Implementing PinADX
Software & Services Group
13
Stack Debugger – Instrumentation
Thread Start:
[…]sub $0x60, %esp
cmp %esi, %edxjle <L1>
size = StackBase - %esp;if (size > MaxStack) MaxStack = size;if (size > StackLimit) TriggerBreakpoint();
StackBase = %esp;MaxStack = 0;
After each stack-changing instruction
Record initial stack
Software & Services Group
14
VOID Instruction(INS ins, VOID *){ if (INS_RegWContain(ins, REG_STACK_PTR)) { IPOINT where = (INS_HasFallThrough(ins)) ? IPOINT_AFTER : IPOINT_TAKEN_BRANCH; INS_InsertCall(ins, where, (AFUNPTR)OnStackChange, IARG_REG_VALUE, REG_STACK_PTR, IARG_THREAD_ID, IARG_CONTEXT, IARG_END); }}VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt){ size_t size = StackBase - sp; if (size > StackMax) StackMax = size; if (size > StackLimit) { ostringstream os; os << "Stopped: Thread uses " << size << " stack bytes."; PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str()); }}
Stack Debugger – ImplementationIn
stru
men
tatio
nAn
alys
is
Instrument only instructions that change $SPCall after each instruction
Software & Services Group
15
int main() { […] PIN_AddDebugInterpreter(HandleDebugCommand, 0);} BOOL HandleDebugCommand(const string &cmd, string *result) { if (cmd == "stats") { ostringstream os; os << "Maximum stack usage: " << StackMax << " bytes.\n"; *result = os.str(); return TRUE; } else if (cmd.find("stackbreak ") == 0) { StackLimit = /* parse limit */; ostringstream os; os << "Break when thread uses " << limit << " stack bytes."; *result = os.str(); return TRUE; } return FALSE; // Unknown command}
Stack Debugger – Implementation
Software & Services Group
16
Visual Studio IDE Extension
Software & Services Group
17
Other Debugger Extensions
• Intel Inspector XE Product– Memory Checker– Thread Checker
• Intel SDE: Instruction emulation• Debug from log file (PinPlay, CGO 2010)• Dynamic slicing (Rajiv Gupta, UC
Riverside)• Cmp$im: Cache simulator• Write your own!
Software & Services Group
18
Rest of the Talk• Introduction / Motivation• Example: Using “Stack Debugger”
extension• Example: Authoring “Stack Debugger”
extension• Implementing PinADX
Software & Services Group
19
Process running under Pin
PinADX Architecture
Tool
Pin
Appl
icat
ion Debugger
PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.
Tool extends debugger via instrumentation.
PinADXcore
GDBor
Microsoft Visual Studio 11
Software & Services Group
20
Communication Details
• Very low level• Symbol processing in debugger• Expression evaluation in debugger• Extension of GDB’s remote debugging protocol
Commands• Read / write registers, memory• Set breakpoints• Continue, single-step, stop
Notifications• Breakpoint triggered• Caught signal• Application exited
DebuggerPinADXcore
Pin
Software & Services Group
21
Communication Details
Breakpoint alternatives• Insert real INT3 trap instruction• Virtualize inside Pin VMSee paper for details
Commands• Read / write registers, memory• Set breakpoints• Continue, single-step, stop
Notifications• Breakpoint triggered• Caught signal• Application exited
DebuggerPinADXcore
Pin
Software & Services Group
set breakpoint at 4continue
Code CacheOriginal Code
Breakpoint
1’
2
3
1
4
5
6
breakpoint notification
Execution stops in PinWaits for GDB to continue
BP
2’
3’
PinADXcore Debugger
22
Software & Services Group
Code CacheOriginal Code
Single Step
1’
2
3
1
4
5
6
step complete notification
Debugger
Execution stops in PinWaits for GDB to continue
do single-stepPinADX
core
23
Software & Services Group
24
Thanks
• Mark Charney – SDE software emulator• Andria Pazarloglou – Created VS11 GUI plugin• Gregg Miskelly – Microsoft VS11 debugger
architect• Robert Cohn – Father of Pin
Software & Services Group
25
Summary
• DBI can implement powerful debugger features
• API allows Pin tools to extend debugger easily
• Multi-platform– Linux: GDB– Windows: Microsoft Visual Studio 11 (soon)
• Works with off-the-shelf debuggers
http://pintool.org