bud17-tr04: kernel debug stories
TRANSCRIPT
![Page 1: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/1.jpg)
![Page 2: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/2.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
![Page 3: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/3.jpg)
ENGINEERS AND DEVICES
WORKING TOGETHER
●○○○
●○○○○○
●○
●
![Page 4: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/4.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●○○○ printk()
●○○○
●○○○○
●○
![Page 5: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/5.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
0x0 0x0000ffffffffffff 256TB
0xffff000000000000 0xffff000008000000 128MB
0xffff000008000000 0xffff00000807ffff 512KB
0xffff000008080000 10MB+
Kernel image specific 0xffff7dffbfff0000 ~126TB
0xffff7dfffe7fd000 0xffff7dfffec00000 4MB + 12KB
0xffff7dfffee00000 0xffff7dffffe00000 16MB
0xffff7e0000000000 0xffff800000000000 2048G
0xffff800000000000 0xffffffffffffffff 128TB
![Page 6: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/6.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
![Page 7: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/7.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●○ ⇒○ ⇒○ ⇒○ ⇒
●○ printk○ git grep printk
●○
○
○
![Page 8: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/8.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●○ FRAME_WARN○ SCHED_STACK_END_CHECK
●○ slub_debug=
■ SLUB_DEBUG
○ DEBUG_PAGEALLOC○ PAGE_POISONING
●○ DEBUG_SPINLOCK○ DEBUG_MUTEXES○ DEBUG_ATOMIC_SLEEP○ DEBUG_LOCK_ALLOC○ LOCKUP_DETECTOR
●
![Page 9: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/9.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●○○○
●○○
●○○ PROVE_LOCKING○ KASAN
![Page 10: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/10.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
PROVE_LOCKING
lock(subsys) lock(driver)lock(driver) lock(subsys)
DEBUG_SPINLOCKDEBUG_MUTEXES
lock(subsys)lock(driver)unlock(driver)unlock(subsys)lock(driver)lock(subsys)
PROVE_LOCKING
PROVE_LOCKING
![Page 11: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/11.jpg)
ENGINEERS AND DEVICES
WORKING TOGETHER
●○○○
●○○○○○
●○
●
![Page 12: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/12.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●
●
●
●●
●
![Page 13: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/13.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
make cscope git grep 'struct foo {'
printk() [ 0.001636] xyz: Found 10 widgets⇒ git grep "Found .* widgets"
I can’t reproduce butmy customer can
![Page 14: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/14.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●
○○○
●○○ ⇒○ quiet
LOG_BUF_SHIFT log_bug_len=● pr_debug()
○ <8> <7>○○ DYNAMIC_DEBUG
I can’t reproduce butmy customer can
![Page 15: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/15.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
I can’t reproduce butmy customer can
SP
LR (r30)
FP (r29)
r19..r28
r18
IP0/IP1 (r16-r17)
r9..r15
r8
r0..r7
![Page 16: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/16.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
git grep REGMAP_ALLOW_WRITE_DEBUGFS
I can’t reproduce butmy customer can
config DEBUG_FSbool "Debug Filesystem"select SRCUhelp debugfs is a virtual file system that kernel developers use to put debugging files into. Enable this option to be able to read and write to these files.
If unsure, say N.
![Page 17: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/17.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
I can’t reproduce butmy customer can
<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30bl ffff000008092ea0 <_mcount>ldp x29, x30, [sp],#16ret
<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30nopldp x29, x30, [sp],#16ret
<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30bl ftrace_callerldp x29, x30, [sp],#16ret
![Page 18: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/18.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●●
○ ftrace=function ftrace_filter=mydrv_* tp_printk○ ftrace=function_graph ftrace_graph_notrace=rcu*,*lock,*spin*
●cd /sys/kernel/debug/tracingecho function > current_tracerecho 1 > tracing_on
●○○○ ftrace_dump_on_oops○
● trace_printk()
I can’t reproduce butmy customer can
![Page 19: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/19.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
/proc/vmcore
console=ttyS0,1115200 … … crashkernel=128M
-p
./kexec -p vmlinux --dtb=xxx.dtb --append="root=/dev/mmcblk0p9 rw 1 maxcpus=1 reset_devices"
I can’t reproduce butmy customer can
![Page 20: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/20.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●
●
●
●
●
●
●printk()
![Page 21: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/21.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●
●●
●○○
○ echo 'sched:*' > /sys/kernel/debug/tracing/set_event
My widget missedits deadline
![Page 22: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/22.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●●●●●●
My widget missedits deadline
![Page 23: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/23.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●●●●
.
![Page 24: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/24.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●●
●
●
●
●
![Page 25: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/25.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●○○ initcall_debug○ initcall_debug clk_disable_unused() clk_ignore_unused○ DEBUG_SPINLOCK DEBUG_MUTEX LOCKUP_DETECTOR○
●○○
○
○○
My board juststopped dead
![Page 26: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/26.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●○
●○
○
○
●○○○
●○○
My board juststopped dead
![Page 27: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/27.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●
●●
○○
○●●
●
My board juststopped dead
![Page 28: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/28.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
My board juststopped dead
[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78[ 277.600833] PCSR of cluster0 cpu2 is 0xffff00000809a934[ 277.609564] PCSR of cluster0 cpu3 is 0xffff00000809a934
![Page 29: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/29.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●
●
●
●
○○○
![Page 30: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/30.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
git bisect start <bad_sha1> <good_sha1>git bisect run compile_kernel_and_run_test.sh
compile_kernel_and_test.sh
compile_kernel_and_test.sh
I’m sure thisused to work
![Page 31: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/31.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
ktest.pl
compile_kernel_and_test.sh
Documentation/ tools/testing/ktest/sample.conftools/testing/ktest/examples
I’m sure thisused to work
![Page 32: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/32.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
●●
●
●
●
●
![Page 33: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/33.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
My board justrandomly failed
![Page 34: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/34.jpg)
ENGINEERS AND DEVICES
WORKING TOGETHER
●○○○
●○○○○○
●○
●
![Page 35: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/35.jpg)
ENGINEERS AND DEVICESWORKING TOGETHER
![Page 36: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/36.jpg)
ENGINEERS AND DEVICES
WORKING TOGETHER
Graphic by Jost, CC0-PD
![Page 37: BUD17-TR04: Kernel Debug Stories](https://reader036.vdocuments.net/reader036/viewer/2022062306/58ce8ebf1a28ab8c3b8b4e3f/html5/thumbnails/37.jpg)
ENGINEERS AND DEVICES
WORKING TOGETHER
●○○○
●○○○○○
●○
●