yvr18-308: kernel bugs and regressions debugging best
TRANSCRIPT
![Page 1: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/1.jpg)
YVR18-308: Kernel Bugs and Regressions Debugging Best Practices
Rafael David TinocoKernel Validation Team
![Page 2: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/2.jpg)
Topics to be covered● Environment (15 min)
○ Work directory “idea”
○ X-compiling & containers
○ Package generation and repository
○ Bug reproduction environment
■ KVM/QEMU guests■ Boards■ Packaged Test Suites
● Eclipse CDT as IDE (15 min)
○ Using Eclipse as IDE
○ Debugging with Eclipse
○ Eclipse as IDE for the Linux Kernel
● Debugging: Real cases (45 min)○ Interpreting Issues
■ Foundations■ Dead locks / Race conditions■ Usual issues w/ Kernel bugs■ Stack traces
○ BUG #3765■ kdump generation
● QEMU● Board
○ BUG #3303■ Using crash
○ BUG #3903■ SystemTap usage
● Collab to Test Suites (15 min)○ LTP & Kselftests: Lessons Learned
![Page 3: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/3.jpg)
Who am I ?
![Page 4: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/4.jpg)
Environment - 15 minYes, I know, you have a better tool or script.
Who doesn’t ?!
![Page 5: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/5.jpg)
Work directory “idea”● Started at previous company: sustaining engineering debugging needs.● Provided an easy way to back port the same fix into multiple envs.● Bug notes could be easily shared when being put @ Launchpad.● Case notes: Debugging thoughts for fast context switching.
![Page 6: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/6.jpg)
X-compiling and containers● Containers
○ QEMU-user-static for arm■ sysctl -a fs.binfmt_misc | grep qemu
○ LXC backed by debootstrap○ lxc-create
-t download --name mytemplate -- -d debian -r sid
-a (i386|amd64|armhf|arm64)○ isc-dhcp-server AND bind9 (dynamic dns)
○ shared mounts between host and LXC
● X-compiling w/ QEMU-user-static○ When gcc/libc cross comp. isn't enough○ Package dependencies are fully met○ chroots + qemu-user-static is enough○ chroots:
■ aren't good for services■ aren't good for "dist-upgrades"
● if mounting VM ext4 loop dev■ won't give you diff namespaces
○ LXC has some issues, but… it is usable■ armhf or arm64 container on x86
○ QEMU static is faster than QEMU VM
![Page 7: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/7.jpg)
Package generation and repository
https://www.debian.org/doc/debian-policy/policy.pdf
https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf
![Page 8: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/8.jpg)
Package generation and repository
Credits for Slide Content: Lucas Nussbaum ([email protected])
● Debian package workflow: ● Basic idea:
$ apt-get install build-essential$ apt-get install devscripts$ apt-get install ubuntu-dev-tools
$ apt-get build-dep <package>
● And then…
$ cd <package_dir>$ fakeroot debian/rules binary
or$ debuild -b -us -uc
![Page 9: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/9.jpg)
BUG reproduction environment: Boards: Pkg generation and repo
![Page 10: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/10.jpg)
BUG reproduction environment: KVM & QEMU guests
![Page 11: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/11.jpg)
BUG reproduction environment: KVM & QEMU guests
![Page 12: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/12.jpg)
BUG reproduction environment: Boards
● Board Environment○ workstation connected to:
■ hikey 960■ dragonboard 410c■ beagleboard x15
○
Cork is your friend
![Page 13: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/13.jpg)
BUG reproduction environment: Boards● SD cards w/ Debian installed● Upgrading kernel w/ .deb pkgs● Upgrading might be challenging:
○ Different recovery mechanisms○ Different boot loaders○ Removing SD cards manually
● QEMU/KVM, libvirt & LXC ready● .deb test suites easily installed
● Straightforward reproduction based on LKFT output most of the times.
![Page 14: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/14.jpg)
Eclipse CDT as IDE - 15 minI know… I know…
But, for real, give it a try once or twice, you might not regret.It happened to me...
![Page 15: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/15.jpg)
Using Eclipse as IDE
![Page 16: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/16.jpg)
Using Eclipse as IDE
![Page 17: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/17.jpg)
Using Eclipse as IDE
![Page 18: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/18.jpg)
Using Eclipse as IDE
![Page 19: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/19.jpg)
Debugging with Eclipse
![Page 20: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/20.jpg)
Eclipse as IDE for the Linux Kernel
![Page 21: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/21.jpg)
Debugging: Real Cases - 45 minEvery Bug is a learning opportunity
For Real, Every Bug is a learning opportunityTrust me, Every Bug is a HUGE learning opportunity
![Page 22: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/22.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: LInux Kernel Development (Robert Love; Third Edition)
![Page 23: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/23.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 24: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/24.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: LInux Kernel Development (Robert Love; Third Edition)
![Page 25: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/25.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 26: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/26.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 27: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/27.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 28: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/28.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 29: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/29.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 30: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/30.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 31: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/31.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 32: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/32.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 33: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/33.jpg)
Interpreting Issues:Foundations
Credits for Slide Content: https://myaut.github.io/dtrace-stap-book/ (Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License)
![Page 34: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/34.jpg)
Interpreting Issues:Stack traces
![Page 35: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/35.jpg)
Interpreting Issues:Deadlocks / Race conditions
![Page 36: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/36.jpg)
Interpreting Issues:Usual issues with Kernel bugs
![Page 37: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/37.jpg)
Debugging: Real CasesBUG #3765
![Page 38: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/38.jpg)
Debugging: Real CasesBUG #3765 (QEMU kdump generation)
![Page 39: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/39.jpg)
Debugging: Real CasesBUG #3765 (Board kdump generation)
![Page 40: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/40.jpg)
Debugging: Real CasesBUG #3303
![Page 41: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/41.jpg)
Debugging: Real CasesBUG #3303 (Using crash)
![Page 42: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/42.jpg)
Debugging: Real CasesBUG #3903
![Page 43: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/43.jpg)
Debugging: Real CasesBUG #3903 (Systemtap Usage)
![Page 44: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/44.jpg)
Collab to Test Suites - 15 minLessons Learned
![Page 45: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/45.jpg)
LTP & Kselftests: Lessons Learned
![Page 46: YVR18-308: Kernel Bugs and Regressions Debugging Best](https://reader033.vdocuments.net/reader033/viewer/2022061400/629e0b641d819a39dd63ff2a/html5/thumbnails/46.jpg)
LTP & Kselftests: Lessons Learned