Joel Fernandes’ 2023 Linux Foundation webinar, Linux Kernel Debugging Tricks of the Trade, is a practical guide to investigating kernel crashes, hangs, warnings and suspected memory errors. It focuses on kernel-specific techniques—not general debugging fundamentals—and is aimed at people who already have some programming and Linux experience.
What is the webinar?
The Linux Foundation recorded the session on September 12, 2023. Its presenter, Joel Agnel Fernandes, is identified by the event page as a Google Staff Software Engineer; the event biography notes his Linux kernel and RCU subsystem maintenance work. The slide deck identifies him as a Kernel RCU Co-Maintainer. The event page describes the intended audience as both seasoned kernel developers and people beginning Linux kernel development, but the slides say the talk skips introductory software-debugging material and moves directly to kernel-specific topics.
The Linux Foundation event page links to the presentation slides and the demo kernel repository. The talk’s examples and configurations reflect the 2023 presentation; consult current kernel documentation before using any boot parameter or command in a live environment.
How to choose a debugging approach
The deck frames kernel debugging as investigative work rather than a universal sequence. Its slide deck says, “Usually no magic formula, requires creative detective work.” The useful starting point is the failure you have, the evidence you can collect, and what your environment supports.
#1 Best Overall
| Failure or question | Approach discussed | What it can reveal | Prerequisites or trade-offs |
|---|---|---|---|
| A reproducible issue where you need to inspect execution | Live debugging with QEMU and GDB; the talk also discusses KGDB/KDB and remote-debugging alternatives. | Code flow, data structures, assembly and execution state. | A virtualized or otherwise debuggable target and suitable symbols are useful. The problem may not reproduce, the investigator may not know what to inspect, or GDB may not be available on the target. |
| A crash that has already occurred | Examine a crash dump with GDB or use the available stack and trace information. | Source-level context and call paths, depending on the dump and build information. | Debug information and usable crash data affect how much detail is available. A live debugging session is not required for GDB to be useful with a crash dump. |
| A hang or suspected lockup | Inspect per-CPU execution and backtraces; enable lockup detectors where appropriate. | Call paths that help locate where execution is stuck, or evidence of an interrupt storm. | Stack quality and the relevant kernel configuration matter; interpreting the evidence requires understanding the system’s state. |
| A warning, oops or panic where recent execution history matters | Configure ftrace to dump trace data around the event. | A record of activity that can provide context around the failure. | Tracing must be configured in advance. An oops may leave the kernel running, while a panic means the kernel cannot recover and must halt or reboot. |
| Suspected memory corruption | Use KASAN, the Kernel Address Sanitizer. | Reports that can help detect errors such as use-after-free and out-of-bounds access. | It requires deliberate configuration and carries a performance cost, as the presentation notes. |
What the talk demonstrates
Symbols, code locations and ASLR
Debug information helps connect kernel execution to source lines. The slides contrast a build without line information with one that includes debug information, and explain that address-space layout randomization (ASLR) can complicate locating code. If a trace or debugger output does not map cleanly to source, build details and address layout are part of the investigation—not just the debugger command.
GDB with QEMU and crash dumps
A live debugger can let you follow execution, inspect data structures and assembly, and investigate a hang. The presentation pairs QEMU with GDB and discusses KGDB/KDB and remote debugging. It also cautions that live debugging is not always practical: a failure may not recur, the target may not support GDB, or it may be unclear where to look. GDB can also be used to inspect a crash dump when a live session is not available.
Stacks, hangs and lockups
The deck recommends considering CONFIG_FRAME_POINTERS to improve stack traces. For a hang, it demonstrates switching among CPU threads and examining backtraces to understand what each CPU was doing. Lockup detectors can help investigate cases such as interrupt storms. These methods turn a general symptom into more specific evidence: a call path, a CPU’s state or a pattern of interrupt activity.
Tracing and kernel diagnostics
The talk covers configuring ftrace to preserve or dump trace data around warnings, oopses and panics. It also discusses lockup detection and KASAN among the in-kernel tools available for particular classes of failure. These tools are not interchangeable: tracing captures activity, stack inspection helps expose call paths, and KASAN targets memory errors. Each depends on configuration appropriate to the problem.
Recommended Free Tools
Rank #3
Where to get the slides and demo code
Open the Linux Foundation webinar page for the downloadable slides and the linked kernel repository with demo code. The material is useful for following the examples and seeing how the presenter applies debugging techniques, but treat its 2023 setup details as examples rather than guaranteed current instructions.
Quick Recap
Rank #4
- Used Book in Good Condition
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
