Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
There is no single best Linux kernel debugger. Start by asking what evidence you need, whether the problem reproduces consistently, whether instrumentation changes its timing, and how much access you have to the target. Then choose the least disruptive method likely to answer the question: selected debug messages, execution traces, performance evidence, a post-crash dump, or live inspection.
Start with the failure, not the debugger
Before enabling tools, establish what you can reproduce and what you can safely change. A reliable, repeatable failure invites a different approach from a rare race on a system you cannot stop. The Linux kernel debugging guide frames two useful questions: “Do I need to debug with limited access?” and “Is timing a factor?”
- Reproducibility: Record the steps and conditions that trigger the failure, and whether repeated attempts produce the same result.
- Timing sensitivity: Note whether adding logging, tracing, or a breakpoint changes the symptom. Instrumentation can alter execution timing and hide a race.
- Target access: Establish whether you have root access, can change the kernel or module, can connect a development host, and can interrupt the target.
- Evidence needed: Decide whether you need a specific message, a call sequence, performance data, state captured at a crash, or interactive source-level inspection.
- Build and hardware: Check the kernel configuration, architecture, available debug symbols, and any supported debugger I/O such as a serial connection.
These methods can be combined. Begin with the lowest-cost evidence that can test your current hypothesis, then add instrumentation if the result is inconclusive.
Choose a method by the evidence you need
| Need | Start with | What to check |
|---|---|---|
| Messages from selected kernel code | Dynamic debug | The relevant debug-print callsites must already be compiled in; dynamic debug support and root access are needed. |
| Function execution or kernel events | ftrace and event tracing | Available functions, events, tracefs controls, and required kernel configuration depend on the build. |
| An unexplained slowdown or system-wide issue | Ordinary performance tools, then perf |
Narrow the investigation using system load, CPU, I/O, memory, and per-process evidence. |
| State captured when the kernel crashes | Kdump | A crash dump must be captured and available for later analysis; deployment steps depend on distribution and architecture. |
| Live inspection of a stopped target | KGDB or KDB | Plan for a development host and target setup, matching symbols for KGDB, and suitable debugger I/O. |
When do dynamic debug and printk help?
Use dynamic debug for selected messages
Dynamic debug can enable or disable supported kernel debug-print callsites at runtime. If the target exposes /proc/dynamic_debug/control, its catalog can be queried by source filename, function, line, module, format string, or declared class. It only controls callsites already present in the built kernel; it does not add debug instrumentation to arbitrary code. The kernel documentation describes dynamic debug as a way to obtain additional kernel information by changing which debug-print code is enabled.
#1 Best Overall
Root access and a kernel built with suitable dynamic-debug support are required. Enabling a callsite does not guarantee that its message will appear on the console: kernel log-level settings also affect console visibility. Check the target’s kernel documentation and logging setup before concluding that a callsite did not run.
Take care when timing matters
Ordinary printk() logging can affect timing. If the failure changes or disappears when messages are added, the kernel debugging guidance points to trace_printk() as an alternative that logs to the trace file. Treat a changed symptom as evidence that the observation method may be affecting the failure, not as proof that the underlying issue is gone.
Rank #2
When should you use ftrace and event tracing?
Use tracing when you need to understand function execution or correlate kernel events rather than simply turn on selected messages. The tracing family includes ftrace, tracepoints and event tracing, as well as facilities such as Kprobes, Fprobes, Eprobes, boot-time tracing, histograms, and hardware or performance tracing. Which options are available depends on the kernel build and subsystem; use the documentation for the specific tracing feature you need.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →The kernel’s userspace tracing guide names CONFIG_DYNAMIC_FTRACE as a prerequisite for its ftrace guidance. Tracefs exposes controls and buffers, including function filters, stack traces, and the trace_pipe interface. Configuration and available functions or events vary, so verify what the running kernel provides instead of assuming that a feature documented for another build is present. KernelShark can visualize trace-cmd output, but the core tracing controls are exposed through tracefs.
Tracing is useful for narrowing a sequence of events or functions before committing to an interactive debugger. It is still instrumentation: if the failure is timing-sensitive, evaluate whether the tracing setup changes its behavior.
When is performance analysis a better first step?
If the problem is not localized, or interrupting the system is undesirable, begin with ordinary system performance tools and move to perf for deeper analysis. System load, CPU use, I/O, memory, and per-process measurements can help identify where to focus before you add kernel-specific instrumentation. This approach is especially useful for unexplained slowdown, where stopping execution to inspect a live state may not answer the original question.
When should you use Kdump?
Kdump is the Linux kernel’s kexec-based crash-dump solution. Use it when a kernel crash must be investigated from captured state after the event, rather than as a live interactive debugger. Its usefulness depends on having a dump available to examine.
Setup commands, service configuration, and supported details depend on the distribution and architecture. Follow the current documentation for the distribution and target system in question; do not assume one set of deployment steps applies everywhere.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When do KGDB and KDB make sense?
KGDB: source-level inspection with GDB
KGDB connects a development host running GDB to the target kernel under investigation. The kernel documentation states: “Two machines are required for using kgdb.” GDB needs the target’s matching, symbol-bearing vmlinux; a compressed boot image is not a substitute. Enable CONFIG_DEBUG_INFO to provide debug information. CONFIG_FRAME_POINTER is advised for more accurate stack backtraces, but is not required. Exact configuration labels and available options can vary with kernel version and architecture.
For serial debugging, the documentation describes kgdboc as the main mechanism for connecting GDB to KGDB and interacting with KDB. kgdbwait and early debugging require the KGDB I/O driver to be built into the kernel. A tty already serving as an active system console cannot be used with kgdboc plus kgdbcon. These constraints make it important to plan the target’s console and I/O arrangement before attempting a session.
A USB-to-serial adapter may help connect a development host to a target using serial KGDB, but compatibility depends on the target UART and board. Check the board documentation for the required connector and electrical compatibility before choosing an adapter.
KDB: console-based inspection and basic run control
KDB is a shell-like console debugger for examining system state and basic run control. It can be useful for inspecting memory, registers, processes, logs, and basic breakpoints when a console-based interface fits the task. It is distinct from GDB’s source-level interface, although the two frontends can interoperate through the shared debug core. The kernel documentation requires KGDB support to be enabled for KDB.
What can prevent a method from working?
- Missing dynamic-debug output: Confirm that the kernel includes dynamic debug and that the callsite exists in the build. Check the control catalog and account for log-level settings when judging console output.
- Unavailable trace options: Verify the running kernel’s configuration and the functions or events it exposes. The presence of tracefs alone does not establish that every tracing feature is available.
- Unhelpful KGDB backtraces: Check that GDB is using the matching symbol-bearing
vmlinuxand that debug information is available. Frame pointers may improve backtrace accuracy when supported. - Serial debugger conflicts: Check which tty is assigned to the system console and whether the KGDB I/O driver is built in when using
kgdbwaitor early debugging. - No crash evidence: Kdump is for later analysis of a captured crash state; it cannot provide that evidence if no usable dump is available.
A practical escalation path
- Characterize the symptom. Record what triggers it, whether it repeats, whether timing changes it, and whether the target can be interrupted or modified.
- Choose the least disruptive evidence source. For selected existing debug messages, check dynamic debug; for an unexplained slowdown, start with system performance tools; for execution flow or events, inspect the tracing facilities available in the build.
- Escalate only when the evidence calls for it. Use Kdump when post-crash state is required. Use KGDB or KDB when live state inspection justifies the setup and interruption.
- Match the tools to the exact target. Confirm configuration, symbols, architecture, console arrangement, and debugger I/O against documentation for the kernel and distribution actually under test.
Kernel documentation is version-sensitive: its tracing guide includes rolling “latest” material, while the userspace debugging guide cited here is versioned for Linux v7.0. Check documentation matching the kernel under test because configuration, available trace events, architectures, and debugger I/O support can differ by release and distribution.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
