***************************************************************************** * Kernel debugging, part 3: "Debugging a live kernel" * * * * Copyright (c) 2002 Daniel P. Bovet, Marco Cesati, and Cosimo Comella * * Permission is granted to copy, distribute and/or modify this document * * under the terms of the GNU Free Documentation License, Version 1.1, * * published by the Free Software Foundation; with no Invariant Sections, * * with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the * * license is included in the file named LICENSE. * * * * (version 1.0) * ***************************************************************************** A few techniques are in the toolbox of any kernel hacker: enabling a console on the serial line, and building a LILO-enabled boot floppy disk. *** Part one: enabling a console on the serial line *** When debugging a faulty kernel, the console on the serial line is quite useful. Even if your test machine crash badly, you can read and save any kernel message oops. Even if your X server crashed, you might be open a remote command shell through the serial line. The serial console is also essential for hackers porting Linux to embedded devices without keyboard and screen. Step 1 --- Configure the kernel Proceed as usual. Just make sure to: * in the "Character devices" section - answer "y" to "Standard generic (8250/16550 and compatible UARTs) serial support" - answer "y" to "Support for console on serial port" Step 2 --- Compile the kernel Proceed as usual. Step 3 --- Install the kernel You must specify an option when booting the kernel: "console=ttySX,NNNNPB" where "ttySX" denotes the serial port (ttyS0, ttyS1, ...) "NNNN" is the speed (300,1200,2400,4800,9600,19200,38400,57600,115200) "P" is the type of parity bit ("n", "o", "e") "B" is the number of data bits ("5" ... "8") *** Part two: preparing a boot floppy disk with LILO *** Let's install our new kernel on a floppy disk. We cannot use the native boot loader included in the kernel image, because we need to pass an option to the kernel. Rather, we have to install the LILO boot loader on the floppy disk, together with the kernel image. A few HOWTO's explain how to create a boot and/or root floppy disk with LILO (see for instance the "Bootdisk HOWTO"). We would suggest to refer to those documents, as well as to the LILO manual pages. However, here is a simple receipt to create a *minimal* boot floppy, which may be used to load our hacked kernel images passing to it any boot option. Step 1: format a floppy disk and put an Ext2 filesystem on it. # superformat /dev/fd0 # mke2fs /dev/fd0 (This procedure would work also with a MSDOS/VFAT filesystem. However, installing LILO corrupts the FAT sector, therefore you will be no longer able to mount the diskette.) Step 2: mount the floppy disk. # mount -t auto /dev/fd0 /flp Step 3: copy the kernel image on the floppy. # cat arch/i386/boot/bzImage > /flp/vmlinuz Step 4: copy the boot.b loader into the floppy. All distributions put the LILO files in the /boot directory. However, you don't need to create a boot directory on the floppy. # cat /boot/boot.b > /flp/boot.b Step 5: create a temporary lilo configuration file # cat > /tmp/lilo.conf << EOF lba32 compact serial="0,115200n8" boot=/dev/fd0 install=/flp/boot.b map=/flp/map backup=/dev/null image=/flp/vmlinuz label=test root=/dev/hda3 read-only append="console=ttyS0,115200n8 console=vga" EOF You might omit the "lba32" and/or "compact" option if the floppy fails to boot. Be sure to specify the "/flp" prefix in the "install", "map", and "image" sections. Also, be sure to put the floppy device driver /dev/fd0 in the "boot" section. You must also specify in the "root" section the disk partition where the root filesystem of the test machine is. The "serial" line enables LILO to use the serial port. The "append" section includes the line that enables the kernel support on the serial line. The second "console" option also enables the standard console on the virtual terminal. Step 6: run lilo specifying the temporary configuration file. # lilo -v -C /tmp/lilo.conf Step 7: unmount the floppy disk. # umount /flp (This step is important, do not forget it! Well, I suppose you know why already... ;-) Step 8: start minicom (or another terminal emulator for the serial port) and configure it to use /dev/ttyS0 (or whatever) with the given protocol (speed, parity, number of data bits) # minicom (hit AO and configure the serial port) Step 9: connect a test machine to the development machine with a serial cable. Insert the floppy disk in the drive of the test machine. Power-up the test machine and watch the LILO prompt and the kernel booting messages. Step 10: You might want a getty program that monitors the serial line on the test machine. In the /etc/inittab file of the test machine include a line similar to the following: ======== excerpt from /etc/inittab (on test machine) ======================= s1:12345:respawn:/sbin/agetty 115200 ttyS0 vt100 ============================================================================ If the getty program is present, a login prompt should appear on the minicom window when initialization completes. *** Part three: using a kernel debugger *** In the past lectures we discussed a few techniques that allowed us to get a kernel oops for to a problematic process. We also showed how to make sense of the data in the oops in order to reconstruct the last "kernel history" of the process: the trace of the function calls, the arguments of the functions, the register contents, and so on. This information is quite often more than enough to discover and fix a kernel bug. However, these techniques are similar to the job of a coroner that analyzes a cadaver. Most often, when you read the kernel oops, the corresponding process is already died. This is quite dissimilar from what an application programmer would do: he/she runs a debugger on the program, which allows to put breakpoints on the source code, to define watchdogs on the data structures, to inspect and modify the value of any variable, and so on. Thus, using a debugger is more similar to the job of a physician. Of course, running a debugger on the kernel poses some problems. A traditional debugger is a normal process that monitors the execution of another process by means of special hardware features (like software-generated exceptions) and system calls. Running such a debugger on the vanilla kernel is impossible, because the kernel is not a process by itself. Rather, any process is both kernel and application, depending on whether it is running in Kernel Mode or User Mode, respectively. Thus, a standard debugger (the User Mode side of the debugger process) cannot monitor the kernel (the Kernel Mode side of the debugger process). However, kernel debuggers do exist. There are two main types of them: 1. Integrated kernel debuggers: the kernel code defines some procedures that allow the user to monitor the execution of the kernel itself (for instance, Keith Owens's kdb patch for Linux, see below) 2. Remote kernel debuggers: the kernel includes code that allows a remote debugger to monitor it (for instance kgdb, see below). Real kernel hackers do not like debuggers too much. Although they turn out to be useful in some occasions, they can also be dangerous. As Linus once said: I'm afraid that I've seen too many people fix bugs by looking at debugger output, and that almost inevitably leads to fixing the symptoms rather than the underlying problems. [...] "Use the Source, Luke, use the Source. Be one with the code.". Think of Luke Skywalker discarding the automatic firing system when closing on the deathstar, and firing the proton torpedo (or whatever) manually. _Then_ do you have the right mindset for fixing kernel bugs. However, in a few cases a kernel debugger is invaluable. There are a whole class of kernel bugs that do not produce kernel oops: the lookup of a CPU due to a deadlock. In these cases, being able to snoop inside a live kernel could be quite useful. ========== The kgdb kernel debugger We shall now discuss the usage of a remote debugger. Specifically, we introduce kgdb, a kernel patch that allows using the standard GNU debugger gdb on a live kernel. The main drawback of this technique is that we need two machines: a "monitor" (or "development") machine and a "target" (or "test") machine. The monitor machine runs the debugger as a normal process, while the target machine runs the kernel being debugged. The two machines communicate through the serial port. Step 1: get the kgdb kernel patch. The kgdb kernel patch can be fetched at http://kgdb.sourceforge.net/downloads.html The home page of kgdb is http://kgdb.sourceforge.net/ . Here you might find documentation, hints, and useful macros for the gdb debugger The patch is against a few specific kernel versions. Get the version nearest to your hacked kernel version. For instance, if you kernel version is 2.4.17, get the linux-2.4.16-kgdb-1.3.patch Step 2: patch the kernel. Apply the patch as follows: # patch -p0 < linux-2.4.16-kgdb-1.3.patch You might have some rejected hunks. Look for them: # find linux/ -name \*.rej -ls For each of them, apply the hunk "by hand". By the way, linux-2.4.16-kgdb-1.3.patch works flawlessy for linux-2.4.17 Step 3: configure the kernel. # cd linux # make menuconfig (or config, oldconfig, xconfig) In "Kernel Hacking" section: - answer "yes" to "KGDB: Remote (serial) kernel debugging with gdb" - answer "yes" to "KGDB: Thread analysis" - answer "yes" to "KGDB: Console messages through gdb" Step 4: compile the kernel. # make dep clean bzImage Notice that the option compiler "-g" is always defined. This option tells the compiler to include the source code symbols in the executable code, so that the debugger might relate them with the linear addresses. Step 5: install the kernel on the test machine Be sure to pass the following options to the kernel: "gdb gdbttyS=ttyS0 gdbbaud=115200" For instance, if preparing a boot floppy as in the previous example, replace the "append" section in the lilo.conf file with the following line: append="gdb gdbttyS=0 gdbbaud=115200" Be sure to remove the append="console=ttyS0,115200n8 console=vga" line used when enabling the console on the serial line. Console on serial line and kgdb do not mix. Step 6: boot the test machine with the new kernel. The kernel messages should flow normally through the serial port. The "gdb" option, however, forces the test kernel to wait for a connection from a remote debugger: +Waiting for connection from remote gdb... Step 7: launch the debugger. Go in the main source directory of the patched kernel, and give the command: # gdb vmlinux GNU gdb 5.0 Copyright 2000 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-slackware-linux"... (gdb) Step 8: configure the serial port on the development machine. (gdb) set remotebaud 115200 Step 9: connect to the test machine. Give the "target" command on the development machine: (gdb) target remote /dev/ttyS0 Remote debugging using /dev/ttyS0 0xc010e7a1 in breakpoint () at gdbstub.c:1158 1158 BREAKPOINT(); (gdb) Step 10: complete the initialization of the test kernel. Give the "continue" command on the development machine: (gdb) continue (or simply "c") Continuing. PCI: PCI BIOS revision 2.10 entry at 0xfb4c0, last bus=0 PCI: Using configuration type 1 PCI: Probing PCI hardware Limiting direct PCI/PCI transfers. Linux NET4.0 for Linux 2.4 Based upon Swansea University Computer Society NET3.039 Initializing RT netlink socket Starting kswapd Detected PS/2 Mouse Port. pty: 256 Unix98 ptys configured Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled ttyS01 at 0x02f8 (irq = 3) is a 16550A block: 64 slots per queue, batch=16 Uniform Multi-Platform E-IDE driver Revision: 6.31 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx PIIX4: IDE controller on PCI bus 00 dev 39 PIIX4: chipset revision 1 PIIX4: not 100% native mode: will probe irqs later ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:pio, hdb:pio ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:pio, hdd:pio hda: FUJITSU M1624TAU, ATA DISK drive [snipped] Before allowing the kernel to complete the initialization, you might wanto to set up some breakpoints. Step 11: use the debugger. Now the debugger is monitoring the kernel on the test machine. In order to stop the kernel execution, hit C: (C pressed) Program received signal SIGTRAP, Trace/breakpoint trap. 0xc010e7a1 in breakpoint () at gdbstub.c:1158 1158 BREAKPOINT(); (gdb) The "backtrace" command shows the stack of nested function calls: (gdb) backtrace #0 0xc010e7a1 in breakpoint () at gdbstub.c:1158 #1 0xc015ab24 in gdb_interrupt (irq=4, dev_id=0x0, regs=0xc01f3f98) at gdbserial.c:143 #2 0xc0108072 in handle_IRQ_event (irq=4, regs=0xc01f3f98, action=0xc109b280) at irq.c:451 #3 0xc01081de in do_IRQ (regs={ebx = -1071702016, ecx = -1056292864, edx = -1056292240, esi = -1072672400, edi = -8192, ebp = -1071693876, eax = 0, xds = 24, xes = 24, orig_eax = -252, eip = -1072672362, xcs = 16, eflags = 582, esp = -1071693856, xss = -1072672249}) at irq.c:624 #4 0xc01becc8 in call_do_IRQ () at af_packet.c:1882 #5 0xc0105207 in cpu_idle () at process.c:135 #6 0xc010502a in rest_init () at init/main.c:539 #7 0xc01f4895 in start_kernel () at init/main.c:631 #8 0xc0100197 in L6 () at af_packet.c:1889 Cannot access memory at address 0x8e000 Suppose we wish to continue execution until the kernel returns at the cpu_idle() function. We select that frame by typing: (gdb) frame 5 #5 0xc0105207 in cpu_idle () at process.c:135 135 idle(); We can easily get the source code listing of this portion of kernel: (gdb) list 130 while (1) { 131 void (*idle)(void) = pm_idle; 132 if (!idle) 133 idle = default_idle; 134 while (!current->need_resched) 135 idle(); 136 schedule(); 137 check_pgt_cache(); 138 } 139 } (gdb) Let us define a breakpoint so that the kernel stops right after returning from the the idle() function: (gdb) break process.c:136 Breakpoint 9 at 0xc0105213: file process.c, line 13 (gdb) To continue execution, press "c" (continue) (gdb) c Breakpoint 9, cpu_idle () at process.c:137 137 check_pgt_cache(); To work at Assembly language level: (gdb) disassembly Dump of assembler code for function cpu_idle: 0xc01051c8 : push %ebp 0xc01051c9 : mov %esp,%ebp 0xc01051cb : push %edi 0xc01051cc : push %esi 0xc01051cd : push %ebx 0xc01051ce : call 0xc01fc7a0 0xc01051d3 : mov $0xffffe000,%edx 0xc01051d8 : mov %edx,%eax 0xc01051da : and %esp,%eax 0xc01051dc : movl $0x14,0x24(%eax) 0xc01051e3 : movl $0xffffff9c,0x20(%eax) 0xc01051ea : mov %edx,%edi 0xc01051ec : lea 0x0(%esi,1),%esi 0xc01051f0 : mov 0xc020c864,%esi 0xc01051f6 : test %esi,%esi 0xc01051f8 : jne 0xc01051ff 0xc01051fa : mov $0xc0105170,%esi 0xc01051ff : mov %edi,%ebx 0xc0105201 : and %esp,%ebx 0xc0105203 : jmp 0xc0105207 0xc0105205 : call *%esi 0xc0105207 : mov 0x14(%ebx),%eax 0xc010520a : test %eax,%eax 0xc010520c : je 0xc0105205 0xc010520e : call 0xc0106e80 0xc0105213 : call 0xc011e6c8 0xc0105218 : jmp 0xc01051f0 End of assembler dump. (gdb) info register eip eip 0xc0105213 0xc0105213 Working with high-level source code: The gdb debugger has a huge number of commands and features, be sure to skim the online documentation ("help" command). (gdb) info symbol 0xc01e4340 uts_sem in section .data (gdb) p *(struct rw_semaphore *)0xc01e4340 $2 = {count = -196607, wait_lock = {gcc_is_buggy = 0}, wait_list = {next = 0xc01f9f88, prev = 0xc1103f88}} ============ The kdb kernel debugger The Silicon Graphics' kernel debugger can be obtained from http://www.oss.sgi.com/projects/kdb/ It is also integrated in the IKD (Integrated Kernel Debug) patch, from Keith Owens, now maintained by Andrea Arcangeli (ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/) It usually comes as two patches: an architecture-independent patch (like kdb-v2.1-2.4.17-common-1.bz2) and an architecture-dependent patch (like kdb-v2.1-2.4.17-i386-1.bz2). In order to use it, you must patch a vanilla kernel, and reconfigure it by enabling in the "Kernel hacking" section: "Built-in Kernel Debugger support" (answer "y") "Compile the kernel with frame pointers" (answer "y") Once recompiled and reinstalled, you enter the kernel debugger with the PAUSE key. Most important commands: help give help go continue execution ps list of processes bp,bph,bd,bc handle breakpoints bt,btp,bta show stack(s) md,mds,mm memory manipulation rd,rm registers manipulation ss,ssb single step id disassemble You can also use the kdb on the serial port. To do this, just enable the console on the serial line. To activate the debugger, hit A (twice on minicom, because it is the menu key) Documentation for kdb can be found in Documentation/kbd under the root directory of the kernel (once the patches has been applied).