11.1 Debugging Applications with gdb

The GNU debugger (gdb) is the symbolic debugger of the GNU project and is arguably the most important debugging tool for any Linux system. It has been around for over 10 years, and many non-Linux embedded systems already use it in conjunction with what is known as gdb stubs to debug a target remotely.^[1] Because the Linux kernel implements the ptrace( ) system call, however, you don't need gdb stubs to debug embedded applications remotely. Instead, a gdb server is provided with the gdb package. This server is a very small application that runs on the target and executes the commands it receives from the gdb debugger running on the host. Hence, any application can be debugged on the target without having the gdb debugger actually running on the target. This is very important, because as we shall see, the actual gdb binary is fairly large.

^[1] gdb stubs are a set of hooks and handlers placed in a target's firmware or operating system kernel in order to allow interaction with a remote debugger. The gdb manual explains the use of gdb stubs.

This section discusses the installation and use of gdb in a host/target configuration, not the actual use of gdb to debug an application. To learn how to set breakpoints, view variables, and view backtraces, for example, read one of the various books or manuals that discuss the use of gdb. In particular, have a look at Chapter 14 of Running Linux (O'Reilly) and the gdb manual available both within the gdb package and online at http://www.gnu.org/manual/.

11.1.1 Building and Installing gdb Components

The gdb package is available from ftp://ftp.gnu.org/gnu/gdb/ under the terms of the GPL. Download and extract the gdb package in your ${PRJROOT}/debug directory. For my control module, for example, I used gdb Version 5.2.1. As with the other GNU toolset components I described in Chapter 4, it is preferable not to use the package's directory to build the actual debugger. Instead, create a build directory, move to it, and build gdb:

$ mkdir ${PRJROOT}/debug/build-gdb
$ cd ${PRJROOT}/debug/build-gdb
$ ../gdb-5.2.1/configure --target=$TARGET --prefix=${PREFIX}
$ make
$ make install

These commands build the gdb debugger for handling target applications. As with other GNU toolset components, the name of the binary depends on the target. For my control module, for example, the debugger is powerpc-linux-gdb. This binary and the other debugger files are installed within the $PREFIX directory. The build process proper takes from 5 to 10 minutes on my hardware, and the binary generated is fairly large. For a PPC target, for example, the stripped binary is 4 MB in size when linked dynamically. This is why the gdb binary can't be used as-is on the target and the gdb server is used instead.

At the time of this writing, the gdb built for the target cannot handle target core files. Instead, the faulty program must be run on the target using the gdb server to catch the error as it happens natively. There has been discussion regarding adding cross-platform core file reading capabilities to gdb on the gdb mailing lists, and a few patches are already available. Support for reading cross-platform core files in gdb may therefore be available by the time your read this.

The gdb server wasn't built earlier because it has to be cross-compiled for the target using the appropriate tools. To do so, create a directory to build the gdb server, move to it, and build the gdb server:

$ mkdir ${PRJROOT}/debug/build-gdbserver
$ cd ${PRJROOT}/debug/build-gdbserver
$ chmod +x ../gdb-5.2.1/gdb/gdbserver/configure
$ CC=powerpc-linux-gcc ../gdb-5.2.1/gdb/gdbserver/configure \
> --host=$TARGET --prefix=${TARGET_PREFIX}
$ make
$ make install

The gdb server binary, gdbserver, has now been installed in your ${TARGET_PREFIX}/bin directory. The dynamically linked gdbserver is 25 KB in size when stripped. Compared to gdb, the size of gdbserver is much more palatable.

Once built, copy gdbserver to your target's root filesystem:

$ cp ${TARGET_PREFIX}/bin/gdbserver ${PRJROOT}/rootfs/usr/bin

There are no additional steps required to configure the use of the gdb server on your target. I will cover its use in the next section.

Debugging Information, Symbol Tables, and strip

Most modern Linux binaries are in the ELF format. As with binaries of other formats, ELF binaries contain a number of sections, each with a different name and role. The actual executable code for the binary is usually in the .text section. There are also other sections such as .data for initialized data and .bss for uninitialized data. Debugging information is usually in the .stab and .stabstr sections. These sections are formatted according to the Stabs (short for symbol table) debug format and contain information such as line numbers, paths to source files, paths to include files, variables declarations, types declarations, and so on.

Both objdump and readelf can be used to view the sections of an ELF binary. Here is a sample output generated by running readelf on the unstripped gdbserver binary:

$ powerpc-linux-readelf -S gdbserver 
There are 32 section headers, starting at offset 0x1aca4:
Section Headers:
  [Nr] Name            Type           Addr     Off    Size   ES Flg Lk Inf Al
 ...
  [12] .text           PROGBITS       10000b48 000b48 003008 00  AX   0   0 4
 ...
  [17] .data           PROGBITS       10015470 005470 000914 00  WA   0   0 4
 ...
  [25] .bss            NOBITS         10016114 005e54 00236c 00  WA   0   0 16
  [26] .stab           PROGBITS       00000000 005e54 00798c 0c      27   0 4
  [27] .stabstr        STRTAB         00000000 00d7e0 00d1d5 00       0   0 1
  [28] .comment        PROGBITS       00000000 01a9b5 0001ee 00       0   0 1
  [29] .shstrtab       STRTAB         00000000 01aba3 0000ff 00       0   0 1
  [30] .symtab         SYMTAB         00000000 01b1a4 000f40 10      31  6e 4
  [31] .strtab         STRTAB         00000000 01c0e4 000d19 00       0   0 1
Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings)
              I (info), L (link order), O (extra OS processing required)
              o (os specific), p (processor specific) x (unknown)

When the strip command is used, the sections containing the debugging information, .stab and .stabstr, are removed from the binary along with .symtab and .strtab, while the rest of the sections, except .shstrtab, remain unchanged. The only section that changes is the section header string table, .shstrtab, which shrinks in size since there are fewer sections in the binary. Here is the output generated by running readelf on the stripped gdbserver binary:

$ powerpc-linux-readelf -S gdbserver 
There are 28 section headers, starting at offset 0x6134:
Section Headers:
  [Nr] Name            Type           Addr     Off    Size   ES Flg Lk Inf Al
 ...
  [12] .text           PROGBITS       10000b48 000b48 003008 00  AX   0   0 4
 ...
  [17] .data           PROGBITS       10015470 005470 000914 00  WA   0   0 4
 ...
  [25] .bss            NOBITS         10016114 005e54 00236c 00  WA   0   0 16
  [26] .comment        PROGBITS       00000000 005e54 0001ee 00       0   0 1
  [27] .shstrtab       STRTAB         00000000 006042 0000f0 00       0   0 1
Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings)
              I (info), L (link order), O (extra OS processing required)
              o (os specific), p (processor specific) x (unknown)

For more information on binary formats, including ELF, have a look at John Levine's Linkers & Loaders (Morgan Kaufmann). For information on the Stabs format, have a look at the The "stabs" debug format manual provided in the gdb/doc directory of the gdb package and available online at http://sources.redhat.com/gdb/current/onlinedocs/stabs.html.

11.1.2 Using the gdb Components to Debug Target Applications

Before you can debug an application using gdb, you need to compile your application using the appropriate flags. Mainly, you need to add the -g option to the gcc command line. This option adds the debugging information to the object files generated by the compiler. To add even more debugging information, use the -ggdb option. The information added by both debugging options is thereafter found in the application's binary. Though this addition results in a larger binary, you can still use a stripped binary on your target, granted you have the original unstripped version with the debugging information on your host. To do so, build your application on your host with complete debugging symbols. Copy the resulting binary to your target's root filesystem and use strip to reduce the size of the binary you just copied by removing all symbolic information, including debugging information. On the target, use the stripped binary with gdbserver. On the host, use the original unstripped binary with gdb. Though the two gdb components are using different binary images, the target gdb running on the host is able to find and use the appropriate debug symbols for your application, because it has access to the unstripped binary.

Here are the relevant portions of my command daemon's Makefile that changed (see Chapter 4 for the original Makefile):

...
DEBUG         = -g
CFLAGS        = -O2 -Wall $(DEBUG)
...

Though gcc allows us to use both the -g and -O options in the same time, it is often preferable not to use the -O option when generating a binary for debugging, because the optimized binary may contain some subtle differences when compared to your application's original source code. For instance, some unused variables may not be incorporated into the binary, and the sequence of instructions actually executed in the binary may differ in order from those contained in your original source code.

There are two ways for the gdb server running on the target to communicate with the gdb debugger running on the host: using a crossover serial link or a TCP/IP connection. Though these communication interfaces differ in many respects, the syntax of the commands you need to issue is very similar. Starting a debug session using a gdb server involves two steps: starting the gdb server on the target and connecting to it from the gdb debugger on the host.

Once you are ready to debug your application, start the gdb server on your target with the means of communication and your application name as parameters. If your target has a configured TCP/IP interface available, you can start the gdb server and configure it to run over TCP/IP:

# gdbserver 192.168.172.50:2345 command-daemon

In this example, the host's IP address^[2] is 192.168.172.50 and the port number used locally to listen to gdb connections is 2345. Note that the protocol used by gdb to communicate between the host and the target doesn't include any form of authentication or security. Hence, I don't recommend that you debug applications in this way over the public Internet. If you need to debug applications in this way, you may want to consider using SSH port forwarding to encrypt the gdb session. The book SSH, The Secure Shell: The Definitive Guide (O'Reilly) explains how to implement SSH port forwarding.

^[2] At the time of this writing, this field is actually ignored by gdbserver.

As I said earlier, the command-daemon being passed to gdbserver can be a stripped copy of the original command-daemon built on the host.

If you are using a serial link to debug your target, use the following command line on your target:

# gdbserver /dev/ttyS0 command-daemon

In this example, the target's serial link to the host is the first serial port, /dev/ttyS0.

Once the gdb server is started on the target, you can connect to it from the gdb debugger on the host using the target remote command. If you are connected to the target using a TCP/IP network, use the following command:

$ powerpc-linux-gdb command-daemon
(gdb) target remote 192.168.172.10:2345
Remote debugging using 192.168.172.10:2345
0x10000074 in _start (  )

In this case, the target is located at IP 192.168.172.10 and the port number specified is the same one we used above to start the gdb server on the target. Unlike the gdb server on the target, the command-daemon used here has to be the unstripped copy of the binary. Otherwise, gdb will be of little use to try debugging the application.

If the program exits on the target or is restarted, you do not need to restart gdb on the host. Instead, you need to issue the target remote command anew once gdbserver is restarted on the target.

If your host is connected to your target through a serial link, use the following command:

$ powerpc-linux-gdb progname
(gdb) target remote /dev/ttyS0
Remote debugging using /dev/ttyS0
0x10000074 in _start (  )

Though both the target and the host are using /dev/ttyS0 to link to each other in this example, this is only a coincidence. The target and the host can use different serial ports to link to each other. The device being specified for each is the local serial port where the serial cable is connected.

With the target and the host connected, you can now set breakpoints and do anything you would normally do in a symbolic debugger.

There are a few gdb commands that are you are likely to find particularly useful when debugging an embedded target such as we are doing here. Here are some of these commands and summaries of their purposes:

file: Sets the filename of the binary being debugged. Debug symbols are loaded from that file.
dir: Adds a directory to the search path for the application's source code files.
target: Sets the parameters for connecting to the remote target, as we did earlier. This is actually not a single command but rather a complete set of commands. Use help target for more details.
set remotebaud: Sets the speed of the serial port when debugging remote applications through a serial cable.
set solib-absolute-prefix: Sets the path for finding the shared libraries used with the binary being debugged.

The last command is likely to be the most useful when your binaries are linked dynamically. Whereas the binary running on your target finds its shared libraries starting from / (the root directory), the gdb running on the host doesn't know how to locate these shared libraries. You need to use the following command to tell gdb where to find the correct target libraries on the host:

(gdb) set solib-absolute-prefix ../../tools/powerpc-linux/

Unlike the normal shell, the gdb command line doesn't recognize environment variables such as ${TARGET_PREFIX}. Hence, the complete path must be provided. In this case, the path is provided relative to the directory where gdb is running, but we could use an absolute path, too.

If you want to have gdb execute a number of commands each time it starts, you may want to use a .gdbinit file. For an explanation on the use of such files, have a look at the "Command files" subsection in the "Canned Sequences of Commands" section of the gdb manual.

To get information regarding the use of the various debugger commands, you can use the help command within the gdb environment, or look in the gdb manual.

11.1.3 Interfacing with a Graphical Frontend

Many developers find it difficult or counter-intuitive to debug using the plain gdb command line. Fortunately for these developers, there are quite a few graphical interfaces that hide much of gdb's complexity by providing user-friendly mechanisms for setting breakpoints, viewing variables, and tending to other common debugging tasks. Examples include DDD (http://www.gnu.org/software/ddd/), KDevelop and other IDEs we discussed in Chapter 4. Much like your host's debugger, the cross-platform gdb we built earlier for your target can very likely be used by your favorite debugging interface. Each frontend has its own way for allowing the name of the debugger binary to be specified. Have a look at your frontend's documentation for this information. In the case of my control module, I would need to configure the frontend to use the powerpc-linux-gdb debugger.