In the previous chapter, we explored the use of GDB for debugging kernel code and code resident in Flash, such as bootloader code. In this chapter, we continue our coverage of GDB for debugging application code in user space. We extend our coverage of remote debugging and the tools and techniques used for this peculiar debugging environment.
We already explored several important debugging tools in Chapter 13, "Development Tools." strace and ltrace can be used to observe and characterize a process's behavior and often isolate problems. dmalloc can help isolate memory leaks and profile memory usage. ps and top are both useful for examining the state of processes. These relatively small tools are designed to run directly on the target hardware.
Debugging Linux application code on an embedded system has its own unique challenges. Resources on your embedded target are often limited. RAM and nonvolatile storage limitations might prevent you from running target-based development tools. You might not have an Ethernet port or other high-speed connection. Your target embedded system might not have a graphical display, keyboard, or mouse.
This is where your cross-development tools and an NFS root mount environment can yield large dividends. Many tools, especially GDB, have been architected to execute on your development host while actually debugging code on a remote target. GDB can be used to interactively debug your target code or to perform a postmortem analysis of a core file generated by an application crash. We covered the details of application core dump analysis in Chapter 13.
Cross-development tools were developed primarily to overcome the resource limitations of embedded platforms. A modest-size application compiled with symbolic debug information can easily exceed several megabytes. With cross-debugging, the heavy lifting can be done on your development host. When you invoke your cross-version of GDB on your development host, you pass it an ELF file compiled with symbolic debug information. On your target, there is no reason you can't strip[100] the ELF file of all unnecessary debugging info to keep the resulting image to its minimum size.
We introduced the readelf utility in Chapter 13. In Chapter 14, "Kernel Debugging Techniques," we used it to examine the debug information in an ELF file compiled with symbolic debugging information. Listing 15-1 contains the output of readelf for a relatively small web server application compiled for the ARM architecture.
Listing 15-1. ELF File Debug Info for Example Program
$ xscale_be-readelf -S websdemo
There are 39 section headers, starting at offset 0x3dfd0:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .interp PROGBITS 00008154 000154 000013 00 A 0 0 1
[ 2] .note.ABI-tag NOTE 00008168 000168 000020 00 A 0 0 4
[ 3] .note.numapolicy NOTE 00008188 000188 000074 00 A 0 0 4
[ 4] .hash HASH 000081fc 0001fc 00022c 04 A 5 0 4
[ 5] .dynsym DYNSYM 00008428 000428 000460 10 A 6 1 4
[ 6] .dynstr STRTAB 00008888 000888 000211 00 A 0 0 1
[ 7] .gnu.version VERSYM 00008a9a 000a9a 00008c 02 A 5 0 2
[ 8] .gnu.version_r VERNEED 00008b28 000b28 000020 00 A 6 1 4
[ 9] .rel.plt REL 00008b48 000b48 000218 08 A 5 11 4
[10] .init PROGBITS 00008d60 000d60 000018 00 AX 0 0 4
[11] .plt PROGBITS 00008d78 000d78 000338 04 AX 0 0 4
[12] .text PROGBITS 000090b0 0010b0 019fe4 00 AX 0 0 4
[13] .fini PROGBITS 00023094 01b094 000018 00 AX 0 0 4
[14] .rodata PROGBITS 000230b0 01b0b0 0023d0 00 A 0 0 8
[15] .ARM.extab PROGBITS 00025480 01d480 000000 00 A 0 0 1
[16] .ARM.exidx ARM_EXIDX 00025480 01d480 000008 00 AL 12 0 4
[17] .eh_frame_hdr PROGBITS 00025488 01d488 00002c 00 A 0 0 4
[18] .eh_frame PROGBITS 000254b4 01d4b4 00007c 00 A 0 0 4
[19] .init_array INIT_ARRAY 0002d530 01d530 000004 00 WA 0 0 4
[20] .fini_array FINI_ARRAY 0002d534 01d534 000004 00 WA 0 0 4
[21] .jcr PROGBITS 0002d538 01d538 000004 00 WA 0 0 4
[22] .dynamic DYNAMIC 0002d53c 01d53c 0000d0 08 WA 6 0 4
[23] .got PROGBITS 0002d60c 01d60c 000118 04 WA 0 0 4
[24] .data PROGBITS 0002d728 01d728 0003c0 00 WA 0 0 8
[25] .bss NOBITS 0002dae8 01dae8 0001c8 00 WA 0 0 4
[26] .comment PROGBITS 00000000 01dae8 000940 00 0 0 1
[27] .debug_aranges PROGBITS 00000000 01e428 0004a0 00 0 0 8
[28] .debug_pubnames PROGBITS 00000000 01e8c8 001aae 00 0 0 1
[29] .debug_info PROGBITS 00000000 020376 013d27 00 0 0 1
[30] .debug_abbrev PROGBITS 00000000 03409d 002ede 00 0 0 1
[31] .debug_line PROGBITS 00000000 036f7b 0034a2 00 0 0 1
[32] .debug_frame PROGBITS 00000000 03a420 003380 00 0 0 4
[33] .debug_str PROGBITS 00000000 03d7a0 000679 00 0 0 1
[34] .note.gnu.arm.ide NOTE 00000000 03de19 00001c 00 0 0 1
[35] .debug_ranges PROGBITS 00000000 03de35 000018 00 0 0 1
[36] .shstrtab STRTAB 00000000 03de4d 000183 00 0 0 1
[37] .symtab SYMTAB 00000000 03e5e8 004bd0 10 38 773 4
[38] .strtab STRTAB 00000000 0431b8 0021bf 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)
$
You can see from Listing 15-1 that there are many sections containing debug information. There is also a .comment section that contains more than 2KB (0x940) of information that is not necessary for the application to function. The size of this example file, including debug information, is more than 275KB.
$ ls -l websdemo
-rwxrwxr-x 1 chris chris 283511 Nov 8 18:48 websdemo
If we strip this file using the strip utility, we can minimize its size to preserve resources on our target system. Listing 15-2 shows the results.
Listing 15-2. Strip Target Application
$ xscale_be-strip -s -R .comment -o websdemo-stripped websdemo
$ ls -l websdemo*
-rwxrwxr-x 1 chris chris 283491 Apr 9 09:19 websdemo
-rwxrwxr-x 1 chris chris 123156 Apr 9 09:21 websdemo-stripped
$
Here we strip both the symbolic debug information and the .comment section from the executable file. We specify the name of the stripped binary using the -o command line switch. You can see that the resulting size of the stripped binary is less than half of its original size. Of course, for larger applications, this space savings can be even more significant. A recent Linux kernel compiled with debug information was larger than 18MB. After stripping as in Listing 15-2, the resulting binary was slightly larger than 2MB!
For debugging in this fashion, you place the stripped version of the binary on your target system and keep a local unstripped copy on your development workstation containing symbolic information needed for debugging. You use gdbserver on your target board to provide an interface back to your development host where you run the full-blown version of GDB on your nonstripped binary.
Using gdbserver allows you to run GDB from a development workstation rather than on the target embedded Linux platform. This configuration has obvious benefits. For starters, it is common that your development workstation has far more CPU power, memory, and hard-drive storage than the embedded platform. In addition, it is common for the source code for your application under debug to exist on the development workstation and not on the embedded platform.
gdbserver is a small program that runs on the target board and allows remote debugging of a process on the board. It is invoked on the target board specifying the program to be debugged, as well as an IP address and port number on which it will listen for connection requests from GDB. Listing 15-3 shows the startup sequence on the target board.
Listing 15-3. Starting gdbserver on Target Board
$ gdbserver localhost:2001 websdemo-stripped
Process websdemo-stripped created; pid = 197
Listening on port 2001
This particular example starts gdbserver configured to listen for an Ethernet TCP/IP connection on port 2001, ready to debug our stripped binary program called websdemo-stripped.
From our development workstation, we launch GDB, passing it the name of the binary executable containing symbolic debug information that we want to debug as an argument. After GDB starts up, we issue a command to connect to the remote target board. Listing 15-4 shows this sequence.
Listing 15-4. Starting Remote GDB Session
$ xscale_be-gdb -q websdemo
(gdb) target remote 192.168.1.141:2001
Remote debugging using 192.168.1.141:2001
0x40000790 in ?? ()
(gdb) p main <<<< display address of main function
$1 = {int (int, char **)} 0x12b68 <main>
(gdb) b main <<<< Place breakpoint at main()
Breakpoint 1 at 0x12b80: file main.c, line 72.
(gdb)
The sequence in Listing 15-4 invokes cross-gdb on your development host. When GDB is running, we issue the gdb target remote command. This command causes GDB to initiate a TCP/IP connection from your development workstation to your target board, with the indicated IP address on port 2001. When gdbserver accepts the connection request, it prints a line similar to this:
Remote debugging from host 192.168.0.10
Now GDB is connected to the target board's gdbserver process, ready to accept commands from GDB. The rest of the session is exactly the same as if you were debugging an application locally. This is a powerful tool, allowing you to use the power of your development workstation for the debug session, leaving only a small, relatively unobtrusive GDB stub and your program being debugged on the target board. In case you were wondering, gdbserver for this particular ARM target is only 54KB.
root@coyote:~# ls -l /usr/bin/gdbserver
-rwxr-xr-x 1 root root 54344 Jul 23 2005 /usr/bin/gdbserver
There is one caveat, and it is the subject of a frequently asked question (FAQ) on many mailing lists. You must be using a GDB on your development host that was configured as a cross-debugger. It is a binary program that runs on your development workstation but understands binary executable images compiled for another architecture. This is an important and frequently overlooked fact. You cannot debug a PowerPC target with a native GDB such as that found in a typical Red Hat Linux installation. You must have a GDB configured for your host and target combination.
When GDB is invoked, it displays a banner consisting of several lines of information and then displays its compiled configuration. Listing 15-5 is an example of the GDB used for some examples in this book, which is part of an embedded Linux distribution provided by MontaVista Software configured for PowerPC cross-development.
Listing 15-5. Invocation of cross-gdb
$ ppc_82xx-gdb
GNU gdb 6.0 (MontaVista 6.0-8.0.4.0300532 2003-12-24)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=powerpc-hardhat-linux".
(gdb)
Notice the last lines of this GDB startup message. This is the configuration compiled into this version of GDB. It was compiled to execute on a Pentium (i686) PC host running GNU/Linux and to debug binary programs compiled for a PowerPC processor running GNU/Linux. This is specified by the --host and --target variables displayed by the banner text, and is also a part of the configuration string passed to ./configure when building GDB.
Now that you understand how to invoke a remote debug session using GDB on the host and gdbserver on the target, we turn our attention to the complexities of shared libraries and debug symbols. Unless your application is a statically linked executable (linked with the -static linker command line switch), many symbols in your application will reference code outside your application. Obvious examples include the use of standard C library routines such as fopen, printf, malloc, and memcpy. Less obvious examples might include calls to application-specific functions, such as jack_transport_locate() (a routine from the JACK low-latency audio server), which calls a library function outside the standard C libraries.
To have symbols from these routines available, you must satisfy two requirements for GDB :
• You must have debug versions of the libraries available.
• GDB must know where to find them.
If you don't have debug versions of the libraries available, you can still debug your application; you just won't have any debug information available for library routines called by your application. Often this is perfectly acceptable, unless, of course, you are developing a shared library object as part of your embedded project.
Look back at Listing 15-4, where we invoked GDB on a remote target. After GDB connected via the target remote command, GDB issued a two-line response:
Remote debugging using 192.168.1.141:2001
0x40000790 in ?? ()
This confirms that GDB connected to our target at the indicated IP address and port. GDB then reports the location of the program counter as 0x40000790. Why do we get question marks instead of a symbolic location? Because this is the Linux dynamic loader (ld-x.y.z.so), and on this particular platform, we do not have debug symbols available for this shared library. How do we know this?
Recall our introduction of the /proc file system from Chapter 9, "File Systems." One of the more useful entries was the maps entry (see Listing 9-16, in Chapter 9) in the per-process directory structure. We know the process ID (PID) of our target application from the gdbserver output in Listing 15-3. Our process was assigned PID 197. Given that, we can see the memory segments in use right after process startup, as shown in Listing 15-6.
Listing 15-6. Initial Target Memory Segment Mapping
root@coyote:~# cat /proc/197/maps
00008000-00026000 r-xp 00000000 00:0e 4852444 ./websdemo-stripped
0002d000-0002e000 rw-p 0001d000 00:0e 4852444 ./websdemo-stripped
40000000-40017000 r-xp 00000000 00:0a 4982583 /lib/ld-2.3.3.so
4001e000-40020000 rw-p 00016000 00:0a 4982583 /lib/ld-2.3.3.so
bedf9000-bee0e000 rwxp bedf9000 00:00 0 [stack]
root@coyote:~#
Here we see the target websdemo-stripped application occupying two memory segments. The first is the read-only executable segment at 0x8000, and the second is a read-write data segment at 0x2d000. The third memory segment is the one of interest. It is the Linux dynamic linker's executable code segment. Notice that it starts at address 0x40000000. If we investigate further, we can confirm that GDB is actually sitting at the first line of code for the dynamic linker, before any code from our own application has been executed. Using our cross version of readelf, we can confirm the starting address of the linker as follows:
# xscale_be-readelf -S ld-2.3.3.so | grep \.text
[ 9] .text PROGBITS 00000790 000790 012c6c 00 AX 0 0 16
From this data, we conclude that the address GDB reports on startup is the first instruction from ld-2.3.3.so, the Linux dynamic linker/loader. You can use this technique to get rough ideas of where your code is if you don't have symbolic debug information for a process or shared library.
Remember that we are executing this cross readelf command on our development host. Therefore, the ld-2.3.3.so file, itself an XScale binary object, must be accessible to your development host. Most typically, this file resides on your development host, and is a component of your embedded Linux distribution installed on your host.
GDB can alert you to shared library events. This can be useful for understanding your application's behavior or the behavior of the Linux loader, or for setting breakpoints in shared library routines you want to debug or step through. Listing 15-7 illustrates this technique. Normally, the complete path to the library is displayed. This listing has been edited for better readability.
Listing 15-7. Stopping GDB on Shared Library Events
$ xscale_be-gdb -q websdemo
(gdb) target remote 192.168.1.141:2001
Remote debugging using 192.168.1.141:2001
0x40000790 in ?? ()
(gdb) i shared <<<Display loaded shared libs
No shared libraries loaded at this time.
(gdb) b main <<<Break at main
Breakpoint 1 at 0x12b80: file main.c, line 72.
(gdb) c
Continuing.
Breakpoint 1, main (argc=0x1, argv=0xbec7fdc4) at main.c:72
72 int localvar = 9;
(gdb) i shared
From To Syms Read Shared Object Library
0x40033300 0x4010260c Yes /opt/mvl/.../lib/tls/libc.so.6
0x40000790 0x400133fc Yes /opt/mvl/.../lib/ld-linux.so.3
(gdb) set stop-on-solib-events 1
(gdb) c
Continuing.
Stopped due to shared library event
(gdb) i shared
From To Syms Read Shared Object Library
0x40033300 0x4010260c Yes /opt/mvl/.../lib/tls/libc.so.6
0x40000790 0x400133fc Yes /opt/mvl/.../lib/ld-linux.so.3
0x4012bad8 0x40132104 Yes /opt/mvl/.../libnss_files.so.2
(gdb)
When the debug session is first started, of course, no shared libraries are loaded. You can see this with the first i shared command. This command displays the shared libraries that are currently loaded. Setting a breakpoint at our application's main() function, we see that two shared libraries are now loaded. These are the Linux dynamic linker/loader and the standard C library component libc.
From here, we issue the set stop-on-solib-event command and continue program execution. When the application tries to execute a function from another shared library, that library is loaded. In case you are wondering, the gethostbyname() function is encountered and causes the next shared object load.
This example illustrates an important cross-development concept. The binary application (ELF image) running on the target contains information on the libraries it needs to resolve its external references. We can view this information easily using the ldd command introduced in Chapter 11, "BusyBox," and detailed in Chapter 13. Listing 15-8 shows the output of ldd invoked from the target board.
Listing 15-8. ldd Executed on Target Board
root@coyote:/workspace# ldd websdemo
libc.so.6 => /lib/tls/libc.so.6 (0x40020000)
/lib/ld-linux.so.3 (0x40000000)
root@coyote:/workspace#
Notice that the paths to the shared libraries on the target are absolute paths starting at /lib on the root file system. But GDB running on your host development workstation cannot use these paths to find the libraries. You should realize that to do so would result in your host GDB loading libraries from the wrong architecture. Your host is likely x86, whereas, in this example, the target is ARM XScale.
If you invoke your cross version of ldd, you will see the paths that were preconfigured into your toolchain. Your toolchain must have knowledge of where these files exist on your host development system.[101] Listing 15-9 illustrates this. Again, we have edited the listing for readability; long paths have been abbreviated.
Listing 15-9. ldd Executed on Development Host
$ xscale_be-ldd websdemo
libc.so.6 => /opt/mvl/.../xscale_be/target/lib/libc.so.6 (0xdead1000)
ld-linux.so.3 => /opt/mvl/.../xscale_be/target/lib/ld-linux.so.3 (0xdead2000)
$
Your cross toolchain should be preconfigured with these library locations. Not only does your host GDB need to know where they are located, but, of course, your compiler and linker also need this knowledge.[102] GDB can tell you where it is configured to look for these libraries using the show solib-absolute-prefix command:
(gdb) show solib-absolute-prefix
Prefix for loading absolute shared library symbol files is
"/opt/mvl/pro/devkit/arm/xscale_be/target".
(gdb)
You can set or change where GDB searches for shared libraries using the GDB commands set solib-absolute-prefix and set solib-search-path. If you are developing your own shared library modules or have custom library locations on your system, you can use solib-search-path to instruct GDB where to look for your libraries. For more details about these and other GDB commands, consult the online GDB manual referenced at the end of this chapter in Section 15.6.1, "Suggestions for Additional Reading."
One final note about ldd. You might have noticed the addresses from Listing 15-8 and 15-9 associated with the libraries. ldd displays the load address for the start of these code segments as they would be if the program were loaded by the Linux dynamic linker/loader. Executed on the target, the addresses in Listing 15-5 make perfect sense, and we can correlate these with the /proc/<pid>/maps listing of the running process on the target. Listing 15-10 displays the memory segments for this target process after it is completely loaded and running.
Listing 15-10. Memory Segments from /proc/<pid>/maps on Target
root@coyote:~# cat /proc/197/maps
00008000-00026000 r-xp 00000000 00:0e 4852444 /workspace/websdemo-stripped
0002d000-0002e000 rw-p 0001d000 00:0e 4852444 /workspace/websdemo-stripped
0002e000-0005e000 rwxp 0002e000 00:00 0 [heap]
40000000-40017000 r-xp 00000000 00:0a 4982583 /lib/ld-2.3.3.so
40017000-40019000 rw-p 40017000 00:00 0
4001e000-4001f000 r--p 00016000 00:0a 4982583 /lib/ld-2.3.3.so
4001f000-40020000 rw-p 00017000 00:0a 4982583 /lib/ld-2.3.3.so
40020000-4011d000 r-xp 00000000 00:0a 4982651 /lib/tls/libc-2.3.3.so
4011d000-40120000 ---p 000fd000 00:0a 4982651 /lib/tls/libc-2.3.3.so
40120000-40124000 rw-p 000f8000 00:0a 4982651 /lib/tls/libc-2.3.3.so
40124000-40126000 r--p 000fc000 00:0a 4982651 /lib/tls/libc-2.3.3.so
40126000-40128000 rw-p 000fe000 00:0a 4982651 /lib/tls/libc-2.3.3.so
40128000-4012a000 rw-p 40128000 00:00 0
4012a000-40133000 r-xp 00000000 00:0a 4982652 /lib/tls/libnss_files-2.3.3.so
40133000-4013a000 ---p 00009000 00:0a 4982652 /lib/tls/libnss_files-2.3.3.so
4013a000-4013b000 r--p 00008000 00:0a 4982652 /lib/tls/libnss_files-2.3.3.so
4013b000-4013c000 rw-p 00009000 00:0a 4982652 /lib/tls/libnss_files-2.3.3.so
becaa000-becbf000 rwxp becaa000 00:00 0 [stack]
root@coyote:~#
Notice the correlation of the target ldd output from Listing 15-8 to the memory segments displayed in the /proc file system for this process. The start (beginning of .text segment) of the Linux loader is 0x40000000 and the start of libc is at 0x40020000. These are the virtual addresses where these portions of the application have been loaded, and are reported by the target invocation of ldd. However, the load addresses reported by the cross version of ldd in Listing 15-9 (0xdead1000 and 0xdead2000) are there to remind you that these libraries cannot be loaded on your host system (they are ARM architecture binaries), and the load addresses are simply placeholders.
Generally the developer is presented with two different debugging scenarios when dealing with multiple threads of execution. Processes can exist in their own address space or can share an address space (and other system resources) with other threads of execution. The former (independent processes not sharing common address space) must be debugged using separate independent debug sessions. Nothing prevents you from using gdbserver on multiple processes on your target system, and using a separate invocation of GDB on your development host to coordinate a debug session for multiple cooperating but independent processes.
When a process being debugged under GDB uses the fork() system call[103] to spawn a new process, GDB can take two courses of action. It can continue to control and debug the parent process, or it can stop debugging the parent process and attach to the newly formed child process. You can control this behavior using the set follow-fork-mode command. The two modes are follow parent and follow child. The default behavior is for GDB to follow the parent. In this case, the child process executes immediately upon a successful fork.
Listing 15-11 reproduces a snippet of a simple program that forks multiple processes from its main() routine.
Listing 15-11. Using fork() to Spawn a Child Process
...
for(i=0; i<MAX_PROCESSES; i++) {
/* Creating child process */
pid[i] = fork(); /* Parent gets non-zero PID */
if (pid[i] == -1) {
perror("fork failed");
exit(1);
}
if (pid[i] == 0) { /* Indicates child's code path */
worker_process(); /* The forked process calls this */
}
}
/* Parent's main control loop */
while (1) {
...
}
This simple loop creates MAX_THREADS new processes using the fork() system call. Each newly spawned process executes a body of code defined by the function worker_process(). When run under GDB in the default mode, GDB detects the creation of the new threads of execution (processes) but remains attached to the parent's thread of execution. Listing 15-12 illustrates this GDB session.
Listing 15-12. GDB in follow-fork-mode = parent
(gdb) target remote 192.168.1.141:2001
0x40000790 in ?? ()
(gdb) b main
Breakpoint 1 at 0x8888: file forker.c, line 104.
(gdb) c
Continuing.
[New Thread 356]
[Switching to Thread 356]
Breakpoint 1, main (argc=0x1, argv=0xbe807dd4) at forker.c:104
104 time(&start_time);
(gdb) b worker_process
Breakpoint 2 at 0x8784: file forker.c, line 45.
(gdb) c
Continuing.
Detaching after fork from child process 357.
Detaching after fork from child process 358.
Detaching after fork from child process 359.
Detaching after fork from child process 360.
Detaching after fork from child process 361.
Detaching after fork from child process 362.
Detaching after fork from child process 363.
Detaching after fork from child process 364.
Notice that eight child processes were spawned, with PID values from 357 to 364. The parent process was instantiated with PID 356. When the breakpoint in main() was hit, we entered a breakpoint at the worker_process() routine, which each child process executes upon fork(). Letting the program continue from main, we see each of the new processes spawned and detached by the debugger. They never hit the breakpoint because GDB is attached to the main process, which never executes the worker_process() routine.
If you need to debug each process, you must execute a separate independent GDB session and attach to the child process after it is forked(). The GDB documentation referenced at the end of this chapter outlines a useful technique to place a call to sleep() in the child process, giving you time to attach a debugger to the new process. Attaching to a new process is explained in Section 15.5.2, "Attaching to a Running Process."
If you simply need to follow the child process, set the follow-fork-mode to follow child before your parent reaches the fork() system call. Listing 15-13 shows this.
Listing 15-13. GDB in follow-fork-mode = child
(gdb) target remote 192.168.1.141:2001
0x40000790 in ?? ()
(gdb) set follow-fork-mode child
(gdb) b worker_process
Breakpoint 1 at 0x8784: file forker.c, line 45.
(gdb) c
Continuing.
[New Thread 401]
Attaching after fork to child process 402.
[New Thread 402]
[Switching to Thread 402]
Breakpoint 1, worker_process () at forker.c:45
45 int my_pid = getpid();
(gdb) c
Continuing.
Here we see the parent process being instantiated as PID 401. When the first child is spawned by the fork() system call, GDB detaches silently from the parent thread of execution and attaches to the newly spawned child process having PID 402. GDB is now in control of the first child process and honors the breakpoint set at worker_process(). Notice, however, that the other child processes spawned by the code snippet from Listing 15-11 are not debugged and continue to run to their own completion.
In summary, using GDB in this fashion, you are limited to debugging a single process at a time. You can debug through the fork() system call, but you have to decide which thread of execution to follow through the fork() call, either the parent or the child. As mentioned in the introduction to this section, you can use multiple independent GDB sessions if you must debug more than one cooperating process at a time.
If your application uses the POSIX thread library for its threading functions, GDB has additional capabilities to handle concurrent debugging of a multithreaded application. The Native Posix Thread Library (NPTL) has become the de facto standard thread library in use on Linux systems, including embedded Linux systems. The rest of this discussion assumes that you are using this thread library.
For this section, we use a demonstration program that spawns a number of threads using the pthread_create() library function in a simple loop. After the threads are spawned, the main() routine simply waits for keyboard input to terminate the application. Each thread displays a short message on the screen and sleeps for a predetermined time. Listing 15-14 shows the startup sequence on the target board.
Listing 15-14. Target Threads Demo Startup
root@coyote:/workspace # gdbserver localhost:2001 ./tdemo
Process ./tdemo created; pid = 671
Listening on port 2001
Remote debugging from host 192.168.1.10
^^^^^ Previous three lines displayed by gdbserver
tdemo main() entered: My pid is 671
Starting worker thread 0
Starting worker thread 1
Starting worker thread 2
Starting worker thread 3
As in our previous examples, gdbserver prepares the application for running and waits for a connection from our host-based cross-gdb. When GDB connects, gdbserver reports the connection with the Remote debugging... message. Now we start GDB on the host and connect. Listing 15-15 reproduces this half of the session.
Listing 15-15. Host GDB Connecting to Target Threads Demo
$ xscale_be-gdb -q tdemo
(gdb) target remote 192.168.1.141:2001
0x40000790 in ?? ()
(gdb) b tdemo.c:97
Breakpoint 1 at 0x88ec: file tdemo.c, line 97.
(gdb) c
Continuing.
[New Thread 1059]
[New Thread 1060]
[New Thread 1061]
[New Thread 1062]
[New Thread 1063]
[Switching to Thread 1059]
Breakpoint 1, main (argc=0x1, argv=0xbefffdd4) at tdemo.c:98
98 int c = getchar();
(gdb)
Here we connect to the target (resulting in the " Remote debugging... " message in Listing 15-14), set a breakpoint just past the loop where we spawned the new threads, and continue. When the new thread is created, GDB displays a notice along with the thread ID. Thread 1059 is the TDemo application, doing its work directly from the main() function. Threads 1060 through 1063 are the new threads created from the call to pthread_create().
When GDB hits the breakpoint, it displays the message [Switching to Thread 1059], indicating that this was the thread of execution that encountered the breakpoint. It is the active thread for the debugging session, referred to as the current thread in the GDB documentation.
GDB enables us to switch between threads and perform the usual debugging operations such as setting additional breakpoints, examining data, displaying a backtrace, and working with the individual stack frames within the current thread. Listing 15-16 provides examples of these operations, continuing directly with our debugging session started in Listing 15-15.
Listing 15-16. GDB Operations on Threads
...
(gdb) c
Continuing.
<<< Ctl-C to interrupt program execution
Program received signal SIGINT, Interrupt.
0x400db9c0 in read () from /opt/mvl/.../lib/tls/libc.so.6
(gdb) i threads
5 Thread 1063 0x400bc714 in nanosleep ()
from /opt/mvl/.../lib/tls/libc.so.6
4 Thread 1062 0x400bc714 in nanosleep ()
from /opt/mvl/.../lib/tls/libc.so.6
3 Thread 1061 0x400bc714 in nanosleep ()
from /opt/mvl/.../lib/tls/libc.so.6
2 Thread 1060 0x400bc714 in nanosleep ()
from /opt/mvl/.../lib/tls/libc.so.6
* 1 Thread 1059 0x400db9c0 in read ()
from /opt/mvl/.../lib/tls/libc.so.6
(gdb) thread 4 <<< Make Thread 4 the current thread
[Switching to thread 4 (Thread 1062)]
#0 0x400bc714 in nanosleep ()
from /opt/mvl/.../lib/tls/libc.so.6
(gdb) bt
#0 0x400bc714 in nanosleep ()
from /opt/mvl/.../lib/tls/libc.so.6
#1 0x400bc4a4 in __sleep (seconds=0x0) at sleep.c:137
#2 0x00008678 in go_to_sleep (duration=0x5) at tdemo.c:18
#3 0x00008710 in worker_2_job (random=0x5) at tdemo.c:36
#4 0x00008814 in worker_thread (threadargs=0x2) at tdemo.c:67
#5 0x40025244 in start_thread (arg=0xfffffdfc) at pthread_create.c:261
#6 0x400e8fa0 in clone () at../sysdeps/unix/sysv/linux/arm/clone.S:82
#7 0x400e8fa0 in clone () at../sysdeps/unix/sysv/linux/arm/clone.S:82
(gdb) frame 3
#3 0x00008710 in worker_2_job (random=0x5) at tdemo.c:36
36 go_to_sleep(random);
(gdb) l <<< Generate listing of where we are
31 }
32
33 static void worker_2_job(int random)
34 {
35 printf("t2 sleeping for %d\n", random);
36 go_to_sleep(random);
37 }
38
39 static void worker_3_job(int random)
40 {
(gdb)
A few points are worth mentioning. GDB assigns its own integer value to each thread and uses these values to reference the individual threads. When a breakpoint is hit in a thread, all threads within the process are halted for examination. GDB marks the current thread with an asterisk (*). You can set unique breakpoints within each threadassuming, of course, that they exist in a unique context. If you set a breakpoint in a common portion of code where all threads execute, the thread that hits the breakpoint first is arbitrary.
The GDB user documentation referenced at the end of this chapter contains more useful information related to debugging in a multithreaded environment.
Debugging Flash resident code presents its own unique challenges. The most obvious limitation is the way in which GDB and gdbserver cooperate in setting target breakpoints. When we discussed the GDB remote serial protocol in Chapter 14, you learned how breakpoints are inserted into an application.[104] GDB replaces the opcode at the breakpoint location with an architecture-specific opcode that passes control to the debugger. However, in ROM or Flash, GDB cannot overwrite the opcode, so this method of setting breakpoints is useless.
Most modern processors contain some number of debug registers that can be used to get around this limitation. These capabilities must be supported by architecture-and processor-specific hardware probes or stubs. The most common technique for debugging Flash and ROM resident code is to use JTAG hardware probes. These probes support the setting of processor-specific hardware breakpoints. This topic was covered in detail in Chapter 14. Refer back to Section 14.4.2, "Debugging with a JTAG Probe," for details.
Sometimes you might want to use a serial port for remote debugging. For other tasks, you might find it useful to attach the debugger to a process that is already running. These simple but useful operations are detailed here.[105]
Debugging via serial port is quite straightforward. Of course, you must have a serial port available on your target that is not being used by another process, such as a serial console. The same limitation applies to your host. A serial port must be available. If both of these conditions can be met, simply replace the IP:Port specification passed to gdbserver with a serial port specification. Use the same technique when connecting to your target from your host-based GDB.
On your target:
root@coyote:/workspace # gdbserver /dev/ttyS0 ./tdemo
Process ./tdemo created; pid = 698
Remote debugging using /dev/ttyS0
From your host:
$ xscale_be-gdb -q tdemo
(gdb) target remote /dev/ttyS1
Remote debugging using /dev/ttyS1
0x40000790 in ?? ()
It is often advantageous to connect to a process to examine its state while it is running instead of killing the process and starting it again. With gdbserver, it is trivial:
root@coyote:/workspace # ps ax | grep tdemo
1030 pts/0 Sl+ 0:00 ./tdemo
root@coyote:/workspace # gdbserver localhost:2001 --attach 1030
Attached; pid = 1030
Listening on port 2001
When you are finished examining the process under debug, you can issue the gdb detach command. This detaches the gdbserver from the application on the target and terminates the debug session. The application continues where it left off. This is a very useful technique for examining a running program. Be aware, though, that when you attach to the process, it halts, waiting for instructions from you. It will not resume execution until instructed to do so, using either the continue command or the detach command. Also note that you can use the detach command at almost any time to end the debug session and leave the application running on the target.
• Remote (cross) debugging enables symbolic debugging using host development workstation resources for the heavy lifting, preserving often scarce target resources.
• gdbserver runs on the target system and acts as the glue between the cross-gdb running on a development host and the process being debugged on the target.
• GDB on the host typically uses IP connections via Ethernet to send and receive commands to gdbserver running on the target. The GDB remote serial protocol is used between GDB and gdbserver.
• GDB can halt on shared library events and can automatically load shared library symbols when available. Your toolchain should be configured for the default paths on your cross-development system. Alternatively, you can use GDB commands to set the search paths for shared library objects.
• GDB can be used to debug multiple independent processes via multiple concurrent GDB sessions.
• GDB can be configured to follow a forked process on a fork() system call. Its default mode is to continue to debug the parentthat is, the caller of fork().
• GDB has features to facilitate debugging multithreaded applications written to POSIX thread APIs. The current default Linux thread library is the Native Posix Threads Library (NPTL).
• GDB supports attaching to and detaching from an already running process.
GDB: The GNU Project Debugger
Online Documentation
http://sourceware.org/gdb/onlinedocs/
GDB Pocket Reference
Arnold Robbins
O'Reilly Media, 2005
Remember to use your cross-version of strip, for example ppc_82xx-strip.
It is certainly possible to pass these locations to your compiler, linker, and debugger for every invocation, but any good embedded Linux distribution will configure these defaults into the toolchain as a convenience to the developer.
Of course, your compiler also needs to know the location of target files such as architecture-specific system and library header files.
We will use the term system call, but fork() in this context is actually the C library function which in turn calls the Linux sys_fork() system call.
Refer back to Listing 14-5 in Chapter 14.
Refer back to Listing 14-5 in Chapter 13