Tech Sujhav : How to debug threads with GDB debugger

Here we shall discuss how to debug a thread with GDB-debugger. Here are the list of items that are discussed with example.

How to connect GDB with a multi-threaded binary or a core file generated after crash?
How to connect GDB with a running multi-threaded process?
Check threads status at the time of crash, breakpoint and watchpoint.IMPORTANT
Check activity of a given thread. IMPORTANT
How to find a thread with thread-id.
How to find a thread with thread-id returned by get_id().
Suggestion: For better learning please do suggested step by yourself.
Copy, Paste and Run example code.

NOTE: - GDB COMMANDS in red, [Description in blue]

How to connect GDB with a multi-threaded binary or a core file generated after crash?

we are suppose to generate binary with one of the -g, -g2, -g3 option. Let check how our example shall be compiled

Compile Binary

$g++ gdbThread.cpp -o gdbThread -g -lpthread

Attach binary with gdb

$gdb ./<absolute-binary-name>

$gdb ./gdbThread

Attach core file with gdb

$gdb -c <absolute-core-file-name> <absolute-binary-name>

$gdb -c crash1932.core gdbThread

How to connect GDB with a running multi-threaded process?

Get process id of a running process with following Linux command

$ ps –ef |grep -i <running_process_executable_name>

$ ps -ef |grep -i gdbThread

TechSujhav 10640 10237 99 06:20 pts/0 00:00:31 ./gdbThread

TechSujhav 10645 10403 0 06:21 pts/1 00:00:00 grep --color=auto -i gdbThread

2. Attach process with gdb

$gdb -p 10640

Code used for explanation

#include <iostream>
#include <thread>
#include <string>
using namespace std;
class Suggest
{
   unsigned int counter;
public:
   Suggest():counter(1){}
   void suggestActivity()
   {
       while(++counter)
       {
           if(counter%500000000 ==0)
               cout<<"Suggest::suggestActivity = "<<counter<<endl;;
           if(counter == 4294967295)
               counter=0;
       }
   }
};
void increment()
{
   unsigned int counter=0;
   while(++counter)
   {
       if(counter%300000000 ==0)
           cout<<"increment = "<<counter<<endl;
       if(counter == 2147483647)
           counter=0;
   }
}
void decrement()
{
   unsigned int counter=4294967295;
   while(--counter)
       if(counter%100000000 ==0)
           cout<<"decrement = "<<counter<<endl;
}
int main()
{
   //thread id of main thread
   cout<<"main() Thread Id=0x"<<std::hex <<std::this_thread::get_id()<<std::dec<<" ::"<<std::this_thread::get_id()<<endl;
//First thread is created
   std::thread thrdObj(increment);
   cout<<"increament() Thread Id=0x"<<std::hex <<thrdObj.get_id()<<std::dec<<" ::"<<thrdObj.get_id()<<endl;
   thrdObj.detach();
//Second Thread is created
   std::thread thrdObj1(decrement);
   cout<<"decrement() Thread Id=0x"<<std::hex <<thrdObj1.get_id()<<std::dec<<" ::"<<thrdObj1.get_id()<<endl;
   thrdObj1.detach();
  //Third Thread is created
   Suggest sObj;
   std::thread thrdObj2(&Suggest::suggestActivity,sObj);
   cout<<"Suggest::suggestActivity() Thread Id=0x"<<std::hex <<thrdObj2.get_id()<<std::dec<<" ::"<<thrdObj2.get_id()<<endl<<endl;
   thrdObj2.join();
   return 0;
}

Put breakpoint for a given thread logically.

gdb ./gdbThread

GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./gdbThread...done.

(gdb) b gdbThread.cpp:16 [Breakpoint added on line-16]

Breakpoint 1 at 0x18d9: file gdbThread.cpp, line 16.

(gdb) r [Press r to run binary]

Starting program: /home/TechSujhav/thread/gdbThread
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
main() Thread Id=0x7ffff7fe3740 ::140737354020672
[New Thread 0x7ffff6e85700 (LWP 11079)]
increament() Thread Id=0x7ffff6e85700 ::140737335809792

[New Thread 0x7ffff6684700 (LWP 11080)]
decrement() Thread Id=0x7ffff6684700 ::140737327417088

[New Thread 0x7ffff5e83700 (LWP 11081)]
Suggest::suggestActivity() Thread Id=0x7ffff5e83700 ::140737319024384

decrement = 4200000000
decrement = 4100000000
decrement = 4000000000
increment = 300000000
decrement = 3900000000
decrement = 3800000000
decrement = 3700000000

[Switching to Thread 0x7ffff5e83700 (LWP 11081)][Program stops at breakpoint]
Thread 4 "gdbThread" hit Breakpoint 1, Suggest::suggestActivity (this=0x55555576c528) at gdbThread.cpp:16

16 cout<<"Suggest::suggestActivity = "<<counter<<endl;;

Check threads status at the time of crash, breakpoint and watchpoint.

***IMPORTANT***

Here we are going to check what other threads are doing once a thread hit breakpoint, watchpoint or crash . We shall use “info thread” command to check thread status.

(gdb) info thread

n Number of threads running at the moment.

Here number of thread 4. Strange while we have created on 3 threads. For operating system process that is performing main() function is also a thread.

1-main() + 3 user created.

n What is thread id associated with a thread or LWP – lightweight process

Here process id shown is the process Id given by operating system it not the one returned by get_id () function of thread class.

n Which code statement a given thread is performing.

Let check in our code, it showing all thread

(gdb) info thread [4-Threads are visible,check above point-1]

Id Target Id Frame

1 Thread 0x7ffff7fe3740 (LWP 11075) "gdbThread" 0x00007ffff7bbed2d in __GI___pthread_timedjoin_ex (threadid=140737319024384,

thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89

2 Thread 0x7ffff6e85700 (LWP 11079) "gdbThread" 0x000055555555517e in increment () at gdbThread.cpp:27

3 Thread 0x7ffff6684700 (LWP 11080) "gdbThread" 0x00005555555551f9 in decrement () at gdbThread.cpp:37

* 4 Thread 0x7ffff5e83700 (LWP 11081) "gdbThread" Suggest::suggestActivity (this=0x55555576c528) at gdbThread.cpp:16

Check activity of a given thread.

***IMPORTANT***

It is clearly visible that main() is waiting on join(). Check following line of our code

in main () at gdbThread.cpp:57

(gdb) thread 1 [check thread 1]

[Switching to thread 1 (Thread 0x7ffff7fe3740 (LWP 11075))]

#0 0x00007ffff7bbed2d in __GI___pthread_timedjoin_ex (threadid=140737319024384, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89

89 pthread_join_common.c: No such file or directory.

(gdb) bt [Back-trace of thread 1 -it is on line-57]

#0 0x00007ffff7bbed2d in __GI___pthread_timedjoin_ex (threadid=140737319024384, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89

#1 0x00007ffff78ea933 in std::thread::join() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

#2 0x000055555555550e in main () at gdbThread.cpp:57

(gdb) thread 3 [check thread 3]

[Switching to thread 3 (Thread 0x7ffff6684700 (LWP 11080))]
#0 0x00005555555551f9 in decrement () at gdbThread.cpp:37

37 if(counter%100000000 ==0)

(gdb) bt [Check the status of thread 3 -it is on line-37]

#0 0x00005555555551f9 in decrement () at gdbThread.cpp:37
#1 0x0000555555555d6d in std::__invoke_impl<void, void (*)()> (__f=@0x55555576c3d8: 0x5555555551d1 <decrement()>) at /usr/include/c++/7/bits/invoke.h:60

#2 0x0000555555555a06 in std::__invoke<void (*)()> (__fn=@0x55555576c3d8: 0x5555555551d1 <decrement()>) at /usr/include/c++/7/bits/invoke.h:95

#3 0x000055555555678a in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x55555576c3d8) at /usr/include/c++/7/thread:234

#4 0x00005555555566fa in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x55555576c3d8) at /usr/include/c++/7/thread:243

#5 0x0000555555556682 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x55555576c3d0) at /usr/include/c++/7/thread:186

#6 0x00007ffff78ea6df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

#7 0x00007ffff7bbd6db in start_thread (arg=0x7ffff6684700) at pthread_create.c:463

#8 0x00007ffff7345a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

How to find a thread

(gdb) thread find 11081

Thread 4 has target id 'Thread 0x7ffff5e83700 (LWP 11081)'

(gdb) thread find 0x7ffff6e85700

Thread 2 has target id 'Thread 0x7ffff6e85700 (LWP 11079)'

(gdb)

How to find thread with thread-id returned by get_id()

Just convert returned value of get_id() to HEX digits using calculator then we can find that information in the output of “info thread”.

(gdb) info thread

Id Target Id Frame

1 Thread 0x7ffff7fe3740 (LWP 11075) "gdbThread" 0x00007ffff7bbed2d in __GI___pthread_timedjoin_ex (threadid=140737319024384, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89

2 Thread 0x7ffff6e85700 (LWP 11079) "gdbThread" 0x000055555555517e in increment () at gdbThread.cpp:27

3 Thread 0x7ffff6684700 (LWP 11080) "gdbThread" 0x00005555555551f9 in decrement () at gdbThread.cpp:37

* 4 Thread 0x7ffff5e83700 (LWP 11081) "gdbThread" Suggest::suggestActivity (this=0x55555576c528) at gdbThread.cpp:16

Binary output:

main() Thread Id=0x7ffff7fe3740 ::140737354020672

increament() Thread Id=0x7ffff6e85700 ::140737335809792

decrement() Thread Id=0x7ffff6684700 ::140737327417088

Suggest::suggestActivity() Thread Id=0x7ffff5e83700 ::140737319024384

PREV:: Frequently Asked Interview Questions on Threads

Main Page

Thread Information Linux Commands ::NEXT

Your Comments /Suggestions & Questions are always welcome.