Is there a way to attach a debugger to a multi-threaded Python process?

Question

I'm trying to debug a deadlock in a multi-threaded Python application after it has locked up. Is there a way to attach a debugger to inspect the state of the process?

Edit: I'm attempting this on Linux, but it would be great if there were a cross-platform solution. It's Python after all :)

Answer 1

What platform are you attempting this on? Most debuggers allow you to attach to a running process by using the process id. You can either output the process id via logging or using something like Task Manager. Once that is achieved it will be possible to inspect individual threads and their call stacks.

EDIT: I don't have any experience with GNU Debugger (GDB), which is cross platform, however I found this link and it may start you on the right path. It explains how to add debug symbols (handy for reading stack traces) and how to instruct gdb to attach to a running python process.

Answer 2

Yeah, gdb is good for lower level debugging.

You can change threads with the thread command.

e.g

(gdb) thr 2
[Switching to thread 2 (process 6159 thread 0x3f1b)]
(gdb) backtrace
....

You could also check out Python specific debuggers like Winpdb, or pydb. Both platform independent.

Answer 3

If you mean the pydb, there is no way to do it. There was some effort in that direction: see the svn commit, but it was abandoned. Supposedly winpdb supports it.

Answer 4

My experience debugging multi-threaded programs in PyDev (Eclipse on Windows XP) is, threads created using thread.start_new_thread could not be hooked, but thread created using threading.Thread could be hooked. Hope the information is helpful.

Answer 5

You can attach a debugger to a multi-threaded Python process, but you need to do it at the C level. To make sense of what's going on, you need the Python interpreter to be compiled with symbols. If you don't have one, you need to download source from python.org and build it yourself:

./configure --prefix=/usr/local/pydbg
make OPT=-g
sudo make install
sudo ln -s /usr/local/pydbg/bin/python /usr/local/bin/dbgpy

Make sure your workload is running on that version of the interpreter. You can then attach to it with GDB at any time. The Python folks have included a sample ".gdbinit" in their Misc directory, which has some useful macros. However it's broken for multi-threaded debugging (!). You need to replace lines like this

while $pc < Py_Main || $pc > Py_GetArgcArgv

with the following:

while ($pc < Py_Main || $pc > Py_GetArgcArgv) && ($pc < t_bootstrap || $pc > thread_PyThread_start_new_thread)

Otherwise commands like pystack won't terminate on threads other than the main thread. With this stuff in place, you can do stuff like

gdb> attach <PID>
gdb> info threads
gdb> thread <N>
gdb> bt
gdb> pystack
gdb> detach

and see what's going on. Kind of.

You can parse what the objects are with the "pyo" macro. Chris has some examples on his blog.

Good luck.

(Shoutout for Dan's blog for some key information for me, notably the threading fix!)

Answer 6

Use Winpdb. It is a platform independent graphical GPL Python debugger with support for remote debugging over a network, multiple threads, namespace modification, embedded debugging, encrypted communication and is up to 20 times faster than pdb.

Features:

• GPL license. Winpdb is Free Software.
• Compatible with CPython 2.3 through 2.6 and Python 3000
• Compatible with wxPython 2.6 through 2.8
• Platform independent, and tested on Ubuntu Gutsy and Windows XP.
• User Interfaces: rpdb2 is console based, while winpdb requires wxPython 2.6 or later.