We have a Linux application that makes use of OpenSSL's Python bindings and I suspect it is causing random crashes. Occasionally, we see it crash with the message "Python Fatal Error: GC Object already tracked," which would appear to be either a programming error on the part of the library, or a symptom of memory corruption. Is there any way to know the last line of Python source code it executed, given a core file? Or if it is attached in GDB? I realize it is probably all compiled bytecode, but I'm hoping there someone out there may have dealt with this. Currently it is running with the trace module active and we're hoping it will happen again, but it could be a long while.
+1
A:
If you have mac or sun box kicking around you could use dtrace and a version of python compiled with dtrace to figure out what the application was doing at the time. Note: in 10.5 python is pre-compiled with dtrace which is really nice and handy.
If that isn't available to you, then you can import gc and enable debugging which you can then put out to a log file.
To specifically answer your question regarding debugging with GDB you might want to read "Debugging With GDB" on the python wiki.
RaySl
2008-11-07 18:20:16
+2
A:
Yes, you can do this kind of thing:
(gdb) print PyRun_SimpleString("import traceback; traceback.print_stack()")
File "<string>", line 1, in <module>
File "/var/tmp/foo.py", line 2, in <module>
i**2
File "<string>", line 1, in <module>
$1 = 0
It should also be possible to use the pystack command defined in the python gdbinit file, but it's not working for me. It's discussed here if you want to look into it.
Also, if you suspect memory issues, it's worth noting that you can use valgrind with python, if you're prepared to recompile it. The procedure is described here.
fivebells
2008-11-07 18:36:04
If I recall, the gdbinit file needs to be changed, because the order of functions has changed in the .c files. That was the case the last time I used gdb+python. I should have submitted a patch...
Tony Arkles
2008-11-07 19:01:42
Awesome, will leave it running under gdb and hope it crashes over the weekend.
Matt Green
2008-11-07 19:07:45
Didn't notice before, but the core dumps were in different locations. This is almost certainly memory corruption. Thanks a bunch!
Matt Green
2008-11-07 19:40:07
No problem, Matt. Tony, the pystack command is failing because it's referencing a variable "co" which does not exist in the current frame.
fivebells
2008-11-08 16:03:03
A:
If you're using CDLL to wrap a C library in python, and this is 64-bit linux, there's a good chance that you're CDLL wrapper is misconfigured. CDLL defaults to int return types on all platforms (should be a long long on 64-bit systems) and just expects you to pass the right arguments in. You may need to verify the CDLL wrapper in this case...
Douglas Mayle
2008-11-07 18:59:10
A:
In addition to all above one can quickly implement an adhoc tracer via the trace module.
utku_karatas
2008-11-07 19:03:15