Multithreaded python script silently dies - how to debug

Question

I have a python script that creates and starts 3 threads, and then goes to a KeyboardInterrupt-catching-loop, to send the threads stop signal when ctrl+c is pressed.

The threads' run method has a top level try-except which logs every exception, also the top level code that creates threads is wrapped into try-except to log every exception.

But the script just dies randomly, sometimes after a day, sometimes after an hour, without any exception being logged.

It's driving me crazy, because I have no further ideas how to debug this.

Any ideas guys?

Edit: As Luper suggested a look into syslog indeed revealed

python[27737]: segfault at 0 ip 0808e1d3 sp b662c5e0 error 4 in python2.5[8048000+fb000]

Still no idea how to go from here.

As for the code, it doesnt do anything fancy, some file parsing and copying between directories, and calling some executables via os.system

Answer 1

A segfault in python is usually caused by a bug in a module written in C. There is nothing the interpreter can do.

A quick search revealed that the common problems that cause a segfault are 1) bad memory (but you should see more segfaults - run memcheck from a live CD if you suspect this), 2) corrupted installation (try reinstall python and packages, re-downloading everything) and 3) bugs (duh).

Try if it is 1) or 2) first, then to see where the process stops, can use strace to record all system calls. That may give you some other clue of what is going on (the output file may grow big):

strace -f python my_script.py > strace.out 2>&1