How do you automate the launching/debugging of large scale projects?

Question

Scenario:

There is a complex piece of software that is annoying to launch by hand. What I've done is to create a python script to launch the executable and attach gdb for debugging.

The process launching script:

• ensures an environment variable is set.
• ensures a local build directory gets added to the environment's LD_LIBRARY_PATH variable.
• changes the current working directory to where the executable expects to be (not my design)
• launches the executable with a config file the only command line option
• pipes the output from the executable to a second logging process
• remembers PID of executable, then launches & attaches gdb to running executable.

The script works, with one caveat. ctrl-c doesn't interrupt the debugee and return control to gdb. So if I "continue" with no active breakpoints I can never stop the process again, it has to be killed/interrupted from another shell. BTW, running "kill -s SIGINT <pid>" where <pid> is the debuggee's pid does get me back to gdb's prompt... but it is really annoying to have to do things this way

At first I thought Python was grabbing the SIGINT signal, but this doesn't seem to be the case as I set up signal handlers forward the signal to the debugee and that doesn't fix the problem.

I've tried various configurations to the python script (calling os.spawn* instead of subprocess, etc.) It seems that any way I go about it, if python launched the child process, SIGINT (ctrl-c) signals DO NOT to get routed to gdb or the child process.

Current line of thinking

• This might be related to needing a separate process group id for the debugee & gdb...any credence to this?
• Possible bug with SELinux?

Info:

• gdb 6.8
• Python 2.5.2 (problem present with Python 2.6.1 as well)
• SELinux Environment (bug delivering signals to processes?)

Alternatives I've considered:

• Setting up a .gdbinit file to do as much of what the script does, environment variables and current working directory are a problem with this approach.
• Launching executable and attaching gdb manually (yuck)

Question: How do you automate the launching/debugging of large scale projects?

Update: I've tried Nicholas Riley's examples below, on my Macintosh at home they all allow cntl-c to work to varrying degrees, on the production boxen (which I now to believe may be running SELinux) they don't...

Answer 1

Instead of forwarding the signal to the debuggee from Python, you could try just ignoring it. The following worked for me:

import signal
signal.signal(signal.SIGINT, signal.SIG_IGN)

import subprocess
cat = subprocess.Popen(['cat'])
subprocess.call(['gdb', '--pid=%d' % cat.pid])

With this I was able to ^C repeatedly inside GDB and interrupt the debuggee without a problem, however I did see some weird behavior.

Incidentally, I also had no problem when forwarding the signal to the target process.

import subprocess
cat = subprocess.Popen(['cat'])

import signal, os
signal.signal(signal.SIGINT,
              lambda signum, frame: os.kill(cat.pid, signum))

subprocess.call(['gdb', '--pid=%d' % cat.pid])

So, maybe something else is going on in your case? It might help if you posted some code that breaks.

Answer 2

Your comment notes that you're sshing in with putty... do you have a controlling tty? With openssh you would want to add the -T option, I don't know how/if putty will do this the way you're using it.

Also: you might try using cygwin's ssh instead of putty.

Answer 3

if you already have a current script set up to do this, but are having problems automating part of it, maybe you can just grab expect and use it to provide the setup, then drop back into interactive mode in expect to launch the process. Then you can still have your ctrl-c available to interrupt.