ansaurus

Question

Troubleshooting SIGTERMs with tee on a cluster within SGE jobs

Answer 1

+1 A:

I don't know anything about why your particular case is failing, but one option might be to make $GMSCOMMAND do it's own logging. (Effectively put the tee inside the app). I guess this option depends on cost of changing the legacy app.

Failing that you could wrap the 'legacy app' with your own script/application to do the redirection/duplication.

Douglas Leeder 2009-11-23 11:01:18

Thanks Douglas, it would have been a good idea if I were allowed to modify the legacy app, but as a matter of policy I can't do this.

tramdas 2009-11-23 12:17:56

Ok, could you try replacing the `tee` command with `cat` - to see if it's the pipe that's the problem?

Douglas Leeder 2009-11-23 13:23:48

Indeed, `| cat` fails the same way `| tee` does, so it does seem like the pipe is the problem.With this in mind I'm starting to wonder about somehow executing `$GMSCOMMAND > $outputfile` detached, and then streaming $outputfile to STDOUT until EOF.But before attempting that I really need to properly look through the Sun Grid Engine documentation to see if there's some relevant setting information.Thanks for your help thus far, if you have any other ideas, I'm all ears...

tramdas 2009-11-23 16:30:47

As pipe is the problem - I suggest writing your own wrapper.

Douglas Leeder 2009-11-23 17:51:38

Does it fail when piping to tee if you run it interactively, not from SGE?@Douglas Leeder: SGE runs your job script with the shell, so it's highly unlikely that having that script run another script would make any difference. Hmm, unless it's using something other than /bin/sh. I also had the idea of finding out what file stdout is going to, and copying it after $GMSCOMMAND exits. SGE sets quite a few variables in the job's environment...

Peter Cordes 2009-12-08 06:04:21

First of all, sorry for the amazingly late reply... for the last few months I've been occupied with other more pressing work.I've discovered that it's an issue with my specific legacy app; using other apps, the pipe doesn't cause a SIGPIPE/SIGTERM or whatever... and on that note, I didn't get to discover exactly what signal it was because I could not run strace - it's not installed on the slave nodes.Anyway, I wrote a wrapper that utilizes forkpty in an obvious way. It's pretty standard usage of forkpty, so I won't post the code here unless someone asks.Thanks all.

tramdas 2010-03-21 17:24:45

Answer 2

A:

If pipes are your problem perhaps you can get around this by using a 'while/read' loop with process substitution. Does this work for you?

while read line; do
    echo "$line"
    echo "$line" >> ${SCRATCHDIR}/gamess_output.log
done <(${GMSCOMMAND})

SiegeX 2009-12-16 11:20:49

ansaurus

tags:

views:

answers:

Troubleshooting SIGTERMs with tee on a cluster within SGE jobs

related questions