Writing a starter program which aborts hanging programs

Question

I have a script which needs to periodically start programs out of a array with program names via Perl on Linux. The problem is that from time to time one of the programs takes too long/hangs and needs to be aborted.

Currently I am starting the program using qx/$cmd/ in a seperate thread which reads from a shared start queue. The main thread enqueues every x seconds a new element into the queue. If there are elements in the queue, the main thread kills the child thread and starts a new child.

This works fine from a functional perspective, but now I have realised that this leads to a memory leak. How would you design such a program ? Is there any CPAN module which can help? Please let me know if you need further code to understand the problem.

The main thread looks like this:

     if (!$startQueue->pending) {
   $startQueue->enqueue($programList[$i++]);
 } else {
   $log->warn("Aborting hanging execution");
   $starterThread->kill('KILL')->detach();
   $log->info("Creating new thread");
   $starterThread=threads->create("starterThread");
 }

The child thread like this:

sub starterThread{
  $SIG{'KILL'}=sub{threads->exit();};
  $log->info("Starter Thread started");
  while() {
    my $programName=$startQueue->dequeue();
    $log->debug("programName:$programName");
    qx/$programName/;
  }
}

Answer 1

An alternative to a manager process like that is a PID file with a timestamp (i.e. contents are pid timestamp), like sendmail uses. Then you start up a new copy of the process every minute from cron, or something, and if an old process is there, the new process either dies (if the timestamp is recent) or kills the old process (if the timestamp is old).

I don't actually know why your manager process should necessarily lead to a memory leak, though. Have you firmly established that this is the case? What's your reasoning?

Answer 2

You could have a look at Parallel::ForkManager which lets you manage a fixed number of child processes at any given time. Does the program name array remain static through the life of the script or does it update itself continuously?

The memory leak sounds fishy; have you profiled your script and determined absolutely that the kill/enqueue leads to the memory leak?

Answer 3

Normally use ALARM signal + self slaying within the program itself rather than a chaser program.