Multi-server monitor/auto restarter in python

Question

I have 2 server programs that must be started with the use of GNU Screen. I'd like to harden these servers against crashes with a Python based program that kicks off each screen session then monitors the server process. If the server process crashes, I need the python code to kill the extraneous screen session and restart the server with screen again.

I'm very new to python but I'm using this opportunity to teach myself. I'm aware this can be done in bash scripting. But I want to build on this code for future features, so it needs to be just python.

The pseudocode is as follows:

thread-one {
     While 1:
     start server 1 using screen
     wait for server to end
     end while
}

thread-two {
     While 1:
     start server 2 using screen
     wait for server to end
     end while
}

Answer 1

You really shouldn't run production software on a screen. If the server gets rebooted, how will You start it up? Manually? Also I think that You are trying to re-invent the wheel. There are already pretty good tools that do the thing You need.

launchtool lets you run a user-supplied command supervising its execution in many ways, such as controlling its environment, blocking signals, logging its output, changing user and group permissions, limiting resource usage, restarting it if it fails, running it continuously, turning it into a daemon, and more.

.

Monit is a free open source utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Answer 2

"need to be multi-threaded to handle the restarting of two separate programs"

Don't see why.

import subprocess

commands = [ ["p1"], ["p2"] ]
programs = [ subprocess.Popen(c) for c in commands ]
while True:
    for i in range(len(programs)):
        if programs[i].returncode is None:
            continue # still running
        else:
            # restart this one
            programs[i]= subprocess.Popen(commands[i])
        time.sleep(1.0)

Answer 3

I wrote a python script I use to start some Team Fortress 2 and Left 4 Dead servers on different screens. When invoked it checks all current screens with 'screen -list' and then runs the different server instances that aren't already running.

I thought you may find it useful, so here it is.