Python: Monitoring and killing/throttling spawned processes based on load, time, etc

Question

I have a queue of workers that spawn external third party apps using subprocess. I'd like to control how much of the overall resources of my server these process consume. Some of these external apps also tend to hang for unknown reasons, fixed with a restart.

What's a good way to:

• Monitor the overall server load (say, load average or equivalent of vmstat) in python?
• Monitor the cpu load of the processes I spawn?
• Kill processes I've spawned if they're taking too long or taking too much cpu?

Basically I need to be able to control the load the I'm placing on my server with my spawned threads.

Hopefully there's a package or library that'll do all this for me?

Answer 1

As for governing CPU, you'll want to use nice to launch your processes.

For monitoring system load and other stats related to currently running processes, you might look into the /proc directory of pseudo-devices.

Answer 2

Functions to get load average and kill process are available in standard python library (os.getloadavg(), os.kill(), subprocess.Popen.kill()). There is a psutil package for the rest (psutil.Process.get_cpu_times(), psutil.Process.get_cpu_percent(), psutil.Process.get_memory_info(), psutil.Process.get_memory_percent() and more)

Answer 3

Monitor the overall server load (say, load average or equivalent of vmstat) in python?

>>> import psutil, subprocess
>>> subp = subprocess.Popen('python', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> proc = psutil.Process(subp.pid)
>>> rss, vms =  proc.get_memory_info()
>>> print "Resident memory: %s KB" %(rss / 1024)
Resident memory: 136 KB
>>> print "Virtual memory: %s KB" %(vms / 1024)
Virtual memory: 356 KB
>>> print proc.get_memory_percent()
0.00324324118077

Monitor the cpu load of the processes I spawn?

>>> proc.get_cpu_percent()
0.0

Kill processes I've spawned if they're taking too long or taking too much cpu?

>>> proc.kill()
>>>