Automatically kill process that consume too much memory or stall on linux

Question

Problem: I would like a "system" that monitors a process and would kill said process if:

• the process exceeds some memory requirements
• the process does not respond to a message from the "system" in some period of time

I assume this "system" could be something as simple as a monitoring process? A code example of how this could be done would be useful. I am of course not averse to a completely different solution to this problem. Thank you.

Answer 1

To limit memory usage of processes, check /etc/security/limits.conf

Answer 2

For the first requirement, you might want to look into either using ulimit, or tweaking the kernel OOM-killer settings on your system.

Monitoring daemons exist for this sort of thing as well. God is a recent example.

Answer 3

Are the monitored processes ones you're writing, or just any process?

If they're arbitrary processes then it might be hard to monitor for responsiveness. Unless the process is already set up to handle and respond to events that you can send it, then I doubt you'll be able to monitor them. If they're processes that you're writing, you'd need to add some kind of message handling that you can use the check against.

Answer 4

Try Process Resource Monitor for a classic, easy-to-use process monitor. Code available under the GPL.

There's a few other monitoring scripts there you might find interesting too.

Answer 5

If you want to set up a fairly comprehensive monitoring system, check out monit. It can be very (VERY VERY VERY VERY) chatty at times, but it will do a lot of monitoring, restart services, alert you, etc.

That said, don't be surprised if you're getting dozens of e-mails a day until you get used to configuring it and telling it what not to bug you about.