queque system that manages load

Question

I need a queque system that runs as fast as possible but doesn't slow down the server.

I have a web app (php) that converts different pieces of media (photos,video,etc). When someone wants to convert a file the command to convert goes into a database. I need a program that can run the commands in a way that will not crash my server.

I need the following:

-Something that always is running on the server. -Looks at a postgres database for new content -Program advances to the next command in the database once the previous command has finished -When a command finishes it updates the postgres table with either success/failure and timestamp -The program should use as much cpu as needed but not be the priority...I want the process to be fast but I don't want have my server crawl to a stop. -Program needs to be able to limit different programs (so if app 1 is running and someone submits for app 2...app 2 doesn't run unless the server can handle it.)

Anyone know any existing scripts available or ideas? I have played with cpulimit but that just limits it doesn't let the server use its full potential when there is no server load. I have tried -nice -n but that doesn't seem to work.

Answer 1

A daemon will do well. Consider a client server model - the server dispatches conversion tasks to a child process or calls pthread_create to create a client thread to handle the task. At the time of fork()/exec or pthread_create you can determine the number of active clients, and hence the load on the system. Then you can adjust priority accordingly. This gives you dynamic control of the resource consumption.

I fail to see why nice can not work for you. What exactly does "doesn't seem to work" mean in your context?