Is there a way to use the CPU of a remote machine to convert in ffmpeg?

Question

My site is built in PHP. I have a WWW server, where all the uploads end up for processing, and then they get rsynced to one of the 4 media servers. If there is a slow and steady stream of uploads, the WWW server converts them all reasonably quickly, but if a bunch of people upload something at the same time, they queue up, and it may take several hours for a file to be processed.

The media servers are typically idle, since serving files off SSD drives results in no iowait, so the CPU is just sitting there, and I wanted to utilize it for conversions.

What would be a good (simple) way to do that?

Answer 1

Have the WWW server copy the file to the media server, and run a continuous process there that converts them, and then informs the web server somehow.

Here's an example using a database for state communication:

1. Web server receives upload
2. Web server copies file to one of the media servers and creates a database entry with the state "NEW", and assigns it to the media server
3. The media server in question notices the new entry while polling the database every 10 seconds or so
4. The media server converts the video and updates the database entry to "PROCESSED" state
5. Video is now visible on the website.

The assignment could either be handled by handing the files over to the media servers in round-robin fashion, or you could even make them report their current workload so that the least busy server can be used.

(Also you have SSD storage for videos? I want some of that...)

Answer 2

Why is the webserver doing the ffmpeg conversion in the first place? it seems the media servers should be doing that anyway.

When a upload arrives, copy the upload the media server, round robin. You could detect the load on the media servers and copy to the least used one. Signal the server copied to do the ffmpeg conversion. When complete the media server copies the processed file to the rsync location for the rest of the servers.

seems fairly simple to me.

Answer 3

You could use ssh -e none hostname command < infile > outfile to execute the conversion process on the media server from the web server, piping the file in via stdin and getting the output file via stdout.

If your network is properly secured, you could use rsh instead, and avoid the computation overhead of the encryption, but you need to be very certain nobody else can get access to your media server, since rsh is a textbook example of a remote-execution exploit.