Very large uploads with PHP

Question

Hi guys,

I want to allow uploads of very large files into our PHP application (hundred of megs - 8 gigs). There are a couple of problems with this however.

Browser:

• HTML uploads have crappy feedback, we need to either poll for progress (which is a bit silly) or show no feedback at all
• Flash uploader puts entire file into memory before starting the upload

Server:

• PHP forces us to set post_max_size, which could result in an easily exploitable DOS attack. I'd like to not set this setting globally.
• The server also requires some other variables to be there in the POST vars, such as an secret key. We'd like to be able to refuse the request right away, instead of after the entire file is uploaded.

Requirements:

• HTTP is a must.
• I'm flexible with client-side technology, as long as it works in a browser.
• PHP is not a requirement, if there's some other technology that will work well on a linux environment, that's perfectly cool.

Answer 1

How about a Java applet? That's how we had to do it at a company I previously worked for. I know applets suck, especially in this day and age with all our options available, but they really are the most versatile solution to desktop-like problems encountered in web development. Just something to consider.

Answer 2

Have you looked into using APC to check the progress and total file size. Here is a good blog post about it. It might help.

Answer 3

Maybe you could use Webdav and Javascript in the browser

AJAX Big file upload, with progress, to WebDAV

http://www.webdavsystem.com/ajax/programming/upload_progress

A simple library

http://debris.demon.nl/projects/davclient.js/doc/README.html

You can then get the JS to redirect the user to a success page. Secret keys and what-not can be handled in a PHP prelude before handing off the JS Client->WebDAV

Answer 4

You can set the post_max_size for just scripts in 1 directory. Place your upload script there, and allow only that script to handle large sizes. It's still possible for that script to be attacked with large/useless files, but it avoids setting it globally.

Use that with APC and you might be able to work out something good: IBM Developer works article on APC

Answer 5

I would look into FTP, SSH or SCP this allows you to upload a large file and still have access control over the file as well. This might take a little longer to implement but its probably the most secure way I could think of.

Answer 6

I know it sucks to add another dependency but in my experience, most websites that are doing something like this are using flash on the client side, and uploading the large file as chunks

adobe as a howto on flash file uploads

I also found this tutorial on codeproject:

Multiple File Upload With Progress Bar Using Flash and ASP.NET

PS - I know you're using PHP and not .net, I figured the important part was the flash ;)

Answer 7

Python Handler?

Using a Python POST handler instead of PHP. Generate a unique identifier from your PHP app that the client can put in the HTTP headers. With mod_python to reject or accept the large upload before the entire POST body is transmitted.

I think http://www.modpython.org/live/current/doc-html/dir-handlers-hph.html

Allows you to check headers and decline the rest of the POST input. I haven't tried it but might be the right path?

Looking at the source of mod_python, the buffering of the input via read() seems to allow bit-at-a-time evaluation of the HTTP input. Headers are first.

https://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk/src/filterobject.c

Answer 8

upload_max_filesize can be set on a per-directory basis; the same goes for post_max_size

e.g.:

<Directory /uploadpath/>
  php_value upload_max_filesize 10G
  php_value post_max_size 10G
</IfModule>

Answer 9

I've had success with uploadify, and I would recommend it. It's a jQuery/Flash script that handles large uploads, and you can pass extra parameters to it (like the secret key). To solve the server-side issues, simply use the following code. The changes take affect just for the script they're called in:

//Check to see if the key is there
if(!isset($_POST['secret_key']) || !isValid($_POST['secret_key']))
{
    exit("Invalid request");
}
function isValid($key)
{
    //Put your validation code here.
}

//This line changes the timeout.
//Give it a value in seconds (3600 = 1 hour)
set_time_limit(3600);

//Set these amounts to whatever you need.
ini_set("post_max_size","8192M");
ini_set("upload_max_filesize","8192M");

//Generally speaking, the memory_limit should be higher
//than your post size.  So make sure that's right too.
ini_set("memory_limit","8200M");

EDIT In response to your comment:

Given what you've said, I'm afraid you may not be able to meet your requirements over http. All of the solutions out there are code that add features to http that it was never designed for.

Like you said yourself, it's a simple protocol. Apart from writing your own client software that runs outside of the browser, a java applet, or using a different protocol (like FTP, which was designed for this), you might not get what you want.

I've done the best I could within the given constraints. Sorry I couldn't do better.

Answer 10

Tried all of this... this is by far the best I have used yet...

http://www.uploadify.com/

Answer 11

Take a look at jumploader.com

A good java-applet for uploading.

I've used it for uploading images and it works fine. Haven't tried with bigger files than 10MB, but i should work for really big files too.

Answer 12

Try JFileUpload. It supports HTTP, FTP with PHP, JSP or ASP.NET script to handle large uploads. It can split files in chunks and recompose files from chunks on server-side.
http://www.jfileupload.com/products/jfileupload/index.html