Streaming large file uploads to ASP.NET MVC

Question

For an application I'm working on, I need to allow the user to upload very large files--i.e., potentially many gigabytes--via our website. Unfortunately, ASP.NET MVC appears to load the entire request into RAM before beginning to service it--not exactly ideal for such an application. Notably, trying to circumvent the issue via code such as the following:

if (request.Method == "POST")
{
    request.ContentLength = clientRequest.InputStream.Length;
    var rgbBody = new byte[32768];

    using (var requestStream = request.GetRequestStream())
    {
        int cbRead;
        while ((cbRead = clientRequest.InputStream.Read(rgbBody, 0, rgbBody.Length)) > 0)
        {
            fileStream.Write(rgbBody, 0, cbRead);
        }
    }
}

fails to circumvent the buffer-the-request-into-RAM mentality. Is there an easy way to work around this behavior?

Answer 1

ASP.NET MVC is no magic. It does share the same limitations 'regular' ASP.NET have, including direct uploading of huge files.

If you want to accept gigs of data, you should explore better solutions, involving a client that can chop the sent file into chunks, (and gain better recoverability, scalability etc.)

Answer 2

Sure, you can do this. See RESTful file uploads with HttpWebRequest and IHttpHandler. I have been using this method for a few years and have a site that has been tested with files of at least several gigabytes. Essentially, you want to create your own IHttpHandler, which is easier than it sounds.

In a nutshell, you create a class that implements the IHttpHandler interface, meaning you have to support the IsReusable property and the ProcessRequest method. On top of that, there is a minor change to your web.config, and it works like a charm. At this stage in the life cycle of the request, the entire file being uploaded does not get loaded into memory, so it neatly steps around out of memory issues.

Note that in the web.config,

<httpHandlers>
 <add verb="*" path="DocumentUploadService.upl" validate="false" type="TestUploadService.FileUploadHandler, TestUploadService"/>
</httpHandlers>

the file referenced, DocumentUploadService.upl, doesn't actually exist. That is just there to give an alternate extension so that the request is not intercepted by the standard handler. You point your file upload form to that path, but then your FileUploadHandler class kicks in and actually receives the file.

Update: Actually, the code I use is different from that article, and I think I stumbled on the reason it works. I use the HttpPostedFile class, in which "Files are uploaded in MIME multipart/form-data format. By default, all requests, including form fields and uploaded files, larger than 256 KB are buffered to disk, rather than held in server memory."

if (context.Request.Files.Count > 0)
{
    string tempFile = context.Request.PhysicalApplicationPath;
    for(int i = 0; i < context.Request.Files.Count; i++)
    {
        HttpPostedFile uploadFile = context.Request.Files[i];
        if (uploadFile.ContentLength > 0)
        {
            uploadFile.SaveAs(string.Format("{0}{1}{2}",
              tempFile,"Upload\\", uploadFile.FileName));
        }
    }
}

Answer 3

It turns out that my initial code was basically correct; the only change required was to change

request.ContentLength = clientRequest.InputStream.Length;

to

request.ContentLength = clientRequest.ContentLength;

The former streams in the entire request to determine the content length; the latter merely checks the Content-Length header, which only requires that the headers have been sent in full. This allows IIS to begin streaming the request almost immediately, which completely eliminates the original problem.