Django return large file

Question

I am trying to find the best way (most efficient way) to return large files from Django back to an http client.

• receive http get request
• read large file from disk
• return the content of that file

I don't want to read the file then post the response using HttpResponse as the file content is first stored in RAM if I am correct. How can I do that efficiently ?

Laurent

Answer 1

My preference for all of this is to synthesize django with your http server so that when you want to serve static files, you simply refer them to a path that will never reach django. The strategy will look something like this:

1. Configure http server so that some requests go to django and some go to a static document root
2. link to static documents from any web pages that obviously need the static documents (e.g. css, javascript, etc.)
3. for any non-obvious return of a static document, use an HttpRedirect("/web-path/to/doc").
4. If you need to include the static document inside a dynamic document (maybe a page-viewer wrapping a large text or binary file), then return a wrapper page that populates a div with an ajax call to your static document.

Answer 2

Look into mod_xsendfile on Apache (or equivalents for nginx, etc) if you like to use Django for authentication. Otherwise, there's no need to hit django, and just server straight from Apache.

Answer 3

My suggestion is to transfer the file over the network in chunks instead of all at once.

Answer 4

There is a ticket that aims to deal with this problem here: http://code.djangoproject.com/ticket/2131

It adds an HttpResponseSendFile class that uses sendfile() to send the file, which transparently sends the file as it's read.

However, the standard HttpResponse is implemented as an iterator, so if you pass it a file-like object, it will follow its iteration semantics, so presumably you could create a file-like object wrapper that chunks the file in small enough pieces before sending them out.

I believe the semantics of iterating over a standard file object in python is that it reads line-by-line, which most likely won't solve your problem if you're dealing with binary files.

Of course, you could always put the static files in another location and serve that with a normal web server, unless you require intricate control (like access control requiring knowledge of the Django database)