I need to upload a potentially huge plain-text file to a very simple wsgi-app without eating up all available memory on the server. How do I accomplish that? I want to use standard python modules and avoid third-party modules if possible.
A:
Has python know how to deal with gzip annd zip file I would suggest you to compress your file (using a zip or gzip compliant application like 7-zip) on your client and then upload it to your server.
Even better : write a script that automaticaly compress your file and upload it. This is possible with the standard library.
thomas
2009-07-09 13:47:35
+3
A:
wsgi.input should be a file like stream object. You can read from that in blocks, and write those blocks directly to disk. That shouldn't use up any significant memory.
Or maybe I misunderstood the question?
Lennart Regebro
2009-07-09 13:49:23
these days the file has to be really huge to make memory a real problem anyway.
Aaron Watters
2009-07-16 01:32:17
When you run on a server, even if it has a couple of GB of memory, you also have loads of services up and running, etc, you don't want any file, even 100MB, to suddenly eat up a chunk of memory, because that means, even as a best case, that you had to throw out 100MB of your cache somewhere, which will slow down the servers. It may not crash anything, but a busy server might slow down just because of the upload.
Lennart Regebro
2009-07-16 07:39:45
+1
A:
If you use the cgi module to parse the input (which most frameworks use, e.g., Pylons, WebOb, CherryPy) then it will automatically save the uploaded file to a temporary file, and not load it into memory.
ianb
2009-07-30 21:32:04