Django upload failing on request data read error

Question

Hi All,
I've got a Django app that accepts uploads from jQuery uploadify, a jQ plugin that uses flash to upload files and give a progress bar.

Files under about 150k work, but bigger files always fail and almost always at around 192k (that's 3 chunks) completed, sometimes at around 160k. The Exception I get is below.

exceptions.IOError
request data read error
File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 171, in _get_post
    self._load_post_and_files()
File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 137, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META,     self.environ[\'wsgi.input\'])
File "/usr/lib/python2.4/site-packages/django/http/__init__.py", line 124, in parse_file_upload
    return parser.parse()
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 192, in parse
    for chunk in field_stream:
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 314, in next
    output = self._producer.next()
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 468, in next
    for bytes in stream:
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 314, in next
    output = self._producer.next()
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 375, in next
    data = self.flo.read(self.chunk_size)
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 405, in read
    return self._file.read(num_bytes)

When running locally on the Django development server, big files work.
I've tried setting my FILE_UPLOAD_HANDLERS = ("django.core.files.uploadhandler.TemporaryFileUploadHandler",) in case it was the memory upload handler, but it made no difference.

Does anyone know how to fix this?

Answer 1

The problem could be:

• your internet connection (some firewall blocking large uploads, just try uploading to some other service to test that).
• your webbrowser (try disabling flash to see if that works)
• your webserver (perhaps your webserver prohibits large uploads?)

So... to be a bit more helpful you'll have to give us some info about which part is the problem (should be fairly easy to test) and if it's the webserver (which I'm guessing) what kind of webserver you're running, what kind of Python handler (mod_wsgi, mod_python, fastcgi, etc..)

Answer 2

I'm currently having the same issue. But for much larger files. Running Django 1.1.1, mod_wsgi 2.6 (prefork MPM) and apache 2.2 all on Ubuntu. On the front end I'm seeing a timeout somewhere between 15 and 30 min (uploading files over 100MB). I've seen consistent failures almost every time on IE7 and IE8. Only failed once on FF. But FF survived a 250MB upload well over 15 min. In my apache logs I'm seeing the same "IOError: request data read error". I had a thread for this, but I killed it in favor of this fresher one with more activity. (Sorry this is posted as an answer, can't leave comments).

Answer 3

This is common with MediaTemple -- I encountered this problem, fixed it, and somehow it's broken again. It has to deal with something about NFS locking, and the system calls used by django's storage modules.

... (10 minutes later) ...

Found it:

Anyone trying to run the latest 1.1 svn release will have noticed that the above fix no longer works since TemporaryFileUploaderHandler now uses the locking code also. A workaround until mediatemple and/or django devs figure this out is to edit django.core.files.locks and change the two calls to fcntl.lockf to fcntl.flock

Here's a relevant bug report for anyone who's interested: http://code.djangoproject.com/ticket/9400

http://osdir.com/ml/DjangoUsers/2009-06/msg01014.html -> https://forums.mediatemple.net/viewtopic.php?id=2514