Alternative upload method for Django FileField

Question

Hi

I have some django models that use a FileField, and users have been uploading files in the admin interface. We now have a problem that some files are quite big (1-3G), which makes http upload tricky.

There is a "container" class, and then "file" classes with a FK to the container. I have used inlines in the admin gui to make this easier for the users (container class form, with file class inlines)

I have found FilePathField, which I guess will be useful to let users scp / rsync files over to the server and then browse for and add details to them, but how can I get this in the admin? Do I need to just build a new form / view that creates the objects? Is there any way to somehow override the model in a custom admin form (and thus keep all the free admin goodness), that replaces filefield with filepathfield? Any other suggestions?

Thanks for any suggestions!

Answer 1

I'm not sure if I understood what you want to include in the admin area. Are you looking for a way automate the file upload process with a transfer protocol other than HTTP?

If so, you could create a model with a basic CharField (or maybe a URLField), and then perform a rsync or scp copy when the object is saved. For example:

from django.db import models

class File(models.Model):
   path = models.CharField()
   def save(self):
      import os
      # WARNING! The path given by the user should be sanitized; this is
      # example code only. This is a security vulnerability.

      # Attempt to rsync the target file from a remote machine
      exit_code = os.system("rsync %s /incoming/files/path/" % self.path)
      # Make sure it worked before saving
      if exit_code == 0:
         super(File, self).save() # Call the "real" save() method
      else:
         # Raise exception

This is a very rough example, but it should help you figure something out. You'll need to make sure that your routine is secure (malformed paths could allow a user to run anything they want, for example). Also, this will likely block the Django save routine while copying over a file, so you should look at running your preferred command in a separate process.

Good luck!

Answer 2

I'm no expert in transferring huge files, but if your users need to be able do that without "UNIX incantations", I would suggest looking into creating a Java applet for it. It would be a lot of work, but then you could have proper progress bar etc.

Answer 3

You mention that FilePathField works, but needs to have the webserver restarted to see new files. This happens because the options are gathered FilePathField.__init__, which is called once when the module is imported.

A fix would be to re-call the field's __init__ in the form's __init__:

def __init__(self, **kwargs):
    super(MyForm, self).__init__(**kwargs)
    self.fields['file'].__init__(path)

(That way the directory is scanned each time the form is instanced.)