Validating Uploaded Files in Django

Question

A Django app that I am working has an Event model. An Event may have associated photos, static html files and pdf files.

I would like to allow trusted users to upload these files, but I am wary about security, especially having read the following in the Django docs (link).

Note that whenever you deal with uploaded files, you should pay close attention to where you're uploading them and what type of files they are, to avoid security holes. Validate all uploaded files so that you're sure the files are what you think they are. For example, if you blindly let somebody upload files, without validation, to a directory that's within your Web server's document root, then somebody could upload a CGI or PHP script and execute that script by visiting its URL on your site. Don't allow that.

How can I validate the different types of files? I would be interested to hear anyone's experience of dealing with this kind of thing, or links for further reading. I have a gut feeling that html files may be too risky, in which case I'll restrict upload permissions to the administrator.

Answer 1

The first thing you want to do with the uploaded content is store it in a directory which is not directly accessible for downloading. If your app exists in ~/www/ consider putting your data in '~/data/`.

The second thing, you need to determine what kind of file the user uploaded, and then create rules for each file type.

You can't trust the file based on the extension, so use something like Fileinfo. Then for each mime type, create a validator. ImageMagick can validate image files. For higher security, you may have to run a virus scanner over files like pdf's and flash files. For html, you may want to consider limit to a subset of tags.

I can't find a Python equivalent of the Fileinfo module, though it's always possible to exec /usr/bin/file -i . Most system that allow uploads then create a content name or id. They then use mod_rewrite to parse the URL, and find the content on disk. Once the content is found, it's returned to the user using sendfile, or something similar. For example, until the content is approved, maybe only the user who uploaded it is allowed to view it.

Answer 2

For images you might be able to just use Python Imaging Library (PIL).

Image.open(filepath)

If the file is not an image, an exception will be thrown. I'm pretty new to Python/Django so someone else might have a better way of validating images.

Answer 3

you can validate html files with BeautifulSoup

Answer 4

'trusted users' is a subjective term. Is it people that you know in person or only someone who has created an account on your app? Don't give access to your filesystem to people that you don't know in person.

Giving the ability to someone to upload a file is in any case a bit dangerous and I think that it should be avoided. I was facing a similar problem last week with the automatic upload of html code and I've decided to store it in the database. I think that in most cases, you can use the database rather than the file system.

One problem with the validation is that you'll have to write a new validator for any type of files. It can be a limitation in the future and be a big task in some cases.

So, I would recommend to reconsider a database-based design.

Answer 5

This is a little bit specific to your hosting environment, but here is what I do:

Serve all user uploaded content with Nginx instead of apache, and serve it all as static content (it will not run any of the php or cgi, even if the users upload it)

Answer 6

All the answers are focusing on validating files. This is pretty much impossible.

The Django devs aren't asking you to validate whether files can be executed as cgi files. They are just telling you not to put them in a place where they will be executed.

You should put all Django stuff in a specially Django directory. That Django code directory should not contain static content. Don't put user files in the Django source repository.

If you are using Apache2, check out the basic cgi tutorial: http://httpd.apache.org/docs/2.0/howto/cgi.html

Apache2 might be setup to run any files in the ScriptAlias folder. Don't put user files in the /cgi-bin/ or /usr/local/apache2/cgi-bin/ folders.

Apache2 might be set to server cgi files, depending on the AddHandler cgi-script settings. Don't let the users submit files with extensions like .cgi or .pl.

However, you do need to sanitize user submitted files so they are safe to run on other clients' machines. Submitted HTML is unsafe to other users. It won't hurt your server. Your server will just spit it back at whoever requests it. Get a HTML sanitizer.

Also, SVG may be unsafe. It's had bugs in the past. SVG is an XML document with javascript in it, so it can be malicious.

PDF is ... tricky. You could convert it to an image (if you really had to), or provide an image preview (and let users download at their own risk), but it would be a pain for people trying to use it.

Consider a white-list of files that are OK. A virus embedded in a gif, jpeg or png file will just look like a corrupt picture (or fail to display). If you want to be paranoid, convert them all to a standard format using PIL (hey, you could also check sizes). Sanitized HTML should be OK (stripping out script tags isn't rocket science). If the sanitization is sucking cycles (or you're just cautious), you could put it on a separate server, I guess.