What is the best way to store big files in Plone 3?

Question

I want to serve a lot of big files in a Plone site. By big files I mean around 5MB (music) and a lot of them. I've already do it straight to the ZODB, not a good idea. I'm running Plone 3.1.1 and Zope 2.10.6.

Answer 1

As you may already know, the long-term solution for this is supposed to be the ZODB BLOB support. Ticket 6805 is probably the most authorative source on this. Unfortunately, the milestone is set to 4.0, and running it in production on an older release is perhaps not a good solution.

There has, historically, existed a lot of Plone products for storing files externally, keeping only metadata in the ZODB. I have tried several of them, and from my experience, there is not a single one that works well with current Plone/Zope releases. Don't trust me on this, though, I have not tried any products of this type the last year or so.

Personally, I would go for a solution that is as simple as possible and doesn't involve Plone more than neccesary. Storing the music files on disk, serving them directly from apache/whatever web server you use, keeping only metadata in Plone - in a product you write yourself, will give you a robust solution with good performance. That is, your product should produce links to a path on your web server where the music files are available.

If you require authorization for download of the music files and assuming that you run lighthttpd or apache in front of your Zope, looking at a solution based on X-sendfile is probably the best option. With X-sendfile, you keep the files on disk, and add a header (X-sendfile) to the response when a music file should be sent to the client browser. The web server will pick this header up and send the file to the client, without Plone being involved.

Some pointers:

• http://tn123.ath.cx/mod_xsendfile/ (The apache module)

• http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/ (Ruby example)

Answer 2

Zodb blob support is the best, most integrated way to deal with large files. Big files are stored transparently on the filesytem instead of in the zodb object database. "Transparently" in this case means that you won't notice it in your actual programming work after initial configuration.

The blob functionality has been backported to current (halfway 2008) zope versions and can be used in plone 3. Use plone.app.blob in your project for this: http://plone.org/products/plone.app.blob.

Answer 3

Yeah, you shouldn't use anything else than the ZODB BLOB support at this point. It works fine with the 3.x series of releases.

More information in ticket #6805

— Alexander Limi, Plone co-founder

Answer 4

I have plone.app.blob installed on some low-traffic sites and installable (ready to roll, if you like) for my busier production sites in the same instance.

There's the 4.0 milestone but I'll certainly review (and probably click the install button for plone.app.blob on my production sites) around 3.4 time.

A couple of references:

http://n2.nabble.com/PLIPs-I%27d-love-to-see-for-Plone-3.3-tp1123218p1130015.html

http://dev.plone.org/plone/ticket/8629#comment:2 highlight

… 3.4, when we'll probably have blob filestorage specification support added to plone.recipe.zeoserver and zope2instance. That will give us a standard location for whatever owner/permission fixups the installers need to make.

In context: I'm playing roughly with plone.app.blob and a very mixed bag of other add-on products with versions 3.1.7 and 3.2a1 of Plone based on standard and experimental installers. In these environments, without me treating things with kid gloves, Plone sies behave remarkably well and when (as expected) experiments lead to oddities, the support from the community is paced and proper.

Answer 5

Clarifying, to the best of my knowledge:

• from various candidate technologies in a PLIP (Plone Immprovement Proposal), plone.app.blob is the lead contender with widespread support

  -- for exceptional use cases, we sometimes find something other than BLOBs recommended

• 4.0 is currently the most likely milestone for plone.app.blob to become a product within Plone core

• in the meantime plone.app.blob is a recommended add-on product for current 3.x versions of Plone

  -- for use cases that suggest BLOB-like technologies.