On the fly zip or tar creation?

Question

I may NOT bother with this but if its very simple i may consider it. The site i am working on by design is to hold hundreds of thousands of files. I dont know if we'll have only one download or multiple. Right now the choices are A) Just the file B) An archive that has the file + license and conditions.

I am trying to figure out it can be efficient to offer both and use something like file.open/read prefixing an archive header before it and after it which contains the license and other zip contents. My biggest worries are doing file open/read will not be as efficient as letting the server transmit the file and if its hard to generate and change the contents of the zip dynamically (if a user wishes to change the license or if we want to add other data such as author description, author URL and a permalink on the site)

Is it efficient and how would i create the file dynamically only the original file and data pulled from the database?

PS: I am using debian/apache/asp.net using xsp.net and mono.

Answer 1

SharpZipLib is a very nice stream based library that you can use to create archive files.

Answer 2

You can use zip libraries (System.IO.Packaging.ZipPackage, dotnetzip, SharpZipLib) or event command line programs (say 7zip) for compressing the file . The library should offer better performance.

However, important thing will be to add an caching layer i.e. zip files should be cached in file system so that they can be served directly if request comes for the same.