Compression types for a specific use in Python

Question

I actually partially asked this in this thread http://stackoverflow.com/questions/3494020/organizing-files-in-tar-bz2-file-with-python But it mad eme rethink what I was doing.

Now I think it might be a good idea to change the compression method I am using. I am currently using tar.bz2 compression. What I need to be able to do is access the compressed files one at a time, and specifically by name, to build them into a different file. Zip compression is supposed to support this well, but the number of files I am compressing, 200,000 txt files, seem to be to large for that file type. Zips can only support 64k files, or something like that. Does anyone have any idea of what compression type I should use?

Also, it would help if there was a python library built for it, such as tarfile, zipfile etc.

Answer 1

You could compress the individual files and put those into a tar (or ar) container.