Alternative to ZIP as a project file format. SQLite or Other?

Question

My Java application is currently using ZIP as a project file format. The project files contain a few XML files and many image and sound files.

The project files are getting pretty big, and since I can't find a way with the java.util.zip classes to write to a ZIP file without recreating it, my file saves are becoming very slow. So for example, if I just want to update one XML file, I need to rewrite the entire ZIP.

Is there some other Java ZIP library that will allow me to do random writes to a ZIP file?

I know switching to something like SQLite solves the random write issue. Would using SQLite just to write XML, Sound and Images as blobs be an appropriate use?

I suppose I could come up with my own file format and use RandomAccessFile but then there would be a lot of bookkeeping I'd have to write.

Update...

My file format is very much like Office Open XML. It is a ZIP file containing XML and other resources.

Someone must have solved the problem of how to do random writes to update a ZIP file. Does anyone know how?

Answer 1

First of all I would separate your app's resources in those that are static (such as images) and those that can be changed (the xml files you mentioned). Since the static files won't be re-written, you can continue to store them in a zip file, which IMHO is a good approach to deploy any resources.

Now you have 2 options:

• Since the non-static files are probably not too big (the xml files are likely to be smaller than images+sounds), you can stick with your current solution (zip file) and simply maintain 2 zip files, of which only one (the smaller one with the changeable files) can/will be re-written.

• You could use a in-memory-database (such as hsqldb) to store the changeable files and only persist them (transferring from the database to a file on the drive) when your application shuts down or that operation is explicitly needed.

Answer 2

sqlite is not always fast (at least in my experience). I would suggest individually compressing the XML files -- you'll still get decent compression, and just use the file system to save them. You could experiment with btrfs, or just go with ext4. If you're not on Linux, then this should still work okay, but it might not be as fast until things are cached in memory.

the idea is that if you do not have redundancy between XML files, then you don't get that much saving by compressing them in one "solid" archive.

Answer 3

There exist so-called single-file virtual file systems, that let you create file-based containers and provide file-system like structure and APIs. One of the samples is SolFS (it has C-written core with JNI wrapper) and some other C- and Delphi-written solutions (I don't remember their names at the moment). I guess there exist similar native Java solutions as well.

Answer 4

Before offering another answer along the lines of using properly structured JARs, I have to ask -- why does the project need to be encapsulated in one file? How do you distribute the program to users to run?

Answer 5

If you must keep a project contained within a single file and be able to replace resources efficiently, yes I would say SQLite is a good choice.

If you do choose to use SQLite, also consider converting some of the XML schemas to one or more SQL tables rather than storing large XML documents as BLOBs.