Java (ME) library for fixed-length record files

Question

I am looking for a library that can run on Java ME (Foundation Profile 1.1, CDC) and allows me to basically do something along the lines of

FILE OF type;

in Pascal.

Background: I need to have a largish (approx. 100MB) set of around 500.000 records for lookups by a known index value quickly. Do I really have to write this myself? Databases like Derby are way too big and bring lots of features (stored procedures, anyone?) I do not need.

Ideally I would just like to define a class with a few fields based on primitive types and Strings as a value holder object and persist these in a file I could - should the need arise - manually recover. That's why I am not too much into serialization. From the past I have fought several occasions of corrupted binary data files which could not be recovered at all.

Answer 1

Your biggest problem here is establishing a correspondence between field names and columns in the file, as you really shouldn't assume that the class layout matches the field ordering in the source file.

If the file were to contain a header row then it's a simple matter of using reflection/introspection and shouldn't take more than a day to implement yourself.

Alternatively, you'll have to use an annotation of some sort to specify, for each field, where it appears in the file.

Have you instead considered alternative text serialization methods, such as CSV, JSON or XML using XStream? These avoid the risks of binary corruption and would get you up and running faster, but might also impose a higher memory footprint which could be an issue as you're targeting a mobile device.

Answer 2

After looking around for quite some time, I have finally come to xBaseJ from SourceForge. It relies on java.nio, which is normally not included in the JavaME CDC profile, but we had a contractor port the relevant parts to the mobile J9 VM. Armed with this, we are now building our application on top of DBase III compatible files. Apart from being pretty reasonably fast, even on the mobile platform, this gives us access to a plethora of tools that can handle this format, without having to teach non-tech folks about a JDBC based DB admin tool they do not feel comfortable with.

There has just been a recent release of a whole eBook, called "The Minimum You Need To Know About xBaseJ", which is available for free from the project's website, too.