Incremental interchange format

Question

I need a format to transfer data from program A to program B, and obviously I'd prefer to use a standard format. On the face of it XML would be perfect for this job, or possibly JSON.

The fly in the ointment is that I need the format to be incremental: program A appends data to the file from time to time, then program B slurps the whole lot when it is ready. From what I've read, neither XML nor JSON can do this, because they both require exactly one top-level element in a file.

Is there anything I'm missing?

Clarification:

I need structure of the kind XML provides and CSV doesn't. (Well I could make a mutant variant of CSV that nothing else would be able to read, but that's what I'm trying to avoid.)

A doesn't know when it will be finished (depends on user actions, availability of network connections and other such unpredictable variables), so A can't say "now I will write the last end tag and handover to B" because A might be called again before B.

Answer 1

How about CSV? Easy to write - easy to parse if you know the format and don't need extra metainformation.

Answer 2

If I'm understanding you correctly, the workflow is as follows:

• Over some period of time, A is building up data to put in a file
• Once that is ready, B consumes the whole thing.

In this case, both XML and JSON are fine, since there's a distinct point where A is working and B doesn't start consuming the file until it is processed (as opposed to something like an always-open stream that continually feeds data).

In something like .NET, you could create an XmlDocument with a root element, and continue to add nodes to it until you are done.

Answer 3

So A always needs to write a well-formed file such that B can read it at any time.

If you use XML (and that seems appropriate) you will always have to write a well-formed document (e.g. containing one root node and subnodes within). So you'll have to maintain a DOM object in your process. Depending on memory consumption, that may well be fine.

A can write this whenever an update comes in. The problem here is that writing a file will take a finite amount of time, and you don't want B to read a partially-written file. Some form of signalling that the file is complete will be required here (using temp files as semaphores, renaming etc.)