What are some best-practices for writing documents with leading header data?

Question

I know this is likely to be an open-ended kind of question, but I'll take the best-regarded answers and consolidate them here for reference.

I'm working on a set of tools to read and write Mobipocket files (a.k.a. Kindle eBooks) using C#, and am running into a design question when it comes to writing new files. The underlying file format starts with a header that includes offset/size information for all of the following "blocks" of data. This leaves me with the following two options:

1. Collect all of the file data (blocks) in advance before starting to write the file. This may mean that large amounts of document content are in memory, but has the advantage of allowing me to write the file in a single forward-only pass.

2. Write placeholder data in the header, followed by the not-known-in-advance file data, and then seek back into the header and fix up the offset/size data just before the file is closed. This seems more complex, but has the advantage of allowing the document content to be streamed in as it's generated, potentially reducing the memory footprint.

There may be other designs as well, which I'd love to hear about. I'm currently leaning towards #1 because I'm not anticipating particularly large documents and I think the programming model may be easier to deal with. If you've had any experience either positive or negative with one of these designs (or another proposed suggestion), please let me know!

Answer 1

I would opt for the slightly more complex second design. It doesn't require too much more effort, and it will ensure that your code still runs smoothly if someone decides to throw a 1000GB file at it a few years from now.

I too would be curious to see what other approaches might be recommended.