Appending to an ObjectOutputStream

Question

Is it not possible to append to an ObjectOutputStream?

I am trying to append to a list of objects. Following snippet is a function that is called whenever a job is finished.

FileOutputStream fos = new FileOutputStream
           (preferences.getAppDataLocation() + "history" , true);
ObjectOutputStream out = new ObjectOutputStream(fos);

out.writeObject( new Stuff(stuff) );
out.close();

But when I try to read it I only get the first in the file. Then I get java.io.StreamCorruptedException.

To read I am using

FileInputStream fis = new FileInputStream
     ( preferences.getAppDataLocation() + "history");
ObjectInputStream in = new ObjectInputStream(fis);    

try{
    while(true)
        history.add((Stuff) in.readObject());
}catch( Exception e ) { 
    System.out.println( e.toString() );
}

I do not know how many objects will be present so I am reading while there are no exceptions. From what Google says this is not possible. I was wondering if anyone knows a way?

Answer 1

Because of the precise format of the serialized file, appending will indeed corrupt it. You have to write all objects to the file as part of the same stream, or else it will crash when it reads the stream metadata when it's expecting an object.

You could read the Serialization Specification for more details, or (easier) read this thread where Roedy Green says basically what I just said.

Answer 2

As the API says, the ObjectOutputStream constructor writes the serialization stream header to the underlying stream. And this header is expected to be only once, in the beginning of the file. So calling

new ObjectOutputStream(fos);

multiple times on the FileOutputStream that refers to the same file will write the header multiple times and corrupt the file.

Answer 3

The easiest way to avoid this problem is to keep the OutputStream open when you write the data, instead of closing it after each object. Calling reset() might be advisable to avoid a memory leak.

The alternative would be to read the file as a series of consecutive ObjectInputStreams as well. But this requires you to keep count how many bytes you read (this can be implementd with a FilterInputStream), then close the InputStream, open it again, skip that many bytes and only then wrap it in an ObjectInputStream().

Answer 4

Here's the trick: subclass ObjectOutputStream and override the writeStreamHeader method:

public class AppendableObjectOutputStream extends ObjectOutputStream {

  public AppendableObjectOutputStream(OutputStream out) {
    super(out);
  }

  @Override
  protected void writeStreamHeader() throws IOException {
    // do not write a header
  }

}

To use it, just check whether the history file exists or not and instantiate either this appendable stream (in case the file exists = we append = we don't want a header) or the original stream (in case the file does not exist = we need a header).

Edit

I wasn't happy with the first naming of the class. This one's better: it describes the 'what it's for' rather then the 'how it's done'

Answer 5

I ran into this issue and was in a situation where I had to append to the same file - across app resets/crashes/etc. I really do not like mucking with the derived class idea as it may NOT be future proof - nice hack but still a hack. BTW: prevayler logging works this way...

If you must append (there's a very good chance you can work around it for most things) to existing file but cannot do it in one "open", then I have a write up with 2 decent solutions .