How can I implement an OutputStream that I can rewind?

Question

After writing out some processed content to an output stream, I need to revisit the beginning of the stream and write out some content metadata. The data I'm writing is very large, as much as 4Gb, and may be written either directly to a file or to an in-memory buffer, depending on various environmental factors.

How can I implement an OutputStream that allows me to write out headers after completing the writing of content?

Answer 1

If you know the size of the header, you can write a blank header initially, then go back to fix it up with RandomAccessFile at the end. If you don't know the size of the header then you have a fundamental that filesystems generally don't allow you to insert data. So you need to write to a temporary file and then write the real file.

Answer 2

Lucene seems to have an implementation; and the api look ok.

getFilePointer()
void seek(long pos)

http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/store/OutputStream.html

I guess they wrap a RandomAccessFile

Answer 3

One way would be to write initial contents to a memory buffer first, then headers into 'real' output stream, followed by flush of buffered contents, and from thereon just write to the non-buffered stream. It sounds like initial segment wouldn't be all that long, to make buffering reasonable. As to implementing it, you can use ByteArrayOutputStream for buffering, and then have your OutputStream class take "real" output stream as argument; and just switch between the two as necessary. You may need to extend OutputStream API to allow for defining what is metadata to write, as that triggers switch from buffered mode.

As mentioned by the other answer, RandomAccessFile would work too, although wouldn't implement OutputStream.

Answer 4

Here's a random access file output stream.

Note that if using it for a large amount of streamed output you can temporarily wrap it in a BufferedOutputStream to avoid lots of small writes (just be very sure to flush it before discarding the wrapper or using the underlying stream directly).

import java.io.*;

/**
 * A positionable file output stream.
 * <p>
 * Threading Design : [x] Single Threaded  [ ] Threadsafe  [ ] Immutable  [ ] Isolated
 */

public class RandomFileOutputStream
extends OutputStream
{

// *****************************************************************************
// INSTANCE PROPERTIES
// *****************************************************************************

protected RandomAccessFile              randomFile;                             // the random file to write to
protected boolean                       sync;                                   // whether to synchronize every write

// *****************************************************************************
// INSTANCE CONSTRUCTION/INITIALIZATON/FINALIZATION, OPEN/CLOSE
// *****************************************************************************

public RandomFileOutputStream(String fnm) throws IOException {
    this(fnm,false);
    }

public RandomFileOutputStream(String fnm, boolean syn) throws IOException {
    this(new File(fnm),syn);
    }

public RandomFileOutputStream(File fil) throws IOException {
    this(fil,false);
    }

public RandomFileOutputStream(File fil, boolean syn) throws IOException {
    super();

    File                                par;                                    // parent file

    fil=fil.getAbsoluteFile();
    if((par=fil.getParentFile())!=null) { IoUtil.createDir(par); }
    randomFile=new RandomAccessFile(fil,"rw");
    sync=syn;
    }

// *****************************************************************************
// INSTANCE METHODS - OUTPUT STREAM IMPLEMENTATION
// *****************************************************************************

public void write(int val) throws IOException {
    randomFile.write(val);
    if(sync) { randomFile.getFD().sync(); }
    }

public void write(byte[] val) throws IOException {
    randomFile.write(val);
    if(sync) { randomFile.getFD().sync(); }
    }

public void write(byte[] val, int off, int len) throws IOException {
    randomFile.write(val,off,len);
    if(sync) { randomFile.getFD().sync(); }
    }

public void flush() throws IOException {
    if(sync) { randomFile.getFD().sync(); }
    }

public void close() throws IOException {
    randomFile.close();
    }

// *****************************************************************************
// INSTANCE METHODS - RANDOM ACCESS EXTENSIONS
// *****************************************************************************

public long getFilePointer() throws IOException {
    return randomFile.getFilePointer();
    }

public void setFilePointer(long pos) throws IOException {
    randomFile.seek(pos);
    }

public long getFileSize() throws IOException {
    return randomFile.length();
    }

public void setFileSize(long len) throws IOException {
    randomFile.setLength(len);
    }

public FileDescriptor getFD() throws IOException {
    return randomFile.getFD();
    }

} // END PUBLIC CLASS