Bytes consumed by StreamReader

Question

Is there a way to know how many bytes of a stream have been used by StreamReader?

I have a project where we need to read a file that has a text header followed by the start of the binary data. My initial attempt to read this file was something like this:

private int _dataOffset;
void ReadHeader(string path) 
{
    using (FileStream stream = File.OpenRead(path)) 
    {
        StreamReader textReader = new StreamReader(stream);

        do 
        {
            string line = textReader.ReadLine();
            handleHeaderLine(line);
        } while(line != "DATA") // Yes, they used "DATA" to mark the end of the header

        _dataOffset = stream.Position;
    }
}

private byte[] ReadDataFrame(string path, int frameNum) 
{
    using (FileStream stream = File.OpenRead(path)) 
    {
        stream.Seek(_dataOffset + frameNum * cbFrame, SeekOrigin.Begin);

        byte[] data = new byte[cbFrame];
        stream.Read(data, 0, cbFrame);

        return data;
    }
    return null;
}

The problem is that when I set _dataOffset to stream.Position, I get the position that the StreamReader has read to, not the end of the header. As soon as I thought about it this made sense, but I still need to be able to know where the end of the header is and I'm not sure if there's a way to do it and still take advantage of StreamReader.

Answer 1

So your last line contains 'DATA' + an unknown amount of data bytes. You could extract the position by using IndexOf() with your last read line. Then readjust the stream.Position.

But I am not sure if you should use ReadLine() at all in this case. Maybe it would be better to read byte by byte until you reach the 'DATA' mark.

Answer 2

So the data is utf8 (the default encoding for StreamReader). This is a multibyte encoding, so IndexOf would be inadvisable. You could:

Encoding.UTF8.GetByteCount(string)

on your data so far, adding 1 or 2 bytes for the missing line ending.

Answer 3

You can find out how many bytes the StreamReader has actually returned (as opposed to read from the stream) in a number of ways, none of them too straightforward I'm afraid.

1. Get the result of textReader.CurrentEncoding.GetByteCount(totalLengthOfAllTextRead) and then seek to this position in the stream.
2. Use some reflection hackery to retrieve the value of the private variable of the StreamReader object that corresponds to the current byte position within the internal buffer (different from that with the stream - usually behind, but no more than equal to of course). Judging by .NET Reflector, the this variable seems to be named bytePos.
3. Don't bother using a StreamReader at all but instead implement your custom ReadLine function built on top of the Stream or BinaryReader even (BinaryReader is guaranteed never to read further ahead than what you request). This custom function must read from the stream char by char, so you'd actually have to use the low-level Decoder object (unless the encoding is ASCII/ANSI, in which case things are a bit simpler due to single-byte encoding).

Option 1 is going to be the least efficient I would imagine (since you're effectively re-encoding text you just decoded), and option 3 the hardest to implement, though perhaps the most elegant. I'd probably recommend against using the ugly reflection hack (option 2), even though it's looks tempting, being the most direct solution and only taking a couple of lines. (To be quite honest, the StreamReader class really ought to expose this variable via a public property, but alas it does not.) So in the end, it's up to you, but either method 1 or 3 should do the job nicely enough...

Hope that helps.

Answer 4

If you're needing to count bytes, I'd go with the BinaryReader. You can take the results and cast them about as needed, but I find its idea of its current position to be more reliable (in that since it reads in binary, its immune to character-set problems).

Answer 5

The line breaks are easily identifiable without needing to decode the stream first (except for some encodings rarely used for text files like EBCDIC, UTF-16, UTF-32), so you can just read each line as bytes and then decode the entire line:

using (FileStream stream = File.OpenRead(path)) {
   List<byte> buffer = new List<byte>();
   bool hasCr = false;
   bool done = false;
   while (!done) {
      int b = stream.ReadByte();
      if (b == -1) throw new IOException("End of file reached in header.");
      if (b == 13) {
         hasCr = true;
      } else if (b == 10 && hasCr) {
         string line = Encoding.UTF8.GetString(buffer.ToArray(), 0, buffer.Count);
         if (line == "DATA") {
            done = true;
         } else {
            HandleHeaderLine(line);
         }
         buffer.Clear();
         hasCr = false;
      } else {
         if (hasCr) buffer.Add(13);
         hasCr = false;
         buffer.Add((byte)b);
      }
   }
   _dataOffset = stream.Position;
}

Instead of closing the stream and open it again, you could of course just keep on reading the data.