.NET Stream Decoders behavior

Question

Hello,

I've got a process which attempts to decode different encodings of strings from a binary stream. I get some behavior which does not quite add up in my mind when I step through it. Specifically, what I do is:

• obtain the maximum number of bytes which would be used to encode a character in the given encoding
• grab the amount of bytes from the stream
• use Encoding.GetCharCount to determine just how many characters might have been encoded in those bytes (could be 0 one or two...)
• if its not zero i use Encoding.GetString to grab the characters out of the byte array
• i then figure out how many bytes were used to encode the extracted characters and advance the stream index by that amount
• if the number of decodable bytes turns out to be zero i advance the index by one byte and try the whole thing again...in this fashion i expect not to miss any decodable characters

BTW, if anyone notices any incorrect assumption made in the above, feel free to say so...

I have my decoders set to throw DedcoderFallbackExceptions when they cannot decode a given set of bytes. What confuses me is that some times the exception arises when I call GetCharCount and other times it occurs when I call GetString. Is there any reason this should be happening? Is this in fact expected? I would like to be able to reliably check for the presence of printable characters in as few places as possible - currently I'm doing it in several places.

Any thoughts?

thanks, brian

BIG UPDATE: It seems that my initial description of the problem is lacking a bit. Let me add a few more premises to the problem:

• the stream could be extremely large - it will not fit in memory for most users
• at any given place in the stream i don't know for sure that I am at the beginning of text, in the middle of text
• at any given place in the stream i don't know if i am in the middle or beginning of a multi byte character
• the stream will contain much material that is in fact not text of any sort, as well as a smattering of different encodings

Hopefully this clarifies some of the issues. Responses so far have been very helpful! Please do continue!

Answer 1

Encodings like UTF8 use variable number of bytes for characters so you can't simply multiply the maximum number of bytes to fetch from the stream. The last byte might fall at the middle of a character (and might be invalid in isolation) or the last character might represent a completely different character on that specific encoding.

Answer 2

Wow. Sounds like mighty overkill. Have you tried using the GetDecoder method of your encoding? It hands you a Decoder with a GetChars method that you feed a byte array and a char array to and it fills the char array with the available characters decoded from the byte array.

If there's any overshoot (i.e. spare bytes) these are saved in the state of the decoder for the next time that you call GetChars with fresh bytes.

You can use a StringBuilder to assemble the result.

A little simpler than your method.

Answer 3

If I understand your question correctly, you trying to read char data from byte stream with unknown encoding?

If my assumption is right, then you need first detect encoding, and read read byte stream using TextReader with this encoding, then you will not need to worry about different char sizes, TextReader will do all work for you.

I know two approaches to detect encoding from byte stream:

1. Ude is a C# port of Mozilla Universal Charset Detector.
2. IE MultiLang services