Base64 String throwing invalid character error.

Question

I keep getting a Base64 invalid character error even though I shouldn't.

The program takes an XML file and exports it to a document. If the user wants, it will compress the file as well. The compression works fine and returns a Base64 String which is encoded into UTF-8 and written to a file.

When its time to reload the document into the program I have to check whether its compressed or not, the code is simply:

byte[] gzBuffer = System.Convert.FromBase64String(text);
return "1F-8B-08" == BitConverter.ToString(new List<Byte>(gzBuffer).GetRange(4, 3).ToArray());

It checks the beginning of the string to see if it has GZips code in it.

Now the thing is, all my tests work. I take a string, compress it, decompress it, and compare it to the original. The problem is when I get the string returned from an ADO Recordset. The string is exactly what was written to the file (with the addition of a "\0" at the end, but I don't think that even does anything, even trimmed off it still throws). I even copy and pasted the entire string into a test method and compress/decompress that. Works fine.

The tests will pass but the code will fail using the exact same string? The only difference is instead of just declaring a regular string and passing it in I'm getting one returned from a recordset.

Any ideas on what am I doing wrong?

Answer 1

You say

The string is exactly what was written to the file (with the addition of a "\0" at the end, but I don't think that even does anything).

In fact, it does do something (it causes your code to throw a FormatException:"Invalid character in a Base-64 string") because the Convert.FromBase64String does not consider "\0" to be a valid Base64 character.

  byte[] data1 = Convert.FromBase64String("AAAA\0"); // Throws exception
  byte[] data2 = Convert.FromBase64String("AAAA");   // Works

Solution: Get rid of the zero termination. (Maybe call .Trim("\0"))

Notes:

The MSDN docs for Convert.FromBase64String say it will throw a FormatException when

The length of s, ignoring white space characters, is not zero or a multiple of 4.

-or-

The format of s is invalid. s contains a non-base 64 character, more than two padding characters, or a non-white space character among the padding characters.

and that

The base 64 digits in ascending order from zero are the uppercase characters 'A' to 'Z', lowercase characters 'a' to 'z', numerals '0' to '9', and the symbols '+' and '/'.

Answer 2

Whether null char is allowed or not really depends on base64 codec in question. Given vagueness of Base64 standard (there is no authoritative exact specification), many implementations would just ignore it as white space. And then others can flag it as a problem. And buggiest ones wouldn't notice and would happily try decoding it... :-/

But it sounds c# implementation does not like it (which is one valid approach) so if removing it helps, that should be done.

One minor additional comment: UTF-8 is not a requirement, ISO-8859-x aka Latin-x, and 7-bit Ascii would work as well. This because Base64 was specifically designed to only use 7-bit subset which works with all 7-bit ascii compatible encodings.

Answer 3

Daniel is correct in that you can just remove the \0, but I would say that you should attempt to determine why the null character is showing up in the first place. In C strings are a collection of characters with a '\0' at the end to specify the end of the string. C# does not use the '\0' to specify the end of the string, and can contain the null character.

Somewhere in the processing of your data the extra character is getting added. I'm sure you stored and read plenty of strings to a database just as I have, but I've never seen the string become null terminated after doing so. What database engine are you using?

Answer 4

If removing \0 from the end of string is impossible, you can add your own character for each string you encode, and remove it on decode.