Hello! I'm currently working on an application to validate and parse CSV-files. The CSV files have to be encoded in UTF-8, although sometimes we get files in a false encoding. The CSV-files most likely contain special characters of the German alphabet (Ä, Ö, Ü, ß) as most of the texts within the CSV file are in German language.
For the part of the validator, i need to make sure, the file is UTF-8 encoded. As long as there are no special characters present, there is most likely no problem with parsing.
What i have tried so far is to read the file as bytes and use some libraries to detect (or guess) the encoding. I tried most of possibilities of this blog post: http://fredeaker.blogspot.com/2007/01/character-encoding-detection.html
But all libraries I tried returned not the correct encoding and therefore I couldn't parse the special characters.
Now to my question: Is there a way to determine for a given Character Encoding like UTF-8 to detect characters that are not encoded correctly? So basically the characters that are displayed in the (Eclipse) console as quesion marks.
Or is there any other way to correctly determine the character encoding? I just need to know if it's UTF-8 or not.
Thank you all in advance for your help! :)
Best Regards, Robert