How to read non-english texts in java? They are represented in wrong encoding.

Question

I use apache HttpClient. And when I'm trying to "read site", all non-english content is represented wrongly.

Actually, it's represented in windows-1252 but it should be in UTF-8. How can I fix this?

I tried to use InputStreamReader (inputStream, Charset.forName ("UTF-8")), but it didn't help (wrong symbols transformed into ????????).

Answer 1

Finding the correct character encoding can be a bit of a nightmare. Depending on what the content of your site is, the following might be useful. One thing I've done in the past is rely on a class that will use multiple methods for determining the correct character encoding:

The XmlReader from the rome project will use the UTF byte order mark and/or XML declarations to determine the correct encoding.

So you could use the following construct:

new BufferedReader(new XmlReader(inputStream))

to get to the content.

Answer 2

If the file is in Windows-1252, then telling it to use UTF-8 isn't going to work. Give it Windows-1252 as the charset name, and then you can read the correct data. Knowing what format data should be in isn't nearly as useful as knowing what format it's actually in :)

It's up to you whether you then rewrite it in UTF-8...

Answer 3

If the page has encoding in "Content-Type" header, HttpClient will honor it. If not, it will assume Latin-1, not Windows-1252. Are you sure you are getting Windows-1252? You can check encoding like this,

String encoding = method.getResponseCharSet();

If you know the response indeed uses UTF-8 but the header didn't specify it, you can force it to read UTF-8 like this,

byte[] body = method.getResponseBody();
String response = new String(body, "UTF-8");