UTF-8 Encoding in java, retrieving data from website

Question

I'm trying to get data from website which is encoded in UTF-8 and insert them into the database (MYSQL). Database is also encoded in UTF-8.

This is the method I use to download data from specific site.

public String download(String url) throws java.io.IOException {
        java.io.InputStream s = null;
        java.io.InputStreamReader r = null;
        StringBuilder content = new StringBuilder();
        try {
            s = (java.io.InputStream)new URL(url).getContent();

            r = new java.io.InputStreamReader(s, "UTF-8");

            char[] buffer = new char[4*1024];
            int n = 0;
            while (n >= 0) {
                n = r.read(buffer, 0, buffer.length);
                if (n > 0) {
                    content.append(buffer, 0, n);
                }
            }
        }
        finally {
            if (r != null) r.close();
            if (s != null) s.close(); 
        }
        return content.toString();
    }

If encoding is set to 'UTF-8' (r = new java.io.InputStreamReader(s, "UTF-8"); ) data inserted into database seems to look OK, but when I try to display it, I am getting something like this: C�te d'Ivoire, instead of Côte d'Ivoire.

All my websites are encoded in UTF-8.

Please help.

If encoding is set to 'windows-1252' (r = new java.io.InputStreamReader(s, "windows-1252"); ) everything works fine and I am getting Côte d'Ivoire on my website (), but in java this title looks like 'C?´te d'Ivoire' what breaks other things, such as for example links. What does it mean ?

Answer 1

Java

The problem seems to lie in the HttpServletResponse , if you have a servlet or jsp page. Make sure to set your HttpServletResponse encoding to UTF-8.

In a jsp page or in the doGet or doPost of a servlet, before any content is sent to the response, just do :

response.setCharacterEncoding("UTF-8");

PHP

In PHP, try to use the utf8-encode function after retrieving from the database.

Answer 2

I would consider using commons-io, they have a function doing what you want to do:link

That is replace your code with this:

public String download(String url) throws java.io.IOException {
    java.io.InputStream s = null;
    String content = null;
    try {
        s = (java.io.InputStream)new URL(url).getContent();
        content = IOUtils.toString(s, "UTF-8")

    }
    finally {
        if (s != null) s.close(); 
    }
    return content.toString();
}

if that nots doing start looking into if you can store it to file correctly to eliminate the possibility that your db isn't set up correctly.

Answer 3

Is your database encoding set to UTF-8 for both server, client, connection and have the tables been created with that encoding? Check 'show variables' and 'show create table <one-of-the-tables>'

Answer 4

If encoding is set to 'UTF-8' (r = new java.io.InputStreamReader(s, "UTF-8"); ) data inserted into database seems to look OK, but when I try to display it, I am getting something like this: C�te d'Ivoire, instead of Côte d'Ivoire.

Thus, the encoding during the display is wrong. How are you displaying it? As per the comments, it's a PHP page? If so, then you need to take two things into account:

1. Write them to HTTP response output using the same encoding, thus UTF-8.
2. Set content type to UTF-8 so that the webbrowser knows which encoding to use to display text.

As per the comments, you have apparently already done 2. Left behind 1, in PHP you need to install mb_string and set mbstring.http_output to UTF-8 as well. I have found this cheatsheet very useful.