parse CSV files that contain unicode character using OpenCSV

Question

hi everybody, i parse a .csv file with OpenCSV in netbeans 6.0.1. my file contains some unicode character and when i write it in output the character appears in other form(like (HJ1'-E/;) when i open this file in notepad, it is ok.

the code that i used:

CSVReader reader=new CSVReader(new FileReader("d:\\a.csv"),',','\'',1);
    String[] line;
    while((line=reader.readNext())!=null){
        StringBuilder stb=new StringBuilder(400);
        for(int i=0;i<line.length;i++){
            stb.append(line[i]);
            stb.append(";");
        }
        System.out.println( stb);
    }

Thanks very much in advance.

Answer 1

First you need to know what encoding your file is in, such as UTF-8 or UTF-16. What's generating this file to start with?

After that, it's relatively straightforward - you need to create a FileInputStream wrapped in an InputStreamReader instead of just a FileReader. (FileReader always uses the default encoding for the system.) Specify the encoding to use when you create the InputStreamReader, and if you've picked the right one, everything should start working.

Note that you don't need to use OpenCSV to check this - you could just read the text of the file yourself and print it all out. I'm not sure I'd trust System.out to be able to handle non-ASCII characters though - you may want to find a different way of examining strings, such as printing out the individual values of characters as integers (preferably in hex) and then comparing them with the charts at unicode.org. On the other hand, you could try the right encoding and see what happens to start with...

EDIT: Okay, so if you're using UTF-8:

CSVReader reader=new CSVReader(
    new InputStreamReader(new FileInputStream("d:\\a.csv"), "UTF-8"), 
    ',', '\'', 1);
String[] line;
while ((line = reader.readNext()) != null) {
    StringBuilder stb = new StringBuilder(400);
    for (int i = 0; i < line.length; i++) {
         stb.append(line[i]);
         stb.append(";");
    }
    System.out.println(stb);
}

(I hope you have a try/finally block to close the file in your real code.)