File is not saved in UTF-8 encoding even when I set encoding to UTF-8

Question

When I check my file with Notepad++ it's in ANSI encoding. What I am doing wrong here?

OutputStreamWriter out = new OutputStreamWriter(new FileOutputStream(file), "UTF8");
try
     {   
      out.write(text);
      out.flush();
     } finally
     {
      out.close();
     }

UPDATE:

This is solved now, reason for jboss not understanding my xml wasn't encoding, but it was naming of my xml. Thanx all for help, even there really wasn't any problem...

Answer 1

The IANA registered type is "UTF-8", not "UTF8". However, Java should throw an exception for invalid encodings, so that's probably not the problem.

I suspect that Notepad is the problem. Examine the text using a hexdump program, and you should see it properly encoded.

Answer 2

If there is no BOM (and Java doesn't output one for UTF8, it doesn't even recognize it), the text is identical in ANSI and UTF8 encoding as long as only characters in the ASCII range are being used. Therefore Notepad++ cannot detect any difference.

(And there seems to be an issue with UTF8 in Java anyways...)

Answer 3

UTF-8 is designed to be, in the common case, rather indistinguishable from ANSI. So when you write text to a file and encode the text with UTF-8, in the common case, it looks like ANSI to anyone else who opens the file.

• UTF-8 is 1-byte-per-character for all ASCII characters, just like ANSI.
• UTF-8 has all the same bytes for the ASCII characters as ANSI does.
• UTF-8 does not have any special header characters, just as ANSI does not.

It's only when you start to get into the non-ASCII codepoints that things start looking different.

But in the common case, byte-for-byte, ANSI and UTF-8 are identical.

Answer 4

If you're creating an XML file (as your comments imply), I would strongly recommend that you use the XML libraries to output this and write the correct XML encoding header. Otherwise your character encoding won't conform to XML standards and other tools (like your JBoss instance) will rightfully complain.

    // Prepare the DOM document for writing
    Source source = new DOMSource(doc);

    // Prepare the output file
    File file = new File(filename);
    Result result = new StreamResult(file);

    // Write the DOM document to the file
    Transformer xformer = TransformerFactory.newInstance().newTransformer();
    xformer.transform(source, result);

Answer 5

There's no such thing as plain text. The problem is that an application is decoding character data without you telling it which encoding the data uses.

Although many Microsoft apps rely on the presence of a Byte Order Mark to indicate a Unicode file, this is by no means standard. The Unicode BOM FAQ says more.

You can add a BOM to your output by writing the character '\uFEFF' at the start of the stream. More info here. This should be enough for applications that rely on BOMs.

Answer 6

Did you try to write a BOM at the beginning of the file? BOM is the only thing that can tell the editor the file is in UTF-8. Otherwise, the UTF-8 file can just look like Latin-1 or extended ANSI.

You can do it like this,

public final static byte[] UTF8_BOM = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
...
OutputStream os = new FileOutputStream(file);
os.write(UTF8_BOM);
os.flush();
OutputStreamWriter out = new OutputStreamWriter(os, "UTF8");
try
    {                       
            out.write(text);
            out.flush();
    } finally
    {
            out.close();
    }