Any good tool to convert HTML entities in HTML documents to plain UTF characters?

Question

I have many HTML documents containing many HTML entities of Unicode code point representation, e.g. بروح

Is there a good tool to convert HTML entities in multiple HTML documents to plain UTF-8/UTF-16/UTF-32 characters?

I want an offline converter tool that can do a batch job for this purpose.

Answer 1

I don't know of such a tool, but you could easily write one. This C# code for example would convert all html files in the current folder:

foreach (string name in Directory.GetFiles(".", "*.html")) {
  string s = File.ReadAllText(name);
  s = Regex.Replace(
    s,
    @"&#(\d+);",
    m => ((char)Int32.Parse(m.Groups[1].Value)).ToString()
  );
  File.WriteAllText(name, s);
}

Answer 2

The GNU utility "recode" will do this, with the invocation

recode HTML..UTF-16LE < old.html > new.html

(or UTF-16BE, of course.)

http://ftp.gnu.org/gnu/recode/recode-3.6.tar.gz

It's use of HTML as a character set is a bit of a hack and is treated as either ASCII or LATIN-1, when it should be treated as a "surface" for any character set. If there are any UTF-8 characters, it can break, so I'm now withdrawing my recommendation. Use the first.

(You might expect recode UTF-8..HTML,HTML..UTF-16LE to work, but this first encodes the ampersands...)