Why does anyone use an encoding other than UTF-8?

Question

I want to know why any developer would need to use an encoding other than UTF-8.

Answer 1

Sometimes they are restricted due to historical/unsupported reasons (I'm developing on Windows using Zend Studio on a Samba share on a Linux box: and something in that mix means I keep reverting to Cp1512 instead of UTF8).

Sometimes you don't need to use UTF-8 (for example when storing a md5 hash in a database: you only need the hexadecimal range 0-9 A-F: why make it a UTF-8 field, which will take at least a byte extra storage instead of normal ASCII).

Sometimes it's just laziness learning the UTF-8 functions for a particular language.

Answer 2

One legitimate reason is when you need to deal with legacy documents, software or hardware that are not Unicode compatible.

Another legitimate reason is that you need to use a programming language / libraries that do not support UTF8 / Unicode well ... or at all.

Other answers mention that UTF-16 is more compact than UTF-8 for Asian languages / characters.

And of course there are reasons like short-sightedness, ignorance, laziness ... and deadlines.

Answer 3

Wikipedia lists advantages and disadvantages of UTF-8 as compared to a variety of other encodings:

http://en.wikipedia.org/wiki/UTF-8#Advantages_and_disadvantages

The most important disadvantages are IMHO that UTF-8 might use significantly more space especially in Asian languages such as Chinese, Japanese or Hindi and that not all code points have the same size which makes measurements more difficult and many string operations such as search inefficient.

Answer 4

Because you sometimes want to operate easily on codepoints -- then you'd choose f.e. UCS-2 or UCS-4.

Answer 5

In UTF-8 code points between 0800 and FFFF take up three bytes in UTF-8 but only two in UTF-16. See the wikipedia comparison for more details, but basically if text heavily uses code points in this range (say, if it's Chinese), UTF-8 files will be larger than UTF-16 files with the same content.

Answer 6

Its also worth remembering that in some circumstances (where a non-latin set of characters are needed) UTF-8 can actually bloat larger than the 16 bit Unicode encoding. In those cases ucs-2 or utf-16 would be a better choice.

Answer 7

Unicode certainly is a good place to work from in most cases, but a developer should be familiar with many different types of character encoding. Certainly ASCII might be used if the set of characters is limited.

What if you're a developer and receiving data from a source that doesn't send UTF-8? There could be lots of interface issues if you don't understand your input.

Joel's article on the must-knows for character encoding is good and worth reading.

Answer 8

UTF-8 is very efficient at encoding plain English text (same as ASCII). If your user base is likely to be mostly, say, Chinese, you will be much better off using UTF-16.

For more information, see The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets.

Answer 9

Many APIs require other Unicode encodings - mostly UTF-16. For instance, Java, .NET, Win32.

Answer 10

Well, some do it because their tools are archaic or flawed. Some do it because they don't see a need to support anything other than ASCII. Some do it because they don't know any better.

Those are the usual excuses for not using Unicode.

As for not using UTF-8 specifically there are different reasons. Some systems, like Windows¹ (and stemming from that, .NET) and Java came to be in a time where Unicode was a strict 16-bit code. Therefore, there was really only one encoding: UCS-2, encoding code points directly as 16-bit words.

Later Unicode was expanded to 21 bits because 65536 code points weren't enough anymore. This caused encodings such as UTF-32 and UTF-16 to appear. For systems previously working with UCS-2 the transition to UTF-16 was the easiest and most sensible choice. Windows did that transition back in Ye Olde Days of Windows 2000.

So while I think that nearly all application nowadays should support Unicode I don't think it is entirely necessary for them to specifically use UTF-8. There are historic reasons for that and no real benefit in converting existing systems from UTF-16 to UTF-8.

---

¹ NT.

Answer 11

Mostly for historical reasons.

Answer 12

At my previous employer we used iso-8859-1 for some of our ASP pages to match the collation of our SQL Server, which as you can guess was not Unicode. I wanted to change the collation, but the manager said to wait till we upgrade our SQL Server to do it. Needless to say it never happened - I haven't been with them for a little over a year now, so I don't know if they finally did it.

Answer 13

In Western Europe, the ISO-8859-1 (a.k.a. "Latin1") encoding is quite common, it was used in the DOS and early Windows days, and many places (databases, service calls) you will still find this encoding at times. So when interfacing with such a legacy system, you're likely to encounter that encoding.

Not that I would recommend it's use - UTF-8 is just so much easier to use and causes so much less trouble and friction.

Marc

Answer 14

Because outside the English-speaking world, people have been using various encodings that predate Unicode and are tailored for their respective languages for decades. These language-specific encodings have become ingrained everywhere and are pretty much a standard. If you want to have any hope of interfacing with legacy systems, you have to use them, so all systems have to support them and usually use them as default even if they by now support UTF-8 as well. There may even be multiple legacy encodings traditionally used for different purposes.

Examples:

• ISO-8859-1 in western Europe - actually outdated there as well, as you need ISO-8859-15 for the Euro sign
• ISO-2022-JP in Japan for emails, Shift JIS for websites
• Big5 in Taiwan
• GB2312 in China

The last two examples show that encodings can even be a political issue.

Answer 15

http://www.personal.psu.edu/ejp10/blogs/gotunicode/2007/02/cjk-unicode-angst-in-japan-and.html has a good summary + links about the difficulty Japanese users have with Unicode.

http://www.jbrowse.com/text/unij.html

http://www.hastingsresearch.com/net/04-unicode-limitations.shtml

http://www.mojikyo.org/html/abroad/abroad%5Ftop.html

Apparently Unicode is moving away from unification due to such complaints.

Answer 16

The reasons for using non-Unicode 8-bit character sets / encodings are all back compatibility of some kind, and/or inertia. For that matter, the most frequent reasons for using UTF-8 are compatibility with standards like XML that mandate or prefer UTF-8.

Differences in the number of bytes you think text will take up in different encodings, especially in storage, are mostly theoretical. In real world situations, compatibility requirements are more important. If compression is used, the size differences go away anyway. Even if compression is not used, total text size is hard to predict and is rarely a deciding factor.

When converting legacy code that used non-Unicode 8-bit encodings, using UTF-16 can be a tool for making sure all code has been converted, because mismatches can be caught as compile-time type errors. Many languages, runtimes and libraries like Javascript, JVM, .NET, ICU use 16-bit strings and UTF-16, even though storage and Internet protocols are usually 8-bit.

Answer 17

Related to the subject, when using MySQL, as if it wasn't complex enough, you get the option the choose which kind of UTF-8 collation you want to use. So what would you use? "UTF-8 general ci" or "UTF-8 unicode ci"? (I tend to use the UTF-8 variant that is used for the database connection)