Why are "control" characters illegal in XML?

Question

There are a variety of characters that are not legally encodeable in XML, e.g. U+0007 ('bell') and U+001B ('escape'). Most of the interesting ones are non-whitespace 'control' characters.

It's clear from (e.g.) this question and others that it's the XML spec that's the issue -- but can anyone illuminate me as to why the XML spec forbids these characters?

It seems like it could have been required that they be encoded in escapes, e.g. as  and  respectively, but perhaps there's a practical reason that the characters were forbidden rather than required to be escaped?

Answerers have suggested that there is some motivation towards avoiding transmission control characters, but Unicode includes many other control-like characters (consider U+200C "zero width non joiner"). I recognize there may be no good reason for this behavior, but I would still like to understand it better.

It's particularly frustrating because when those character values appear in other encodings data formats, I end up "double-escaping" new XML documents that need to encode this.

Answer 1

My understanding is that this range is barred on the grounds that a markup language should not have any need to support transmission and flow control characters and including them would create a problem for any editors and parsers in binary conversion.

I'm struggling to find anything ex cathedra on this from Tim Bray et al though.

edit: some discussion of control chars and a vague admission it wasn't exactly over-engineered

Answer 2

XML was designed specially around Unicode (specifically UTF-8 and UTF-16) and ISO/IEC 10646, both of which (I'm not quite positive about ISO 10646) contain the transmission/flow control characters which were left over from ASCII and the days of character-based terminals. While those characters still have uses, they don't belong in a format like XML.

As for these new encodings that use those codes for something else, well, it seems that the XML spec may need to adapt.

Answer 3

It seems like it could have been required that they be encoded in escapes, e.g. as  and 

You can do exactly that in XML 1.1, for all but \0.

Answer 4

That was a long time ago, but my best recollection was that they have no graphical representation and also no agreed-upon semantics. Picking a couple at random we see U+0006 "Acknowledge" or U+0016 "Synchronous idle"... what do those mean? Unicode doesn't say. Even back when everyone claimed to support ASCII, there was no interoperability around this junk. XML is supposed to be about interoperability.

The experience has been that people who want to use these things really want to jam binary data into their XML elements (and the next thing they want is to include U+0000 NULL), which has been an explicit non-goal of XML since day 1. If you want to represent the numbers 0x6 or 0x16, there are lots of good ways to do that which don't muddy the notion of "character".