Sane(r) way to get character-encoding of the CLI in Mac OS X?

Question

Hi all!

I was writing a CLI-Tool for Mac OS X (10.5+) that has to deal with command-line arguments which are very likely to contain non-ASCII characters.

For further processing, I convert these arguments using +[NSString stringWithCString:encoding:].

My problem is, that I couldn't find good information on how to determine the character-encoding used by the shell in which said cli-tool is running in.
What I came up with as a solution is the following:

NSDictionary *environment = [[NSProcessInfo processInfo] environment];
NSString *ianaName = [[environment objectForKey:@"LANG"] pathExtension];
NSStringEncoding encoding = CFStringConvertEncodingToNSStringEncoding(
  CFStringConvertIANACharSetNameToEncoding( (CFStringRef)ianaName ) );

NSString *someArgument = [NSString stringWithCString:argv[someIndex] encoding:encoding];

I find that a little crude, however -- which makes me think that I missed out something obvious...but what?

Is there a saner/cleaner way of achieving essentially the same?

Thanks in advance

D

Answer 1

The answer depends on what the non-asciiness comes from.

1. In OS X, the environment variable LANG does not reflect the choice of language in the GUI. Very few people will set LANG at the command line.
2. The choice of the "system encoding" at the GUI is stored in ~/.CFUserTextEncoding, and can be obtained by CFStringGetSystemEncoding, see this Apple doc.
3. That said, this "system encoding" is rarely used except in a very old, non-unicode aware softwares. Any sane Cocoa program uses just Unicode and nothing else.

4. In particular, the file path at the level of Cocoa is always encoded in (a variant of) UTF-8. So, to get an NSString from a C string, use

    NSString*string=[NSString stirngWithCString:cString encoding:NSUTF8Encoding];
   

   and to get a C-string for the file path from an NSString, use

    char*path=[string fileSystemRepresentation];
   

   Here it is recommended not to use just [string UTF8String], due to the subtlety, see this Apple doc.

5. So, I recommend you not to care about the encoding and just assume UTF-8.

6. That said, there might be a very small number of people who sets LANG on the command line, and you might want to take care of them. Then, what you did is the only thing I can come up with.

Answer 2

Okay, it turns out there seems to be none!

As Yuji pointed out, the underlying encoding of filenames is UTF-8, no matter what. Therefore, one needed to handle two scenarios:

1. Arguments that are typed in, character for character, by the user.
2. Arguments that are tab-completed or the output of commands like ls, as they do not convert any characters.

The second case is simply covered by the assumption of UTF-8.

The first case, however, is problematic:

• On Mac OS 10.6, $LANG contains the IANA-name of the used encoding like de_DE.IANA_NAME.
• Prior to Snow Leopard, this is not the case for charsets other than UTF-8!

I didn't test each and every charset I could think of, but none of the european ones were included. Instead, $LANG only was the language-locale (de_DE in my case)!

Since the results of calling +[NSString stringWithCString:encoding:] with an incorrect encoding are undefined, you cannot safely assume that it will return nil in that case* (if eg. it's ASCII-only, it might work perfectly fine!).

What adds to the overall mess is that $LANG is not guarateed to be around, anyway: There's a checkbox in Terminal.app's preferences, that enables a user to not set $LANG at all (not to speak of X11.app which doesn't seem to handle any non-ASCII input...).

So what's left:

1. Check for presence of $LANG. If it's not set, Goto:4!
2. Check if $LANG contains information on the encoding. If it doesn't, Goto:4!
3. Check if the encoding you find there is UTF-8. If it is Goto:6, else...
4. If argc is greater than 2 and [[NSString stringWithCString: argv[0] encoding: NSUTF8StringEncoding] isEqualToString: yourForceUTFArgumentFlag], print that you are forcing UTF-8 now and Goto 6. If not:
5. Assume you don't know anything, issue a warning that your user should set the Terminal encoding to UTF-8 and may consider passing yourForceUTFArgumentFlag as the first argument and exit().
6. Assume UTF-8 and do what you have to...

Sounds shitty? That's because it is, but I can't think of any saner way of doing it.

---

One further note though: If you are using UTF-8 as an encoding, stringWithCString:encoding: returns nil whenever it encounters non-ASCII characters in a C-String that is not encoded in UTF-8.)