How to get the token type from a CFStringTokenizer in Cocoa?

Question

The CFStringTokenizer documentation has two conflicting statements in CFStringTokenizerAdvanceToNextToken():

CFStringTokenizerAdvanceToNextToken

...

Return Value

The type of the token if the tokenizer succeeded in finding a token and setting it as current token. Returns kCFStringTokenizerTokenNone if the tokenizer failed to find a token. For possible values, see “Token Types.”

...

If a token is found, it is set as the current token and the function returns true; otherwise the current token is invalidates and the function returns false.

The first paragraph (returning a token type) is what I'd like to see: it lets you, for example, check if a token is made up entirely of non-alphanumeric characters. However, the second paragraph (returning true or false) seems to be what is actually happening.

Any ideas why that would be, or how to work around it?

Answer 1

The header comment doesn't mention returning true or false, and when the header and the online docs disagree it's often the header that is correct.

In a simple test I'm able to get return values other than 0 and 1, so the problem you are seeing may be more specific; can you post sample code that's failing?

Answer 2

It's possible they meant “true” and “false” in the broader sense of “non-zero” and “zero”. If it finds a token, the function returns kCFStringTokenizerTokenNormal (which is 1) or some combination of the masks (either way, non-zero/“true”). If it doesn't, the function returns kCFStringTokenizerTokenNone (which is zero/“false”).

It's certainly vague language, though, so please file a documentation bug on the Apple Bug Reporter.