ANTLR: Unicode Character Scanning

Question

Problem: Can't get Unicode character to print correctly.

Here is my grammar:

options { k=1; filter=true;
 // Allow any char but \uFFFF (16 bit -1)
charVocabulary='\u0000'..'\uFFFE'; 
}

ANYCHAR :'$'
|    '_' { System.out.println("Found underscore: "+getText()); }
|    'a'..'z' { System.out.println("Found alpha: "+getText()); }
|    '\u0080'..'\ufffe' { System.out.println("Found unicode: "+getText()); }
; 

Code snippet of main method invoking the lexer:

public static void main(String[] args) {
SimpleLexer simpleLexer = new SimpleLexer(System.in);
while(true) {
try {
Token t = simpleLexer.nextToken();
System.out.println("Token : "+t);

} catch(Exception e) {}

}
}

For input "ठ", I'm getting the following output :

Found unicode: 
Token : ["à",<5>,line=1,col=7]
Found unicode: 
Token : ["¤",<5>,line=1,col=8]
Found unicode:  
Token : [" ",<5>,line=1,col=9]

It appears that the lexer is treating Unicode char "ठ" as three separate character. My aim is to scan and print "ठ".

Answer 1

Your problem is not in the ANTLR generated lexer, but in the Java stream you pass to it. The stream reads bytes only (doesn't interpret them in an encoding), and what you see is an UTF-8 sequence.

If its ANTLR 3, you can use the ANTLRInputStream constructor that takes an ancoding as a parameter:

ANTLRInputStream (InputStream input, String encoding) throws IOException