Hidden token into default channel - AntlrV3

Question

Suppose I'm having white spaces (WS) in the hidden channel. And for a particular rule alone, I want white spaces also to be considered, is it possible to bring WS to the default channel for that particular rule alone in the parser?

Answer 1

Have look at the answer for your path question, notice how I put a '\n' into the parser rule. You should be able to put ' ' as well. Now, do all the options for your WS on the hidden channel need to be in the rule would be the only concern.

eg

rulename : Token1 ' ' Token2 ' ' Token1 {place action here};

Please note that the rule name starts with a lowercase letter and it is a parser rule while the "Token#" start with uppercase letter and are lexer rules. In between the different tokens the rule requires a space in this example, and I suppose you could put something like (' '|'\t'|'\r'|'\n')+ but I have not tried this and will leave that for you to attempt.

Answer 2

You can always query the hidden token stream

ie in C++

myrule: MYTOK { static_cast<antlr::CommonHiddenStreamToken*>(LT(1).get())->getHiddenAfter()->getType() == WS}? MYTOK 

The semantic predicate will check to see if there is a whitespace token after matching the lexical token MYTOK

Answer 3

Lexer rules are evaluated in the order they are listed in your grammar file.

This means you can have something like this:

STRING_LITERAL: '"' NONCONTROL_CHAR* '"';   


fragment NONCONTROL_CHAR: LETTER | DIGIT | UNDERSCORE |  SPACE | BACKSLASH | MINUS | COMMA;
fragment LETTER: LOWER | UPPER;
fragment LOWER: 'a'..'z';
fragment UPPER: 'A'..'Z';
fragment DIGIT: '0'..'9';
fragment SPACE: ' ' | '\t';
fragment UNDERSCORE: '_';   
fragment MINUS:  '-';
fragment BACKSLASH: '\\';

COMMA: ',';     

NEWLINE: ('\r'? '\n')+ { $channel = HIDDEN; };
TERMINATOR  : ';';


WHITESPACE: SPACE+ { $channel = HIDDEN; };

LINE_COMMENT
    :   
    '//' ~('\n'|'\r')*  ('\r\n' | '\r' | '\n') 
    {
        $channel = HIDDEN;
    }
    |   
    '//' ~('\n'|'\r')*     
    {
        $channel = HIDDEN;
    }
    ;   

As you can see a string literal can have space or tabs in it. However a stand alone space or tab will be sent to the HIDDEN channel.