An antlr problem with embedded comments

Question

Hi,

I am trying to implement a nested comment in D.

nestingBlockComment
 : '/+' (options {greedy=false;} :nestingBlockCommentCharacters)* '+/' {$channel=HIDDEN;}; // line 58

  nestingBlockCommentCharacters
   :  (nestingBlockComment| '/'~'+' | ~'/' ) ; //line 61

For me, it would be logical that this should work...

This is the error message I get:
[21:06:34] warning(200): d.g:58:64: Decision can match input such as "'+/'" using multiple alternatives: 1, 2
As a result, alternative(s) 1 were disabled for that input
[21:06:34] warning(200): d.g:61:7: Decision can match input such as "'/+'" using multiple alternatives: 1, 3
As a result, alternative(s) 3 were disabled for that input

Could someone explan those error messages to me and the fix?

Thanks.

Answer 1

AFAIK, the error is because nestingBlockCommentCharacters can match +/ (the ~'/' twice).

Personally, I'd keep the nestingBlockComment as a lexer rule instead of a parser rule. You can do that by adding a little helper method in the lexer class:

public boolean openOrCloseCommentAhead() {
  // return true iff '/+' or '+/' is ahead in the character stream
}

and then in a lexer comment-rule, use a gated semantic predicates with that helper method as the boolean expression inside the predicate:

// match nested comments
Comment
  :  '/+' (Comment | {!openOrCloseCommentAhead()}?=> Any)* '+/'
  ;

// match any character
Any
  :  .
  ;

A little demo-grammar:

grammar DComments;

@lexer::members {
  public boolean openOrCloseCommentAhead() {
    return (input.LA(1) == '+' && input.LA(2) == '/') ||
           (input.LA(1) == '/' && input.LA(2) == '+');
  }
}

parse
  :  token+ EOF
  ;

token
  :  Comment {System.out.println("comment :: "+$Comment.text);}
  |  Any
  ;

Comment
  :  '/+' (Comment | {!openOrCloseCommentAhead()}?=> Any)* '+/'
  ;

Any
  :  .
  ;

and a main class to test it:

import org.antlr.runtime.*;

public class Main {
    public static void main(String[] args) throws Exception {
        ANTLRStringStream in = new ANTLRStringStream(
            "foo /+ comment /+ and +/ comment +/ bar /+ comment +/ baz");
        DCommentsLexer lexer = new DCommentsLexer(in);
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        DCommentsParser parser = new DCommentsParser(tokens);
        parser.parse();
    }
}

Then the following commands:

java -cp antlr-3.2.jar org.antlr.Tool DComments.g 
javac -cp antlr-3.2.jar *.java
java -cp .:antlr-3.2.jar Main

(for Windows, the last command is: java -cp .;antlr-3.2.jar Main)

produce the following output:

comment :: /+ comment /+ and +/ comment +/
comment :: /+ comment +/