What's the matter with this Grammar?

Question

grammar Test;

IDHEAD: ('a'..'z' | 'A'..'Z' | '_');
IDTAIL: (IDHEAD | '0'..'9');
ID:     (IDHEAD IDTAIL*);
fragment
TYPE:   ('text' | 'number' | 'bool');

define: 'define' ID 'as' TYPE;

The problem is that the define rule matches the tokens define, ID, as, but wont match TYPE. I'm yielding a MissingTokenException.

If I inline the TYPE, as follows, it works as I'm intending:

grammar Test;

IDHEAD: ('a'..'z' | 'A'..'Z' | '_');
IDTAIL: (IDHEAD | '0'..'9');
ID:     (IDHEAD IDTAIL*);
fragment
TYPE:   ('text' | 'number' | 'bool');

define: 'define' ID 'as' ('text' | 'number' | 'bool');

---

Update: The fragment keyword was added in an effort to resolve another conflict: The following token definitions can never be matched because prior tokens match the same input: TYPE.

Answer 1

Isn't it because TYPE is defined as fragment?

Can't test right now, but try and remove fragment, and that should do the trick, plus give you a token to boot.

Answer 2

What is the line with fragment supposed to be doing? I think it should work as you expect if you remove it.

Answer 3

I thought the lexer rules have a priority of how they are listed. So if you want token TYPE to actually be created, move it above all the other lexer rules.

grammar Test;

fragment
TYPE:   ('text' | 'number' | 'bool');
IDHEAD: ('a'..'z' | 'A'..'Z' | '_');
IDTAIL: (IDHEAD | '0'..'9');
ID:     (IDHEAD IDTAIL*);

define: 'define' ID 'as' TYPE;

Answer 4

In the combined grammar, place your parser rules above your lexer rules. Also, remember that the lexer runs first and only after it's finished the parser runs. The TYPE (lexer) token must be matched before it's known that the define (parser) rule needs it.

Fragment lexer rules do not create tokens, but they can be composed into non-fragment rules that do create tokens. In your example, IDHEAD and IDTAIL are not tokens - they are just used to describe the parts of ID. As such, TYPE and ID are your non-fragment rules, and IDHEAD and IDTAIL are fragment rules.

grammar Test;

define: 'define' ID 'as' TYPE;

/*
 * Lexer rules only below here
 */

TYPE:   ('text' | 'number' | 'bool');
ID:     (IDHEAD IDTAIL*);

fragment
IDHEAD: ('a'..'z' | 'A'..'Z' | '_');

fragment
IDTAIL: (IDHEAD | '0'..'9');