ANTLR lexing getting confused over '...' and floats

Question

I think the ANTLR lexer is treating my attempt at a range expression "1...3" as a float. The expression "x={1...3}" is coming out of the lexer as "x={.3}" when I used the following token definitions:

FLOAT
    :   ('0'..'9')+   ('.' '0'..'9'+)? EXPONENT?
    |   ('.' '0'..'9')+ EXPONENT?
    ;

AUTO : '...';

When I change FLOAT to just check for integers, as so:

FLOAT   :    ('0'..'9')+;

then the expression "x={1...3}" is tokenized correctly. Can anyone help me to fix this?

Thanks!

Answer 1

For anyone wanting to do this...

http://www.antlr.org/wiki/display/ANTLR3/Lexer+grammar+for+floating+point%2C+dot%2C+range%2C+time+specs

I can just change the language syntax to replace the "..." with a "to" keyword.

Answer 2

I think the lexer is putting your first period into the FLOAT token and then the remain two periods do not make your AUTO token. You will need a predicate to determine if the period should be part of a float or auto token.

So why are you using three periods instead of two, must languages use two periods for a "range" and the language should determine if the period is part of a float or the range based on the following "character".

You probably need to be looking into the Defiitive ANTLR Reference on how to build your predicate for the different rules.

Hope this helps you find the correct way to complete the task.

Answer 3

WayneH hits on your problem. You've allowed floats in the format ".3" (without a leading 0). So, the lexer identifies the last . and the 3 and considers it a floating point number. As a result it doesn't see three dots. It sees two dots and a float.

It's very common for languages to disallow this format for floats and require that there be at least one digit (even if it's a 0) to the left of the decimal. I believe that change to your grammar would fix your problem.

There probably is a way to fix it with a predicate, but I've not yet spent enough time with ANTLR to see an obvious way to do so.