Does anyone know of a way to debug tree grammars in ANTLRWorks

Question

The recommended pattern for ANTLR usage is to have the Parser construct an Abstract Syntax Tree, and then build Tree walkers (AKA tree grammars) to process them.

I'm trying to get to the bottom of why my tree grammar isn't working and would love to use ANTLRWorks' debugger the same way I used it for the parser itself. The input to the parser is the "source code", but the input to a tree parser is the AST result of the parser. I don't see how to make that available as input to test the tree grammar.

It's not clear that there is a way to test a tree grammar in ANTLRWorks. If it can be done, a pointer in the right direction would really be appreciated.

Answer 1

If you're sure that the AST you're building is fine (with ANTLRWORKS debugger), the tree-walking testing is not different than testing any other app. If you're emitting Java Code for example, use Eclipse's debugger to test it, or plain log messages...

Answer 2

The ANTLRWorks debugger should work fine with your tree grammar. If I recall correctly, you need to use the ANTLR code generation tool with the "-debug" flag (I'm using the Java target), then, where you create your tree parser instance, use the debug constructor that takes a port as an argument. In my case, the default port didn't work, so I arbitrarily picked 35505.

Fire up ANTLRWorks, open your tree grammar, click "Run"->"Debug Remote...", set the port to the same value used in the constructor for your tree parser, and you should be able to connect the debugger to your running application. See the ANTLR 3 Debugging FAQ for details.

[Update] Assuming you're using the Java target (let us know if that's not the case), here's more detailed information on getting started:

When you're testing your non-tree parser in ANTLRWorks, there's a behind-the-scenes process that generates Java code from your grammar file, then uses that code to parse your input. When you use your parser in your own application, you have to use ANTLR (specifically, the class org.antlr.Tool) to generate Java code that you can then include in your application. ANTLRWorks has a menu option for this, which should get you started. In my case, I have a target in my ant build file that generates the Java code from my grammars and puts those Java source files in a place where the rest of my application can find them. My ant target looks something like this:

<java classpath="${antlr.tool.classpath}" classname="org.antlr.Tool" failonerror="true">
    <arg value="-o" />
    <arg value="${antlr.out.dir}" />
    <arg value="${grammar.dir}/GrammarName.g" />
</java>

The property antlr.tool.classpath needs to contain stringtemplate.jar and antlr.jar, and antlr.out.dir needs to point to the directory where you want the generated source code to go (e.g., build/antlr/src/org/myorg/antlr/parser, if your parser grammars specify the package org.myorg.antlr.parser).

Then, when you compile the rest of your application, you can use something like:

<javac destdir="${build.classes.dir}" debug="on" optimize="on" deprecation="${javac.deprecation}" source="${javac.source}" target="${javac.target}">
    <classpath refid="stdclasspath"/>
    <src path="${src.dir}" />
    <src path="${antlr.src.dir}" />
</javac>

Here, we compile our application sources (in src.dir) along with the generated ANTLR code (in antlr.src.dir, which in this example would be build/antlr/src).

As far as using the generated code in your application (i.e., outside ANTLRWorks), you'll need to do something like:

String sourceText = "a + b = foo";
ANTLRStringStream inStream = new ANTLRStringStream(sourceText);

// your generated lexer class
MyLexer lexer = new MyLexer(inStream);
CommonTokenStream tokens = new CommonTokenStream(lexer);

// your generated parser class
MyParser parser = new MyParser(tokens);

// run the toplevel rule (in this case, `program`)
MyParser.program_return prog = parser.program();

// get the resulting AST (a CommonTree instance, in this case)
CommonTree tree = (CommonTree) prog.getTree();

// run a tree parser rule on the AST
MyTreeParser treeParser = new MyTreeParser(new CommonTreeNodeStream(tree));
treeParser.program();

I strongly recommend getting a copy of The Definitive ANTLR Reference if you're going to be using ANTLR. All of this is covered pretty thoroughly, with plenty of examples to get you started.