How to write a bison file to automatically use a token enumeration list define in a C header file ?

Question

Hi everyone,

I am trying to build a parser with Bison/Yacc to be able to parse a flow of token done by another module. The token different token id are already listed in a enumeration type as follow:

// C++ header file
enum token_id {
  TokenType1         = 0x10000000,
  TokenType2         = 0x11000000,
  TokenType3         = 0x11100000,
  //... and the list go on with about 200/300 line
};

I have gone through the documentation of bison many time but I couldn't find a better solution than copying each token in the Bison file like this:

/* Bison/Yacc file */
%token TokenType1 0x10000000
%token TokenType2 0x11000000
%token TokenType3 0x11100000
//...

If I have to do it like that, It will become pretty hard to maintain the file if the other module specification change (which happen quite oftenly).

Could you please tell me how to do it, or point me in the good direction (any idea/comment is welcome). It would greatly help me! Thanks in advance.

Answer 1

The obvious method would be a small utility to convert from one format to the other. If you're really making changes quite frequently, you might even consider storing the names and values in something like a SQL database, and write a couple of queries to produce the output in the correct format for each tool.

select token_name, '=' token_number ','
    from token_table

select '%token ', token_name, ' ', token_number
    from token_table

The first would require a bit of massaging, such as adding the 'enum token_id {" to the beginning, and "};" to the end, but you get the general idea. Of course, there are lots of alternatives -- XML, CSV, etc., but the general idea remains the same: store and edit as close to raw data as possible, and automate adding the extra "stuff" necessary to keep the tools happy.

Answer 2

Instead of doing :

/* Bison/Yacc file */
%token TokenType1 0x10000000
%token TokenType2 0x11000000
%token TokenType3 0x11100000
//...

You just need to include the file with the token type in the declaration part

#include "mytoken_enum.h"
// ...
%token TokenType1
%token TokenType2
%token TokenType3 
//...

EDIT: This can not be done:

As you see from the numbers above, Bison just numbers the tokens sequentially, and it is used shifted in parser lookup tables as indices, for speed simply. So Bison does not support that, I feel sure, and it would not be easy to fit with the implementation model.

Just need to wrapper to convert the real token to yacc/bison token (eg: via yylex())