views:

131

answers:

2

I am programming a lexer in C and I read somewhere about the header file tokens.h. Is it there? If so, what is its use?

+1  A: 

tokens.h is a file generated by yacc or bison that contains a list of tokens within your grammar.

Your yacc/bison input file may contain token declarations like:

%token INTEGER
%token ID
%token STRING
%token SPACE

Running this file through yacc/bison will result in a tokens.h file that contains preprocessor definitions for these tokens:

/* Something like this... */
#define INTEGER (1)
#define ID      (2)
#define STRING  (3)
Sean Bright
so how am i supposed to use it?.. do u have any links ?
mekasperasky
Are you using flex?
Sean Bright
no i am not . i am programming my own lex. can you recommend some books on it?
mekasperasky
When did that change? Last I used them, yacc output y.tab.c and y.tab.h files. Bison replaced the 'y' by the source file basename.
AProgrammer
@mekasperasky, yacc (and bison which is the GNU rewriting of yacc) are parser generators. If you write a lexer which isn't supposed to interface with them, you don't have to know about them. "Lex and Yacc" by John R. Levine is a well known book about that tools.
AProgrammer
Right. If you're hand-rolling a lexer, you don't have to worry about tokens.h.
Sean Bright
A: 

Probably, tokens.h is a file generated by the parser generator (Yacc/Bison) containing token definitions so you can return tokens from the lexer to the parser.

With Lex/Flex and Yacc/Bison, it works like this:

parser.y:

%token FOO
%token BAR

%%

start: FOO BAR;

%%

lexer.l:

%{
#include "tokens.h"
%}

%%

foo {return FOO;}
bar {return BAR;}

%%
Zifre