I am programming a lexer in C and I read somewhere about the header file tokens.h
. Is it there? If so, what is its use?
views:
131answers:
2
+1
A:
tokens.h
is a file generated by yacc
or bison
that contains a list of tokens within your grammar.
Your yacc
/bison
input file may contain token declarations like:
%token INTEGER
%token ID
%token STRING
%token SPACE
Running this file through yacc
/bison
will result in a tokens.h
file that contains preprocessor definitions for these tokens:
/* Something like this... */
#define INTEGER (1)
#define ID (2)
#define STRING (3)
Sean Bright
2009-07-31 15:26:12
so how am i supposed to use it?.. do u have any links ?
mekasperasky
2009-07-31 15:28:49
Are you using flex?
Sean Bright
2009-07-31 15:30:44
no i am not . i am programming my own lex. can you recommend some books on it?
mekasperasky
2009-07-31 15:37:23
When did that change? Last I used them, yacc output y.tab.c and y.tab.h files. Bison replaced the 'y' by the source file basename.
AProgrammer
2009-07-31 15:44:11
@mekasperasky, yacc (and bison which is the GNU rewriting of yacc) are parser generators. If you write a lexer which isn't supposed to interface with them, you don't have to know about them. "Lex and Yacc" by John R. Levine is a well known book about that tools.
AProgrammer
2009-07-31 15:46:02
Right. If you're hand-rolling a lexer, you don't have to worry about tokens.h.
Sean Bright
2009-07-31 15:48:09
A:
Probably, tokens.h
is a file generated by the parser generator (Yacc/Bison) containing token definitions so you can return tokens from the lexer to the parser.
With Lex/Flex and Yacc/Bison, it works like this:
parser.y
:
%token FOO
%token BAR
%%
start: FOO BAR;
%%
lexer.l
:
%{
#include "tokens.h"
%}
%%
foo {return FOO;}
bar {return BAR;}
%%
Zifre
2009-07-31 15:35:16