views:

286

answers:

3

Hello,

I'm looking for a hand-written free lexer/scanner for any language, even a toy language written in a language similiar to C/C++. I am a beginner in creating languages and compiler design, I'd like to play around with the source.

For example, theres a hand written lexer on this website before the flex generated lex: http://en.wikipedia.org/wiki/Flex%5Flexical%5Fanalyser

Thanks.

A: 

You probably want flex, which is probably the most widely used lexer on the planet.

timdev
Looking for hand-written lexers.
Shawn Mclean
Oh, missed that bit. Ummmm... good luck, I guess. Is this a homework-related question? I can't easily think of a reason anyone would write a lexer by hand other than they were taking a compiler construction course and it was an assignment. Come to think of it, that sort of assignment would come about right now in the semester.
timdev
@Tim, don't you just *hate* it when people repeat themselves like that (as if we couldn't understand).
pavium
A: 

Lex and Yacc are the "classic" tools used for the purpose of building languages and compilers.

bobbymcr
Looking for hand-written lexers
Shawn Mclean
+4  A: 

I like the LLVM tutorial -- it starts by presenting a hand-written lexer for a toy language called Kaleidoscope, first as a "literate programming" exercise with good textual explanations of each bit; it continues by building a parser that builds an abstract syntax tree in the same style; then the whole thing (400 lines of hand-writted standalone C++ for lexer, parser and AST builder) is shown again as a complete .cpp source file, so you don't have to put it all together by yourself. Following chapters show code generation, JIT, optimization, ... a truly useful little tutorial indeed! But, you can stop with the lexer, if that's all you want to understand for now;-).

Alex Martelli
+1 Excellent little tutorial, and interesting project (LLVM)
timdev