Lexical Analysis of Python Programming Language

views:

391

answers:

+3 Q:

Lexical Analysis of Python Programming Language

Does anyone know where a FLEX or LEX specification file for Python exists? For example, this is a lex specification for the ANSI C programming language: http://www.quut.com/c/ANSI-C-grammar-l-1998.html

FYI, I am trying to write code highlighting into a Cocoa application. Regex won't do it because I also want grammar parsing to fold code and recognize blocks.

+2 A:

grammar.txt is the official, complete Python grammar -- not directly lex compatible, but you should be able to massage it into a suitable form.

ephemient 2009-11-14 00:31:37

I have seen that but I noticed it's very different from the lex syntax I've seen. I'm not partial to flex/lex, just looking for something that works with a grammar parser. Do you have any recommendations for a lexical analyzer using that grammar file?

pokstad 2009-11-14 00:57:54

Have you considered using one of the existing code highlighters, like Pygments?

Ned Deily 2009-11-14 00:35:48

That would solve the first issue, but wouldn't help a lot with the other transformations.

Chris 2009-11-14 00:41:02

Code highlighting is not enough because I need to produce output for a compatible grammar parser.

pokstad 2009-11-14 01:00:09

+2 A:

Lex is typically just used for tokenizing, not full parsing. Projects that use flex/lex for tokenizing typically use yacc/bison for the actual parsing.

You may want to take a look at ANTLR, a more "modern" alternative to lexx & yacc, which has a few Python grammars. (That's 2.5. See their grammar list for others.)

Laurence Gonsalves 2009-11-14 00:44:47

So this will both scan and parse? Awesome, I've heard of this but wasn't sure of how mature it was. Besides spitting out C-Code, do you know how well it plays with Objective-C? Any option to produce diagrams (like railroad diagrams or finite state machine diagrams)?

pokstad 2009-11-14 01:03:42

Yes, ANTLR does both scanning and parsing. It's been a while since I last used it, but I remember liking the fact that the scanning and parsing syntaxes were extremely similar. It apparently even has explicit support for Objective-C these days (in addition to C, C++, Java, C#, and Python). I've never used it, but you might want to try ANTLRWorks (http://antlr.org/works/index.html) for diagrams.

Laurence Gonsalves 2009-11-14 01:32:44

I'll give it a shot, thanks!

pokstad 2009-11-14 02:50:22

ansaurus

tags:

views:

answers:

Lexical Analysis of Python Programming Language

related questions