Tokenizer for C#?

Question

Is there any functionality built into the .NET framework somewhere to tokenize C# code? I'm not looking to build a tokenizer in C#, I'm looking for something that can tokenize C# source code.

Answer 1

No, not built into the framwork.

However, you may want to look at Irony, and C# Parser on CodePlex, as they both provide a parser/lexer for at least simple C#

Answer 2

The only thing that comes to mind is a parser generator like ANTLR, which has C# Sample Grammar available. Bison/Flex also looks like it has pretty decent C# grammar as well. Parsing any language and then actually making sense of it is fairly difficult, so I wish you the best of luck.

Answer 3

The GOLD Parser too has a C# grammar (to parse C#), and run-time engines written in C# (so that you can execute that grammar using C# code).