Tokenizers in .NET

Question

This must be a classic .NET question for anyone migrating from Java.

.NET does not seem to have a direct equivalent to java.io.StreamTokenizer, however the JLCA provides a SupportClass that attempts to implement it. I believe the JLCA also provides a Tokenizer SupportClass that takes a String as the source, which I thought a StreamTokenizer would be derived from, but isn't.

What is the preferred way to Tokenize both a Stream and a String? or is there one? I'd like to have the flexibility that java.io.StreamTokenizer provides. Any thoughts?

Answer 1

To tokenize a string, use string.Split(...).

Answer 2

There's a tokenizer in the Nextem library -- you can see an example here: http://trac.assembla.com/nextem/browser/trunk/Examples/Parsing.n

It's implemented as a Nemerle macro, but you can write this and then use it from C# easily.

Answer 3

Use System.String.Split if you need to split a string based on a collection of specific characters.

Use System.Text.RegularExpressions.RegEx.Split to split based on matching patterns.

Answer 4

I don't think so, for very simple tokenizing have a look at System.String.Split().

More complex tokenizing can be achieved by System.Text.RegularExpressions.Regex.

Answer 5

There isn't anything in .NET that is completely equivalent to StreamTokenizer. For simple cases, you can use String.Split(), but for more advanced token parsing, you'll probably end up using System.Text.RegularExpressions.Regex.

Good luck!

Answer 6

We had the same problem of finding a StreamTokenizer equivalent when porting tuProlog from Java to C#. We ended up writing what as far as I know is a straight conversion of StreamTokenizer which takes a TextReader as a "stream" for input purposes. You will find the code in the download for tuProlog.NET 2.1 (LGPL-licensed) so feel free to reuse and adapt it to your needs.