What is the best way to tokenize a text file in Java?

Question

What is the best way to tokenize a text file in Java, if

1. I want to work with a java.io.Reader, not a String
2. Delimiters should be returned?

I have evaluated the following classes:

• java.util.StringTokenizer fulfills [2.], but not [1.]
• java.util.Scanner fulfills [1.], but not [2.]
• java.io.StreamTokenizer seems quite complicated.

I don't need delimiters, actually. I just need to know, if a certain token is available till the end of the stream, fetch it, and fetch everything that stands before the token. It should be possible to re-specify the token from step to step.

I have looked at Apache Commons, Google Code Search and Stack Overflow, but didn't find anything.

Answer 1

I think in Scanner there is a next(Pattern p) method which may do the job. Just find correct Pattern and it will return delimiters and also you can re-specify tokens.